Ron Li is a Clinical Assistant Professor of Medicine in the Division of Hospital Medicine and Center for Biomedical Informatics Research at Stanford University School of Medicine. His academic interests focus on the "delivery science" of artificial intelligence in healthcare and how to design, implement, and evaluate AI enabled systems of care delivery. As the Medical Informatics Director for Artificial Intelligence Clinical Integration at Stanford Health Care, he also helps lead the integration efforts of clinical AI for the health system. He is also the medical director for the Stanford Emerging Apps Lab (SEAL), which builds novel digital apps for clinicians at Stanford Health Care.
Ron's work spans across multiple disciplines, including clinical medicine, data science, digital health, information technology, design thinking, process improvement, and implementation science. Current areas of focus include using machine learning to improve advance care planning, care of clinically deteriorating patients, and the expansion of virtual care networks. He has consulted for various companies in the digital health and artificial intelligence space, and is leading work in AI and UXR in partnership with Google Health. He is an attending physician on the inpatient medicine teaching service at Stanford Hospital, and is a core faculty for the Stanford Clinical Informatics Fellowship.
- Clinical Informatics
- Hospital Medicine
- Internal Medicine
Clinical Assistant Professor, Medicine
Faculty Affiliate, Institute for Human-Centered Artificial Intelligence (HAI)
Medical Informatics Director for AI Clinical Integration, Stanford Health Care (2019 - Present)
Honors & Awards
Stanford Human Centered AI Seed Grant Recipient, Stanford (2018)
Board Certification: American Board of Preventive Medicine, Clinical Informatics (2021)
Board Certification, American Board of Preventive Medicine, Clinical Informatics (2021)
Fellowship: Stanford University Clinical Informatics Fellowship (2019) CA
Board Certification: American Board of Internal Medicine, Internal Medicine (2018)
Residency: Stanford University Internal Medicine Residency (2017) CA
Medical Education: Northwestern University Feinberg School of Medicine (2014) IL
Internal Medicine Residency, Stanford University
MD, Northwestern University
Rethinking PICO in the Machine Learning Era: ML-PICO.
Applied clinical informatics
2021; 12 (2): 407-416
BACKGROUND: Machine learning (ML) has captured the attention of many clinicians who may not have formal training in this area but are otherwise increasingly exposed to ML literature that may be relevant to their clinical specialties. ML papers that follow an outcomes-based research format can be assessed using clinical research appraisal frameworks such as PICO (Population, Intervention, Comparison, Outcome). However, the PICO frameworks strain when applied to ML papers that create new ML models, which are akin to diagnostic tests. There is a need for a new framework to help assess such papers.OBJECTIVE: We propose a new framework to help clinicians systematically read and evaluate medical ML papers whose aim is to create a new ML model: ML-PICO (Machine Learning, Population, Identification, Crosscheck, Outcomes). We describe how the ML-PICO framework can be applied toward appraising literature describing ML models for health care.CONCLUSION: The relevance of ML to practitioners of clinical medicine is steadily increasing with a growing body of literature. Therefore, it is increasingly important for clinicians to be familiar with how to assess and best utilize these tools. In this paper we have described a practical framework on how to read ML papers that create a new ML model (or diagnostic test): ML-PICO. We hope that this can be used by clinicians to better evaluate the quality and utility of ML papers.
View details for DOI 10.1055/s-0041-1729752
View details for PubMedID 34010977
A framework for making predictive models useful in practice.
Journal of the American Medical Informatics Association : JAMIA
OBJECTIVE: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.MATERIALS AND METHODS: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.RESULTS: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.DISCUSSION: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.CONCLUSION: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.
View details for DOI 10.1093/jamia/ocaa318
View details for PubMedID 33355350
- Annals for Hospitalists Inpatient Notes - Realizing the Promises of Hospital Electronic Order Sets. Annals of internal medicine 2020; 173 (4): HO2–HO3
- Assessment of a Real-Time Locator System to Identify Physician and Nurse Work Locations. JAMA network open 2020; 3 (2): e1920352
Developing a delivery science for artificial intelligence in healthcare.
NPJ digital medicine
2020; 3: 107
Artificial Intelligence (AI) has generated a large amount of excitement in healthcare, mostly driven by the emergence of increasingly accurate machine learning models. However, the promise of AI delivering scalable and sustained value for patient care in the real world setting has yet to be realized. In order to safely and effectively bring AI into use in healthcare, there needs to be a concerted effort around not just the creation, but also the delivery of AI. This AI "delivery science" will require a broader set of tools, such as design thinking, process improvement, and implementation science, as well as a broader definition of what AI will look like in practice, which includes not just machine learning models and their predictions, but also the new systems for care delivery that they enable. The careful design, implementation, and evaluation of these AI enabled systems will be important in the effort to understand how AI can improve healthcare.
View details for DOI 10.1038/s41746-020-00318-y
View details for PubMedID 32885053
View details for PubMedCentralID PMC7443141
- Developing a delivery science for artificial intelligence in healthcare. NPJ digital medicine 2020; 3 (1): 107
When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record.
BMJ quality & safety
Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs.We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set.There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%-18%) and median medication retraction rate was 4% (IQR 2%-10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte.Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow.
View details for DOI 10.1136/bmjqs-2018-008968
View details for PubMedID 31164486
- Impact of problem-based charting on the utilization and accuracy of the electronic problem list JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 2018; 25 (5): 548–54
- The Impact of Big Data on the Physician GUIDE TO BIG DATA APPLICATIONS 2018; 26: 415–48
Discordance Between Apolipoprotein B and LDL-Cholesterol in Young Adults Predicts Coronary Artery Calcification The CARDIA Study
JOURNAL OF THE AMERICAN COLLEGE OF CARDIOLOGY
2016; 67 (2): 193–201
High levels of apolipoprotein B (apoB) have been shown to predict atherosclerotic cardiovascular disease (CVD) in adults even in the context of low levels of low-density lipoprotein cholesterol (LDL-C) or non-high-density lipoprotein cholesterol (non-HDL-C).This study aimed to quantify the associations between apoB and the discordance between apoB and LDL-C or non-HDL-C in young adults and measured coronary artery calcium (CAC) in midlife.Data were derived from a multicenter cohort study of young adults recruited at ages 18 to 30 years. All participants with complete baseline CVD risk factor data, including apoB and year 25 (Y25) CAC score, were entered into this study. Presence of CAC was defined as having a positive, nonzero Agatston score as determined by computed tomography. Baseline apoB values were divided into tertiles of 4 mutually exclusive concordant/discordant groups, based on median apoB and LDL-C or non-HDL-C.Analysis included 2,794 participants (mean age: 25 ± 3.6 years; body mass index: 24.5 ± 5 kg/m(2); and 44.4% male). Mean lipid values were as follows: total cholesterol: 177.3 ± 33.1 mg/dl; LDL-C: 109.9 ± 31.1 mg/dl; non-HDL-C: 124.0 ± 33.5 mg/dl; HDL-C: 53 ± 12.8 mg/dl; and apoB: 90.7 ± 24 mg/dl; median triglycerides were 61 mg/dl. Compared with the lowest apoB tertile, higher odds of developing Y25 CAC were seen in the middle (odds ratio [OR]: 1.53) and high (OR: 2.28) tertiles based on traditional risk factor-adjusted models. High apoB and low LDL-C or non-HDL-C discordance was also associated with Y25 CAC in adjusted models (OR: 1.55 and OR: 1.45, respectively).These data suggest a dose-response association between apoB in young adults and the presence of midlife CAC independent of baseline traditional CVD risk factors.
View details for DOI 10.1016/j.jacc.2015.10.055
View details for Web of Science ID 000368114400011
View details for PubMedID 26791067
Validation of Test Performance and Clinical Time Zero for an Electronic Health Record Embedded Severe Sepsis Alert.
Applied clinical informatics
2016; 7 (2): 560-572
Increasing use of EHRs has generated interest in the potential of computerized clinical decision support to improve treatment of sepsis. Electronic sepsis alerts have had mixed results due to poor test characteristics, the inability to detect sepsis in a timely fashion and the use of outside software limiting widespread adoption. We describe the development, evaluation and validation of an accurate and timely severe sepsis alert with the potential to impact sepsis management.To develop, evaluate, and validate an accurate and timely severe sepsis alert embedded in a commercial EHR.The sepsis alert was developed by identifying the most common severe sepsis criteria among a cohort of patients with ICD 9 codes indicating a diagnosis of sepsis. This alert requires criteria in three categories: indicators of a systemic inflammatory response, evidence of suspected infection from physician orders, and markers of organ dysfunction. Chart review was used to evaluate test performance and the ability to detect clinical time zero, the point in time when a patient develops severe sepsis.Two physicians reviewed 100 positive cases and 75 negative cases. Based on this review, sensitivity was 74.5%, specificity was 86.0%, the positive predictive value was 50.3%, and the negative predictive value was 94.7%. The most common source of end-organ dysfunction was MAP less than 70 mm/Hg (59%). The alert was triggered at clinical time zero in 41% of cases and within three hours in 53.6% of cases. 96% of alerts triggered before a manual nurse screen.We are the first to report the time between a sepsis alert and physician chart-review clinical time zero. Incorporating physician orders in the alert criteria improves specificity while maintaining sensitivity, which is important to reduce alert fatigue. By leveraging standard EHR functionality, this alert could be implemented by other healthcare systems.
View details for DOI 10.4338/ACI-2015-11-RA-0159
View details for PubMedID 27437061
View details for PubMedCentralID PMC4941860
- Psoriasis is associated with decreased plasma adiponectin levels beyond cardiometabolic risk factors Clinical and Experimental Dermatology 2014; 39 (1)
- Dietary vitamin K intake and anticoagulation control during the initiation phase of warfarin therapy: A prospective cohort study THROMBOSIS AND HAEMOSTASIS 2013; 110 (1): 195–96
- Abnormal lipoprotein particles and cholesterol efflux capacity in patients with psoriasis Atherosclerosis 2012; 224 (1)