Ron Li

All Publications

• Clinical Evaluations of Early Warning Scores-Reply. JAMA internal medicine Gallo, R. J., Geldsetzer, P., Li, R. C. 2024

  View details for DOI 10.1001/jamainternmed.2024.3053

  View details for PubMedID 39037786

• Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration. JAMA internal medicine Gallo, R. J., Shieh, L., Smith, M., Marafino, B. J., Geldsetzer, P., Asch, S. M., Shum, K., Lin, S., Westphal, J., Hong, G., Li, R. C. 2024

  Abstract

  Inpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness.To evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference.This cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022.An artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians.The primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization.During the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold.Using a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

  View details for DOI 10.1001/jamainternmed.2024.0084

  View details for PubMedID 38526472

• Design and Implementation of an Electronic Health Record-Integrated Hypertension Management Application. Journal of the American Heart Association Funes Hernandez, M., Babakhanian, M., Chen, T. P., Sarraju, A., Seninger, C., Ravi, V., Azizi, Z., Tooley, J., Chang, T. I., Lu, Y., Downing, N. L., Rodriguez, F., Li, R. C., Sandhu, A. T., Turakhia, M., Bhalla, V., Wang, P. J. 2024; 13 (2): e030884

  Abstract

  High blood pressure affects approximately 116 million adults in the United States. It is the leading risk factor for death and disability across the world. Unfortunately, over the past decade, hypertension control rates have decreased across the United States. Prediction models and clinical studies have shown that reducing clinician inertia alone is sufficient to reach the target of ≥80% blood pressure control. Digital health tools containing evidence-based algorithms that are able to reduce clinician inertia are a good fit for turning the tide in blood pressure control, but careful consideration should be taken in the design process to integrate digital health interventions into the clinical workflow.We describe the development of a provider-facing hypertension management platform. We enumerate key steps of the development process, including needs finding, clinical workflow analysis, treatment algorithm creation, platform design and electronic health record integration. We interviewed and surveyed 5 Stanford clinicians from primary care, cardiology, and their clinical care team members (including nurses, advanced practice providers, medical assistants) to identify needs and break down the steps of clinician workflow analysis. The application design and development stage were aided by a team of approximately 15 specialists in the fields of primary care, hypertension, bioinformatics, and software development.Digital monitoring holds immense potential for revolutionizing chronic disease management. Our team developed a hypertension management platform at an academic medical center to address some of the top barriers to adoption and achieving clinical outcomes. The frameworks and processes described in this article may be used for the development of a diverse range of digital health tools in the cardiovascular space.

  View details for DOI 10.1161/JAHA.123.030884

  View details for PubMedID 38226516

• Redefining Rare Disease Care in the Digital Age: Insights and Key Takeaways from a Digital Health Symposium Focused on Empowering Rare Disease Communities. Biomedicine hub Lewis, E., Dayal, A., Li, R. 2024; 9 (1): 38-44

  Abstract

  At the Stanford-UCB Rare Disease Digital Health Symposium held in Stanford, California, on September 8, 2023, researchers, clinicians, payers, thought leaders, and rare disease caregivers and advocates discussed the current state of care delivery and future perspectives of digitally-enabled care for rare disease patient populations. Digital health aims to improve healthcare delivery through novel ways of providing access to more precise diagnosis, monitoring of disease progression, treatment, prognosis, and care management for rare disease patients. The meeting focused on highlighting challenges and unmet needs, data infrastructure and analytics, the need for targeted and effective personalized therapies, and the importance of digital care transformation. The meeting also covered the social and ethical impact of access to digitally delivered, patient-centered care, as well as views on implementation and patient autonomy and empowerment.

  View details for DOI 10.1159/000536274

  View details for PubMedID 38601364

• Lessons Learned from a Multi-Site, Team-Based Serious Illness Care Program Implementation at an Academic Medical Center. Journal of palliative medicine Seevaratnam, B., Wang, S., Fong, R., Hui, F., Callahan, A., Chobot, S., Gensheimer, M. F., Li, R. C., Nguyen, D., Ramchandran, K., Shah, N. H., Shieh, L., Zeng, J. G., Teuteberg, W. 2023

  Abstract

  Background: Patients with serious illness benefit from conversations to share prognosis and explore goals and values. To address this, we implemented Ariadne Labs' Serious Illness Care Program (SICP) at Stanford Health Care. Objective: Improve quantity, timing, and quality of serious illness conversations. Methods: Initial implementation followed Ariadne Labs' SICP framework. We later incorporated a team-based approach that included nonphysician care team members. Outcomes included number of patients with documented conversations according to clinician role and practice location. Machine learning algorithms were used in some settings to identify eligible patients. Results: Ambulatory oncology and hospital medicine were our largest implementation sites, engaging 4707 and 642 unique patients in conversations, respectively. Clinicians across eight disciplines engaged in these conversations. Identified barriers that included leadership engagement, complex workflows, and patient identification. Conclusion: Several factors contributed to successful SICP implementation across clinical sites: innovative clinical workflows, machine learning based predictive algorithms, and nonphysician care team member engagement.

  View details for DOI 10.1089/jpm.2023.0254

  View details for PubMedID 37935036

• How Chatbots and Large Language Model Artificial Intelligence Systems Will Reshape Modern Medicine: Fountain of Creativity or Pandora's Box? JAMA internal medicine Li, R., Kumar, A., Chen, J. H. 2023

  View details for DOI 10.1001/jamainternmed.2023.1835

  View details for PubMedID 37115531

• Hypophosphatemia, hypokalaemia and rhabdomyolysis associated with a panic attack. BMJ case reports Deverett, B., Li, R. 2023; 16 (3)

  Abstract

  Panic attacks have been associated with hypophosphatemia, which can lead to numerous complications if unrecognised. Here, we present the case of an otherwise-healthy man in his 20s who experienced a panic attack accompanied by hypophosphatemia and hypokalaemia and subsequently developed rhabdomyolysis. This trajectory highlights the clinical significance of panic attack-associated metabolic derangements and their potential for medical complications such as rhabdomyolysis.

  View details for DOI 10.1136/bcr-2022-254362

  View details for PubMedID 36948522

• Targeting Repetitive Laboratory Testing with Electronic Health Records-Embedded Predictive Decision Support: A Pre-Implementation Study. Clinical biochemistry Rabbani, N., Ma, S. P., Li, R. C., Winget, M., Weber, S., Boosi, S., Pham, T. D., Svec, D., Shieh, L., Chen, J. H. 2023

  Abstract

  INTRODUCTION: Unnecessary laboratory testing contributes to patient morbidity and healthcare waste. Despite prior attempts at curbing such overutilization, there remains opportunity for improvement using novel data-driven approaches. This study presents the development and early evaluation of a clinical decision support tool that uses a predictive model to help providers reduce low-yield, repetitive laboratory testing in hospitalized patients.METHODS: We developed an EHR-embedded SMART on FHIR application that utilizes a laboratory test result prediction model based on historical laboratory data. A combination of semi-structured physician interviews, usability testing, and quantitative analysis on retrospective laboratory data were used to inform the tool's development and evaluate its acceptability and potential clinical impact.KEY RESULTS: Physicians identified culture and lack of awareness of repeat orders as key drivers for overuse of inpatient blood testing. Users expressed an openness to a lab prediction model and 13/15 physicians believed the tool would alter their ordering practices. The application received a median System Usability Scale score of 75, corresponding to the 75th percentile of software tools. On average, physicians desired a prediction certainty of 85% before discontinuing a routine recurring laboratory order and a higher certainty of 90% before being alerted. Simulation on historical lab data indicates that filtering based on accepted thresholds could have reduced 22% of repeat chemistry panels.CONCLUSIONS: The use of a predictive algorithm as a means to calculate the utility of a diagnostic test is a promising paradigm for curbing laboratory test overutilization. An EHR-embedded clinical decision support tool employing such a model is a novel and acceptable intervention with the potential to reduce low-yield, repetitive laboratory testing.

  View details for DOI 10.1016/j.clinbiochem.2023.01.002

  View details for PubMedID 36623759

• Team is brain: leveraging EHR audit log data for new insights into acute care processes. Journal of the American Medical Informatics Association : JAMIA Rose, C., Thombley, R., Noshad, M., Lu, Y., Clancy, H. A., Schlessinger, D., Li, R. C., Liu, V. X., Chen, J. H., Adler-Milstein, J. 2022

  Abstract

  OBJECTIVE: To determine whether novel measures of contextual factors from multi-site electronic health record (EHR) audit log data can explain variation in clinical process outcomes.MATERIALS AND METHODS: We selected one widely-used process outcome: emergency department (ED)-based team time to deliver tissue plasminogen activator (tPA) to patients with acute ischemic stroke (AIS). We evaluated Epic audit log data (that tracks EHR user-interactions) for 3052 AIS patients aged 18+ who received tPA after presenting to an ED at three Northern California health systems (Stanford Health Care, UCSF Health, and Kaiser Permanente Northern California). Our primary outcome was door-to-needle time (DNT) and we assessed bivariate and multivariate relationships with six audit log-derived measures of treatment team busyness and prior team experience.RESULTS: Prior team experience was consistently associated with shorter DNT; teams with greater prior experience specifically on AIS cases had shorter DNT (minutes) across all sites: (Site 1: -94.73, 95% CI: -129.53 to 59.92; Site 2: -80.93, 95% CI: -130.43 to 31.43; Site 3: -42.95, 95% CI: -62.73 to 23.17). Teams with greater prior experience across all types of cases also had shorter DNT at two sites: (Site 1: -6.96, 95% CI: -14.56 to 0.65; Site 2: -19.16, 95% CI: -36.15 to 2.16; Site 3: -11.07, 95% CI: -17.39 to 4.74). Team busyness was not consistently associated with DNT across study sites.CONCLUSIONS: EHR audit log data offers a novel, scalable approach to measure key contextual factors relevant to clinical process outcomes across multiple sites. Audit log-based measures of team experience were associated with better process outcomes for AIS care, suggesting opportunities to study underlying mechanisms and improve care through deliberate training, team-building, and scheduling to maximize team experience.

  View details for DOI 10.1093/jamia/ocac201

  View details for PubMedID 36303451

• User-centred design for machine learning in health care: a case study from care management. BMJ health & care informatics Seneviratne, M. G., Li, R. C., Schreier, M., Lopez-Martinez, D., Patel, B. S., Yakubovich, A., Kemp, J. B., Loreaux, E., Gamble, P., El-Khoury, K., Vardoulakis, L., Wong, D., Desai, J., Chen, J. H., Morse, K. E., Downing, N. L., Finger, L. T., Chen, M., Shah, N. 2022; 29 (1)

  Abstract

  OBJECTIVES: Few machine learning (ML) models are successfully deployed in clinical practice. One of the common pitfalls across the field is inappropriate problem formulation: designing ML to fit the data rather than to address a real-world clinical pain point.METHODS: We introduce a practical toolkit for user-centred design consisting of four questions covering: (1) solvable pain points, (2) the unique value of ML (eg, automation and augmentation), (3) the actionability pathway and (4) the model's reward function. This toolkit was implemented in a series of six participatory design workshops with care managers in an academic medical centre.RESULTS: Pain points amenable to ML solutions included outpatient risk stratification and risk factor identification. The endpoint definitions, triggering frequency and evaluation metrics of the proposed risk scoring model were directly influenced by care manager workflows and real-world constraints.CONCLUSIONS: Integrating user-centred design early in the ML life cycle is key for configuring models in a clinically actionable way. This toolkit can guide problem selection and influence choices about the technical setup of the ML problem.

  View details for DOI 10.1136/bmjhci-2022-100656

  View details for PubMedID 36220304

• Impact of telemedicine on clinical practice patterns for patients with chest pain in the emergency department. International journal of medical informatics Ostberg, N., Ip, W., Brown, I., Li, R. 2022; 161: 104726

  Abstract

  BACKGROUND: The outbreak of the COVID-19 pandemic has led to the rapid adoption of novel telemedicine programs within the emergency department (ED) to minimize provider exposure and conserve personal protective equipment (PPE). In this study, we sought to assess how the adoption of telemedicine in the ED impacted clinical order patterns for patients with chest pain. We hypothesize that clinicians would rely more on imaging and laboratory workup for patients receiving telemedicine due to limitation in physical exams.METHODS: A single-center, retrospective, propensity score matched study was designed for patients presenting with chest pain at an ED. The study period was defined between April 1st, 2020 and September 30th, 2020. The frequency of the most frequent lab, imaging, and medication orders were compared. In addition, poisson regression analysis was performed to compare the overall number of orders between the two groups.RESULTS: 455 patients with chest pain who received telemedicine were matched to 455 similar patients without telemedicine with standardized mean difference<0.1 for all matched covariates. The proportion of frequent lab, imaging, and medication orders were similar between the two groups. However, telemedicine patients received more orders overall (RR, 1.19, 95% CI, 1.11, 1.28, p-value<0.001) as well as more imaging, lab, and nursing orders. The number of medication orders between the two groups remained similar.CONCLUSIONS: Frequent labs, imaging, and medications were ordered in similar proportions between the two cohorts. However, telemedicine patients had more orders placed overall. This study is an important objective assessment of the impact that telemedicine has upon clinical practice patterns and can guide future telemedicine implementation after the COVID-19 pandemic.

  View details for DOI 10.1016/j.ijmedinf.2022.104726

  View details for PubMedID 35228006

• Using AI to Empower Collaborative Team Workflows: Two Implementations for Advance Care Planning and Care Escalation NEJM Catalyst Innovations in Care Delivery Li, R. C., Smith, M., Lu, J., Avati, A., Wang, S., Teuteberg, W. G., Shum, K., Hong, G., Seevaratnam, B., Westphal, J., Dougherty, M., Rao, P., Asch, S., Lin, S., Sharp, C., Shieh, L., Shah, N. H. 2022; 3 (4)

  View details for DOI 10.1056/CAT.21.0457

• Building a Learning Health System: Creating an Analytical Workflow for Evidence Generation to Inform Institutional Clinical Care Guidelines. Applied clinical informatics Dash, D., Gokhale, A., Patel, B. S., Callahan, A., Posada, J., Krishnan, G., Collins, W., Li, R., Schulman, K., Ren, L., Shah, N. H. 2022; 13 (1): 315-321

  Abstract

  BACKGROUND: One key aspect of a learning health system (LHS) is utilizing data generated during care delivery to inform clinical care. However, institutional guidelines that utilize observational data are rare and require months to create, making current processes impractical for more urgent scenarios such as those posed by the COVID-19 pandemic. There exists a need to rapidly analyze institutional data to drive guideline creation where evidence from randomized control trials are unavailable.OBJECTIVES: This article provides a background on the current state of observational data generation in institutional guideline creation and details our institution's experience in creating a novel workflow to (1) demonstrate the value of such a workflow, (2) demonstrate a real-world example, and (3) discuss difficulties encountered and future directions.METHODS: Utilizing a multidisciplinary team of database specialists, clinicians, and informaticists, we created a workflow for identifying and translating a clinical need into a queryable format in our clinical data warehouse, creating data summaries and feeding this information back into clinical guideline creation.RESULTS: Clinical questions posed by the hospital medicine division were answered in a rapid time frame and informed creation of institutional guidelines for the care of patients with COVID-19. The cost of setting up a workflow, answering the questions, and producing data summaries required around 300hours of effort and $300,000 USD.CONCLUSION: A key component of an LHS is the ability to learn from data generated during care delivery. There are rare examples in the literature and we demonstrate one such example along with proposed thoughts of ideal multidisciplinary team formation and deployment.

  View details for DOI 10.1055/s-0042-1743241

  View details for PubMedID 35235994

• Considerations in the reliability and fairness audits of predictive models for advance care planning Frontiers in Digital Health Lu, J., Sattler, A., Wang, S., Khaki, A. R., Callahan, A., Fleming, S., Fong, R., Ehlert, B., Li, R., Shieh, L., Ramchandran, K., Gensheimer, M., Chobot, S., Pfohl, S., Li, S., Shum, K., Parikh, N., Desai, P., Seevaratnam, B., Hanson, M., Smith, M., Xu, Y., Gokhale, A., Lin, S., Shah, N. 2022: 943768

  View details for DOI 10.3389/fdgth.2022.943768

• Harnessing the Power of Smart and Connected Health to Tackle COVID-19: IoT, AI, Robotics, and Blockchain for a Better World IEEE INTERNET OF THINGS JOURNAL Firouzi, F., Farahani, B., Daneshmand, M., Grise, K., Song, J., Saracco, R., Wang, L., Lo, K., Angelov, P., Soares, E., Loh, P., Talebpour, Z., Moradi, R., Goodarzi, M., Ashraf, H., Talebpour, M., Talebpour, A., Romeo, L., Das, R., Heidari, H., Pasquale, D., Moody, J., Woods, C., Huang, E. S., Barnaghi, P., Sarrafzadeh, M., Li, R., Beck, K. L., Isayev, O., Sung, N., Luo, A. 2021; 8 (16): 12826-12846

  View details for DOI 10.1109/JIOT.2021.3073904

  View details for Web of Science ID 000682147600037

• Predicting and Responding to Clinical Deterioration in Hospitalized Patients by Using Artificial Intelligence: Protocol for a Mixed Methods, Stepped Wedge Study. JMIR research protocols Holdsworth, L. M., Kling, S. M., Smith, M., Safaeinili, N., Shieh, L., Vilendrer, S., Garvert, D. W., Winget, M., Asch, S. M., Li, R. C. 2021; 10 (7): e27532

  Abstract

  BACKGROUND: The early identification of clinical deterioration in patients in hospital units can decrease mortality rates and improve other patient outcomes; yet, this remains a challenge in busy hospital settings. Artificial intelligence (AI), in the form of predictive models, is increasingly being explored for its potential to assist clinicians in predicting clinical deterioration.OBJECTIVE: Using the Systems Engineering Initiative for Patient Safety (SEIPS) 2.0 model, this study aims to assess whether an AI-enabled work system improves clinical outcomes, describe how the clinical deterioration index (CDI) predictive model and associated work processes are implemented, and define the emergent properties of the AI-enabled work system that mediate the observed clinical outcomes.METHODS: This study will use a mixed methods approach that is informed by the SEIPS 2.0 model to assess both processes and outcomes and focus on how physician-nurse clinical teams are affected by the presence of AI. The intervention will be implemented in hospital medicine units based on a modified stepped wedge design featuring three stages over 11 months-stage 0 represents a baseline period 10 months before the implementation of the intervention; stage 1 introduces the CDI predictions to physicians only and triggers a physician-driven workflow; and stage 2 introduces the CDI predictions to the multidisciplinary team, which includes physicians and nurses, and triggers a nurse-driven workflow. Quantitative data will be collected from the electronic health record for the clinical processes and outcomes. Interviews will be conducted with members of the multidisciplinary team to understand how the intervention changes the existing work system and processes. The SEIPS 2.0 model will provide an analytic framework for a mixed methods analysis.RESULTS: A pilot period for the study began in December 2020, and the results are expected in mid-2022.CONCLUSIONS: This protocol paper proposes an approach to evaluation that recognizes the importance of assessing both processes and outcomes to understand how a multifaceted AI-enabled intervention affects the complex team-based work of identifying and managing clinical deterioration.INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/27532.

  View details for DOI 10.2196/27532

  View details for PubMedID 34255728

• Development and Implementation of a Real-time Bundle-adherence Dashboard for Central Line-associated Bloodstream Infections. Pediatric quality & safety Chemparathy, A., Seneviratne, M. G., Ward, A., Mirchandani, S., Li, R., Mathew, R., Wood, M., Shin, A. Y., Donnelly, L. F., Scheinker, D., Lee, G. M. 2021; 6 (4): e431

  Abstract

  Introduction: Central line-associated bloodstream infections (CLABSIs) are the most common hospital-acquired infection in pediatric patients. High adherence to the CLABSI bundle mitigates CLABSIs. At our institution, there did not exist a hospital-wide system to measure bundle-adherence. We developed an electronic dashboard to monitor CLABSI bundle-adherence across the hospital and in real time.Methods: Institutional stakeholders and areas of opportunity were identified through interviews and data analyses. We created a data pipeline to pull adherence data from twice-daily bundle checks and populate a dashboard in the electronic health record. The dashboard was developed to allow visualization of overall and individual element bundle-adherence across units. Monthly dashboard accesses and element-level bundle-adherence were recorded, and the nursing staff's feedback about the dashboard was obtained.Results: Following deployment in September 2018, the dashboard was primarily accessed by quality improvement, clinical effectiveness and analytics, and infection prevention and control. Quality improvement and infection prevention and control specialists presented dashboard data at improvement meetings to inform unit-level accountability initiatives. All-element adherence across the hospital increased from 25% in September 2018 to 44% in December 2019, and average adherence to each bundle element increased between 2018 and 2019.Conclusions: CLABSI bundle-adherence, overall and by element, increased across the hospital following the deployment of a real-time electronic data dashboard. The dashboard enabled population-level surveillance of CLABSI bundle-adherence that informed bundle accountability initiatives. Data transparency enabled by electronic dashboards promises to be a useful tool for infectious disease control.

  View details for DOI 10.1097/pq9.0000000000000431

  View details for PubMedID 34235355

• Rethinking PICO in the Machine Learning Era: ML-PICO. Applied clinical informatics Liu, X., Anstey, J., Li, R., Sarabu, C., Sono, R., Butte, A. J. 2021; 12 (2): 407-416

  Abstract

  BACKGROUND: Machine learning (ML) has captured the attention of many clinicians who may not have formal training in this area but are otherwise increasingly exposed to ML literature that may be relevant to their clinical specialties. ML papers that follow an outcomes-based research format can be assessed using clinical research appraisal frameworks such as PICO (Population, Intervention, Comparison, Outcome). However, the PICO frameworks strain when applied to ML papers that create new ML models, which are akin to diagnostic tests. There is a need for a new framework to help assess such papers.OBJECTIVE: We propose a new framework to help clinicians systematically read and evaluate medical ML papers whose aim is to create a new ML model: ML-PICO (Machine Learning, Population, Identification, Crosscheck, Outcomes). We describe how the ML-PICO framework can be applied toward appraising literature describing ML models for health care.CONCLUSION: The relevance of ML to practitioners of clinical medicine is steadily increasing with a growing body of literature. Therefore, it is increasingly important for clinicians to be familiar with how to assess and best utilize these tools. In this paper we have described a practical framework on how to read ML papers that create a new ML model (or diagnostic test): ML-PICO. We hope that this can be used by clinicians to better evaluate the quality and utility of ML papers.

  View details for DOI 10.1055/s-0041-1729752

  View details for PubMedID 34010977

• A framework for making predictive models useful in practice. Journal of the American Medical Informatics Association : JAMIA Jung, K., Kashyap, S., Avati, A., Harman, S., Shaw, H., Li, R., Smith, M., Shum, K., Javitz, J., Vetteth, Y., Seto, T., Bagley, S. C., Shah, N. H. 2020

  Abstract

  OBJECTIVE: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.MATERIALS AND METHODS: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.RESULTS: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.DISCUSSION: The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.CONCLUSION: An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

  View details for DOI 10.1093/jamia/ocaa318

  View details for PubMedID 33355350

• Developing a delivery science for artificial intelligence in healthcare. NPJ digital medicine Li, R. C., Asch, S. M., Shah, N. H. 2020; 3 (1): 107

  View details for DOI 10.1038/s41746-020-00318-y

  View details for PubMedID 33597602

• Annals for Hospitalists Inpatient Notes - Realizing the Promises of Hospital Electronic Order Sets. Annals of internal medicine Li, R. C., Chen, J. H. 2020; 173 (4): HO2–HO3

  View details for DOI 10.7326/M20-5164

  View details for PubMedID 32805161

• Assessment of a Real-Time Locator System to Identify Physician and Nurse Work Locations. JAMA network open Li, R. C., Marafino, B. J., Nielsen, D., Baiocchi, M., Shieh, L. 2020; 3 (2): e1920352

  View details for DOI 10.1001/jamanetworkopen.2019.20352

  View details for PubMedID 32022876

• Developing a delivery science for artificial intelligence in healthcare. NPJ digital medicine Li, R. C., Asch, S. M., Shah, N. H. 2020; 3: 107

  Abstract

  Artificial Intelligence (AI) has generated a large amount of excitement in healthcare, mostly driven by the emergence of increasingly accurate machine learning models. However, the promise of AI delivering scalable and sustained value for patient care in the real world setting has yet to be realized. In order to safely and effectively bring AI into use in healthcare, there needs to be a concerted effort around not just the creation, but also the delivery of AI. This AI "delivery science" will require a broader set of tools, such as design thinking, process improvement, and implementation science, as well as a broader definition of what AI will look like in practice, which includes not just machine learning models and their predictions, but also the new systems for care delivery that they enable. The careful design, implementation, and evaluation of these AI enabled systems will be important in the effort to understand how AI can improve healthcare.

  View details for DOI 10.1038/s41746-020-00318-y

  View details for PubMedID 32885053

  View details for PubMedCentralID PMC7443141

• When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record. BMJ quality & safety Li, R. C., Wang, J. K., Sharp, C. n., Chen, J. H. 2019

  Abstract

  Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs.We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set.There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%-18%) and median medication retraction rate was 4% (IQR 2%-10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte.Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow.

  View details for DOI 10.1136/bmjqs-2018-008968

  View details for PubMedID 31164486

• Impact of problem-based charting on the utilization and accuracy of the electronic problem list JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION Li, R. C., Garg, T., Cun, T., Shieh, L., Krishnan, G., Fang, D., Chen, J. H. 2018; 25 (5): 548–54

  View details for DOI 10.1093/jamia/ocx154

  View details for Web of Science ID 000434113200012

• The Impact of Big Data on the Physician GUIDE TO BIG DATA APPLICATIONS Le, E., Iyer, S., Patil, T., Li, R., Chen, J. H., Wang, M., Sobel, E., Srinivasan, S. 2018; 26: 415–48

  View details for DOI 10.1007/978-3-319-53817-4_16

  View details for Web of Science ID 000434632800018

• Discordance Between Apolipoprotein B and LDL-Cholesterol in Young Adults Predicts Coronary Artery Calcification The CARDIA Study JOURNAL OF THE AMERICAN COLLEGE OF CARDIOLOGY Wilkins, J. T., Li, R. C., Sniderman, A., Chan, C., Lloyd-Jones, D. M. 2016; 67 (2): 193–201

  Abstract

  High levels of apolipoprotein B (apoB) have been shown to predict atherosclerotic cardiovascular disease (CVD) in adults even in the context of low levels of low-density lipoprotein cholesterol (LDL-C) or non-high-density lipoprotein cholesterol (non-HDL-C).This study aimed to quantify the associations between apoB and the discordance between apoB and LDL-C or non-HDL-C in young adults and measured coronary artery calcium (CAC) in midlife.Data were derived from a multicenter cohort study of young adults recruited at ages 18 to 30 years. All participants with complete baseline CVD risk factor data, including apoB and year 25 (Y25) CAC score, were entered into this study. Presence of CAC was defined as having a positive, nonzero Agatston score as determined by computed tomography. Baseline apoB values were divided into tertiles of 4 mutually exclusive concordant/discordant groups, based on median apoB and LDL-C or non-HDL-C.Analysis included 2,794 participants (mean age: 25 ± 3.6 years; body mass index: 24.5 ± 5 kg/m(2); and 44.4% male). Mean lipid values were as follows: total cholesterol: 177.3 ± 33.1 mg/dl; LDL-C: 109.9 ± 31.1 mg/dl; non-HDL-C: 124.0 ± 33.5 mg/dl; HDL-C: 53 ± 12.8 mg/dl; and apoB: 90.7 ± 24 mg/dl; median triglycerides were 61 mg/dl. Compared with the lowest apoB tertile, higher odds of developing Y25 CAC were seen in the middle (odds ratio [OR]: 1.53) and high (OR: 2.28) tertiles based on traditional risk factor-adjusted models. High apoB and low LDL-C or non-HDL-C discordance was also associated with Y25 CAC in adjusted models (OR: 1.55 and OR: 1.45, respectively).These data suggest a dose-response association between apoB in young adults and the presence of midlife CAC independent of baseline traditional CVD risk factors.

  View details for DOI 10.1016/j.jacc.2015.10.055

  View details for Web of Science ID 000368114400011

  View details for PubMedID 26791067

• Validation of Test Performance and Clinical Time Zero for an Electronic Health Record Embedded Severe Sepsis Alert. Applied clinical informatics Rolnick, J., Downing, N. L., Shepard, J., Chu, W., Tam, J., Wessels, A., Li, R., Dietrich, B., Rudy, M., Castaneda, L., Shieh, L. 2016; 7 (2): 560-572

  Abstract

  Increasing use of EHRs has generated interest in the potential of computerized clinical decision support to improve treatment of sepsis. Electronic sepsis alerts have had mixed results due to poor test characteristics, the inability to detect sepsis in a timely fashion and the use of outside software limiting widespread adoption. We describe the development, evaluation and validation of an accurate and timely severe sepsis alert with the potential to impact sepsis management.To develop, evaluate, and validate an accurate and timely severe sepsis alert embedded in a commercial EHR.The sepsis alert was developed by identifying the most common severe sepsis criteria among a cohort of patients with ICD 9 codes indicating a diagnosis of sepsis. This alert requires criteria in three categories: indicators of a systemic inflammatory response, evidence of suspected infection from physician orders, and markers of organ dysfunction. Chart review was used to evaluate test performance and the ability to detect clinical time zero, the point in time when a patient develops severe sepsis.Two physicians reviewed 100 positive cases and 75 negative cases. Based on this review, sensitivity was 74.5%, specificity was 86.0%, the positive predictive value was 50.3%, and the negative predictive value was 94.7%. The most common source of end-organ dysfunction was MAP less than 70 mm/Hg (59%). The alert was triggered at clinical time zero in 41% of cases and within three hours in 53.6% of cases. 96% of alerts triggered before a manual nurse screen.We are the first to report the time between a sepsis alert and physician chart-review clinical time zero. Incorporating physician orders in the alert criteria improves specificity while maintaining sensitivity, which is important to reduce alert fatigue. By leveraging standard EHR functionality, this alert could be implemented by other healthcare systems.

  View details for DOI 10.4338/ACI-2015-11-RA-0159

  View details for PubMedID 27437061

  View details for PubMedCentralID PMC4941860

• Psoriasis is associated with decreased plasma adiponectin levels beyond cardiometabolic risk factors Clinical and Experimental Dermatology Li, R. C., Krishnamoorthy, P., DerOhannessian, S., Doveikis, J., Wilcox, M., Rader, D. J., Reilly, M. P., Van Voorhees, A., Gelfand, J. M., Mehta , N. N. 2014; 39 (1)
• Dietary vitamin K intake and anticoagulation control during the initiation phase of warfarin therapy: A prospective cohort study THROMBOSIS AND HAEMOSTASIS Liu, R. C., Finkelman, B. S., Chen, J., Booth, S. L., Bershaw, L., Brensinger, C., Kimmel, S. E. 2013; 110 (1): 195–96

  View details for DOI 10.1160/TH13-02-0111

  View details for Web of Science ID 000321475800027

  View details for PubMedID 23572189

  View details for PubMedCentralID PMC4415268

• Abnormal lipoprotein particles and cholesterol efflux capacity in patients with psoriasis Atherosclerosis Mehta, N. N., Li, R. C., Krishanmoorthy, P., Farver, W., Rodrigues, A., Raper, A., Wilcox, M., Baer, A., DerOhannessian, S., Wolfe, S., Reilly, M. P., Rader, D. J., Van Voorhees, A., Gelfand, J. M. 2012; 224 (1)