Keith Morse, MD, MBA, is a pediatric hospitalist and Medical Director of Clinical Informatics at Stanford Medicine. His work in operational and research informatics focuses on meaningful deployment of machine learning in clinical settings. He serves as Stanford's co-site PI for participation in PEDSnet, an 11-site pediatric research consortium. His academic roles include Associate Program Director for Stanford's Clinical Informatics fellowship.
Clinical Assistant Professor, Pediatrics
Medical Director of Clinical Informatics, Stanford Children's Health (2020 - Present)
Associate Program Director, Clinical Informatics Fellowship, Stanford Medicine (2022 - Present)
Medical Education: Sidney Kimmel Medical College Thomas Jefferson University (2015) PA
Fellowship: Stanford Hospital and Clinics (2020) CA
Board Certification: American Board of Pediatrics, Pediatrics (2018)
Residency: Phoenix Children's Hospital Pediatric Residency (2018) AZ
Fellowship, Stanford University, Clinical Informatics (2020)
Residency, Phoenix Children's Hospital, Pediatrics (2018)
MD, Jefferson Medical College (2015)
MBA, Washington University in St. Louis (2009)
- Applied Clinical Informatics Seminar
BIOMEDIN 208 (Spr)
Prior Year Courses
- Applied Clinical Informatics Seminar
BIOMEDIN 208 (Spr)
- Applied Clinical Informatics Seminar
BIOMEDIN 208 (Spr)
- Applied Clinical Informatics Seminar
A Natural Language Processing Model to Identify Confidential Content in Adolescent Clinical Notes.
Applied clinical informatics
BACKGROUND: The 21st Century Cures Act mandates the immediate, electronic release of health information to patients. However, in the case of adolescents, special consideration is required to ensure that confidentiality is maintained. The detection of confidential content in clinical notes may support operational efforts to preserve adolescent confidentiality while implementing information sharing.OBJECTIVE: Determine if a natural language processing (NLP) algorithm can identify confidential content in adolescent clinical progress notes.METHODS: 1,200 outpatient adolescent progress notes written between 2016 and 2019 were manually annotated to identify confidential content. Labeled sentences from this corpus were featurized and used to train a two-part logistic regression model, which provides both sentence-level and note-level probability estimates that a given text contains confidential content. This model was prospectively validated on a set of 240 progress notes written in May 2022. It was subsequently deployed in a pilot intervention to augment an ongoing operational effort to identify confidential content in progress notes. Note-level probability estimates were used to triage notes for review and sentence-level probability estimates were used to highlight high-risk portions of those notes to aid the manual reviewer.RESULTS: The prevalence of notes containing confidential content was 21% (255/1200) and 22% (53/240) in the train/test and validation cohorts. The ensemble logistic regression model achieved an AUROC of 90% and 88% in the test and validation cohorts. Its use in a pilot intervention identified outlier documentation practices and demonstrated efficiency gains over completely manual note review.DISCUSSION: An NLP algorithm can identify confidential content in progress notes with high accuracy. Its human-in-the-loop deployment in clinical operations augmented an ongoing operational effort to identify confidential content in adolescent progress notes. These findings suggest NLP may be used to support efforts to preserve adolescent confidentiality in the wake of the information blocking mandate.
View details for DOI 10.1055/a-2051-9764
View details for PubMedID 36898410
A machine learning-based phenotype for long COVID in children: an EHR-based study from the RECOVER program.
medRxiv : the preprint server for health sciences
Background: As clinical understanding of pediatric Post-Acute Sequelae of SARS CoV-2 (PASC) develops, and hence the clinical definition evolves, it is desirable to have a method to reliably identify patients who are likely to have post-acute sequelae of SARS CoV-2 (PASC) in health systems data.Methods and Findings: In this study, we developed and validated a machine learning algorithm to classify which patients have PASC (distinguishing between Multisystem Inflammatory Syndrome in Children (MIS-C) and non-MIS-C variants) from a cohort of patients with positive SARS-CoV-2 test results in pediatric health systems within the PEDSnet EHR network. Patient features included in the model were selected from conditions, procedures, performance of diagnostic testing, and medications using a tree-based scan statistic approach. We used an XGboost model, with hyperparameters selected through cross-validated grid search, and model performance was assessed using 5-fold cross-validation. Model predictions and feature importance were evaluated using Shapley Additive exPlanation (SHAP) values.Conclusions: The model provides a tool for identifying patients with PASC and an approach to characterizing PASC using diagnosis, medication, laboratory, and procedure features in health systems data. Using appropriate threshold settings, the model can be used to identify PASC patients in health systems data at higher precision for inclusion in studies or at higher recall in screening for clinical trials, especially in settings where PASC diagnosis codes are used less frequently or less reliably. Analysis of how specific features contribute to the classification process may assist in gaining a better understanding of features that are associated with PASC diagnoses.Funding Source: This research was funded by the National Institutes of Health (NIH) Agreement OT2HL161847-01 as part of the Researching COVID to Enhance Recovery (RECOVER) program of research.Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the RECOVER Program, the NIH or other funders.
View details for DOI 10.1101/2022.12.22.22283791
View details for PubMedID 36597534
User-centred design for machine learning in health care: a case study from care management.
BMJ health & care informatics
2022; 29 (1)
OBJECTIVES: Few machine learning (ML) models are successfully deployed in clinical practice. One of the common pitfalls across the field is inappropriate problem formulation: designing ML to fit the data rather than to address a real-world clinical pain point.METHODS: We introduce a practical toolkit for user-centred design consisting of four questions covering: (1) solvable pain points, (2) the unique value of ML (eg, automation and augmentation), (3) the actionability pathway and (4) the model's reward function. This toolkit was implemented in a series of six participatory design workshops with care managers in an academic medical centre.RESULTS: Pain points amenable to ML solutions included outpatient risk stratification and risk factor identification. The endpoint definitions, triggering frequency and evaluation metrics of the proposed risk scoring model were directly influenced by care manager workflows and real-world constraints.CONCLUSIONS: Integrating user-centred design early in the ML life cycle is key for configuring models in a clinically actionable way. This toolkit can guide problem selection and influence choices about the technical setup of the ML problem.
View details for DOI 10.1136/bmjhci-2022-100656
View details for PubMedID 36220304
Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review.
JAMA network open
2022; 5 (8): e2227779
Importance: Various model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied.Objectives: To assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested.Evidence Review: MEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items.Findings: From 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range, 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex).Conclusions and Relevance: These findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.
View details for DOI 10.1001/jamanetworkopen.2022.27779
View details for PubMedID 35984654
Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.
Applied clinical informatics
2022; 13 (2): 431-438
OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital.METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes.RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p=0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC=0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings.CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
View details for DOI 10.1055/s-0042-1746168
View details for PubMedID 35508197
Ensuring Adolescent Patient Portal Confidentiality in the Age of the Cures Act Final Rule.
The Journal of adolescent health : official publication of the Society for Adolescent Medicine
PURPOSE: Managing confidential adolescent health information in patient portals presents unique challenges. Adolescent patients and guardians electronically access medical records and communicate with providers via portals. In confidential matters like sexual health, ensuring confidentiality is crucial. A key aspect of confidential portals is ensuring that the account is registered to and utilized by the intended user. Inappropriately registered or guardian-accessed adolescent portal accounts may lead to confidentiality breaches.METHODS: We used a quality improvement framework to develop screening methodologies to flag guardian-accessible accounts. Accounts of patients aged 12-17 were flagged via manual review of account emails and natural language processing of portal messages. We implemented a reconciliation program to correct affected accounts' registered email. Clinics were notified about sign-up errors and educated on sign-up workflow. An electronic alert was created to check the adolescent's email prior to account activation.RESULTS: After initial screening, 2,307 of 3,701 (62%) adolescent accounts were flagged as registered with a guardian's email. Those accounts were notified to resolve their logins. After five notifications over 8 weeks, 266 of 2,307 accounts (12%) were corrected; the remaining 2,041 (88%) were deactivated.CONCLUSIONS: The finding that 62% of adolescent portal accounts were used/accessed by guardians has significant confidentiality implications. In the context of the Cures Act Final Rule and increased information sharing, our institution's experience with ensuring appropriate access to adolescent portal accounts is necessary, timely, and relevant. This study highlights ways to improve patient portal confidentiality and prompts institutions caring for adolescents to review their systems and processes.
View details for DOI 10.1016/j.jadohealth.2021.09.009
View details for PubMedID 34666956
Assessment of Prevalence of Adolescent Patient Portal Account Access by Guardians.
JAMA network open
2021; 4 (9): e2124733
Importance: Patient portals can be configured to allow confidential communication for adolescents' sensitive health care information. Guardian access of adolescent patient portal accounts could compromise adolescents' confidentiality.Objective: To estimate the prevalence of guardian access to adolescent patient portals at 3 academic children's hospitals.Design, Setting, and Participants: A cross-sectional study to estimate the prevalence of guardian access to adolescent patient portal accounts was conducted at 3 academic children's hospitals. Adolescent patients (aged 13-18 years) with access to their patient portal account with at least 1 outbound message from their portal during the study period were included. A rule-based natural language processing algorithm was used to analyze all portal messages from June 1, 2014, to February 28, 2020, and identify any message sent by guardians. The sensitivity and specificity of the algorithm at each institution was estimated through manual review of a stratified subsample of patient accounts. The overall proportion of accounts with guardian access was estimated after correcting for the sensitivity and specificity of the natural language processing algorithm.Exposures: Use of patient portal.Main Outcome and Measures: Percentage of adolescent portal accounts indicating guardian access.Results: A total of 3429 eligible adolescent accounts containing 25 642 messages across 3 institutions were analyzed. A total of 1797 adolescents (52%) were female and mean (SD) age was 15.6 (1.6) years. The percentage of adolescent portal accounts with apparent guardian access ranged from 52% to 57% across the 3 institutions. After correcting for the sensitivity and specificity of the algorithm based on manual review of 200 accounts per institution, an estimated 64% (95% CI, 59%-69%) to 76% (95% CI, 73%-88%) of accounts with outbound messages were accessed by guardians across the 3 institutions.Conclusions and Relevance: In this study, more than half of adolescent accounts with outbound messages were estimated to have been accessed by guardians at least once. These findings have implications for health systems intending to rely on separate adolescent accounts to protect adolescent confidentiality.
View details for DOI 10.1001/jamanetworkopen.2021.24733
View details for PubMedID 34529064
A survey of extant organizational and computational setups for deploying predictive models in health systems.
Journal of the American Medical Informatics Association : JAMIA
OBJECTIVE: Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs.MATERIALS AND METHODS: We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care.RESULTS: We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n=10, 50%), IT Department led (n=2, 10%), and AI in Healthcare (AIHC) Team (n=8, 40%).DISCUSSION: No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS.CONCLUSION: Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.
View details for DOI 10.1093/jamia/ocab154
View details for PubMedID 34423364
Quantifying Discharge Medication Reconciliation Errors at 2 Pediatric Hospitals.
Pediatric quality & safety
2021; 6 (4): e436
Introduction: Medication reconciliation errors (MREs) are common and can lead to significant patient harm. Quality improvement efforts to identify and reduce these errors typically rely on resource-intensive chart reviews or adverse event reporting. Quantifying these errors hospital-wide is complicated and rarely done. The purpose of this study is to define a set of 6 MREs that can be easily identified across an entire healthcare organization and report their prevalence at 2 pediatric hospitals.Methods: An algorithmic analysis of discharge medication lists and confirmation by clinician reviewers was used to find the prevalence of the 6 discharge MREs at 2 pediatric hospitals. These errors represent deviations from the standards for medication instruction completeness, clarity, and safety. The 6 error types are Duplication, Missing Route, Missing Dose, Missing Frequency, Unlisted Medication, and See Instructions errors.Results: This study analyzed 67,339 discharge medications and detected MREs commonly at both hospitals. For Institution A, a total of 4,234 errors were identified, with 29.9% of discharges containing at least one error and an average of 0.7 errors per discharge. For Institution B, a total of 5,942 errors were identified, with 42.2% of discharges containing at least 1 error and an average of 1.6 errors per discharge. The most common error types were Duplication and See Instructions errors.Conclusion: The presented method shows these MREs to be a common finding in pediatric care. This work offers a tool to strengthen hospital-wide quality improvement efforts to reduce pediatric medication errors.
View details for DOI 10.1097/pq9.0000000000000436
View details for PubMedID 34345749
Digital Symptom Checker Usage and Triage: Population-Based Descriptive Study in a Large North American Integrated Health System.
Journal of medical Internet research
BACKGROUND: Pressure on the United States (US) healthcare system has been increasing due to a combination of aging populations, rising healthcare expenditures and, most recently, the COVID-19 pandemic. Responses are hindered in part by a reliance on a limited supply of highly trained healthcare professionals, creating a need for scalable technological solutions. Digital symptom checkers are artificial intelligence (AI)-supported software tools that use a conversational "chatbot" format to support rapid diagnosis and consistent triage. The COVID-19 pandemic has brought new attention to these tools, with the need to avoid face-to-face contact and preserve urgent care capacity. However, evidence-based deployment of these chatbots requires an understanding of user demographics and associated triage recommendations generated by a large, general population.OBJECTIVE: In this study we evaluate the user demographics and levels of triage acuity provided by one symptom checker chatbot deployed in partnership with a large integrated health system in the US.METHODS: Population-based descriptive study including all online symptom assessments completed on the website and patient portal of the Sutter Health system (24 hospitals in Northern California) from April 24th, 2019 to February 1st, 2020. User demographics were compared to relevant US Census population data.RESULTS: A total of 26,646 symptom assessments were completed during the study period. Most assessments (17,816/26,646, 66.9%) were completed by female users. Mean user age was 34.3 years (SD: 14.4 years), compared to a median age of 37.3 years of the general population. The most common initial symptom was 'abdominal pain' (2,060/26,646, 7.7%). A substantial portion (12,357/26,646, 46.4%) was completed outside of typical physician office hours. Most users were advised to seek medical care the same day (7,299/26,646, 27.4%) or within 2-3 days (6,301/26,646, 23.6%). Over one quarter of assessments required a high degree of urgency (7,723/26,646, 29.0%).CONCLUSIONS: Users of the symptom checker chatbot were broadly representative of our patient population, though skewed towards younger and female users. Triage recommendations are comparable to those of nurse-staffed phone triage lines. While the emergence of COVID-19 increases the enthusiasm for remote medical assessment tools, it is important to take an evidence-based approach to their deployment.CLINICALTRIAL:
View details for DOI 10.2196/20549
View details for PubMedID 33170799
Estimate the hidden deployment cost of predictive models to improve patient care.
2020; 26 (1): 18–19
View details for DOI 10.1038/s41591-019-0651-8
View details for PubMedID 31932778
Your Patient Has a New Health App? Start With Its Data Source.
Journal of participatory medicine
2019; 11 (2): e14288
Recent regulatory and technological advances have enabled a new era of health apps that are controlled by patients and contain valuable health information. These health apps will be numerous and use novel interfaces that appeal to patients but will likely be unfamiliar to practitioners. We posit that understanding the origin of the health data is the most meaningful and versatile way for physicians to understand and effectively use these apps in patient care. This will allow providers to better support patients and encourage patient engagement in their own care.
View details for DOI 10.2196/14288
View details for PubMedID 33055064
View details for PubMedCentralID PMC7434101
Hospital-Level Variation in Practice Patterns and Patient Outcomes for Pediatric Patients Hospitalized With Functional Constipation.
2017; 7 (6): 320-327
Constipation is a common pediatric condition with a prevalence of 3% to 5% in children aged 4 to 17 years. Currently, there are no evidence-based guidelines for the management of pediatric patients hospitalized with constipation. The primary objective was to evaluate practice patterns and patient outcomes for the hospital management of functional constipation in US children's hospitals.We conducted a multicenter, retrospective cohort study of children aged 0 to 18 years hospitalized for functional constipation from 2012 to 2014 by using the Pediatric Health Information System. Patients were included by using constipation and other related diagnoses as classified by International Classification of Diseases, Ninth Revision. Patients with complex chronic conditions were excluded. Outcome measures included percentage of hospitalizations due to functional constipation, therapies used, length of stay, and 90-day readmission rates. Statistical analysis included means with 95% confidence intervals for individual hospital outcomes.A total of 14 243 hospitalizations were included, representing 12 804 unique patients. The overall percentage of hospitalizations due to functional constipation was 0.65% (range: 0.19%-1.41%, P < .0001). The percentage of patients receiving the following treatment during their hospitalization included: electrolyte laxatives: 40% to 96%; sodium phosphate enema: 0% to 64%; mineral oil enema: 0% to 61%; glycerin suppository: 0% to 37%; bisacodyl 0% to 47%; senna: 0% to 23%; and docusate 0% to 11%. Mean length of stay was 1.97 days (range: 1.31-2.73 days, P < .0001). Mean 90-day readmission rate was 3.78% (range: 0.95%-7.53%, P < .0001).There is significant variation in practice patterns and clinical outcomes for pediatric patients hospitalized with functional constipation across US children's hospitals. Collaborative initiatives to adopt evidence-based best practices guidelines could help standardize the hospital management of pediatric functional constipation.
View details for DOI 10.1542/hpeds.2016-0101
View details for PubMedID 28522604