Keith Morse, MD, MBA, is a pediatric hospitalist and Medical Director of Clinical Informatics at Stanford Medicine. His work in operational and research informatics focuses on meaningful deployment of machine learning in clinical settings. He completed a fellowship in Clinical Informatics at Stanford.
Clinical Assistant Professor, Pediatrics
Medical Director of Clinical Informatics, Stanford Children's Health (2020 - Present)
Fellowship: Stanford Hospital and Clinics (2020) CA
Board Certification: American Board of Pediatrics, Pediatrics (2018)
Residency: Phoenix Children's Hospital Pediatric Residency (2018) AZ
Medical Education: Jefferson Medical College (2015) PA
Fellowship, Stanford University, Clinical Informatics (2020)
Residency, Phoenix Children's Hospital, Pediatrics (2018)
MD, Jefferson Medical College (2015)
MBA, Washington University in St. Louis (2009)
Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.
Applied clinical informatics
2022; 13 (2): 431-438
OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital.METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes.RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p=0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC=0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings.CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.
View details for DOI 10.1055/s-0042-1746168
View details for PubMedID 35508197
Ensuring Adolescent Patient Portal Confidentiality in the Age of the Cures Act Final Rule.
The Journal of adolescent health : official publication of the Society for Adolescent Medicine
PURPOSE: Managing confidential adolescent health information in patient portals presents unique challenges. Adolescent patients and guardians electronically access medical records and communicate with providers via portals. In confidential matters like sexual health, ensuring confidentiality is crucial. A key aspect of confidential portals is ensuring that the account is registered to and utilized by the intended user. Inappropriately registered or guardian-accessed adolescent portal accounts may lead to confidentiality breaches.METHODS: We used a quality improvement framework to develop screening methodologies to flag guardian-accessible accounts. Accounts of patients aged 12-17 were flagged via manual review of account emails and natural language processing of portal messages. We implemented a reconciliation program to correct affected accounts' registered email. Clinics were notified about sign-up errors and educated on sign-up workflow. An electronic alert was created to check the adolescent's email prior to account activation.RESULTS: After initial screening, 2,307 of 3,701 (62%) adolescent accounts were flagged as registered with a guardian's email. Those accounts were notified to resolve their logins. After five notifications over 8 weeks, 266 of 2,307 accounts (12%) were corrected; the remaining 2,041 (88%) were deactivated.CONCLUSIONS: The finding that 62% of adolescent portal accounts were used/accessed by guardians has significant confidentiality implications. In the context of the Cures Act Final Rule and increased information sharing, our institution's experience with ensuring appropriate access to adolescent portal accounts is necessary, timely, and relevant. This study highlights ways to improve patient portal confidentiality and prompts institutions caring for adolescents to review their systems and processes.
View details for DOI 10.1016/j.jadohealth.2021.09.009
View details for PubMedID 34666956
Assessment of Prevalence of Adolescent Patient Portal Account Access by Guardians.
JAMA network open
2021; 4 (9): e2124733
Importance: Patient portals can be configured to allow confidential communication for adolescents' sensitive health care information. Guardian access of adolescent patient portal accounts could compromise adolescents' confidentiality.Objective: To estimate the prevalence of guardian access to adolescent patient portals at 3 academic children's hospitals.Design, Setting, and Participants: A cross-sectional study to estimate the prevalence of guardian access to adolescent patient portal accounts was conducted at 3 academic children's hospitals. Adolescent patients (aged 13-18 years) with access to their patient portal account with at least 1 outbound message from their portal during the study period were included. A rule-based natural language processing algorithm was used to analyze all portal messages from June 1, 2014, to February 28, 2020, and identify any message sent by guardians. The sensitivity and specificity of the algorithm at each institution was estimated through manual review of a stratified subsample of patient accounts. The overall proportion of accounts with guardian access was estimated after correcting for the sensitivity and specificity of the natural language processing algorithm.Exposures: Use of patient portal.Main Outcome and Measures: Percentage of adolescent portal accounts indicating guardian access.Results: A total of 3429 eligible adolescent accounts containing 25 642 messages across 3 institutions were analyzed. A total of 1797 adolescents (52%) were female and mean (SD) age was 15.6 (1.6) years. The percentage of adolescent portal accounts with apparent guardian access ranged from 52% to 57% across the 3 institutions. After correcting for the sensitivity and specificity of the algorithm based on manual review of 200 accounts per institution, an estimated 64% (95% CI, 59%-69%) to 76% (95% CI, 73%-88%) of accounts with outbound messages were accessed by guardians across the 3 institutions.Conclusions and Relevance: In this study, more than half of adolescent accounts with outbound messages were estimated to have been accessed by guardians at least once. These findings have implications for health systems intending to rely on separate adolescent accounts to protect adolescent confidentiality.
View details for DOI 10.1001/jamanetworkopen.2021.24733
View details for PubMedID 34529064
A survey of extant organizational and computational setups for deploying predictive models in health systems.
Journal of the American Medical Informatics Association : JAMIA
OBJECTIVE: Artificial intelligence (AI) and machine learning (ML) enabled healthcare is now feasible for many health systems, yet little is known about effective strategies of system architecture and governance mechanisms for implementation. Our objective was to identify the different computational and organizational setups that early-adopter health systems have utilized to integrate AI/ML clinical decision support (AI-CDS) and scrutinize their trade-offs.MATERIALS AND METHODS: We conducted structured interviews with health systems with AI deployment experience about their organizational and computational setups for deploying AI-CDS at point of care.RESULTS: We contacted 34 health systems and interviewed 20 healthcare sites (58% response rate). Twelve (60%) sites used the native electronic health record vendor configuration for model development and deployment, making it the most common shared infrastructure. Nine (45%) sites used alternative computational configurations which varied significantly. Organizational configurations for managing AI-CDS were distinguished by how they identified model needs, built and implemented models, and were separable into 3 major types: Decentralized translation (n=10, 50%), IT Department led (n=2, 10%), and AI in Healthcare (AIHC) Team (n=8, 40%).DISCUSSION: No singular computational configuration enables all current use cases for AI-CDS. Health systems need to consider their desired applications for AI-CDS and whether investment in extending the off-the-shelf infrastructure is needed. Each organizational setup confers trade-offs for health systems planning strategies to implement AI-CDS.CONCLUSION: Health systems will be able to use this framework to understand strengths and weaknesses of alternative organizational and computational setups when designing their strategy for artificial intelligence.
View details for DOI 10.1093/jamia/ocab154
View details for PubMedID 34423364
Quantifying Discharge Medication Reconciliation Errors at 2 Pediatric Hospitals.
Pediatric quality & safety
2021; 6 (4): e436
Introduction: Medication reconciliation errors (MREs) are common and can lead to significant patient harm. Quality improvement efforts to identify and reduce these errors typically rely on resource-intensive chart reviews or adverse event reporting. Quantifying these errors hospital-wide is complicated and rarely done. The purpose of this study is to define a set of 6 MREs that can be easily identified across an entire healthcare organization and report their prevalence at 2 pediatric hospitals.Methods: An algorithmic analysis of discharge medication lists and confirmation by clinician reviewers was used to find the prevalence of the 6 discharge MREs at 2 pediatric hospitals. These errors represent deviations from the standards for medication instruction completeness, clarity, and safety. The 6 error types are Duplication, Missing Route, Missing Dose, Missing Frequency, Unlisted Medication, and See Instructions errors.Results: This study analyzed 67,339 discharge medications and detected MREs commonly at both hospitals. For Institution A, a total of 4,234 errors were identified, with 29.9% of discharges containing at least one error and an average of 0.7 errors per discharge. For Institution B, a total of 5,942 errors were identified, with 42.2% of discharges containing at least 1 error and an average of 1.6 errors per discharge. The most common error types were Duplication and See Instructions errors.Conclusion: The presented method shows these MREs to be a common finding in pediatric care. This work offers a tool to strengthen hospital-wide quality improvement efforts to reduce pediatric medication errors.
View details for DOI 10.1097/pq9.0000000000000436
View details for PubMedID 34345749
Digital Symptom Checker Usage and Triage: Population-Based Descriptive Study in a Large North American Integrated Health System.
Journal of medical Internet research
BACKGROUND: Pressure on the United States (US) healthcare system has been increasing due to a combination of aging populations, rising healthcare expenditures and, most recently, the COVID-19 pandemic. Responses are hindered in part by a reliance on a limited supply of highly trained healthcare professionals, creating a need for scalable technological solutions. Digital symptom checkers are artificial intelligence (AI)-supported software tools that use a conversational "chatbot" format to support rapid diagnosis and consistent triage. The COVID-19 pandemic has brought new attention to these tools, with the need to avoid face-to-face contact and preserve urgent care capacity. However, evidence-based deployment of these chatbots requires an understanding of user demographics and associated triage recommendations generated by a large, general population.OBJECTIVE: In this study we evaluate the user demographics and levels of triage acuity provided by one symptom checker chatbot deployed in partnership with a large integrated health system in the US.METHODS: Population-based descriptive study including all online symptom assessments completed on the website and patient portal of the Sutter Health system (24 hospitals in Northern California) from April 24th, 2019 to February 1st, 2020. User demographics were compared to relevant US Census population data.RESULTS: A total of 26,646 symptom assessments were completed during the study period. Most assessments (17,816/26,646, 66.9%) were completed by female users. Mean user age was 34.3 years (SD: 14.4 years), compared to a median age of 37.3 years of the general population. The most common initial symptom was 'abdominal pain' (2,060/26,646, 7.7%). A substantial portion (12,357/26,646, 46.4%) was completed outside of typical physician office hours. Most users were advised to seek medical care the same day (7,299/26,646, 27.4%) or within 2-3 days (6,301/26,646, 23.6%). Over one quarter of assessments required a high degree of urgency (7,723/26,646, 29.0%).CONCLUSIONS: Users of the symptom checker chatbot were broadly representative of our patient population, though skewed towards younger and female users. Triage recommendations are comparable to those of nurse-staffed phone triage lines. While the emergence of COVID-19 increases the enthusiasm for remote medical assessment tools, it is important to take an evidence-based approach to their deployment.CLINICALTRIAL:
View details for DOI 10.2196/20549
View details for PubMedID 33170799
- Estimate the hidden deployment cost of predictive models to improve patient care. Nature medicine 2020; 26 (1): 18–19
Your Patient Has a New Health App? Start With Its Data Source.
Journal of participatory medicine
2019; 11 (2): e14288
Recent regulatory and technological advances have enabled a new era of health apps that are controlled by patients and contain valuable health information. These health apps will be numerous and use novel interfaces that appeal to patients but will likely be unfamiliar to practitioners. We posit that understanding the origin of the health data is the most meaningful and versatile way for physicians to understand and effectively use these apps in patient care. This will allow providers to better support patients and encourage patient engagement in their own care.
View details for DOI 10.2196/14288
View details for PubMedID 33055064
View details for PubMedCentralID PMC7434101
Hospital-Level Variation in Practice Patterns and Patient Outcomes for Pediatric Patients Hospitalized With Functional Constipation.
2017; 7 (6): 320-327
Constipation is a common pediatric condition with a prevalence of 3% to 5% in children aged 4 to 17 years. Currently, there are no evidence-based guidelines for the management of pediatric patients hospitalized with constipation. The primary objective was to evaluate practice patterns and patient outcomes for the hospital management of functional constipation in US children's hospitals.We conducted a multicenter, retrospective cohort study of children aged 0 to 18 years hospitalized for functional constipation from 2012 to 2014 by using the Pediatric Health Information System. Patients were included by using constipation and other related diagnoses as classified by International Classification of Diseases, Ninth Revision. Patients with complex chronic conditions were excluded. Outcome measures included percentage of hospitalizations due to functional constipation, therapies used, length of stay, and 90-day readmission rates. Statistical analysis included means with 95% confidence intervals for individual hospital outcomes.A total of 14 243 hospitalizations were included, representing 12 804 unique patients. The overall percentage of hospitalizations due to functional constipation was 0.65% (range: 0.19%-1.41%, P < .0001). The percentage of patients receiving the following treatment during their hospitalization included: electrolyte laxatives: 40% to 96%; sodium phosphate enema: 0% to 64%; mineral oil enema: 0% to 61%; glycerin suppository: 0% to 37%; bisacodyl 0% to 47%; senna: 0% to 23%; and docusate 0% to 11%. Mean length of stay was 1.97 days (range: 1.31-2.73 days, P < .0001). Mean 90-day readmission rate was 3.78% (range: 0.95%-7.53%, P < .0001).There is significant variation in practice patterns and clinical outcomes for pediatric patients hospitalized with functional constipation across US children's hospitals. Collaborative initiatives to adopt evidence-based best practices guidelines could help standardize the hospital management of pediatric functional constipation.
View details for DOI 10.1542/hpeds.2016-0101
View details for PubMedID 28522604