Michael Wornow

All Publications

• Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A. R., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A., Pfeffer, M. A., Shah, N. H. 2024

  Abstract

  Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas.To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty.A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024.Studies evaluating 1 or more LLMs in health care.Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty.Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented.Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.

  View details for DOI 10.1001/jama.2024.21700

  View details for PubMedID 39405325

  View details for PubMedCentralID PMC11480901

• The shaky foundations of large language models and foundation models for electronic health records. NPJ digital medicine Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., Shah, N. H. 2023; 6 (1): 135

  Abstract

  The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.

  View details for DOI 10.1038/s41746-023-00879-8

  View details for PubMedID 37516790

  View details for PubMedCentralID 8371605

• APLUS: A Python Library for Usefulness Simulations of Machine Learning Models in Healthcare. Journal of biomedical informatics Wornow, M., Gyang Ross, E., Callahan, A., Shah, N. H. 2023: 104319

  Abstract

  Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

  View details for DOI 10.1016/j.jbi.2023.104319

  View details for PubMedID 36791900

• EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models Wornow, M., Thapa, R., Steinberg, E., Fries, J. A., Shah, N. H., Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2023

  View details for Web of Science ID 001220818804011

• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution Nguyen, E., Poli, M., Faizi, M., Thomas, A. W., Sykes, C., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., Ermon, S., Baccus, S. A., Re, C., Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2023

  View details for Web of Science ID 001224281505021

• Generating experimentally unrelated target molecule-binding highly functionalized nucleic-acid polymers using machine learning NATURE COMMUNICATIONS Chen, J. C., Chen, J. P., Shen, M. W., Wornow, M., Bae, M., Yeh, W., Hsu, A., Liu, D. R. 2022; 13 (1): 4541

  Abstract

  In vitro selection queries large combinatorial libraries for sequence-defined polymers with target binding and reaction catalysis activity. While the total sequence space of these libraries can extend beyond 1022 sequences, practical considerations limit starting sequences to ≤~1015 distinct molecules. Selection-induced sequence convergence and limited sequencing depth further constrain experimentally observable sequence space. To address these limitations, we integrate experimental and machine learning approaches to explore regions of sequence space unrelated to experimentally derived variants. We perform in vitro selections to discover highly side-chain-functionalized nucleic acid polymers (HFNAPs) with potent affinities for a target small molecule (daunomycin KD = 5-65 nM). We then use the selection data to train a conditional variational autoencoder (CVAE) machine learning model to generate diverse and unique HFNAP sequences with high daunomycin affinities (KD = 9-26 nM), even though they are unrelated in sequence to experimental polymers. Coupling in vitro selection with a machine learning model thus enables direct generation of active variants, demonstrating a new approach to the discovery of functional biopolymers.

  View details for DOI 10.1038/s41467-022-31955-4

  View details for Web of Science ID 000836424200030

  View details for PubMedID 35927274

  View details for PubMedCentralID PMC9352670

• Construction of disease-specific cytokine profiles by associating disease genes with immune responses. PLoS computational biology Liu, T., Wang, S., Wornow, M., Altman, R. B. 2022; 18 (4): e1009497

  Abstract

  The pathogenesis of many inflammatory diseases is a coordinated process involving metabolic dysfunctions and immune response-usually modulated by the production of cytokines and associated inflammatory molecules. In this work, we seek to understand how genes involved in pathogenesis which are often not associated with the immune system in an obvious way communicate with the immune system. We have embedded a network of human protein-protein interactions (PPI) from the STRING database with 14,707 human genes using feature learning that captures high confidence edges. We have found that our predicted Association Scores derived from the features extracted from STRING's high confidence edges are useful for predicting novel connections between genes, thus enabling the construction of a full map of predicted associations for all possible pairs between 14,707 human genes. In particular, we analyzed the pattern of associations for 126 cytokines and found that the six patterns of cytokine interaction with human genes are consistent with their functional classifications. To define the disease-specific roles of cytokines we have collected gene sets for 11,944 diseases from DisGeNET. We used these gene sets to predict disease-specific gene associations with cytokines by calculating the normalized average Association Scores between disease-associated gene sets and the 126 cytokines; this creates a unique profile of inflammatory genes (both known and predicted) for each disease. We validated our predicted cytokine associations by comparing them to known associations for 171 diseases. The predicted cytokine profiles correlate (p-value<0.0003) with the known ones in 95 diseases. We further characterized the profiles of each disease by calculating an "Inflammation Score" that summarizes different modes of immune responses. Finally, by analyzing subnetworks formed between disease-specific pathogenesis genes, hormones, receptors, and cytokines, we identified the key genes responsible for interactions between pathogenesis and inflammatory responses. These genes and the corresponding cytokines used by different immune disorders suggest unique targets for drug discovery.

  View details for DOI 10.1371/journal.pcbi.1009497

  View details for PubMedID 35404985

• Interregional Transfers for Pandemic Surges. Clinical infectious diseases : an official publication of the Infectious Diseases Society of America Michelson, K. A., Rees, C. A., Sarathy, J., VonAchen, P., Wornow, M., Monuteaux, M. C., Neuman, M. I. 2021; 73 (11): e4103-e4110

  Abstract

  BACKGROUND: Hospital inpatient and intensive care unit (ICU) bed shortfalls may arise due to regional surges in volume. We sought to determine how interregional transfers could alleviate bed shortfalls during a pandemic.METHODS: We used estimates of past and projected inpatient and ICU cases of coronavirus disease 2019 (COVID-19) from 4 February 2020 to 1 October 2020. For regions with bed shortfalls (where the number of patients exceeded bed capacity), transfers to the nearest region with unused beds were simulated using an algorithm that minimized total interregional transfer distances across the United States. Model scenarios used a range of predicted COVID-19 volumes (lower, mean, and upper bounds) and non-COVID-19 volumes (20%, 50%, or 80% of baseline hospital volumes). Scenarios were created for each day of data, and worst-case scenarios were created treating all regions' peak volumes as simultaneous. Mean per-patient transfer distances were calculated by scenario.RESULTS: For the worst-case scenarios, national bed shortfalls ranged from 669 to 58 562 inpatient beds and 3208 to 31 190 ICU beds, depending on model volume parameters. Mean transfer distances to alleviate daily bed shortfalls ranged from 23 to 352 miles for inpatient and 28 to 423 miles for ICU patients, depending on volume. Under all worst-case scenarios except the highest-volume ICU scenario, interregional transfers could fully resolve bed shortfalls. To do so, mean transfer distances would be 24 to 405 miles for inpatients and 73 to 476 miles for ICU patients.CONCLUSIONS: Interregional transfers could mitigate regional bed shortfalls during pandemic hospital surges.

  View details for DOI 10.1093/cid/ciaa1549

  View details for PubMedID 33038215

• Cut out the annotator, keep the cutout: better segmentation with weak supervision Hooper, S., Wornow, M., Seah, Y., Kellman, P., Xue, H., Sala, F., Langlotz, C., Re, C. 2021
• In vivo base editing restores sensory transduction and transiently improves auditory function in a mouse model of recessive deafness SCIENCE TRANSLATIONAL MEDICINE Yeh, W., Shubina-Oleinik, O., Levy, J. M., Pan, B., Newby, G. A., Wornow, M., Burt, R., Chen, J. C., Holt, J. R., Liu, D. R. 2020; 12 (546)

  Abstract

  Most genetic diseases arise from recessive point mutations that require correction, rather than disruption, of the pathogenic allele to benefit patients. Base editing has the potential to directly repair point mutations and provide therapeutic restoration of gene function. Mutations of transmembrane channel-like 1 gene (TMC1) can cause dominant or recessive deafness. We developed a base editing strategy to treat Baringo mice, which carry a recessive, loss-of-function point mutation (c.A545G; resulting in the substitution p.Y182C) in Tmc1 that causes deafness. Tmc1 encodes a protein that forms mechanosensitive ion channels in sensory hair cells of the inner ear and is required for normal auditory function. We found that sensory hair cells of Baringo mice have a complete loss of auditory sensory transduction. To repair the mutation, we tested several optimized cytosine base editors (CBEmax variants) and guide RNAs in Baringo mouse embryonic fibroblasts. We packaged the most promising CBE, derived from an activation-induced cytidine deaminase (AID), into dual adeno-associated viruses (AAVs) using a split-intein delivery system. The dual AID-CBEmax AAVs were injected into the inner ears of Baringo mice at postnatal day 1. Injected mice showed up to 51% reversion of the Tmc1 c.A545G point mutation to wild-type sequence (c.A545A) in Tmc1 transcripts. Repair of Tmc1 in vivo restored inner hair cell sensory transduction and hair cell morphology and transiently rescued low-frequency hearing 4 weeks after injection. These findings provide a foundation for a potential one-time treatment for recessive hearing loss and support further development of base editing to correct pathogenic point mutations.

  View details for DOI 10.1126/scitranslmed.aay9101

  View details for Web of Science ID 000539151300002

  View details for PubMedID 32493795

  View details for PubMedCentralID PMC8167884