Kathleen Boyd

All Publications

• Almanac - Retrieval-Augmented Language Models for Clinical Medicine. NEJM AI Zakka, C., Shad, R., Chaurasia, A., Dalal, A. R., Kim, J. L., Moor, M., Fong, R., Phillips, C., Alexander, K., Ashley, E., Boyd, J., Boyd, K., Hirsch, K., Langlotz, C., Lee, R., Melia, J., Nelson, J., Sallam, K., Tullis, S., Vogelsong, M. A., Cunningham, J. P., Hiesinger, W. 2024; 1 (2)

  Abstract

  Large language models (LLMs) have recently shown impressive zero-shot capabilities, whereby they can use auxiliary data, without the availability of task-specific training examples, to complete a variety of natural language tasks, such as summarization, dialogue generation, and question answering. However, despite many promising applications of LLMs in clinical medicine, adoption of these models has been limited by their tendency to generate incorrect and sometimes even harmful statements.We tasked a panel of eight board-certified clinicians and two health care practitioners with evaluating Almanac, an LLM framework augmented with retrieval capabilities from curated medical resources for medical guideline and treatment recommendations. The panel compared responses from Almanac and standard LLMs (ChatGPT-4, Bing, and Bard) versus a novel data set of 314 clinical questions spanning nine medical specialties.Almanac showed a significant improvement in performance compared with the standard LLMs across axes of factuality, completeness, user preference, and adversarial safety.Our results show the potential for LLMs with access to domain-specific corpora to be effective in clinical decision-making. The findings also underscore the importance of carefully testing LLMs before deployment to mitigate their shortcomings. (Funded by the National Institutes of Health, National Heart, Lung, and Blood Institute.).

  View details for DOI 10.1056/aioa2300068

  View details for PubMedID 38343631

  View details for PubMedCentralID PMC10857783

• Almanac: Retrieval-Augmented Language Models for Clinical Medicine. Research square Zakka, C., Chaurasia, A., Shad, R., Dalal, A. R., Kim, J. L., Moor, M., Alexander, K., Ashley, E., Boyd, J., Boyd, K., Hirsch, K., Langlotz, C., Nelson, J., Hiesinger, W. 2023

  Abstract

  Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n= 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.

  View details for DOI 10.21203/rs.3.rs-2883198/v1

  View details for PubMedID 37205549

  View details for PubMedCentralID PMC10187428

• Say My Name: Understanding the Power of Names, Correct Pronunciation, and Personal Narratives. MedEdPORTAL : the journal of teaching and learning resources Dali, S., Atasuntseva, A., Shankar, M., Ayeroff, E., Holmes, M., Johnson, C., Terkawi, A. S., Beadle, B., Chang, J., Boyd, K., Dunn, T. 2022; 18: 11284

  Abstract

  Names are a reflection of identity and often have personal meaning. The chronic mispronunciation of names can undermine one's identity and be experienced as a microaggression. This workshop aims to provide historical context for names as well as resources for correct name pronunciation.We developed a 60-minute interactive virtual workshop with didactics, small-group sharing of personal experiences, and case discussions. We used an anonymous postworkshop survey to evaluate workshop effectiveness.We presented the workshop at one local academic conference and two local educational conferences to learners of all levels from medical students to faculty. We collected postworkshop survey results from 78 participants of diverse racial and ethnic backgrounds. Participants reported learning historical context, ways to ask about correct name pronunciation, correcting name mispronunciation, documenting pronunciation, and sources for applications to practice. The main barriers to implementing workshop lessons included personal and structural factors.This workshop effectively fills an educational gap by addressing the importance of correct name pronunciation in order to provide a more inclusive environment for clinicians and patients alike.

  View details for DOI 10.15766/mep_2374-8265.11284

  View details for PubMedID 36524099

  View details for PubMedCentralID PMC9705275

• Back to the Basics: Community-Acquired Pneumonia in Children. Pediatric annals Boyd, K. 2017; 46 (7): e257-e261

  Abstract

  Community-acquired pneumonia (CAP) is a common childhood infection and often a reason for inpatient admission, especially when a child is hypoxic or in respiratory distress. Despite advances in technology and diagnostics, it remains difficult to accurately differentiate bacterial CAP from a viral process. Most of the laboratory tests routinely done in inpatient medicine, such as complete blood counts and acute phase reactants, do little to differentiate a viral pneumonia from a bacterial pneumonia. Clinicians must rely heavily on the clinical presentation and decide whether to treat empirically with antibiotics. Guidelines published by the Infectious Disease Society of America in 2011 have helped clinicians standardize the diagnosis and treatment of CAP. The guidelines recommend relatively narrow-spectrum antibiotics, such as ampicillin or penicillin, as empiric coverage for the fully immunized child older than age 3 months who requires hospitalization for CAP. [Pediatr Ann. 2017;46(7):e257-e261.].

  View details for DOI 10.3928/19382359-20170616-01

  View details for PubMedID 28697267

• Answer to August 2016 Photo Quiz. Journal of clinical microbiology Relich, R. F., Boyd, K. M., McCoy, M. H., Kaufman, C., Simpson, E. R., Christenson, J. C. 2016; 54 (8): 2213-4

  View details for DOI 10.1128/JCM.02058-14

  View details for PubMedID 27458277

  View details for PubMedCentralID PMC4963496

• Photo Quiz: Fever, Rash, and Polyarthralgia in a 5-Year-Old Male. Journal of clinical microbiology Relich, R. F., Boyd, K. M., McCoy, M. H., Kaufman, C., Simpson, E. R., Christenson, J. C. 2016; 54 (8): 1937

  View details for DOI 10.1128/JCM.02056-14

  View details for PubMedID 27458267

  View details for PubMedCentralID PMC4963485