Kelly Brennan

All Publications

• Engineering of Generative Artificial Intelligence and Natural Language Processing Models to Accurately Identify Arrhythmia Recurrence. Circulation. Arrhythmia and electrophysiology Feng, R., Brennan, K. A., Azizi, Z., Goyal, J., Deb, B., Chang, H. J., Ganesan, P., Clopton, P., Pedron, M., Ruipe Rez-Campillo, S., Desai, Y., De Larochellière, H., Baykaner, T., Perez, M., Rodrigo, M., Rogers, A. J., Narayan, S. M. 2024: e013023

  Abstract

  Large language models (LLMs), such as ChatGPT, excel at interpreting unstructured data from public sources, yet are limited when responding to queries on private repositories, such as electronic health records (EHRs). We hypothesized that prompt engineering could enhance the accuracy of LLMs for interpreting EHR data without requiring domain knowledge, thus expanding their utility for patients and personalized diagnostics.We designed and systematically tested prompt engineering techniques to improve the ability of LLMs to interpret EHRs for nuanced diagnostic questions, referenced to a panel of medical experts. In 490 full-text EHR notes from 125 patients with prior life-threatening heart rhythm disorders, we asked GPT-4-turbo to identify recurrent arrhythmias distinct from prior events and tested 220 563 queries. To provide context, results were compared with rule-based natural language processing and BERT-based language models. Experiments were repeated for 2 additional LLMs.In an independent hold-out set of 389 notes, GPT-4-turbo had a balanced accuracy of 64.3%±4.7% out-of-the-box at baseline. This increased when asking GPT-4-turbo to provide a rationale for its answers, requiring a structured data output, and providing in-context exemplars, rose to a balanced accuracy of 91.4%±3.8% (P<0.05). This surpassed the traditional logic-based natural language processing and BERT-based models (P<0.05). Results were consistent for GPT-3.5-turbo and Jurassic-2 LLMs.The use of prompt engineering strategies enables LLMs to identify clinical end points from EHRs with an accuracy that surpassed natural language processing and approximated experts, yet without the need for expert knowledge. These approaches could be applied to LLM queries for other domains, to facilitate automated analysis of nuanced data sets with high accuracy by nonexperts.

  View details for DOI 10.1161/CIRCEP.124.013023

  View details for PubMedID 39676642

• AUTOMATED, ACCURATE IDENTIFICATION OF VENTRICULAR TACHYCARDIA FROM ELECTRONIC HEALTH RECORDS USING NATURAL LANGUAGE PROCESSING Brennan, K., Azizi, Z., Feng, R., Goyal, J., Liu, X., Ganesan, P., Ruiperez-Campillo, S., Baykaner, T., Badhwar, N., John, R. M., Viswanathan, M., Perino, A., Wang, P. J., Rogers, A. J., Narayan, S. M. ELSEVIER SCIENCE INC. 2024: 2644

  View details for Web of Science ID 001324901502678