Akshay Swaminathan

All Publications

• MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries. Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing Grolleau, F., Alsentzer, E., Keyes, T., Chung, P., Swaminathan, A., Aali, A., Hom, J., Huynh, T., Lew, T., Liang, A., Chu, W., Steele, N., Lin, C., Yang, J., Black, K., Ma, S., Haredasht, F. N., Shah, N. H., Schulman, K., Chen, J. H. 2026; 31: 388-399

  Abstract

  Evaluating factual accuracy in Large Language Model (LLM)-generated clinical text is a critical barrier to adoption, as expert review is unscalable for the continuous quality assurance these systems require. We address this challenge with two complementary contributions. First, we introduce MedFactEval, a framework for scalable, fact-grounded evaluation where clinicians define high-salience key facts and an "LLM Jury"-a multi-LLM majority vote-assesses their inclusion in generated summaries. Second, we present MedAgentBrief, a model-agnostic, multi-step workflow designed to generate high-quality, factual discharge summaries. To validate our evaluation framework, we established a gold-standard reference using a seven-physician majority vote on clinician-defined key facts from inpatient cases. The MedFactEval LLM Jury achieved almost perfect agreement with this panel (Cohen's κ = 81%), a performance statistically non-inferior to that of a single human expert (κ = 67%, P < 0.001). Our work provides both a robust evaluation framework (MedFactEval) and a high-performing generation workflow (MedAgentBrief), offering a comprehensive approach to advance the responsible deployment of generative AI in clinical workflows.

  View details for DOI 10.1142/9789819824755_0027

  View details for PubMedID 41758155

• Holistic evaluation of large language models for medical tasks with MedHELM. Nature medicine Bedi, S., Cui, H., Fuentes, M., Unell, A., Wornow, M., Banda, J. M., Kotecha, N., Keyes, T., Mai, Y., Oez, M., Qiu, H., Jain, S., Schettini, L., Kashyap, M., Fries, J. A., Swaminathan, A., Chung, P., Haredasht, F. N., Lopez, I., Aali, A., Tse, G., Nayak, A., Vedak, S., Jain, S. S., Patel, B., Fayanju, O., Shah, S., Goh, E., Yao, D. H., Soetikno, B., Reis, E., Gatidis, S., Divi, V., Capasso, R., Saralkar, R., Chiang, C. C., Jindal, J., Pham, T., Ghoddusi, F., Lin, S., Chiou, A. S., Hong, H. J., Roy, M., Gensheimer, M. F., Patel, H., Schulman, K., Dash, D., Char, D., Downing, L., Grolleau, F., Black, K., Mieso, B., Zahedivash, A., Yim, W. W., Sharma, H., Lee, T., Kirsch, H., Lee, J., Ambers, N., Lugtu, C., Sharma, A., Mawji, B., Alekseyev, A., Zhou, V., Kakkar, V., Helzer, J., Revri, A., Bannett, Y., Daneshjou, R., Chen, J., Alsentzer, E., Morse, K., Ravi, N., Aghaeepour, N., Kennedy, V., Chaudhari, A., Wang, T., Koyejo, S., Lungren, M. P., Horvitz, E., Liang, P., Pfeffer, M. A., Shah, N. H. 2026

  Abstract

  While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. Here we introduce MedHELM, an extensible evaluation framework with three contributions. First, a clinician-validated taxonomy organizing medical AI applications into five categories that mirror real clinical tasks-clinical decision support (diagnostic decisions, treatment planning), clinical note generation (visit documentation, procedure reports), patient communication (education materials, care instructions), medical research (literature analysis, clinical data analysis) and administration (scheduling, workflow coordination). These encompass 22 subcategories and 121 specific tasks reflecting daily medical practice. Second, a comprehensive benchmark suite of 37 evaluations covering all subcategories. Third, systematic comparison of nine frontier LLMs-Claude 3.5 Sonnet, Claude 3.7 Sonnet, DeepSeek R1, Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o, GPT-4o mini, Llama 3.3 and o3-mini-using an automated LLM-jury evaluation method. Our LLM-jury uses multiple AI evaluators to assess model outputs against expert-defined criteria. Advanced reasoning models (DeepSeek R1, o3-mini) demonstrated superior performance with win rates of 66%, although Claude 3.5 Sonnet achieved comparable results at 15% lower computational cost. These results not only highlight current model capabilities but also demonstrate how MedHELM could enable evidence-based selection of medical AI systems for healthcare applications.

  View details for DOI 10.1038/s41591-025-04151-2

  View details for PubMedID 41559415

  View details for PubMedCentralID 10916499

• MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries Grolleau, F., Alsentzer, E., Keyes, T., Chung, P., Swaminathan, A., Aali, A., Hom, J., Tridu Huynh, Lew, T., Liang, A., Chu, W., Steele, N., Lin, C., Yang, J., Black, K., Ma, S., Haredasht, F. N., Shah, N. H., Schulman, K., Chen, J. H. edited by Altman, R. B., Hunter, L., Ritchie, M. D., Murray, T., Klein, T. E. WORLD SCIENTIFIC PUBL CO PTE LTD. 2026: 388-399

  Abstract

  Evaluating factual accuracy in Large Language Model (LLM)-generated clinical text is a critical barrier to adoption, as expert review is unscalable for the continuous quality assurance these systems require. We address this challenge with two complementary contributions. First, we introduce MedFactEval, a framework for scalable, fact-grounded evaluation where clinicians define high-salience key facts and an "LLM Jury"-a multi-LLM majority vote-assesses their inclusion in generated summaries. Second, we present MedAgentBrief, a model-agnostic, multi-step workflow designed to generate high-quality, factual discharge summaries. To validate our evaluation framework, we established a gold-standard reference using a seven-physician majority vote on clinician-defined key facts from inpatient cases. The MedFactEval LLM Jury achieved almost perfect agreement with this panel (Cohen's κ = 81%), a performance statistically non-inferior to that of a single human expert (κ = 67%, P < 0.001). Our work provides both a robust evaluation framework (MedFactEval) and a high-performing generation workflow (MedAgentBrief), offering a comprehensive approach to advance the responsible deployment of generative AI in clinical workflows.

  View details for Web of Science ID 001796654100027

  View details for PubMedID 41758155

• Evaluating anti-LGBTQIA+ medical bias in large language models. PLOS digital health Chang, C. T., Srivathsa, N., Bou-Khalil, C., Swaminathan, A., Lunn, M. R., Mishra, K., Koyejo, S., Daneshjou, R. 2025; 4 (9): e0001001

  Abstract

  Large Language Models (LLMs) are increasingly deployed in clinical settings for tasks ranging from patient communication to decision support. While these models demonstrate race-based and binary gender biases, anti-LGBTQIA+ bias remains understudied despite documented healthcare disparities affecting these populations. In this work, we evaluated the potential of LLMs to propagate anti-LGBTQIA+ medical bias and misinformation. We prompted 4 LLMs (Gemini 1.5 Flash, Claude 3 Haiku, GPT-4o, Stanford Medicine Secure GPT [GPT-4.0]) with 38 prompts consisting of explicit questions and synthetic clinical notes created by medically-trained reviewers and LGBTQIA+ health experts. The prompts consisted of pairs of prompts with and without LGBTQIA+ identity terms and explored clinical situations across two axes: (i) situations where historical bias has been observed versus not observed, and (ii) situations where LGBTQIA+ identity is relevant to clinical care versus not relevant. Medically-trained reviewers evaluated LLM responses for appropriateness (safety, privacy, hallucination/accuracy, and bias) and clinical utility. We found that all 4 LLMs generated inappropriate responses for prompts with and without LGBTQIA+ identity terms. The proportion of inappropriate responses ranged from 43-62% for prompts mentioning LGBTQIA+ identities versus 47-65% for those without. The most common reason for inappropriate classification tended to be hallucination/accuracy, followed by bias or safety. Qualitatively, we observed differential bias patterns, with LGBTQIA+ prompts eliciting more severe bias. Average clinical utility score for inappropriate responses was lower than for appropriate responses (2.6 versus 3.7 on a 5-point Likert scale). Future work should focus on tailoring output formats to stated use cases, decreasing sycophancy and reliance on extraneous information in the prompt, and improving accuracy and decreasing bias for LGBTQIA+ patients. We present our prompts and annotated responses as a benchmark for evaluation of future models. Content warning: This paper includes prompts and model-generated responses that may be offensive.

  View details for DOI 10.1371/journal.pdig.0001001

  View details for PubMedID 40920790

• Red teaming ChatGPT in medicine to yield real-world insights on model behavior. NPJ digital medicine Chang, C. T., Farah, H., Gui, H., Rezaei, S. J., Bou-Khalil, C., Park, Y. J., Swaminathan, A., Omiye, J. A., Kolluri, A., Chaurasia, A., Lozano, A., Heiman, A., Jia, A. S., Kaushal, A., Jia, A., Iacovelli, A., Yang, A., Salles, A., Singhal, A., Narasimhan, B., Belai, B., Jacobson, B. H., Li, B., Poe, C. H., Sanghera, C., Zheng, C., Messer, C., Kettud, D. V., Pandya, D., Kaur, D., Hla, D., Dindoust, D., Moehrle, D., Ross, D., Chou, E., Lin, E., Haredasht, F. N., Cheng, G., Gao, I., Chang, J., Silberg, J., Fries, J. A., Xu, J., Jamison, J., Tamaresis, J. S., Chen, J. H., Lazaro, J., Banda, J. M., Lee, J. J., Matthys, K. E., Steffner, K. R., Tian, L., Pegolotti, L., Srinivasan, M., Manimaran, M., Schwede, M., Zhang, M., Nguyen, M., Fathzadeh, M., Zhao, Q., Bajra, R., Khurana, R., Azam, R., Bartlett, R., Truong, S. T., Fleming, S. L., Raj, S., Behr, S., Onyeka, S., Muppidi, S., Bandali, T., Eulalio, T. Y., Chen, W., Zhou, X., Ding, Y., Cui, Y., Tan, Y., Liu, Y., Shah, N., Daneshjou, R. 2025; 8 (1): 149

  Abstract

  Red teaming, the practice of adversarially exposing unexpected or undesired model behaviors, is critical towards improving equity and accuracy of large language models, but non-model creator-affiliated red teaming is scant in healthcare. We convened teams of clinicians, medical and engineering students, and technical professionals (80 participants total) to stress-test models with real-world clinical cases and categorize inappropriate responses along axes of safety, privacy, hallucinations/accuracy, and bias. Six medically-trained reviewers re-analyzed prompt-response pairs and added qualitative annotations. Of 376 unique prompts (1504 responses), 20.1% were inappropriate (GPT-3.5: 25.8%; GPT-4.0: 16%; GPT-4.0 with Internet: 17.8%). Subsequently, we show the utility of our benchmark by testing GPT-4o, a model released after our event (20.4% inappropriate). 21.5% of responses appropriate with GPT-3.5 were inappropriate in updated models. We share insights for constructing red teaming prompts, and present our benchmark for iterative model assessments.

  View details for DOI 10.1038/s41746-025-01542-0

  View details for PubMedID 40055532

  View details for PubMedCentralID 10564921

• Against reflexive recalibration: towards a causal framework for addressing miscalibration. Diagnostic and prognostic research Swaminathan, A., Srivastava, U., Tu, L., Lopez, I., Shah, N. H., Vickers, A. J. 2025; 9 (1): 4

  View details for DOI 10.1186/s41512-024-00184-2

  View details for PubMedID 39930530

• Clinical entity augmented retrieval for clinical information extraction. NPJ digital medicine Lopez, I., Swaminathan, A., Vedula, K., Narayanan, S., Nateghi Haredasht, F., Ma, S. P., Liang, A. S., Tate, S., Maddali, M., Gallo, R. J., Shah, N. H., Chen, J. H. 2025; 8 (1): 45

  Abstract

  Large language models (LLMs) with retrieval-augmented generation (RAG) have improved information extraction over previous methods, yet their reliance on embeddings often leads to inefficient retrieval. We introduce CLinical Entity Augmented Retrieval (CLEAR), a RAG pipeline that retrieves information using entities. We compared CLEAR to embedding RAG and full-note approaches for extracting 18 variables using six LLMs across 20,000 clinical notes. Average F1 scores were 0.90, 0.86, and 0.79; inference times were 4.95, 17.41, and 20.08 s per note; average model queries were 1.68, 4.94, and 4.18 per note; and average input tokens were 1.1k, 3.8k, and 6.1k per note for CLEAR, embedding RAG, and full-note approaches, respectively. In conclusion, CLEAR utilizes clinical entities for information retrieval and achieves >70% reduction in token usage and inference time with improved performance compared to modern methods.

  View details for DOI 10.1038/s41746-024-01377-1

  View details for PubMedID 39828800

  View details for PubMedCentralID 4287068

• FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs Munnangi, M., Swaminathan, A., Fries, J., Jindal, J., Narayanan, S., Lopez, I., Tu, L., Chung, P., Omiye, J. A., Kashyap, M., Shah, N. edited by Agrawal, M., Deshpande, K., Engelhard, M., Joshi, S., Tang, S., Urteaga JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2025

  View details for Web of Science ID 001671663700019

• Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A. R., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A., Pfeffer, M. A., Shah, N. H. 2024

  Abstract

  Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas.To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty.A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024.Studies evaluating 1 or more LLMs in health care.Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty.Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented.Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.

  View details for DOI 10.1001/jama.2024.21700

  View details for PubMedID 39405325

  View details for PubMedCentralID PMC11480901

• Extraction of Unstructured Electronic Health Records to Evaluate Glioblastoma Treatment Patterns. JCO clinical cancer informatics Swaminathan, A., Ren, A. L., Wu, J. Y., Bhargava-Shah, A., Lopez, I., Srivastava, U., Alexopoulos, V., Pizzitola, R., Bui, B., Alkhani, L., Lee, S., Mohit, N., Seo, N., Macedo, N., Cheng, W., Wang, W., Tran, E., Thomas, R., Gevaert, O. 2024; 8: e2300091

  Abstract

  Data on lines of therapy (LOTs) for cancer treatment are important for clinical oncology research, but LOTs are not explicitly recorded in electronic health records (EHRs). We present an efficient approach for clinical data abstraction and a flexible algorithm to derive LOTs from EHR-based medication data on patients with glioblastoma multiforme (GBM).Nonclinicians were trained to abstract the diagnosis of GBM from EHRs, and their accuracy was compared with abstraction performed by clinicians. The resulting data were used to build a cohort of patients with confirmed GBM diagnosis. An algorithm was developed to derive LOTs using structured medication data, accounting for the addition and discontinuation of therapies and drug class. Descriptive statistics were calculated and time-to-next-treatment (TTNT) analysis was performed using the Kaplan-Meier method.Treating clinicians as the gold standard, nonclinicians abstracted GBM diagnosis with a sensitivity of 0.98, specificity 1.00, positive predictive value 1.00, and negative predictive value 0.90, suggesting that nonclinician abstraction of GBM diagnosis was comparable with clinician abstraction. Of 693 patients with a confirmed diagnosis of GBM, 246 patients contained structured information about the types of medications received. Of them, 165 (67.1%) received a first-line therapy (1L) of temozolomide, and the median TTNT from the start of 1L was 179 days.We described a workflow for extracting diagnosis of GBM and LOT from EHR data that combines nonclinician abstraction with algorithmic processing, demonstrating comparable accuracy with clinician abstraction and highlighting the potential for scalable and efficient EHR-based oncology research.

  View details for DOI 10.1200/CCI.23.00091

  View details for PubMedID 38857465

• Burn Care Funding in the Era of Price Transparency-Does Verification Signal Bargaining Power? Journal of burn care & research : official publication of the American Burn Association Stanton, E., Pedreira, R., Rizk, N., Swaminathan, A., Sheckter, C. 2024

  Abstract

  The Price Transparency Rule of 2021 forced payors and hospitals to publicly disclose negotiated prices to foster competition and reduce cost. Burn care is costly and concentrated at less than 130 centers in the US. We aimed to analyze geographic price variations for inpatient burn care and measure the effects of American Burn Association (ABA) verification status and market concentration on prices. All available commercial rates for 2021-2022 for burn-related Diagnosis Related Groups (DRG) 927, 928, 929, 933, 934, and 935 were merged with hospital-level variables, ABA verification status, and Herfindahl-Hirschman Index (HHI) data. For the DRG 927 (most intensive burn admission) a linear mixed effects model was fit with cost as the outcome and the following variables as covariates: HHI, plan type, safety net status, profit status, verification status, rural status, teaching hospital status. Random intercepts allowed for individual burn centers. There were 170,738 rates published from 1541 unique hospitals. Commercial reimbursement rates for the same DRG varied by a factor of approximately three within hospitals for all DRGs. Similarly, rates across different hospitals varied by a factor of three for all DRGs, with DRG 927 having the most variation. Burn center status was independently associated with higher reimbursement rates adjusting for facility-level factors for all DRGs except for 935. Notably, HHI was the largest predictor of commercial rates (p<0.001). Negotiated prices for inpatient burn care vary widely. ABA-verified centers garner higher rates along with burn centers in more concentrated/monopolistic markets.

  View details for DOI 10.1093/jbcr/irae078

  View details for PubMedID 38733210

• Emerging Outlook on Personalized Neuromodulation for Depression: Insights from Tractography-Based Targeting. Biological psychiatry. Cognitive neuroscience and neuroimaging Sadeghzadeh, S., Swaminathan, A., Bhanot, P., Steeman, S., Xu, A., Shah, V., Purger, D. A., Buch, V. P. 2024

  Abstract

  Deep brain stimulation (DBS) has shown individual promise in treating treatment resistant depression (TRD), but larger-scale trials have been less successful. Here, we create the largest meta-analysis with individual patient data (IPD) to date to explore if the use of tractography enhances the efficacy of DBS for TRD.We systematically reviewed 1823 articles, selecting 32 that contributed data from 366 patients. We stratified the IPD based on stimulation target and use of tractography. Utilizing two-way type III Analysis of Variance (ANOVA), Welch Two Sample t-tests, and mixed-effects linear regression models, we evaluated changes in depression severity 9-15 months post-surgery (1-Y) and at last follow-up (LFU) (4 weeks - 8 years) as assessed by depression scales.Tractography was used for medial forebrain bundle (MFB, n=17/32), subcallosal cingulate (SCC, n=39/241), and ventral capsule/ventral striatum (VC/VS, n=3/41) targets; and not used for bed nucleus of stria terminalis (n=11), lateral habenula (n=10), and inferior thalamic peduncle (n=1). Across all patients, tractography significantly improved mean depression scores at 1-Y (p<0.001) and LFU (p=0.009). Within the target cohorts, tractography improved depression scores at 1-Y for both MFB and SCC, though significance was only met at the alpha = 0.1 level (SCC: β=15.8%, p=0.09; MFB: β=52.4%, p=0.10). Within the tractography cohort, MFB with tractography patients showed greater improvement than those with SCC with tractography (72.42±7.17% versus 54.78±4.08%) at 1-Y (p=0.044).Our findings underscore the promise of tractography in DBS for TRD as a methodology for personalization of therapy, supporting its inclusion in future trials.

  View details for DOI 10.1016/j.bpsc.2024.04.007

  View details for PubMedID 38679323

• Diversifying cardiac intensive care unit models: Successful example of an operating surgeon-led unit. JTCVS open Choi, P. S., Pines, K. C., Swaminathan, A., Nilkant, R., Mendez, M. A., He, H., Woo, Y. J., Martin, B. 2023; 16: 524-531

  Abstract

  Objective: The intensivist-led cardiovascular intensive care unit model is the standard of care in cardiac surgery. This study examines whether a cardiovascular intensive care unit model that uses operating cardiac surgeons, cardiothoracic surgery residents, and advanced practice providers is associated with comparable outcomes.Methods: This is a single-institution review of the first 400 cardiac surgery patients admitted to an operating surgeon-led cardiovascular intensive care unit from 2020 to 2022. Inclusion criteria are elective status and operations managed by both cardiovascular intensive care unit models (aortic operations, valve operations, coronary operations, septal myectomy). Patients from the surgeon-led cardiovascular intensive care unit were exact matched by operation type and 1:1 propensity score matched with controls from the traditional cardiovascular intensive care unit using a logistic regression model that included age, sex, preoperative mortality risk, incision type, and use of cardiopulmonary bypass and circulatory arrest. Primary outcome was total postoperative length of stay. Secondary outcomes included postoperative intensive care unit length of stay, 30-day mortality, 30-day Society of Thoracic Surgeons-defined morbidity (permanent stroke, renal failure, cardiac reoperation, prolonged intubation, deep sternal infection), packed red cell transfusions, and vasopressor use. Outcomes between the 2 groups were compared using chi-square, Fisher exact test, or 2-sample t test as appropriate.Results: A total of 400 patients from the surgeon-led cardiovascular intensive care unit (mean age 61.2±12.8years, 131 female patients [33%], 346 patients [86.5%] with European System for Cardiac Operative Risk Evaluation II <2%) and their matched controls were included. The most common operations across both units were coronary artery bypass grafting (n=318, 39.8%) and mitral valve repair or replacement (n=238, 29.8%). Approximately half of the operations were performed via sternotomy (n=462, 57.8%). There were 3 (0.2%) in-hospital deaths, and 47 patients (5.9%) had a 30-day complication. The total length of stay was significantly shorter for the surgeon-led cardiovascular intensive care unit patients (6.3 vs 7.0days, P=.028), and intensive care unit length of stay trended in the same direction (2.5 vs 2.9days, P=.16). Intensive care unit readmission rates, 30-day mortality, and 30-day morbidity were not significantly different between cardiovascular intensive care unit models. The surgeon-led cardiovascular intensive care unit was associated with fewer postoperative red blood cell transfusions in the cardiovascular intensive care unit (P=.002) and decreased vasopressor use (P=.001).Conclusions: In its first 2years, the surgeon-led cardiovascular intensive care unit demonstrated comparable outcomes to the traditional cardiovascular intensive care unit with significant improvements in total length of stay, postoperative transfusions in the cardiovascular intensive care unit, and vasopressor use. This early success exemplifies how an operating surgeon-led cardiovascular intensive care unit can provide similar outcomes to the standard-of-care model for patients undergoing elective cardiac surgery.

  View details for DOI 10.1016/j.xjon.2023.09.040

  View details for PubMedID 38204639

• Natural language processing system for rapid detection and intervention of mental health crisis chat messages. NPJ digital medicine Swaminathan, A., Lopez, I., Mar, R. A., Heist, T., McClintock, T., Caoili, K., Grace, M., Rubashkin, M., Boggs, M. N., Chen, J. H., Gevaert, O., Mou, D., Nock, M. K. 2023; 6 (1): 213

  Abstract

  Patients experiencing mental health crises often seek help through messaging-based platforms, but may face long wait times due to limited message triage capacity. Here we build and deploy a machine-learning-enabled system to improve response times to crisis messages in a large, national telehealth provider network. We train a two-stage natural language processing (NLP) system with key word filtering followed by logistic regression on 721 electronic medical record chat messages, of which 32% are potential crises (suicidal/homicidal ideation, domestic violence, or non-suicidal self-injury). Model performance is evaluated on a retrospective test set (4/1/21-4/1/22, N=481) and a prospective test set (10/1/22-10/31/22, N=102,471). In the retrospective test set, the model has an AUC of 0.82 (95% CI: 0.78-0.86), sensitivity of 0.99 (95% CI: 0.96-1.00), and PPV of 0.35 (95% CI: 0.309-0.4). In the prospective test set, the model has an AUC of 0.98 (95% CI: 0.966-0.984), sensitivity of 0.98 (95% CI: 0.96-0.99), and PPV of 0.66 (95% CI: 0.626-0.692). The daily median time from message receipt to crisis specialist triage ranges from 8 to 13min, compared to 9h before the deployment of the system. We demonstrate that a NLP-based machine learning model can reliably identify potential crisis chat messages in a telehealth setting. Our system integrates into existing clinical workflows, suggesting that with appropriate training, humans can successfully leverage ML systems to facilitate triage of crisis messages.

  View details for DOI 10.1038/s41746-023-00951-3

  View details for PubMedID 37990134

• Selective prediction for extracting unstructured clinical data. Journal of the American Medical Informatics Association : JAMIA Swaminathan, A., Lopez, I., Wang, W., Srivastava, U., Tran, E., Bhargava-Shah, A., Wu, J. Y., Ren, A. L., Caoili, K., Bui, B., Alkhani, L., Lee, S., Mohit, N., Seo, N., Macedo, N., Cheng, W., Liu, C., Thomas, R., Chen, J. H., Gevaert, O. 2023

  Abstract

  While there are currently approaches to handle unstructured clinical data, such as manual abstraction and structured proxy variables, these methods may be time-consuming, not scalable, and imprecise. This article aims to determine whether selective prediction, which gives a model the option to abstain from generating a prediction, can improve the accuracy and efficiency of unstructured clinical data abstraction.We trained selective classifiers (logistic regression, random forest, support vector machine) to extract 5 variables from clinical notes: depression (n = 1563), glioblastoma (GBM, n = 659), rectal adenocarcinoma (DRA, n = 601), and abdominoperineal resection (APR, n = 601) and low anterior resection (LAR, n = 601) of adenocarcinoma. We varied the cost of false positives (FP), false negatives (FN), and abstained notes and measured total misclassification cost.The depression selective classifiers abstained on anywhere from 0% to 97% of notes, and the change in total misclassification cost ranged from -58% to 9%. Selective classifiers abstained on 5%-43% of notes across the GBM and colorectal cancer models. The GBM selective classifier abstained on 43% of notes, which led to improvements in sensitivity (0.94 to 0.96), specificity (0.79 to 0.96), PPV (0.89 to 0.98), and NPV (0.88 to 0.91) when compared to a non-selective classifier and when compared to structured proxy variables.We showed that selective classifiers outperformed both non-selective classifiers and structured proxy variables for extracting data from unstructured clinical notes.Selective prediction should be considered when abstaining is preferable to making an incorrect prediction.

  View details for DOI 10.1093/jamia/ocad182

  View details for PubMedID 37769323

• Critically reading machine learning literature in neurosurgery: a reader's guide and checklist for appraising prediction models. Neurosurgical focus Emani, S., Swaminathan, A., Grobman, B., Duvall, J. B., Lopez, I., Arnaout, O., Huang, K. T. 2023; 54 (6): E3

  Abstract

  OBJECTIVE: Machine learning (ML) has become an increasingly popular tool for use in neurosurgical research. The number of publications and interest in the field have recently seen significant expansion in both quantity and complexity. However, this also places a commensurate burden on the general neurosurgical readership to appraise this literature and decide if these algorithms can be effectively translated into practice. To this end, the authors sought to review the burgeoning neurosurgical ML literature and to develop a checklist to help readers critically review and digest this work.METHODS: The authors performed a literature search of recent ML papers in the PubMed database with the terms "neurosurgery" AND "machine learning," with additional modifiers "trauma," "cancer," "pediatric," and "spine" also used to ensure a diverse selection of relevant papers within the field. Papers were reviewed for their ML methodology, including the formulation of the clinical problem, data acquisition, data preprocessing, model development, model validation, model performance, and model deployment.RESULTS: The resulting checklist consists of 14 key questions for critically appraising ML models and development techniques; these are organized according to their timing along the standard ML workflow. In addition, the authors provide an overview of the ML development process, as well as a review of key terms, models, and concepts referenced in the literature.CONCLUSIONS: ML is poised to become an increasingly important part of neurosurgical research and clinical care. The authors hope that dissemination of education on ML techniques will help neurosurgeons to critically review new research better and more effectively integrate this technology into their practices.

  View details for DOI 10.3171/2023.3.FOCUS2352

  View details for PubMedID 37283326

• Post-traumatic growth in PhD students during the COVID-19 pandemic. Psychiatry research communications Tu, A., Restivo, J., O'Neill, K., Swaminathan, A., Choi, K., Lee, H., Smoller, J., Patel, V., Barreira, P., Liu, C., Naslund, J. 2023; 3 (1): 100104

  Abstract

  Throughout the COVID-19 pandemic, graduate students have faced increased risk of mental health challenges. Research suggests that experiencing adversity may induce positive psychological changes, called post-traumatic growth (PTG). These changes can include improved relationships with others, perceptions of oneself, and enjoyment of life. Few existing studies have explored this phenomenon among graduate students. This secondary data analysis of a survey conducted in November 2020 among graduate students at a private R1 University in the northeast United States examined graduate students' levels and correlates of PTG during the COVID-19 pandemic. Students had a low level of PTG, with a mean score of 10.31 out of 50. Linear regression models showed significant positive relationships between anxiety and PTG and between a measure of self-reported impact of the pandemic and PTG. Non-White minorities also had significantly greater PTG than White participants. Experiencing more negative impact due to the pandemic and ruminating about the pandemic were correlated with greater PTG. These findings advance research on the patterns of PTG during the COVID-19 pandemic and can inform future studies of graduate students' coping mechanisms and support efforts to promote pandemic recovery and resilience.

  View details for DOI 10.1016/j.psycom.2023.100104

  View details for PubMedID 36743383

  View details for PubMedCentralID PMC9886426

• Forecasting efforts from prior epidemics and COVID-19 predictions EUROPEAN JOURNAL OF EPIDEMIOLOGY Nadella, P., Swaminathan, A., Subramanian, S. V. 2020; 35 (8): 727-729

  Abstract

  Since the onset of the COVID-19 pandemic, countless disease prediction models have emerged, shaping the focus of news media, policymakers, and broader society. We reviewed the accuracy of forecasts made during prior twenty-first century epidemics, namely SARS, H1N1, and Ebola. We found that while disease prediction models were relatively nascent as a research focus during SARS and H1N1, for Ebola, numerous such forecasts were published. We found that forecasts of deaths for Ebola were often far from the eventual reality, with a strong tendency to over predict. Given the societal prominence of these models, it is crucial that their uncertainty be communicated. Otherwise, we will be unaware if we are being falsely lulled into complacency or unjustifiably shocked into action.

  View details for DOI 10.1007/s10654-020-00661-0

  View details for Web of Science ID 000549228500001

  View details for PubMedID 32676971

  View details for PubMedCentralID PMC7366467

• Proof of Concept of a Gamified DEvelopmental Assessment on an E-Platform (DEEP) Tool to Measure Cognitive Development in Rural Indian Preschool Children FRONTIERS IN PSYCHOLOGY Mukherjee, D., Bhavnani, S., Swaminathan, A., Verma, D., Parameshwaran, D., Divan, G., Dasgupta, J., Sharma, K., Thiagarajan, T. C., Patel, V. 2020; 11: 1202

  Abstract

  Over 250 million children in developing countries are at risk of not achieving their developmental potential, and unlikely to receive timely interventions because existing developmental assessments that help identify children who are faltering are prohibitive for use in low resource contexts. To bridge this "detection gap," we developed a tablet-based, gamified cognitive assessment tool named DEvelopmental assessment on an E-Platform (DEEP), which is feasible for delivery by non-specialists in rural Indian households and acceptable to all end-users. Here we provide proof-of-concept of using a supervised machine learning (ML) approach benchmarked to the Bayley's Scale of Infant and Toddler Development, 3rd Edition (BSID-III) cognitive scale, to predict a child's cognitive development using metrics derived from gameplay on DEEP. Two-hundred children aged 34-40 months recruited from rural Haryana, India were concurrently assessed using DEEP and BSID-III. Seventy percent of the sample was used for training the ML algorithms using a 10-fold cross validation approach and ensemble modeling, while 30% was assigned to the "test" dataset to evaluate the algorithm's accuracy on novel data. Of the 522 features that computationally described children's performance on DEEP, 31 features which together represented all nine games of DEEP were selected in the final model. The predicted DEEP scores were in good agreement (ICC [2,1] > 0.6) and positively correlated (Pearson's r = 0.67) with BSID-cognitive scores, and model performance metrics were highly comparable between the training and test datasets. Importantly, the mean absolute prediction error was less than three points (<10% error) on a possible range of 31 points on the BSID-cognitive scale in both the training and test datasets. Leveraging the power of ML which allows iterative improvements as more diverse data become available for training, DEEP, pending further validation, holds promise to serve as an acceptable and feasible cognitive assessment tool to bridge the detection gap and support optimum child development.

  View details for DOI 10.3389/fpsyg.2020.01202

  View details for Web of Science ID 000543876900001

  View details for PubMedID 32587551

  View details for PubMedCentralID PMC7299081

• The State of School Infrastructure in the Assembly Constituencies of Rural India: Analysis of 11 Census Indicators from Pre-primary to Higher Education INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH Swaminathan, A., Narayanan, M., Blossom, J., Venkataramanan, R., Saunik, S., Kim, R., Subramanian, S. V. 2020; 17 (1)

  Abstract

  In India, assembly constituencies (ACs), represented by elected officials, are the primary geopolitical units for state-level policy development. However, data on social indicators are traditionally reported and analyzed at the district level, and are rarely available for ACs. Here, we combine village-level data from the 2011 Indian Census and AC shapefiles to systematically derive AC-level estimates for the first time. We apply this methodology to describe the distribution of 11 education infrastructures-ranging from pre-primary school to senior secondary school-across rural villages in 3773 ACs. We found high variability in access to higher education infrastructures and low variability in access to lower education variables. For 40.3% (25th percentile) to 79.7% (75th percentile) of villages in an AC, the nearest government senior secondary school was >5 km away, whereas the nearest government primary school was >5 km away in just 0% (25th percentile) to 1.9% (75th percentile) of villages in an AC. The states of Manipur, Arunachal Pradesh, and Bihar showed the greatest within-state variation in access to education infrastructures. We present a novel analysis of access to education infrastructure to inform AC-level policy, and demonstrate how geospatial and Census data can be leveraged to derive AC-level estimates for any population health and development indicators collected in the Census at the village level.

  View details for DOI 10.3390/ijerph17010296

  View details for Web of Science ID 000509391500296

  View details for PubMedID 31906293

  View details for PubMedCentralID PMC6981364

• Association between interpregnancy interval and subsequent stillbirth in 58 low-income and middle-income countries: a retrospective analysis using Demographic and Health Surveys LANCET GLOBAL HEALTH Swaminathan, A., Fell, D. B., Regan, A., Walker, M., Corsi, D. J. 2020; 8 (1): E113-E122

  Abstract

  About 3 million stillbirths occur each year, 98% of which are in low-income and middle-income countries (LMICs). Interpregnancy interval is a key risk factor of interest, because it is modifiable. We aimed to investigate whether there is a causal relationship between the length of interpregnancy interval and risk of subsequent stillbirth.We used Demographic and Health Surveys (2002-18) from 58 LMICs to study reproductive histories of women and to identify livebirths and stillbirths in the preceding 5 years. Countries were selected on the basis of the availability of interpregnancy interval data and other covariates of interest (age, education, urban or rural residence, and wealth) in surveys done since 2002. Exclusion criteria were being nulliparous, having missing parity data, and not having had at least two births (livebirth or stillbirth) in the 5 years before the survey. We combined two analytic approaches: one that analyses intervals between all births and another that analyses intervals within mothers. We report stratified estimates for the first, second, and third intervals, controlling for all past birth outcomes and intervals in a 5-year period, and other socioeconomic covariates. We also explored effect heterogeneity across key cohort subgroups.Between July, 1997, and April, 2018, we identified 716 478 births from 338 223 women in 123 Demographic and Health Surveys from 58 LMICs, of which 9647 were stillbirths. Intervals of less than 6 months were associated with an increased risk of stillbirth in the between-mother models when considering the first interval (risk difference [RD] 0·0096, 95% CI 0·008-0·011). This association was slightly attenuated when considering only the second interval (RD 0·0054, 95% CI 0·0010 to 0·0099) and substantially attenuated when considering only the third interval (0·0007, -0·037 to 0·039). Within-mother modelling showed a null association with intervals of 24-59 months when considering the first and second (RD 0·007, 95% CI -0·001 to 0·016) and first and third (0·040, -0·422 to 0·501) intervals.Although interpregnancy intervals of less than 12 months were associated with increased risk of stillbirth, these effects were attenuated when considering second and third intervals, suggesting the association in the first interval might not be causal. Future studies should use generalisable cohorts with longitudinal data, and report estimates stratified by birth order.Canadian Institutes of Health Research.

  View details for DOI 10.1016/S2214-109X(19)30458-9

  View details for Web of Science ID 000502605800031

  View details for PubMedID 31839126

• Validity of Maternal Report of Birthweight in a Cohort Study and Its Implication on Low Birthweight Rate Using Simulations INDIAN PEDIATRICS Swaminathan, A., Thomas, T., Corsi, D. J., Dwarkanath, P., Subramanian, S. V., Kurpad, A. V. 2019; 56 (11): 923-928

  Abstract

  Maternal recall of birthweight is a convenient and cost-effective way to obtain birthweight measurements when official records are unavailable. It is important to assess the validity of maternal recall of birthweight before using these measurements to draw conclusions about a population.This is secondary analysis of data from a previous cohort study. We analyzed actual and reported birthweights of 200 mother-and-child pairs from Southern India. We validated maternal report of birthweight by generating correlation coefficients, summary statistics, and Bland-Altman plots. We ran simulations to evaluate how misclassification as low or normal birthweight changed with the mean birthweight of the cohort.Reported birthweight was strongly correlated with actual birthweight (r=0.80, P<0.001); 55%, 78.5%, and 93% of subjects reported values within 50 g, 250 g, and 500 g, respectively of actual birthweight. None of sociodemographic covariates was significantly associated with the accuracy of maternal recall of birthweight. 7.5% of children were misclassified as either low or normal birthweight by reported birthweight. Simulations revealed that increasing the reported and actual birthweights by 500g reduces the misclassification rate from 7.5% to 1.5%.Maternal recall is a sufficiently accurate measure of actual birthweight. However, the distribution of actual birthweight in the population must be taken into consideration when classifying babies as low or normal birthweight, especially in populations where mean birthweight is close to 2500g.

  View details for DOI 10.1007/s13312-019-1648-6

  View details for Web of Science ID 000500782800005

  View details for PubMedID 31729322

• Environmentally Optimal, Nutritionally Sound, Protein and Energy Conserving Plant Based Alternatives to US Meat SCIENTIFIC REPORTS Eshel, G., Stainier, P., Shepon, A., Swaminathan, A. 2019; 9: 10345

  Abstract

  Because meat is more resource intensive than vegetal protein sources, replacing it with efficient plant alternatives is potentially desirable, provided these alternatives prove nutritionally sound. We show that protein conserving plant alternatives to meat that rigorously satisfy key nutritional constraints while minimizing cropland, nitrogen fertilizer (Nr) and water use and greenhouse gas (GHG) emissions exist, and could improve public health. We develop a new methodology for identifying nutritional constraints whose satisfaction by plant eaters is challenging, disproportionately shaping the optimal diets, singling out energy, mass, monounsaturated fatty acids, vitamins B3,12 and D, choline, zinc, and selenium. By replacing meat with the devised plant alternatives-dominated by soy, green pepper, squash, buckwheat, and asparagus-Americans can collectively eliminate pastureland use while saving 35-50% of their diet related needs for cropland, Nr, and GHG emission, but increase their diet related irrigation needs by 15%. While widely replacing meat with plants is logistically and culturally challenging, few competing options offer comparable multidimensional resource use reduction.

  View details for DOI 10.1038/s41598-019-46590-1

  View details for Web of Science ID 000479187400001

  View details for PubMedID 31395893

  View details for PubMedCentralID PMC6687707

• Estimating the burden of child malnutrition across parliamentary constituencies in India: A methodological comparison SSM-POPULATION HEALTH Kim, R., Swaminathan, A., Kumar, R., Xu, Y., Blossom, J. C., Venkataramanan, R., Kumar, A., Joe, W., Subramanian, S. 2019; 7: 100375

  Abstract

  In India, data on key developmental indicators used to formulate policies and interventions are routinely available for the administrative unit of districts but not for the political unit of parliamentary constituencies (PC). Recently, Swaminathan et al. proposed two methodologies to generate PC estimates using randomly displaced GPS locations of the sampling clusters ('direct') and by building a crosswalk between districts and PCs using boundary shapefiles ('indirect'). We advance these methodologies by using precision-weighted estimations based on hierarchical logistic regression modeling to account for the complex survey design and sampling variability. We exemplify this application using the latest National Family Health Survey (NFHS, 2016) to generate PC-level estimates for two important indicators of child malnutrition - stunting and low birth weight - that are being monitored by the Government of India for the National Nutrition Mission targets. Overall, we found a substantial variation in child malnutrition across 543 PCs. The different methodologies yielded highly consistent estimates with correlation ranging r = 0.92-0.99 for stunting and r = 0.81-0.98 for low birth weight. For analyses involving data with comparable nature to the NFHS (i.e., complex data structure and possibility to identify a potential PC membership), modeling for precision-weighted estimates and direct methodology are preferable. Further field work and data collection at the PC level are necessary to accurately validate our estimates. An ideal solution to overcome this gap in data for PCs would be to make PC identifiers available in routinely collected surveys and the Census.

  View details for DOI 10.1016/j.ssmph.2019.100375

  View details for Web of Science ID 000477864100065

  View details for PubMedID 30834287

  View details for PubMedCentralID PMC6384327