Agustina D Saenz

All Publications

• ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics. Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing Banerjee, O., Saenz, A., Wu, K., Clements, W., Zia, A., Buensalido, D., Kavnoudias, H., Abi-Ghanem, A. S., Ghawi, N. E., Luna, C., Castillo, P., Al-Surimi, K., Daghistani, R. A., Chen, Y. M., Chao, H. S., Heiliger, L., Kim, M., Haubold, J., Jonske, F., Rajpurkar, P. 2025; 30: 185-198

  Abstract

  Given the rapidly expanding capabilities of generative AI models for radiology, there is a need for robust metrics that can accurately measure the quality of AI-generated radiology reports across diverse hospitals. We develop ReXamine-Global, a LLM-powered, multi-site framework that tests metrics across different writing styles and patient populations, exposing gaps in their generalization. First, our method tests whether a metric is undesirably sensitive to reporting style, providing different scores depending on whether AI-generated reports are stylistically similar to ground-truth reports or not. Second, our method measures whether a metric reliably agrees with experts, or whether metric and expert scores of AI-generated report quality diverge for some sites. Using 240 reports from 6 hospitals around the world, we apply ReXamine-Global to 7 established report evaluation metrics and uncover serious gaps in their generalizability. Developers can apply ReXamine-Global when designing new report evaluation metrics, ensuring their robustness across sites. Additionally, our analysis of existing metrics can guide users of those metrics towards evaluation procedures that work reliably at their sites of interest.

  View details for DOI 10.1142/9789819807024_0014

  View details for PubMedID 39670370

• Establishing responsible use of AI guidelines: a comprehensive case study for healthcare institutions. NPJ digital medicine Saenz, A. D., Centi, A., Ting, D., You, J. G., Landman, A., Mishuris, R. G. 2024; 7 (1): 348

  Abstract

  This report presents a comprehensive case study for the responsible integration of artificial intelligence (AI) into healthcare settings. Recognizing the rapid advancement of AI technologies and their potential to transform healthcare delivery, we propose a set of guidelines emphasizing fairness, robustness, privacy, safety, transparency, explainability, accountability, and benefit. Through a multidisciplinary collaboration, we developed and operationalized these guidelines within a healthcare system, highlighting a case study on ambient documentation to demonstrate the practical application and challenges of implementing generative AI in clinical environments. Our proposed framework ensures continuous monitoring, evaluation, and adaptation of AI technologies, addressing ethical considerations and enhancing patient care. This work contributes to the discourse on responsible AI use in healthcare, offering a blueprint for institutions to navigate the complexities of AI integration responsibly and effectively, thus promoting better, more equitable healthcare outcomes.

  View details for DOI 10.1038/s41746-024-01300-8

  View details for PubMedID 39616269

  View details for PubMedCentralID PMC11608363

• The MAIDA initiative: establishing a framework for global medical-imaging data sharing. The Lancet. Digital health Saenz, A., Chen, E., Marklund, H., Rajpurkar, P. 2024; 6 (1): e6-e8

  View details for DOI 10.1016/S2589-7500(23)00222-4

  View details for PubMedID 37977999

• Autonomous AI systems in the face of liability, regulations and costs. NPJ digital medicine Saenz, A. D., Harned, Z., Banerjee, O., Abramoff, M. D., Rajpurkar, P. 2023; 6 (1): 185

  Abstract

  Autonomous AI systems in medicine promise improved outcomes but raise concerns about liability, regulation, and costs. With the advent of large-language models, which can understand and generate medical text, the urgency for addressing these concerns increases as they create opportunities for more sophisticated autonomous AI systems. This perspective explores the liability implications for physicians, hospitals, and creators of AI technology, as well as the evolving regulatory landscape and payment models. Physicians may be favored in malpractice cases if they follow rigorously validated AI recommendations. However, AI developers may face liability for failing to adhere to industry-standard best practices during development and implementation. The evolving regulatory landscape, led by the FDA, seeks to ensure transparency, evaluation, and real-world monitoring of AI systems, while payment models such as MPFS, NTAP, and commercial payers adapt to accommodate them. The widespread adoption of autonomous AI systems can potentially streamline workflows and allow doctors to concentrate on the human aspects of healthcare.

  View details for DOI 10.1038/s41746-023-00929-1

  View details for PubMedID 37803209

• Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting Yan, B., Liu, R., Kuo, D. E., Adithan, S., Reis, E., Kwak, S., Venugopal, V., O'Connell, C. P., Saenz, A., Rajpurkar, P., Moor, M. edited by Bouamor, H., Pino, J., Bali, K. ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2023: 14676-14688

  View details for Web of Science ID 001378234406049

• RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction Khanna, S., Dejl, A., Yoon, K., Truong, S. Q. H., Hanh Duong, Saenz, A., Rajpurkar, P. edited by Deshpande, K., Fiterau, M., Joshi, S., Lipton, Z., Ranganath, R., Urteaga, Yeung, S. JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2023

  View details for Web of Science ID 001221187500017

• Trends in High-Severity Billing of Hospitalized Medicare Beneficiaries Treated by Hospitalists vs Nonhospitalists. JAMA health forum Saenz, A. D., Tsugawa, Y., Phelan, J., Orav, E. J., Figueroa, J. F. 2022; 3 (3): e220120

  Abstract

  As US hospital expenditures continue to rise, understanding drivers of high-severity billing for hospitalized patients among inpatient physicians is critically important.To evaluate high-severity billing trends of Medicare beneficiaries treated by hospitalists vs nonhospitalists.This cohort study used Medicare fee-for-service claims of hospitalized patients from 2009 through 2018 to compare the proportion of high-severity billing between general medicine physicians classified as hospitalists vs nonhospitalists across initial, subsequent, and discharge hospital encounters. We compared physicians within the same hospital using hospital fixed effects and adjusted for patient demographics and comorbidities. Changes in the billing practices were assessed by investigating differences in slopes using an interaction term between physician type and time. Analyses were conducted between August 2021 and January 2022.Treatment by hospitalists vs nonhospitalists.High-severity billing for initial, subsequent, and discharge hospital encounters.The sample included 3 121 260 and 1 855 678 Medicare beneficiaries treated by hospitalists vs nonhospitalists, respectively. In each year, mean age, proportion female, proportion Black and Hispanic dual status, and mean number of chronic conditions were similar among those treated by hospitalists vs nonhospitalists (standardized mean difference < .01). The number of hospitalists grew by 76%, from 23 390 in 2009 to 41 084 in 2018, whereas nonhospitalists decreased by 43.6% (53 758 to 30 289). The proportion of encounters performed by hospitalists increased for the initial hospital encounters (46.3% to 76%), subsequent encounters (46.8% to 76.7%), and discharge encounters (46.1% to 78.5%) over the 10-year period. The proportion of high-severity billing across the hospital, subsequent, and discharge encounters was consistently higher among hospitalists relative to nonhospitalists across all years. Compared with the trends for nonhospitalists, the proportion of high-severity billing grew by 0.46% per year (95% CI, 0.44% to 0.49%; P < .001) for initial encounters, 0.38% per year (95% CI, 0.37% to 0.39%; P < .001) for subsequent encounters, and by 1.1% per year (95% CI, 1.1% to 1.15%; P < .001) for discharge encounters among hospitalists.In this cohort study of Medicare fee-for-service beneficiaries treated in hospitals, high-severity billing increased over time for hospital encounters at higher rates for hospitalists than for nonhospitalists. These differences do not appear to be explained by patient complexity. The increase in the number of hospitalists over time may be contributing to rising national costs related to hospital care.

  View details for DOI 10.1001/jamahealthforum.2022.0120

  View details for PubMedID 35977285

  View details for PubMedCentralID PMC8933743

• Hospital-Level Care at Home for Acutely Ill Adults: A Randomized Controlled Trial. Annals of internal medicine Levine, D. M., Ouchi, K., Blanchfield, B., Saenz, A., Burke, K., Paz, M., Diamond, K., Pu, C. T., Schnipper, J. L. 2020; 172 (2): 77-85

  Abstract

  Substitutive hospital-level care in a patient's home may reduce cost, health care use, and readmissions while improving patient experience, although evidence from randomized controlled trials in the United States is lacking.To compare outcomes of home hospital versus usual hospital care for patients requiring admission.Randomized controlled trial. (ClinicalTrials.gov: NCT03203759).Academic medical center and community hospital.91 adults (43 home and 48 control) admitted via the emergency department with selected acute conditions.Acute care at home, including nurse and physician home visits, intravenous medications, remote monitoring, video communication, and point-of-care testing.The primary outcome was the total direct cost of the acute care episode (sum of costs for nonphysician labor, supplies, medications, and diagnostic tests). Secondary outcomes included health care use and physical activity during the acute care episode and at 30 days.The adjusted mean cost of the acute care episode was 38% (95% CI, 24% to 49%) lower for home patients than control patients. Compared with usual care patients, home patients had fewer laboratory orders (median per admission, 3 vs. 15), imaging studies (median, 14% vs. 44%), and consultations (median, 2% vs. 31%). Home patients spent a smaller proportion of the day sedentary (median, 12% vs. 23%) or lying down (median, 18% vs. 55%) and were readmitted less frequently within 30 days (7% vs. 23%).The study involved 2 sites, a small number of home physicians, and a small sample of highly selected patients (with a 63% refusal rate among potentially eligible patients); these factors may limit generalizability.Substitutive home hospitalization reduced cost, health care use, and readmissions while increasing physical activity compared with usual hospital care.Partners HealthCare Center for Population Health and internal departmental funds.

  View details for DOI 10.7326/M19-0600

  View details for PubMedID 31842232