Michael Adam Pfeffer

All Publications

• Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., Hong, H. J., Kashyap, M., Chaurasia, A. R., Shah, N. R., Singh, K., Tazbaz, T., Milstein, A., Pfeffer, M. A., Shah, N. H. 2024

  Abstract

  Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas.To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty.A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024.Studies evaluating 1 or more LLMs in health care.Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty.Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented.Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.

  View details for DOI 10.1001/jama.2024.21700

  View details for PubMedID 39405325

  View details for PubMedCentralID PMC11480901

• Perspectives on Artificial Intelligence-Generated Responses to Patient Messages. JAMA network open Kim, J., Chen, M. L., Rezaei, S. J., Liang, A. S., Seav, S. M., Onyeka, S., Lee, J. J., Vedak, S. C., Mui, D., Lal, R. A., Pfeffer, M. A., Sharp, C., Pageler, N. M., Asch, S. M., Linos, E. 2024; 7 (10): e2438535

  View details for DOI 10.1001/jamanetworkopen.2024.38535

  View details for PubMedID 39412810

• Development of Secure Infrastructure for Advancing Generative AI Research in Healthcare at an Academic Medical Center. Research square Ng, M. Y., Helzer, J., Pfeffer, M. A., Seto, T., Hernandez-Boussard, T. 2024

  Abstract

  The increasing interest in leveraging generative AI models in healthcare necessitates secure infrastructure at academic medical centers. Without an all-encompassing secure system, researchers may create their own insecure microprocesses, risking the exposure of protected health information (PHI) to the public internet or its inadvertent incorporation into AI model training. To address these challenges, our institution implemented a secure pathway to the Azure OpenAI Service using our own private OpenAI instance which we fully control to facilitate high-throughput, secure LLM queries. This pathway ensures data privacy while allowing researchers to harness the capabilities of LLMs for diverse healthcare applications. Our approach supports compliant, efficient, and innovative AI research in healthcare. This paper discusses the implementation, advantages, and use cases of this secure infrastructure, underscoring the critical need for centralized, secure AI solutions in academic medical environments.

  View details for DOI 10.21203/rs.3.rs-5095287/v1

  View details for PubMedID 39399679

• The Need for Continuous Evaluation of Artificial Intelligence Prediction Algorithms. JAMA network open Shah, N. H., Pfeffer, M. A., Ghassemi, M. 2024; 7 (9): e2433009

  View details for DOI 10.1001/jamanetworkopen.2024.33009

  View details for PubMedID 39264634

• Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages. JAMA network open Garcia, P., Ma, S. P., Shah, S., Smith, M., Jeong, Y., Devon-Sand, A., Tai-Seale, M., Takazawa, K., Clutter, D., Vogt, K., Lugtu, C., Rojo, M., Lin, S., Shanafelt, T., Pfeffer, M. A., Sharp, C. 2024; 7 (3): e243201

  Abstract

  The emergence and promise of generative artificial intelligence (AI) represent a turning point for health care. Rigorous evaluation of generative AI deployment in clinical practice is needed to inform strategic decision-making.To evaluate the implementation of a large language model used to draft responses to patient messages in the electronic inbox.A 5-week, prospective, single-group quality improvement study was conducted from July 10 through August 13, 2023, at a single academic medical center (Stanford Health Care). All attending physicians, advanced practice practitioners, clinic nurses, and clinical pharmacists from the Divisions of Primary Care and Gastroenterology and Hepatology were enrolled in the pilot.Draft replies to patient portal messages generated by a Health Insurance Portability and Accountability Act-compliant electronic health record-integrated large language model.The primary outcome was AI-generated draft reply utilization as a percentage of total patient message replies. Secondary outcomes included changes in time measures and clinician experience as assessed by survey.A total of 197 clinicians were enrolled in the pilot; 35 clinicians who were prepilot beta users, out of office, or not tied to a specific ambulatory clinic were excluded, leaving 162 clinicians included in the analysis. The survey analysis cohort consisted of 73 participants (45.1%) who completed both the presurvey and postsurvey. In gastroenterology and hepatology, there were 58 physicians and APPs and 10 nurses. In primary care, there were 83 physicians and APPs, 4 nurses, and 8 clinical pharmacists. The mean AI-generated draft response utilization rate across clinicians was 20%. There was no change in reply action time, write time, or read time between the prepilot and pilot periods. There were statistically significant reductions in the 4-item physician task load score derivative (mean [SD], 61.31 [17.23] presurvey vs 47.26 [17.11] postsurvey; paired difference, -13.87; 95% CI, -17.38 to -9.50; P < .001) and work exhaustion scores (mean [SD], 1.95 [0.79] presurvey vs 1.62 [0.68] postsurvey; paired difference, -0.33; 95% CI, -0.50 to -0.17; P < .001).In this quality improvement study of an early implementation of generative AI, there was notable adoption, usability, and improvement in assessments of burden and burnout. There was no improvement in time. Further code-to-bedside testing is needed to guide future development and organizational strategy.

  View details for DOI 10.1001/jamanetworkopen.2024.3201

  View details for PubMedID 38506805

• MEDALIGN: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records Fleming, S. L., Lozano, A., Haberkorn, W. J., Jindal, J. A., Reis, E., Thapa, R., Blankemeier, L., Genkins, J. Z., Steinberg, E., Nayak, A., Patel, B., Chiang, C., Callahan, A., Huo, Z., Gatidis, S., Adams, S., Fayanju, O., Shah, S. J., Savage, T., Goh, E., Chaudhari, A. S., Aghaeepour, N., Sharp, C., Pfeffer, M. A., Liang, P., Chen, J. H., Morse, K. E., Brunskill, E. P., Fries, J. A., Shah, N. H., Wooldridge, M., Dy, J., Natarajan, S. ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE. 2024: 22021-22030

  View details for Web of Science ID 001239985800017

• Balancing Innovation and Cybersecurity in Medical Schools and Their Related Academic Health Systems. Academic medicine : journal of the Association of American Medical Colleges Halaas, M., Pfeffer, M. A., Roberts, L. W. 2023; 98 (11): 1233-1234

  View details for DOI 10.1097/ACM.0000000000005436

  View details for PubMedID 37881963

• Evaluating the predictive ability of natural language processing in identifying tertiary/quaternary cases in prioritization workflows for interhospital transfer. JAMIA open Lee, T., Lukac, P. J., Vangala, S., Kowsari, K., Vu, V., Fogelman, S., Pfeffer, M. A., Bell, D. S. 2023; 6 (3): ooad069

  Abstract

  Tertiary and quaternary (TQ) care refers to complex cases requiring highly specialized health services. Our study aimed to compare the ability of a natural language processing (NLP) model to an existing human workflow in predictively identifying TQ cases for transfer requests to an academic health center.Data on interhospital transfers were queried from the electronic health record for the 6-month period from July 1, 2020 to December 31, 2020. The NLP model was allowed to generate predictions on the same cases as the human predictive workflow during the study period. These predictions were then retrospectively compared to the true TQ outcomes.There were 1895 transfer cases labeled by both the human predictive workflow and the NLP model, all of which had retrospective confirmation of the true TQ label. The NLP model receiver operating characteristic curve had an area under the curve of 0.91. Using a model probability threshold of ≥0.3 to be considered TQ positive, accuracy was 81.5% for the NLP model versus 80.3% for the human predictions (P = .198) while sensitivity was 83.6% versus 67.7% (P<.001).The NLP model was as accurate as the human workflow but significantly more sensitive. This translated to 15.9% more TQ cases identified by the NLP model.Integrating an NLP model into existing workflows as automated decision support could translate to more TQ cases identified at the onset of the transfer process.

  View details for DOI 10.1093/jamiaopen/ooad069

  View details for PubMedID 37600073

  View details for PubMedCentralID PMC10435371

• The Stanford Medicine data science ecosystem for clinical and translational research. JAMIA open Callahan, A., Ashley, E., Datta, S., Desai, P., Ferris, T. A., Fries, J. A., Halaas, M., Langlotz, C. P., Mackey, S., Posada, J. D., Pfeffer, M. A., Shah, N. H. 2023; 6 (3): ooad054

  Abstract

  To describe the infrastructure, tools, and services developed at Stanford Medicine to maintain its data science ecosystem and research patient data repository for clinical and translational research.The data science ecosystem, dubbed the Stanford Data Science Resources (SDSR), includes infrastructure and tools to create, search, retrieve, and analyze patient data, as well as services for data deidentification, linkage, and processing to extract high-value information from healthcare IT systems. Data are made available via self-service and concierge access, on HIPAA compliant secure computing infrastructure supported by in-depth user training.The Stanford Medicine Research Data Repository (STARR) functions as the SDSR data integration point, and includes electronic medical records, clinical images, text, bedside monitoring data and HL7 messages. SDSR tools include tools for electronic phenotyping, cohort building, and a search engine for patient timelines. The SDSR supports patient data collection, reproducible research, and teaching using healthcare data, and facilitates industry collaborations and large-scale observational studies.Research patient data repositories and their underlying data science infrastructure are essential to realizing a learning health system and advancing the mission of academic medical centers. Challenges to maintaining the SDSR include ensuring sufficient financial support while providing researchers and clinicians with maximal access to data and digital infrastructure, balancing tool development with user training, and supporting the diverse needs of users.Our experience maintaining the SDSR offers a case study for academic medical centers developing data science and research informatics infrastructure.

  View details for DOI 10.1093/jamiaopen/ooad054

  View details for PubMedID 37545984

  View details for PubMedCentralID PMC10397535

• Creation and Adoption of Large Language Models in Medicine. JAMA Shah, N. H., Entwistle, D., Pfeffer, M. A. 2023

  Abstract

  Importance: There is increased interest in and potential benefits from using large language models (LLMs) in medicine. However, by simply wondering how the LLMs and the applications powered by them will reshape medicine instead of getting actively involved, the agency in shaping how these tools can be used in medicine is lost.Observations: Applications powered by LLMs are increasingly used to perform medical tasks without the underlying language model being trained on medical records and without verifying their purported benefit in performing those tasks.Conclusions and Relevance: The creation and use of LLMs in medicine need to be actively shaped by provisioning relevant training data, specifying the desired benefits, and evaluating the benefits via testing in real-world deployments.

  View details for DOI 10.1001/jama.2023.14217

  View details for PubMedID 37548965

• The shaky foundations of large language models and foundation models for electronic health records. NPJ digital medicine Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., Shah, N. H. 2023; 6 (1): 135

  Abstract

  The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.

  View details for DOI 10.1038/s41746-023-00879-8

  View details for PubMedID 37516790

  View details for PubMedCentralID 8371605

• Assessment of Adherence to Reporting Guidelines by Commonly Used Clinical Prediction Models From a Single Vendor: A Systematic Review. JAMA network open Lu, J. H., Callahan, A., Patel, B. S., Morse, K. E., Dash, D., Pfeffer, M. A., Shah, N. H. 2022; 5 (8): e2227779

  Abstract

  Importance: Various model reporting guidelines have been proposed to ensure clinical prediction models are reliable and fair. However, no consensus exists about which model details are essential to report, and commonalities and differences among reporting guidelines have not been characterized. Furthermore, how well documentation of deployed models adheres to these guidelines has not been studied.Objectives: To assess information requested by model reporting guidelines and whether the documentation for commonly used machine learning models developed by a single vendor provides the information requested.Evidence Review: MEDLINE was queried using machine learning model card and reporting machine learning from November 4 to December 6, 2020. References were reviewed to find additional publications, and publications without specific reporting recommendations were excluded. Similar elements requested for reporting were merged into representative items. Four independent reviewers and 1 adjudicator assessed how often documentation for the most commonly used models developed by a single vendor reported the items.Findings: From 15 model reporting guidelines, 220 unique items were identified that represented the collective reporting requirements. Although 12 items were commonly requested (requested by 10 or more guidelines), 77 items were requested by just 1 guideline. Documentation for 12 commonly used models from a single vendor reported a median of 39% (IQR, 37%-43%; range, 31%-47%) of items from the collective reporting requirements. Many of the commonly requested items had 100% reporting rates, including items concerning outcome definition, area under the receiver operating characteristics curve, internal validation, and intended clinical use. Several items reported half the time or less related to reliability, such as external validation, uncertainty measures, and strategy for handling missing data. Other frequently unreported items related to fairness (summary statistics and subgroup analyses, including for race and ethnicity or sex).Conclusions and Relevance: These findings suggest that consistent reporting recommendations for clinical predictive models are needed for model developers to share necessary information for model deployment. The many published guidelines would, collectively, require reporting more than 200 items. Model documentation from 1 vendor reported the most commonly requested items from model reporting guidelines. However, areas for improvement were identified in reporting items related to model reliability and fairness. This analysis led to feedback to the vendor, which motivated updates to the documentation for future users.

  View details for DOI 10.1001/jamanetworkopen.2022.27779

  View details for PubMedID 35984654

• Considerations in the reliability and fairness audits of predictive models for advance care planning Frontiers in Digital Health Lu, J., Sattler, A., Wang, S., Khaki, A. R., Callahan, A., Fleming, S., Fong, R., Ehlert, B., Li, R., Shieh, L., Ramchandran, K., Gensheimer, M., Chobot, S., Pfohl, S., Li, S., Shum, K., Parikh, N., Desai, P., Seevaratnam, B., Hanson, M., Smith, M., Xu, Y., Gokhale, A., Lin, S., Shah, N. 2022: 943768

  View details for DOI 10.3389/fdgth.2022.943768

• Lower Severe Acute Respiratory Syndrome Coronavirus 2 Viral Shedding Following Coronavirus Disease 2019 Vaccination Among Healthcare Workers in Los Angeles, California OPEN FORUM INFECTIOUS DISEASES Adamson, P. C., Pfeffer, M. A., Arboleda, V. A., Garner, O. B., de St Maurice, A., von Bredow, B., Flint, J., Kruglyak, L., Currier, J. S. 2021; 8 (11)

  View details for DOI 10.1093/ofid/ofab526

  View details for Web of Science ID 000732746200048

• Low prevalence (0.13%) of COVID-19 infection in asymptomatic pre-operative/pre-procedure patients at a large, academic medical center informs approaches to perioperative care SURGERY Singer, J. S., Cheng, E. M., Murad, D. A., Maurice, A., Hines, O., Uslan, D. Z., Garner, O., Pregler, J., Bukata, S. V., Pfeffer, M. A., Cherry, R. A. 2020; 168 (6): 980-986

  Abstract

  The coronavirus disease 2019 (COVID-19) pandemic has resulted in reduced performance of elective surgeries and procedures at medical centers across the United States. Awareness of the prevalence of asymptomatic disease is critical for guiding safe approaches to operative/procedural services. As COVID-19 polymerase chain reaction (PCR) testing has been limited largely to symptomatic patients, health care workers, or to those in communal care centers, data regarding asymptomatic viral disease carriage are limited.In this retrospective observational case series evaluating UCLA Health patients enrolled in pre-operative/pre-procedure protocol COVID-19 reverse transcriptase (RT)-PCR testing between April 7, 2020 and May 21, 2020, we determine the prevalence of COVID-19 infection in asymptomatic patients scheduled for surgeries and procedures.Primary outcomes include the prevalence of COVID-19 infection in this asymptomatic population. Secondary data analysis includes overall population testing results and population demographics. Eighteen of 4,751 (0.38%) patients scheduled for upcoming surgeries and high-risk procedures had abnormal (positive/inconclusive) COVID-19 RT-PCR testing results. Six of 18 patients were confirmed asymptomatic and had positive test results. Four of 18 were confirmed asymptomtic and had inconclusive results. Eight of 18 had positive results in the setting of recent symptoms or known COVID-19 infection. The prevalence of asymptomatic COVID-19 infection was 0.13%. More than 90% of patients had residential addresses within a 67-mile geographic radius of our medical center, the median age was 58, and there was equal male/female distribution.These data demonstrating low levels (0.13% prevalence) of COVID-19 infection in an asymptomatic population of patients undergoing scheduled surgeries/procedures in a large urban area have helped to inform perioperative protocols during the COVID-19 pandemic. Testing protocols like ours may prove valuable for other health systems in their approaches to safe procedural practices during COVID-19.

  View details for DOI 10.1016/j.surg.2020.07.048

  View details for Web of Science ID 000594548300004

  View details for PubMedID 33008615

  View details for PubMedCentralID PMC7427530

• Radiology's Information Architecture Could Migrate to One Emulating That of Smartphones JOURNAL OF THE AMERICAN COLLEGE OF RADIOLOGY Enzmann, D. R., Arnold, C. W., Zaragoza, E., Siegel, E., Pfeffer, M. A. 2020; 17 (10): 1299-1306

  Abstract

  Diagnostic radiology (DxR), having had successful serial co-evolutions with imaging equipment and PACS, is faced with another. With a backdrop termed "globotics transition," it should create an IT and informatics infrastructure capable of integrating artificial intelligence (AI) into current critical communication functions of PACS and incorporating functions currently residing in balkanized products. DxR will face the challenge of adopting sustaining and disruptive AI innovations simultaneously. In this co-evolution, a major selection force for AI will be increasing the flow of information and patients; "increasing" means faster flow over larger areas defined by geography and content. Larger content includes a broader spectrum of imaging and nonimaging information streams that facilitate medical decision making. Evolution to faster flow will gravitate toward a hierarchical IT architecture consisting of many small channels feeding into fewer larger channels, something potentially difficult for current PACS. Smartphone-like architecture optimized for communication and integration could provide a large-channel backbone and many smaller feeding channels for basic functions, as well as those needing to innovate rapidly. New, more flexible architectures stimulate market competition in which DxR could act as an artificial selection force to influence development of faster increased flow in current PACS companies, in disruptors such as consolidated AI companies, or in entirely new entrants like Apple or Google. In this co-evolution, DxR should be able to stimulate design of a modern communication medium that increases the flow of information and decreases the time and energy necessary to absorb it, thereby creating even more indispensable clinical value for itself.

  View details for DOI 10.1016/j.jacr.2020.03.032

  View details for Web of Science ID 000577970400018

  View details for PubMedID 32387372

• Excess Patient Visits for Cough and Pulmonary Disease at a Large US Health System in the Months Prior to the COVID-19 Pandemic: Time-Series Analysis JOURNAL OF MEDICAL INTERNET RESEARCH Elmore, J. G., Wang, P., Kerr, K. F., Schriger, D. L., Morrison, D. E., Brookmeyer, R., Pfeffer, M. A., Payne, T. H., Currier, J. S. 2020; 22 (9): e21562

  Abstract

  Accurately assessing the regional activity of diseases such as COVID-19 is important in guiding public health interventions. Leveraging electronic health records (EHRs) to monitor outpatient clinical encounters may lead to the identification of emerging outbreaks.The aim of this study is to investigate whether excess visits where the word "cough" was present in the EHR reason for visit, and hospitalizations with acute respiratory failure were more frequent from December 2019 to February 2020 compared with the preceding 5 years.A retrospective observational cohort was identified from a large US health system with 3 hospitals, over 180 clinics, and 2.5 million patient encounters annually. Data from patient encounters from July 1, 2014, to February 29, 2020, were included. Seasonal autoregressive integrated moving average (SARIMA) time-series models were used to evaluate if the observed winter 2019/2020 rates were higher than the forecast 95% prediction intervals. The estimated excess number of visits and hospitalizations in winter 2019/2020 were calculated compared to previous seasons.The percentage of patients presenting with an EHR reason for visit containing the word "cough" to clinics exceeded the 95% prediction interval the week of December 22, 2019, and was consistently above the 95% prediction interval all 10 weeks through the end of February 2020. Similar trends were noted for emergency department visits and hospitalizations starting December 22, 2019, where observed data exceeded the 95% prediction interval in 6 and 7 of the 10 weeks, respectively. The estimated excess over the 3-month 2019/2020 winter season, obtained by either subtracting the maximum or subtracting the average of the five previous seasons from the current season, was 1.6 or 2.0 excess visits for cough per 1000 outpatient visits, 11.0 or 19.2 excess visits for cough per 1000 emergency department visits, and 21.4 or 39.1 excess visits per 1000 hospitalizations with acute respiratory failure, respectively. The total numbers of excess cases above the 95% predicted forecast interval were 168 cases in the outpatient clinics, 56 cases for the emergency department, and 18 hospitalized with acute respiratory failure.A significantly higher number of patients with respiratory complaints and diseases starting in late December 2019 and continuing through February 2020 suggests community spread of SARS-CoV-2 prior to established clinical awareness and testing capabilities. This provides a case example of how health system analytics combined with EHR data can provide powerful and agile tools for identifying when future trends in patient populations are outside of the expected ranges.

  View details for DOI 10.2196/21562

  View details for Web of Science ID 000579497400005

  View details for PubMedID 32791492

  View details for PubMedCentralID PMC7485935

• Cybersecurity implications for hospital quality HEALTH SERVICES RESEARCH Dameff, C., Pfeffer, M. A., Longhurst, C. A. 2019; 54 (5): 969-970

  View details for DOI 10.1111/1475-6773.13202

  View details for Web of Science ID 000485290700001

  View details for PubMedID 31506957

  View details for PubMedCentralID PMC6736916

• Physicians Voluntarily Using an EHR-Based CDS Tool Improved Patients' Guideline-Related Statin Prescription Rates: A Retrospective Cohort Study APPLIED CLINICAL INFORMATICS Chang, T. S., Buchipudi, A., Fonarow, G. C., Pfeffer, M. A., Singer, J. S., Cheng, E. M. 2019; 10 (3): 421-445

  Abstract

  In 2013, the American College of Cardiology (ACC) and the American Heart Association (AHA) released a revised guideline on statin therapy initiation. The guideline included a 10-year risk calculation based on regression modeling, which made hand calculation infeasible. Compliance to the guideline has been suboptimal, as many patients were recommended but not prescribed statin therapy. Clinical decision support (CDS) tools may improve statin guideline compliance. Few statin guideline CDS tools evaluated clinical outcome.We determined if use of a CDS tool, the statin macro, was associated with increased 2013 ACC/AHA statin guideline compliance at the level of statin prescription versus no statin prescription. We did not determine if each patient's statin prescription met ACC/AHA 2013 therapy intensity recommendations (high vs. moderate vs. low).The authors developed a clinician-initiated, EHR-embedded statin macro command ("statin macro") that displayed the 2013 ACC/AHA statin guideline recommendation in the electronic health record documentation. We included patients who had a primary care visit during the study period (January 1-June 30, 2016), were eligible for statin therapy based on the ACC/AHA guideline prior to the study period, and were not prescribed statin therapy prior to the study period. We tested the association of macro usage and statin therapy prescription during the study period using relative risk and mixed effect logistic regression.Subjects included 11,877 patients seen in primary care, who were retrospectively recommended statin therapy at study initiation based on the ACC/AHA guideline, but who had not received statin therapy. During the study period, 125 clinicians used the statin macro command for 389 of the 11,877 patients (3.2%). Of the 389 patients for whom that statin macro was used, 108 patients (28%) had a statin prescribed during the study period. Of the 11,488 for whom the statin macro was not used, 1,360 (13%) patients received a clinician-prescribed statin (relative risk 2.3, p < 0.001). Controlling for patient covariates and clinicians, statin macro usage was significantly associated with statin therapy prescription (odds ratio 2.86, p < 0.001).Although the statin macro had low uptake, its use was associated with a greater rate of statin prescriptions (dosage not determined) for patients whom 2013 ACC/AHA guidelines required statin therapy.

  View details for DOI 10.1055/s-0039-1692186

  View details for Web of Science ID 000482343000005

  View details for PubMedID 31216590

  View details for PubMedCentralID PMC6584145

• Feasibility study of an EHR-integrated mobile shared decision making application INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS Day, F. C., Pourhomayoun, M. B., Keeves, D., Lees, A. F., Sarrafzadeh, M., Bell, D., Pfeffer, M. A. 2019; 124: 24-30

  Abstract

  Integrating mobile applications (apps) into users' standard electronic health record (EHR) workflows may be valuable, especially for apps that both read and write data. This report details the lessons learned during the integration of a patient decision aid - prostate specific antigen (PSA) testing for prostate cancer screening - into our users' standard EHR workflow for a small usability assessment.This feasibility study included two steps. First we enabled realtime, secure bidirectional data exchange between the mobile app and EHR for 14 data elements, and second we pilot tested the production environment app with 9 primary care patients aged 60-65 years. Our primary usability metric was a net promoter score (NPS), based on users' recommendation of the app to a friend or family member; we also assessed the proportion of users who 1) updated their prostate cancer risk factor information present in the EHR and 2) submitted more than one unique response regarding their preference to have PSA testing.The seven web services necessary to read and write data required considerable configuration, but successfully delivered risk factor-specific educational content and recorded patients' values and decision preference directly within the EHR. Seven of the 9 patients (78%) would recommend this app to a friend/family member (NPS = 55.6%), one patient used the app to update risk factor information, and 4/9 (44%) changed their decision preference while using the app.It is feasible to implement a decision aid directly into users' standard EHR workflow for limited usability testing. Broad scale implementation may have a positive effect on patient engagement and improve shared decision making, but several challenges exist with proprietary EHR vendor application programming interfaces (API)s.

  View details for DOI 10.1016/j.ijmedinf.2019.01.008

  View details for Web of Science ID 000458863600004

  View details for PubMedID 30784423

• Impact of Open Access to Physician Notes on Radiation Oncology Patients: Results from an Exploratory Survey PRACTICAL RADIATION ONCOLOGY Shaverdian, N., Chang, E. M., Chu, F., Morasso, E., Pfeffer, M. A., Cheng, E. M., Wu, A., McCloskey, S. A., Raldow, A. C., Steinberg, M. L. 2019; 9 (2): 102-107

  Abstract

  There is an increasing effort to allow patients open access to their physician notes through electronic medical record portals. However, limited data exist on the impact of such access on oncology patients, and concerns remain regarding potential harms. Therefore, we determined the baseline perceptions and impact of open access to oncology notes on radiation oncology patients.Patients receiving radiation therapy were provided instructional materials on accessing oncology notes at the time of their initial evaluation. Patients were prospectively surveyed to evaluate baseline interest and expectations before access and to determine the actual usage and impact at the end of their radiation treatment course.A total of 220 patients were surveyed; 136 (62%) completed the baseline survey, of which 88 (40%) completed the final survey. The majority of participants were age >60 years (n = 83; 61%), and 70 were male (51%). Before accessing the notes, the majority of patients agreed that open access to oncology notes would improve understanding of diagnosis (99%), understanding of treatment side effects (98%), reassurance about treatment goals (96%), and communication with family (99%). All patients who accessed the notes found them to be useful. After accessing the notes, approximately 96%, 94%, and 96% of patients reported an improved understanding of their diagnosis, an improved understanding of treatment side effects, and feeling more reassured about their treatment, respectively. Approximately 11%, 6%, and 4% of patients noted increased worry, increased confusion, and finding information they now regret reading, respectively. Patient age, sex, and specific cancer diagnoses were not predictive of experiencing negative effects from accessing the notes.Radiation oncology patients have a strong interest in open access to their physician notes, and the majority of patients expect and actually report meaningful benefits. These data support strategies to allow more patients with cancer access to their physicians' notes.

  View details for DOI 10.1016/j.prro.2018.10.004

  View details for Web of Science ID 000460044800029

  View details for PubMedID 30342179

• How Patients Use a Patient Portal: An Institutional Case Study of Demographic and Usage Patterns APPLIED CLINICAL INFORMATICS Tsai, R., Bell, E. J., Woo, H., Baldwin, K., Pfeffer, M. A. 2019; 10 (1): 96-102

  Abstract

  Given the widespread electronic health record adoption, there is increasing interest to leverage patient portals to improve care.To determine characteristics of patient portal users and the activities they accessed in the patient portal.We performed a retrospective analysis of patient portal usage at University of California, Los Angeles, Health from July 2014 to May 2015. A total dataset of 505,503 patients was compiled with 396,303 patients who did not register for the patient portal and 109,200 patients who registered for a patient portal account. We compared patients who did not register for the online portal to the top 75th percentile of users based on number of logins, which was done to exclude those who only logged in to register. Finally, to avoid doing statistical analysis on too large of a sample and overpower the analysis, we performed statistical tests on a random sample of 300 patients in each of the two groups.Patient portal users tended to be older (49.45 vs. 46.22 years in the entire sample, p = 0.008 in the random sample) and more likely female (62.59 vs. 54.91% in the entire sample, p = 0.035 in the random sample). Nonusers had more monthly emergency room (ER) visits on average (0.047 vs. 0.014, p < 0.001). The most frequently accessed activity on the portal was viewing laboratory results (79.7% of users looked at laboratory results).There are differences between patient portal users and nonusers, and further understanding of these differences can serve as foundation for further investigation and possible interventions to drive patient engagement and health outcomes.

  View details for DOI 10.1055/s-0038-1677528

  View details for Web of Science ID 000459170000002

  View details for PubMedID 30727003

  View details for PubMedCentralID PMC6365289

• Characteristics of the National Applicant Pool for Clinical Informatics Fellowships (2016-2017). AMIA ... Annual Symposium proceedings. AMIA Symposium Dell, D. S., Baldwin, K., Bell, E. J., Lehmann, C. U., Webber, E. C., Mohan, V., Leu, M. G., Hofmann, J. M., Kaelber, D. C., Landman, A. B., Hron, J., Silverman, H. D., Levy, B., Elkin, P. L., Poon, E., Luberti, A. A., Finnell, J. T., Safran, C., Palma, J. P., Forman, B. H., Kileen, J., Arvin, D., Pfeffer, M. 2018; 2018: 225–31

  Abstract

  We conducted a national study to assess the numbers and diversity of applicants for 2016 and 2017 clinical informatics fellowship positions. In each year, we collected data on the number of applications that programs received from candidates who were ultimately successful vs. unsuccessful. In 2017, we also conducted an anonymous applicant survey. Successful candidates applied to an average of 4.2 and 5.5 programs for 2016 and 2017, respectively. In the survey, unsuccessful candidates reported applying to fewer programs. Assuming unsuccessful candidates submitted between 2-5 applications each, the total applicant pool numbered 42-69 for 2016 (competing for 24 positions) and 52-85 for 2017 (competing for 30 positions). Among survey respondents (n=33), 24% were female, 1 was black and none were Hispanic. We conclude that greater efforts are needed to enhance interest in clinical informatics among medical students and residents, particularly among women and members of underrepresented minority groups.

  View details for PubMedID 30815060

• The UCLA Health Resident Informaticist Program - A Novel Clinical Informatics Training Program JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION Singer, J. S., Cheng, E. M., Baldwin, K., Pfeffer, M. A., UCLA Hlth Phys Informaticist Comm 2017; 24 (4): 832-840

  Abstract

  Few opportunities exist for physician trainees to gain exposure to, and training in, the field of clinical informatics, an Accreditation Council for Graduate Medical Education-accredited, recently board-certified specialty. Currently, 21 approved programs exist nationwide for the formal training of fellows interested in pursuing careers in this discipline. Residents and fellows training in medical and surgical fields, however, have few avenues available to gain experience in clinical informatics. An early introduction to clinical informatics brings an opportunity to generate interest for future career trajectories. At University of California Los Angeles (UCLA) Health, we have developed a novel, successful, and sustainable program, the Resident Informaticist Program, with the goals of exposing physician trainees to the field of clinical informatics and its academic nature and providing opportunities to expand the clinical informatics workforce. Herein, we provide an overview of the development, implementation, and current state of the UCLA Health Resident Informaticist Program, with a blueprint for development of similar programs.

  View details for DOI 10.1093/jamia/ocw174

  View details for Web of Science ID 000405618200020

  View details for PubMedID 28115427

  View details for PubMedCentralID PMC7651961

• Managing Scale and Innovation in Health IT JOURNAL OF THE AMERICAN COLLEGE OF RADIOLOGY Enzmann, D. R., Pfeffer, M. 2016; 13 (9): 1135-1138

  Abstract

  Given the high-intensity interaction between radiology and IT, radiology leadership should understand IT's new, somewhat conflicting, dual roles. Managing large-scale and small-scale projects concurrently has become an important challenge for leaders of health IT (HIT). Historical parallels of this challenge can be drawn from transportation and communication systems, in which a large-scale mind-set is needed to build the initial network, whereas a small-scale mind-set is more useful to develop the content that will traverse this network. Innovation and creativity is a cornerstone of content small-scale thinking, and in HIT, that is what is needed to extract the value from it. However, unlike the early historical transportation and communication examples, the time between the development of the infrastructure and the follow-on, value-rich content is shortened greatly because it has become nearly simultaneous in HIT. Weaving the ability to concomitantly manage both large- and small-scale projects into the fabric of the organizational HIT culture will be critical for its success.

  View details for DOI 10.1016/j.jacr.2016.02.024

  View details for Web of Science ID 000383313400027

  View details for PubMedID 27039000

• A Systematic Approach to Creation of a Perioperative Data Warehouse ANESTHESIA AND ANALGESIA Hofer, I. S., Gabel, E., Pfeffer, M., Mahbouba, M., Mahajan, A. 2016; 122 (6): 1880-1884

  Abstract

  Extraction of data from the electronic medical record is becoming increasingly important for quality improvement initiatives such as the American Society of Anesthesiologists Perioperative Surgical Home. To meet this need, the authors have built a robust and scalable data mart based on their implementation of EPIC containing data from across the perioperative period. The data mart is structured in such a way so as to first simplify the overall EPIC reporting structure into a series of Base Tables and then create several Reporting Schemas each around a specific concept (operating room cases, obstetrics, hospital admission, etc.), which contain all of the data required for reporting on various metrics. This structure allows centralized definitions with simplified reporting by a large number of individuals who access only the Reporting Schemas. In creating the database, the authors were able to significantly reduce the number of required table identifiers from >10 to 3, as well as to correct errors in linkages affecting up to 18.4% of cases. In addition, the data mart greatly simplified the code required to extract data, making the data accessible to individuals who lacked a strong coding background. Overall, this infrastructure represents a scalable way to successfully report on perioperative EPIC data while standardizing the definitions and improving access for end users.

  View details for DOI 10.1213/ANE.0000000000001201

  View details for Web of Science ID 000376463000025

  View details for PubMedID 27195633

• A Bad Case of Good's Syndrome INFECTIOUS DISEASES AND THERAPY Tachdjian, R., Keller, J. J., Pfeffer, M. 2014; 3 (2): 333-337

  Abstract

  Good's syndrome is a relatively rare immunodeficiency condition that presents in the fourth or fifth decade of life and is defined by hypogammaglobulinemia in the setting of a thymoma. The humoral defect may be severe enough to cause an absence in B cells, with a consequent recurrence of sinopulmonary disease, chronic non-infectious diarrhea and opportunistic infections. The prognosis in patients with Good's syndrome appears to be worse than in those with X-linked agammaglobulinemia (XLA) and common variable immune deficiency (CVID). There have only been three cases of Good's syndrome associated with mycobacterium, and only one case with a cavitary lesion in the lungs. We present here a unique case of Good's syndrome with a non-mycobacterial cavitary lesion.

  View details for DOI 10.1007/s40121-014-0045-7

  View details for Web of Science ID 000215319300021

  View details for PubMedID 25287948

• Acute Aortic Dissection HOSPITAL MEDICINE CLINICS Pfeffer, M. A. 2012; 1 (1): E1-E11

  View details for DOI 10.1016/j.ehmc.2011.10.002

  View details for Web of Science ID 000219487100003

• Liquid crystal surface anchoring of mesophase pitch CARBON Jian, K. Q., Shim, H. S., Tuhus-Dubrow, D., Bernstein, S., Woodward, C., Pfeffer, M., Steingart, D., Gournay, T., Sachsmann, S., Crawford, G. P., Hurt, R. H. 2003; 41 (11): 2073-2083

  View details for DOI 10.1016/S0008-6223(03)00203-3

  View details for Web of Science ID 000184889200006