Bio


Holly Caretta-Weyer is currently Associate Dean for Admissions and Director of Assessment at the Stanford University School of Medicine. She is additionally the Director of Assessment for the Department of Emergency Medicine and Chair of the Clinical Competency Committee. Dr. Caretta-Weyer attended medical school at the University of Wisconsin School of Medicine and Public Health where she graduated Alpha Omega Alpha with Honors in Research. She loved being a Badger so much that she stayed for her Emergency Medicine Residency at the University of Wisconsin where she was also Chief Resident. Dr. Caretta-Weyer then moved to the West Coast where she completed her Medical Education Scholarship Fellowship at Oregon Health & Science University (OHSU) and completed her Masters in Health Professions Education (MHPE) at the University of Illinois-Chicago. She is currently a PhD candidate at Maastricht University studying postgraduate selection in a competency-based system with an anticipated completion date in 2025.

While at OHSU, Dr. Caretta-Weyer worked as a member of the Association of American Medical Colleges (AAMC) Core Entrustable Professional Activities for Entering Residency pilot team and was a founding member of the OHSU undergraduate medical education entrustment committee. She continues to be involved with the national AAMC Core EPA Pilot through her continued collaboration with the OHSU team. Through this process she has gained valuable experience in working to define programmatic assessment, formulate summative entrustment decisions, and more seamlessly bridge the transition from undergraduate to graduate medical education, all of which are key initiatives within medical education.

Dr. Caretta-Weyer is also the PI on a $1.3M AMA Reimagining Residency Grant focused on implementing competency-based education and redesigning assessment across the continuum of emergency medicine training and introducing predictive learning analytics to the process. She is a former Visiting Scholar with the American Board of Medical Specialties (ABMS) examining summative entrustment decision-making by competency committees and its implications for initial certification. She is additionally a member of the International Competency-Based Medical Education (ICBME) Collaborators, a group that seeks to further research on CBME around the world. Finally, Dr. Caretta-Weyer was recently elected as the inaugural Chair of the CBME Task Force for Emergency Medicine. Her work led the Royal College of Physicians and Surgeons of Canada to recognize her as the International Medical Educator of the Year in 2022. She also recently was a keynote speaker for the American Board of Medical Specialties and the Society for Academic Emergency Medicine's annual meetings.

Dr. Caretta-Weyer's education research interests focus on the implementation of competency-based education and assessment across the continuum of medical education, summative entrustment and promotion decision-making processes, residency selection in a competency-based system, and the development of learner handovers to span key transitions in the educational continuum. When not focusing on her administrative work and education research, Dr. Caretta-Weyer can be found kayaking, hiking, cycling, playing volleyball, or cheering on her favorite sports teams including the Marquette Golden Eagles and Milwaukee Brewers.

Clinical Focus


  • Emergency Medicine

Academic Appointments


Administrative Appointments


  • Associate Dean of Admissions, Stanford School of Medicine (2023 - Present)
  • Director of Assessment, Stanford School of Medicine (2023 - Present)
  • Clinical Competency Committee Chair, Stanford University Emergency Medicine Residency Program (2018 - Present)
  • Director of Evaluation and Assessment, Stanford University Emergency Medicine Residency Program (2018 - Present)

Honors & Awards


  • International Educator of the Year, Royal College of Physicians and Surgeons of Canada (2022)
  • Young Alumna Award, Marquette University (2022)
  • Early Educator Award, Society for Academic Emergency Medicine (2022)
  • Top Education Research Paper, International Conference on Residency Education (ICRE) (2020)
  • Top 10 Education Research Abstract, Accreditation Council for Graduate Medical Education (ACGME) (2020)
  • Outstanding Educator Award, Stanford University Department of Emergency Medicine (2019)
  • Outstanding Peer Reviewer, Academic Emergency Medicine Education and Training (2019)
  • Academic Instructor of the Year Award, Oregon Health and Science University Department of Emergency Medicine (2018)
  • Resident Advocate Award, Oregon Health and Science University Department of Emergency Medicine (2018)
  • Best Poster Award - Education Research Category, Oregon Health & Science University Symposium on Educational Excellence (2018)
  • Academic Instructor of the Year Award, Oregon Health and Science University Department of Emergency Medicine (2017)
  • Gold Standard Peer Review Award, Western Journal of Emergency Medicine (WestJEM) Education Supplement (2017)
  • The Ripple Award for Excellence in Leadership and Service, University of Wisconsin Department of Emergency Medicine (2016)
  • Award for Excellence in Scholarship, Wisconsin Chapter of the American College of Emergency Physicians (2015)
  • Alpha Omega Alpha Honor Society Induction, University of Wisconsin School of Medicine (2012)

Boards, Advisory Committees, Professional Organizations


  • Editorial Board Member, Academic Emergency Medicine Education and Training (2020 - Present)
  • Director of Grants, ARMED MedEd - SAEM (2020 - Present)
  • Member-at-Large, International Competency-Based Medical Education Collaborators (2019 - Present)
  • Grant Reviewer, Society for Academic Emergency Medicine (2019 - Present)
  • Section Editor, Western Journal of Emergency Medicine - Education Scholarship Section (2018 - Present)

Professional Education


  • PhD Candidate, Maastricht University, Medical Education
  • MHPE, University of Illinois - Chicago, Health Professions Education (2021)
  • Fellowship: Oregon Health and Science University Emergency Medicine Fellowships (2018) OR
  • Board Certification: American Board of Emergency Medicine, Emergency Medicine (2017)
  • Residency: University of Wisconsin Emergency Medicine Residency (2016) WI
  • Medical Education: University of Wisconsin School of Medicine and Public Health (2013) WI

2023-24 Courses


All Publications


  • Better together: A multistakeholder approach to developing specialty-wide entrustable professional activities in emergency medicine. AEM education and training Caretta-Weyer, H. A., Sebok-Syer, S. S., Morris, A. M., Schnapp, B. H., Fant, A. L., Scott, K. R., Pirotte, M., Gisondi, M. A., Yarris, L. M. 2024; 8 (2): e10974

    Abstract

    Entrustable professional activities (EPAs) are a widely used framework for curriculum and assessment, yet the variability in emergency medicine (EM) training programs mandates the development of EPAs that meet the needs of the specialty as a whole. This requires eliciting and incorporating the perspectives of multiple stakeholders (i.e., faculty, residents, and patients) in the development of EPAs. Without a shared understanding of what a resident must be able to do upon graduation, we run the risk of advancing ill-prepared residents that may provide inconsistent care.In an effort to address these challenges, beginning in February 2020, the authors assembled an advisory board of 25 EM faculty to draft and reach consensus on a final list of EPAs that can be used across all training programs within the specialty of EM. Using modified Delphi methodology, the authors came to consensus on an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors, residents, and patients for refinement. The authors collated and analyzed feedback from focus groups of residents and patients using thematic analysis. The EPAs were subsequently refined based on this feedback.Stakeholders in EM residency training endorsed a final revised list of 22 EPAs. Stakeholder focus groups highlighted two main thematic considerations that helped shape the finalized list of EM EPAs: attention to the meaningful nuances of EPA language and contextualizing the EPAs and viewing them developmentally.To foreground all key stakeholders within the EPA process for EM, the authors chose within the development process to draft; come to consensus; and refine EPAs for EM in collaboration with relevant faculty, patient, and resident stakeholders. Each stakeholder group contributed meaningfully to the content and intended implementation of the EPAs. This process may serve as a model for others in developing stakeholder-responsive EPAs.

    View details for DOI 10.1002/aet2.10974

    View details for PubMedID 38532740

    View details for PubMedCentralID PMC10962124

  • Development of entrustable professional activities for emergency medicine medical education fellowships: A modified Delphi study AEM EDUCATION AND TRAINING Villa, S., Caretta-Weyer, H., Yarris, L. M., Clarke, S. O., Coates, W. C., Sokol, K. A., Jurvis, A., Papanagnou, D., Ahn, J., Hillman, E., Camejo, M., Deiorio, N., Fischer, K. M., Wolff, M., Estes, M., Dimeo, S., Jordan, J. 2024; 8 (1)

    View details for DOI 10.1002/aet2.10944

    View details for Web of Science ID 001169064600001

  • Development of entrustable professional activities for emergency medicine medical education fellowships: A modified Delphi study. AEM education and training Villa, S., Caretta-Weyer, H., Yarris, L. M., Clarke, S. O., Coates, W. C., Sokol, K. A., Jurvis, A., Papanagnou, D., Ahn, J., Hillman, E., Camejo, M., Deiorio, N., Fischer, K. M., Wolff, M., Estes, M., Dimeo, S., Jordan, J. 2024; 8 (1): e10944

    Abstract

    It is essential that medical education (MedEd) fellows achieve desired outcomes prior to graduation. Despite the increase in postgraduate MedEd fellowships in emergency medicine (EM), there is no consistently applied competency framework. We sought to develop entrustable professional activities (EPAs) for EM MedEd fellows.From 2021 to 2022, we used a modified Delphi method to achieve consensus for EPAs. EM education experts generated an initial list of 173 EPAs after literature review. In each Delphi round, panelists were asked to make a binary choice of whether to include the EPA. We determined an inclusion threshold of 70% agreement a priori. After the first round, given the large number of EPAs meeting inclusion threshold, panelists were instructed to vote whether each EPA should be included in the "20 most important" EPAs for a MedEd fellowship. Modifications were made between rounds based on expert feedback. We calculated descriptive statistics.Seventeen experts completed four Delphi rounds each with 100% response. After Round 1, 87 EPAs were eliminated and two were combined. Following Round 2, 46 EPAs were eliminated, seven were combined, and three were included in the final list. After the third round, one EPA was eliminated and 13 were included. After the fourth round, 11 EPAs were eliminated. The final list consisted of 16 EPAs in domains of career development, education theory and methods, research and scholarship, and educational program administration.We developed a list of 16 EPAs for EM MedEd fellowships, the first step in implementing competency-based MedEd.

    View details for DOI 10.1002/aet2.10944

    View details for PubMedID 38504805

    View details for PubMedCentralID PMC10950010

  • The Next Era of Assessment: Building a Trustworthy Assessment System. Perspectives on medical education Caretta-Weyer, H. A., Smirnova, A., Barone, M. A., Frank, J. R., Hernandez-Boussard, T., Levinson, D., Lombarts, K. M., Lomis, K. D., Martini, A., Schumacher, D. J., Turner, D. A., Schuh, A. 2024; 13 (1): 12-23

    Abstract

    Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

    View details for DOI 10.5334/pme.1110

    View details for PubMedID 38274558

  • Finding Medicine's Moneyball: How Lessons from Major League Baseball Can Advance Assessment in Precision Education. Academic medicine : journal of the Association of American Medical Colleges Kinnear, B., Caretta-Weyer, H., Lam, A. C., Tang, B., Ginsburg, S., Wong, B. M., Kelleher, M., Schumacher, D. J., Warm, E. J. 2023

    Abstract

    Precision education (PE) leverages longitudinal data and analytics to tailor educational interventions to improve patient, learner, and system-level outcomes. At present, few programs in medical education can accomplish this goal as they must develop new data streams transformed by analytics to drive trainee learning and program improvement. Other professions, such as Major League Baseball (MLB), have already developed extremely sophisticated approaches to gathering large volumes of precise data points to inform assessment of individual performance.In this perspective, the authors argue that medical education-whose entry into precision assessment is fairly nascent-can look to MLB to learn the possibilities and pitfalls of precision assessment strategies. They describe 3 epochs of player assessment in MLB: observation, analytics (sabermetrics), and technology (Statcast). The longest tenured approach, observation, relies on scouting and expert opinion. Sabermetrics brought new approaches to analyzing existing data in a way that better predicted which players would help the team win. Statcast created precise, granular data about highly attributable elements of player performance while helping to account for non-player factors that confound assessment such as weather, ballpark dimensions, and the performance of other players. Medical education is progressing through similar epochs marked by workplace-based assessment, learning analytics, and novel measurement technologies. The authors explore how medical education can leverage intersectional concepts of MLB player and medical trainee assessment to inform present and future directions of PE.

    View details for DOI 10.1097/ACM.0000000000005600

    View details for PubMedID 38109661

  • Lessons from Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment. Academic medicine : journal of the Association of American Medical Colleges Caretta-Weyer, H. A., Schumacher, D. J., Kinnear, B. 2023

    Abstract

    In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that four key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

    View details for DOI 10.1097/ACM.0000000000005578

    View details for PubMedID 38011041

  • Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values. Academic medicine : journal of the Association of American Medical Colleges Caretta-Weyer, H. A., Eva, K. W., Schumacher, D. J., Yarris, L. M., Teunissen, P. W. 2023; 98 (11S): S98-S107

    Abstract

    The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

    View details for DOI 10.1097/ACM.0000000000005365

    View details for PubMedID 37983402

  • Educator's blueprint: A primer on consensus methods in medical education research. AEM education and training Gottlieb, M., Caretta-Weyer, H., Chan, T. M., Humphrey-Murto, S. 2023; 7 (4): e10891

    Abstract

    Consensus methods such as the Delphi and nominal group techniques are increasingly utilized within medical education research. This educator's blueprint paper provides practical strategies regarding five key steps for ensuring best practices when using consensus methods. These strategies include deciding which consensus method is best, developing the initial questionnaire, identifying the participants, determining the number of rounds and consensus threshold, and describing and justifying any modifications. These strategies will help guide education researchers on their next study using consensus methods.

    View details for DOI 10.1002/aet2.10891

    View details for PubMedID 37448627

  • Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values. Academic medicine : journal of the Association of American Medical Colleges Caretta-Weyer, H. A., Eva, K. W., Schumacher, D. J., Yarris, L. M., Teunissen, P. W. 2023

    Abstract

    The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature in order to explore practical implications and present a research agenda.Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.Five distinct perspectives or value statements were identified in the thematic analysis: (1) Utilizing available metrics to predict performance in postgraduate training; (2) Identifying the best applicants via competitive comparison; (3) Seeking alignment between applicant and program in the selection process; (4) Ensuring diversity, mitigation of bias, and equity in the selection process; and (5) Optimizing the logistics or mechanics of the selection process.This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

    View details for DOI 10.1097/ACM.0000000000005365

    View details for PubMedID 37556804

  • Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs. BMC medical education Williams, S. R., Sebok-Syer, S. S., Caretta-Weyer, H., Katznelson, L., Dohn, A. M., Park, Y. S., Gisondi, M. A., Tekian, A. 2023; 23 (1): 434

    Abstract

    Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions.Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses.687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame.Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

    View details for DOI 10.1186/s12909-023-04355-5

    View details for PubMedID 37312085

  • Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School. Academic medicine : journal of the Association of American Medical Colleges Ryan, M. S., Lomis, K. D., Deiorio, N. M., Cutrer, W. B., Pusic, M. V., Caretta-Weyer, H. A. 2023

    Abstract

    Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root-cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root-cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram, and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

    View details for DOI 10.1097/ACM.0000000000005220

    View details for PubMedID 36972129

  • Learner Education Handover: Moving Beyond Educational Silos. Academic medicine : journal of the Association of American Medical Colleges Shaw, T., Mistry, N. P., Caretta-Weyer, H., Humphrey-Murto, S. 2023

    View details for DOI 10.1097/ACM.0000000000005201

    View details for PubMedID 36862632

  • The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency. Teaching and learning in medicine Caretta-Weyer, H. A., Park, Y. S., Tekian, A., Sebok-Syer, S. S. 2023: 1-9

    Abstract

    Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

    View details for DOI 10.1080/10401334.2023.2178438

    View details for PubMedID 36794363

  • TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs): A Scalable Approach for Linking Education to Patient Care. Perspectives on medical education Burk-Rafel, J., Sebok-Syer, S. S., Santen, S. A., Jiang, J., Caretta-Weyer, H. A., Iturrate, E., Kelleher, M., Warm, E. J., Schumacher, D. J., Kinnear, B. 2023; 12 (1): 149-159

    Abstract

    Competency-based medical education (CBME) is an outcomes-based approach to education and assessment that focuses on what competencies trainees need to learn in order to provide effective patient care. Despite this goal of providing quality patient care, trainees rarely receive measures of their clinical performance. This is problematic because defining a trainee's learning progression requires measuring their clinical performance. Traditional clinical performance measures (CPMs) are often met with skepticism from trainees given their poor individual-level attribution. Resident-sensitive quality measures (RSQMs) are attributable to individuals, but lack the expeditiousness needed to deliver timely feedback and can be difficult to automate at scale across programs. In this eye opener, the authors present a conceptual framework for a new type of measure - TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs) - attuned to both automation and trainee attribution as the next evolutionary step in linking education to patient care. TRACERs have five defining characteristics: meaningful (for patient care and trainees), attributable (sufficiently to the trainee of interest), automatable (minimal human input once fully implemented), scalable (across electronic health records [EHRs] and training environments), and real-time (amenable to formative educational feedback loops). Ideally, TRACERs optimize all five characteristics to the greatest degree possible. TRACERs are uniquely focused on measures of clinical performance that are captured in the EHR, whether routinely collected or generated using sophisticated analytics, and are intended to complement (not replace) other sources of assessment data. TRACERs have the potential to contribute to a national system of high-density, trainee-attributable, patient-centered outcome measures.

    View details for DOI 10.5334/pme.1013

    View details for PubMedID 37215538

  • Filling the Core EPA 10 assessment void: A framework for individual assessment of Core Entrustable professional activity 10 competencies in medical students. AEM education and training Miller, D. T., Gibb, W., Caretta-Weyer, H., Ng, K., Sebok-Syer, S. S., Gisondi, M. A. 2022; 6 (6): e10787

    Abstract

    Objectives: The goal of this study was to develop and evaluate a novel curriculum and assessment tool for Core Entrustable Professional Activity (EPA) 10 competencies and entrustment scoring in a cohort of medical students in their emergency medicine (EM) clerkship using a framework of individualized, ad hoc, formative assessment. Core EPA 10 is an observable workplace-based activity for graduating medical students to recognize a patient requiring urgent or emergent care and initiate evaluation and management.Methods: This is a prospective, pretest-posttest study of medical students during their EM clerkship. Using the Thomas and Kern framework, we created a curriculum of simulation cases about chest pain/cardiac arrest and respiratory distress, which included novel assessment checklists, and instructional videos about recognizing and managing emergencies. Students were individually pretested on EPA 10 competencies using the simulation cases. Two raters scored students using standardized checklists. Students then watched instructional videos, underwent a posttest with the simulation cases, and were scored again by the two raters using the checklists. Differences between pretest and posttest scores were analyzed using paired t-tests and Wilcoxon signed-rank tests.Results: Seventy-three out of 85 (86%) students completed the curriculum. Mean scores from pretest to final posttest in the chest pain/cardiac arrest and respiratory distress cases significantly improved from 14.8/19 (SD 1.91), to 17.1/19 (SD=1.00), t(68)=10.56, p<0.001, and 8.5/13 (SD 1.79), to 11.1/13(SD 0.89), t(67)=11.15, p<0.001, respectively. The kappa coefficients were 0.909 (n=2698, p<0.001) and 0.931 (n=1872, p<0.001). Median modified Chen entrustment scores improved from 1b (i.e., "Watch me do this") to 2b (i.e., "I'll watch you") for the chest pain/cardiac arrest case (p<0.001) and 1b/2a (i.e., "Watch me do this"/ "Let's do this together") to 3a (i.e. "You go ahead, and I'll double-check all of your findings") for the respiratory distress case (p<0.001).Conclusion: A new directed curriculum of standardized simulation cases and asynchronous instructional videos improved medical student performance in EPA 10 competencies and entrustment scores. This study provides a curricular framework to support formative individualized assessments for EPA 10.

    View details for DOI 10.1002/aet2.10787

    View details for PubMedID 36389650

  • The next generation of researchers: One-year outcome data from the SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program. AEM education and training Gottlieb, M., Chan, T. M., Sebok-Syer, S. S., Krzyzaniak, S., Dubosh, N. M., Santen, S., Caretta-Weyer, H., Yarris, L., Coates, W. C. 2022; 6 (6): e10818

    Abstract

    As the field of medical education evolves, there is a need to increase the quality of education scholarship and develop a cadre of research scholars; however, clinician educators (CEs) considering this career transition have limited formal training in education research methodology to heed this call. Therefore, a program that provides more advanced training in education scholarship for CEs without the financial and resource barriers of fellowships and masters programs is needed.The SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program is a longitudinal program for the beyond-beginner CE, seeking advanced training in education research. The program was created using a comprehensive needs assessment and included longitudinal training; small-group projects; dedicated project mentors; and integrated diversity, equity, and inclusion initiatives. Program participants applied for a grant upon program completion.Twenty-one participants completed the course with 100% completing the baseline survey and 67% (14/21) completing the end-of-program survey. Participants reported improved perception of knowledge across all of the topics with a medium to large effect size, ranging from 0.40 to 0.62. When asked about impact on their network of potential collaborators, participants reported a median of 7 (interquartile range [IQR] 5-8) out of 9. When asked about the impact on their community of practice, participants reported a median of 7 (IQR 5-7) out of 9. When asked about the impact on their professional identity, participants reported a median of 7 (IQR 4-9) out of 9. Participants also reported an increase in both the quantity (mean of 2 ± 1 new mentors) and the quality (median score 7 [IQR 5-8] out of 9) of new research mentorship as a result of the program. Open-ended feedback was generally positive, with 100% reporting they would advise others to take this program.The SAEM ARMED MedEd program represents a proof of concept for an advanced education research program seeking to fill the research training gap for the beyond-beginner Clinician educators.

    View details for DOI 10.1002/aet2.10818

    View details for PubMedID 36562028

    View details for PubMedCentralID PMC9763966

  • The next generation of researchers: One-year outcome data from the SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program AEM EDUCATION AND TRAINING Gottlieb, M., Chan, T. M., Sebok-Syer, S. S., Krzyzaniak, S., Dubosh, N. M., Santen, S., Caretta-Weyer, H., Yarris, L., Coates, W. C., Emergency Med Hlth Profession 2022; 6 (6)

    View details for DOI 10.1002/aet2.10818

    View details for Web of Science ID 000887953700001

  • Bridging the gap: Development of an experiential learning-based health disparities curriculum. AEM education and training Caretta-Weyer, H. A., Hess, J. M. 2022; 6 (6): e10820

    Abstract

    Background: The increasing number of vulnerable populations served by the emergency department (ED) calls for developing and implementing curricula aimed at training residents to deliver quality care for the most marginalized groups.Objective: We aimed to address this by developing and piloting a curriculum to introduce the unique challenges and disparities encountered by our diverse ED patient population using an experiential learning approach.Methods: We engaged community partners in designing and implementing a curriculum for incoming interns. This curriculum addresses specific populations encountered within the ED including patients with psychiatric illness, patients with cognitive disabilities, intoxicated patients, violent patients, patients of various cultural backgrounds, non-English-speaking patients, and LGBTQ patients. Experiential and active learning sessions with content experts and site visits to area organizations were arranged. Pre-, post-, and time-delayed surveys were deployed to evaluate the pilot of this curriculum.Results: Thirteen incoming interns participated in the orientation curriculum. Residents' comfort with each of these various patient populations as well as familiarity with community and ED resources was assessed before, after, and 1year delayed from each session (response 13/13, 100%). Their scores increased significantly from baseline in the postsurvey and were maintained 1year later (p<0.05). Residents additionally provided narrative responses regarding what they learned and what was most helpful after completing their intern year.Conclusions: This pilot curriculum demonstrates that implementing an experiential learning curriculum and engaging community leaders and resources is key to training residents to address health disparities within their unique ED patient population. As such, it is imperative that we seek to immersively introduce trainees to the unique needs of the patient population they will serve early in training.

    View details for DOI 10.1002/aet2.10820

    View details for PubMedID 36518232

  • Beyond competence: rethinking continuing professional development in the age of competence-based medical education. CJEM Miller, S., Caretta-Weyer, H., Chan, T. 2022

    View details for DOI 10.1007/s43678-022-00372-3

    View details for PubMedID 36071320

  • Creating Community and Exploring Identity: Integrating a Virtual "Museum Tour" Into Intern Orientation. Journal of graduate medical education Balhara, K. S., Weygandt, P. L., Caretta-Weyer, H., Krzyzaniak, S. M., Regan, L., Irvin, N. 2022; 14 (3): 335-336

    View details for DOI 10.4300/JGME-D-21-00994.1

    View details for PubMedID 35754653

    View details for PubMedCentralID PMC9200248

  • The Purpose, Design, and Promise of Medical Education Research Labs. Academic medicine : journal of the Association of American Medical Colleges Gisondi, M. A., Michael, S., Li-Sauerwine, S., Brazil, V., Caretta-Weyer, H. A., Issenberg, B., Giordano, J., Lineberry, M., Olson, A. S., Burkhardt, J. C., Chan, T. M. 2022

    Abstract

    Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

    View details for DOI 10.1097/ACM.0000000000004746

    View details for PubMedID 35612923

  • A mixed-methods needs assessment to identify pharmacology education objectives for emergency medicine residents. Journal of the American College of Emergency Physicians open Rider, A. C., Dang, B. T., Caretta-Weyer, H. A., Schertzer, K. A., Gisondi, M. A. 2022; 3 (2): e12682

    Abstract

    Objectives: Medication errors represent a significant threat to patient safety. Pharmacotherapy is one of the 23 Accreditation Council of Graduate Medical Education milestones for emergency medicine, yet there is minimal understanding of what content should be prioritized during training. The study aim was to develop objectives for a patient-safety focused pharmacology curriculum for emergency medicine residents.Methods: We incorporated data from a de-identified safety event database and survey responses of 30 faculty and clinical pharmacists at a single-site suburban university hospital with 24-hour emergency medicine pharmacists and an annual volume of approximately 70,000. We reviewed the database to quantify types and severity of medication errors over a 5-year period for a total of 370 errors. Anonymous surveys included categorical items that we analyzed with descriptive statistics and short answer questions that underwent thematic analysis by 2 coders. We summarized all data sources to identify curriculum gaps.Results: Common medication errors reported in our database were wrong dose (43%) and computer order entry errors (14%). Knowledge gaps were medication cost (63%), pregnancy risk information (60%), antibiotic stewardship (53%), interactions (47%), and side effects (47%). Qualitative analysis revealed the need to optimize computer order entry, understand the scope of critical medications, use references, and consult pharmacists. Integration of data suggested specific medications should be covered in curricular efforts, including antibiotics, analgesics, sedatives, and insulin.Conclusion: We developed objectives of pharmacology topics to prioritize during emergency medicine training to enhance prescribing safety. This study is limited due to its small sample and single institution source of data. Future studies should investigate the impact of pharmacology curriculum on minimizing clinical errors.

    View details for DOI 10.1002/emp2.12682

    View details for PubMedID 35310405

  • Josiah Macy Jr. Foundation Conference on COVID-19 and the Impact on Medical and Nursing Education: Conference Recommendations Report ACADEMIC MEDICINE Bickford, E., Boazak, M., Cain, R. A., Camp-Spivey, L., Caretta-Weyer, H., Carruth, M., Carter, K., Chandra, S., Chang, L., Clark, A. K., Cook, T., Dias, J., Directo, L., Fair, M., Farrell, C., Foronda, C., Frazier, L., Gielissen, K., Green, M., Head, M., Henson, R., Hughes, M., Humphrey, H. J., Hurtado, A., Jeffries, P. R., Kinnear, B., Kirk, L. M., Leaver, C. A., Lee, S., Levinson, D., Lucey, C. R., Lypson, M. L., McDougle, L., Muller, D., Murray, T. L., Papanagnou, D., Parks, A. K., Poitevien, P., Power, B., Rushton, C., Ryan, M., Schoenbaum, S. C., Sharp-McHenry, L., Stanley, J. M., Taggart, H., Thrall, C., Turner, D., Veenema, T., Whelan, A. J., Williams, M., Yacht, A. C., Yau-Wang, E. W., Young, J., Kourt, K., Larson, T., Legendre, Y., Mostek, L., Snijdewind, H. 2022; 97 (3S): S3-S11

    View details for DOI 10.1097/ACM.0000000000004506

    View details for Web of Science ID 000760942500002

    View details for PubMedID 34736279

  • An Outcomes-Oriented Approach to Residency Selection: Implementing Novel Processes to Align Residency Programs and Applicants. Academic medicine : journal of the Association of American Medical Colleges Caretta-Weyer, H. A. 1800

    Abstract

    Residency application numbers have skyrocketed in the last decade, and stakeholders have scrambled to identify and deploy methods of reducing the number of applications submitted to each program. These interventions have traditionally focused on the logistics of the application submission and review process, neglecting many of the drivers of overapplication. Implementing application caps, preference signaling as described by Pletcher and colleagues in this issue, or an early Match does not address the fear of not matching that applicants hold, the lack of transparent data available for applicants to assess their alignment with a specific program, or issues of inequity in the residency selection process. Now is the time to reconsider the residency selection process itself. As competency-based medical education emerges as the predominant educational paradigm, residency selection practices must also shift to align with societal, specialty, and program outcomes. The field of industrial and organizational psychology offers a multitude of tools (e.g., job analysis) by which to define the necessary outcomes of residency training. These tools also provide programs with the infrastructure around which to scaffold an outcomes-oriented approach to the residency selection process. Programs then can connect residency selection to training outcomes, longitudinal assessment modalities, and the evolving learning environment. To achieve an outcomes-oriented residency selection process, stakeholders at all levels will need to invest in coproducing novel ways forward. These solutions range from defining program priorities to implementing national policy. Focusing on outcomes will facilitate a more transparent residency selection process while also allowing logistics-level interventions to be successful, as applicants will be empowered to better assess their alignment with each program and apply accordingly.

    View details for DOI 10.1097/ACM.0000000000004614

    View details for PubMedID 35108236

  • The Effects of Leadership Curricula With and Without Implicit Bias Training on Graduate Medical Education: A Multicenter Randomized Trial. Academic medicine : journal of the Association of American Medical Colleges Hansen, M., Harrod, T., Bahr, N., Schoonover, A., Adams, K., Kornegay, J., Stenson, A., Ng, V., Plitt, J., Cooper, D., Scott, N., Chinai, S., Johnson, J., Conlon, L. W., Salva, C., Caretta-Weyer, H., Huynh, T., Jones, D., Jorda, K., Lo, J., Mayersak, R., Pare, E., Hughes, K., Ahmed, R., Patel, S., Tsao, S., Wang, E., Ogburn, T., Guise, J. 1800

    Abstract

    PURPOSE: To determine whether a brief leadership curriculum including high-fidelity simulation can improve leadership skills among resident physicians.METHOD: This was a double-blind randomized controlled trial among obstetrics and gynecology (OB/GYN) and emergency medicine (EM) residents across 5 academic medical centers from different geographic areas of the United States, 2015-2017. Participants were assigned to 1 of 3 study arms: the LEADS (Leadership Education Advanced During Simulation) curriculum, a shortened TeamSTEPPS (Team Strategies and Tools to Enhance Performance and Patient Safety) curriculum, or as active controls (no leadership curriculum). Active controls were recruited from a separate site and not randomized in order to limit any unintentional introduction of materials from the leadership curricula. The LEADS curriculum was developed in partnership with the Council on Resident Education in Obstetrics and Gynecology and Council of Residency Directors in Emergency Medicine as a novel way to provide a leadership toolkit. Both LEADS and the abbreviated TeamSTEPPS were designed as six 10-minute interactive web-based modules.The primary outcome of interest was the leadership performance score from the validated Clinical Teamwork Scale instrument measured during standardized high-fidelity simulation scenarios. Secondary outcomes were 9 key components of leadership from the detailed leadership evaluation measured on 5-point Likert scales. Both outcomes were rated by a blinded clinical video reviewer.RESULTS: One hundred and ten OB/GYN and EM residents participated in this 2-year trial. Participants in both LEADS and TeamSTEPPS had statistically significant improvement in leadership scores from "average" to "good" ranges both immediately and at the 6-month follow-up, while controls remained unchanged in the "average" category throughout the study. There were no differences between the LEADS and TeamSTEPPS curricula with respect to the primary outcome.CONCLUSIONS: Residents who participated in a brief structured leadership training intervention had improved leadership skills that were maintained at 6-month follow-up.

    View details for DOI 10.1097/ACM.0000000000004573

    View details for PubMedID 34966032

  • Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study. JMIR medical education Peng, C. R., Schertzer, K. A., Caretta-Weyer, H. A., Sebok-Syer, S. S., Lu, W., Tansomboon, C., Gisondi, M. A. 2021; 7 (4): e32356

    Abstract

    BACKGROUND: The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment.OBJECTIVE: In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform.METHODS: Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis.RESULTS: All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface.CONCLUSIONS: This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

    View details for DOI 10.2196/32356

    View details for PubMedID 34787582

  • Impact of the COVID-19 pandemic on emergency medicine education: Insights from faculty and residents. AEM education and training Weygandt, P. L., Jordan, J., Caretta-Weyer, H., Osborne, A., Grabow Moore, K. 2021; 5 (3): e10603

    Abstract

    Objectives: The COVID-19 pandemic continues to impact health systems across the United States and worldwide in an unprecedented way; however, its influence on frontline medical trainees' educational experiences is unknown. Our objective was to determine the effects of COVID-19 on emergency medicine (EM) training programs and residents.Methods: We performed a mixed-methods cross-sectional survey study of faculty and residents at programs registered with Foundations of Emergency Medicine. Participants completed an online survey consisting of closed and open-ended response items. We reported descriptive statistics for discrete and continuous data. Free-response data were analyzed qualitatively using a thematic approach.Results: Ninety-two percent of faculty (119/129) and 47% (1,965/4,154) of residents responded to our survey. We identified three major themes related to effects on learning: 1) impact on clinical training, 2) impact on didactic education, and 3) impact on the trainee. Nearly all residencies (96%, 111/116) allowed residents to work with patients suspected of having COVID-19, although fewer (83%, 96/115) allowed residents to intubate them. We found that 99% (1918/1928) of residents experienced virtual didactics. Faculty and trainees noted multiple educational challenges and strategies for adaptation. Trainees also expressed concerns about stress and safety.Conclusion: COVID-19 has impacted EM education in many ways including clinical training, didactic education, and trainee emotional state and concentration. Challenges and suggested solutions for learning in the virtual environment were also identified. While the pandemic continues to evolve and impact EM residents in various ways, our results may inform strategies to support medical educators and trainees during pandemics or other periods of significant disruption or crisis.

    View details for DOI 10.1002/aet2.10603

    View details for PubMedID 34141998

  • Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment. Academic medicine : journal of the Association of American Medical Colleges Kinnear, B., Warm, E. J., Caretta-Weyer, H., Holmboe, E. S., Turner, D. A., van der Vleuten, C., Schumacher, D. J. 2021; 96 (7S): S56-S63

    Abstract

    Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

    View details for DOI 10.1097/ACM.0000000000004108

    View details for PubMedID 34183603

  • Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade. Academic medicine : journal of the Association of American Medical Colleges Ten Cate, O., Balmer, D. F., Caretta-Weyer, H., Hatala, R., Hennus, M. P., West, D. C. 2021; 96 (7S): S96-S104

    Abstract

    To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

    View details for DOI 10.1097/ACM.0000000000004106

    View details for PubMedID 34183610

  • Missing the mark: Alternative sources of variation in graduate milestone achievement. AEM education and training Caretta-Weyer, H. A., Schumacher, D. J. 2021; 5 (3): e10618

    View details for DOI 10.1002/aet2.10618

    View details for PubMedID 34222752

  • Diagnosing conflict: Conflicting data, interpersonal conflict, and conflicts of interest in clinical competency committees. Medical teacher Chan, T., Oswald, A., Hauer, K. E., Caretta-Weyer, H. A., Nousiainen, M. T., Cheung, W. J., ICBME Collaborators 2021: 1-9

    Abstract

    Clinical competency committees (CCCs) are increasingly used within health professions education as their decisions are thought to be more defensible and fairer than those generated by previous training promotion processes. However, as with most group-based processes, it is inevitable that conflict will arise. In this paper the authors explore three ways conflict may arise within a CCC: (1) conflicting data submissions that are presented to the committee, (2) conflicts between members of the committee, and (3) conflicts of interest between a specific committee member and a trainee. The authors describe each of these conflict situations, dissect out the underlying problems, and explore possible solutions based on the current literature.

    View details for DOI 10.1080/0142159X.2021.1925101

    View details for PubMedID 34182879

  • Capturing outcomes of competency-based medical education: The call and the challenge. Medical teacher Van Melle, E., Hall, A. K., Schumacher, D. J., Kinnear, B., Gruppen, L., Thoma, B., Caretta-Weyer, H., Cooke, L. J., Frank, J. R., ; On behalf of the ICBME Collaborators 2021: 1-7

    Abstract

    There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

    View details for DOI 10.1080/0142159X.2021.1925640

    View details for PubMedID 34121596

  • Outcomes of competency-based medical education: A taxonomy for shared language. Medical teacher Hall, A. K., Schumacher, D. J., Thoma, B., Caretta-Weyer, H., Kinnear, B., Gruppen, L., Cooke, L. J., Frank, J. R., Van Melle, E., ICBME Collaborators 2021: 1-6

    Abstract

    As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.

    View details for DOI 10.1080/0142159X.2021.1925643

    View details for PubMedID 34038673

  • The deliberately developmental organization: A conceptual framework for CBME. Medical teacher Thoma, B., Caretta-Weyer, H., Schumacher, D. J., Warm, E., Hall, A. K., Hamstra, S. J., Cavalcanti, R., Chan, T. M., ICBME Collaborators 2021: 1-9

    Abstract

    Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

    View details for DOI 10.1080/0142159X.2021.1925100

    View details for PubMedID 34033512

  • If we could turn back time: Imagining time-variable, competency-based medical education in the context of COVID-19. Medical teacher Caretta-Weyer, H. A., Chan, T., Bigham, B. L., Kinnear, B., Huwendiek, S., Schumacher, D. J., ICBME Collaborators 2021: 1-6

    Abstract

    The COVID-19 pandemic has exposed a paradox in historical models of medical education: organizations responsible for applying consistent standards for progression have needed to adapt to training environments marked by inconsistency and change. Although some institutions have maintained their traditional requirements, others have accelerated their programs to rush nearly graduated trainees to the front lines. One interpretation of the unplanned shortening of the duration of training programs during a crisis is that standards have been lowered. But it is also possible that these trainees were examined according to the same standards as usual and were judged to have already met them. This paper discusses the impacts of the COVID-19 pandemic on the current workforce, provides an analysis of how competency-based medical education (CBME) in the context of the pandemic might have mitigated wide-scale disruption, and identifies structural barriers to achieving an ideal state. The paper further calls upon universities, health centres, governments, certifying bodies, regulatory authorities, and health care professionals to work collectively on a truly time-variable model of CBME. The pandemic has made clear that time variability in medical education already exists and should be adopted widely and formally. If our systems today had used a framework of outcome competencies, sequenced progression, tailored learning, focused instruction, and programmatic assessment, we may have been even more nimble in changing our systems to care for our patients with COVID-19.

    View details for DOI 10.1080/0142159X.2021.1925641

    View details for PubMedID 34027813

  • On the validity of summative entrustment decisions. Medical teacher Touchie, C., Kinnear, B., Schumacher, D., Caretta-Weyer, H., Hamstra, S. J., Hart, D., Gruppen, L., Ross, S., Warm, E., Ten Cate, O., ICBME Collaborators 2021: 1-8

    Abstract

    Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

    View details for DOI 10.1080/0142159X.2021.1925642

    View details for PubMedID 34020576

  • Clarifying essential terminology in entrustment MEDICAL TEACHER Schumacher, D. J., ten Cate, O., Damodaran, A., Richardson, D., Hamstra, S. J., Ross, S., Hodgson, J., Touchie, C., Molgaard, L., Gofton, W., Carraccio, C., ICBME Collaborators 2021: 1-8

    Abstract

    With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

    View details for DOI 10.1080/0142159X.2021.1924365

    View details for Web of Science ID 000650606100001

    View details for PubMedID 33989100

  • Curated collection for clinician educators: Six key papers on residency recruitment. AEM education and training Schnapp, B. H., Alvarez, A., Bianchi, R., Caretta-Weyer, H., Jewell, C., Kalantari, A., Lee, E., Miller, D., Quinn, A. 2021; 5 (2): e10597

    Abstract

    Introduction: All emergency medicine (EM) residency programs must recruit new medical school graduates each year. The process is often overwhelming, with each program receiving far more applicants than available positions. We searched for evidence-based best practices to guide residency programs in screening, interviewing, and ranking applicants to ensure a high-performing and diverse residency class.Methods: A literature search was conducted on the topic of residency recruitment, utilizing a call on social media as well as multiple databases. After identifying relevant articles, we performed a modified Delphi process in three rounds, utilizing junior educators as well as more senior faculty.Results: We identified 51 relevant articles on the topic of residency recruitment. The Delphi process yielded six articles that were deemed most highly relevant over the three rounds. Transparency with selection criteria, holistic application review, standardized letters of evaluation, and blinding applicant files for interviewers were among noted best practices.Conclusions: Well-supported evidence-based practices exist for residency recruitment, and programs may benefit from understanding which common recruitment practices offer the most value. The articles discussed here provide a foundation for faculty looking to improve their program's recruiting practices.

    View details for DOI 10.1002/aet2.10597

    View details for PubMedID 33969251

  • Development, Implementation, and Evaluation of an Open Access, Level-Specific, Core Content Curriculum for Emergency Medicine Residents. Journal of graduate medical education Grabow Moore, K., Ketterer, A., Wheaton, N., Weygandt, P. L., Caretta-Weyer, H. A., Berberian, J., Jordan, J. 2021; 13 (5): 699-710

    Abstract

    Leaders in graduate medical education must provide robust clinical and didactic experiences to prepare residents for independent practice. Programs traditionally create didactic experiences individually, requiring tremendous resources with variable content exposure and quality.We sought to create and implement a free, open access, learner-centric, level-specific, emergency medicine (EM) residency curriculum.We developed Foundations of Emergency Medicine (FoEM) Foundations I and II courses using Kern's model of curriculum development. Fundamental topics were identified through content guidelines from the American Board of Emergency Medicine. We incorporated learner-centric strategies into 2 flipped classroom, case-based courses targeting postgraduate year (PGY) 1 and PGY-2 residents. The curriculum was made freely available online in 2016. Faculty and resident users were surveyed annually for feedback, which informed iterative refinement of the curriculum.Between 2016 and 2020, registration for FoEM expanded from 2 sites with 36 learners to 154 sites and 4453 learners. In 2019, 98 of 102 (96%) site leaders and 1618 of 2996 (54%) learners completed the evaluative survey. One hundred percent of responding leaders and 93% of learners were "satisfied" or "very satisfied" with FoEM content. Faculty and residents valued FoEM's usability, large volume of content, quality, adaptability, organization, resident-faculty interaction, and resident-as-teacher opportunities. Challenges to implementation included resident attendance, conference structure, technology limitations, and faculty engagement.We developed and implemented a learner-centric, level-specific, national EM curriculum that has been widely adopted in the United States.

    View details for DOI 10.4300/JGME-D-21-00067.1

    View details for PubMedID 34721800

    View details for PubMedCentralID PMC8527934

  • Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic. Medical teacher Schumacher, D. J., Caretta-Weyer, H., Busari, J., Carraccio, C., Damodaran, A., Gruppen, L. D., Hall, A. K., Kinnear, B., Warm, E., Ten Cate, O. 2021: 1-13

    Abstract

    Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

    View details for DOI 10.1080/0142159X.2021.1925098

    View details for PubMedID 34038645

  • Demonstrating Your Work: A Guide to Educators' Portfolios for Graduate Medical Educators. Journal of graduate medical education Mayersak, R. J., Yarris, L. M., Tuttle, R. C., Jones, D. C., Nelson, A. M., Bengtzen, R. R., Kornegay, J. G., Caretta-Weyer, H. A. 2021; 13 (5): 635-639

    View details for DOI 10.4300/JGME-D-21-00375.1

    View details for PubMedID 34721790

    View details for PubMedCentralID PMC8527930

  • Trends in medical students' stress, physical, and emotional health throughout training. Medical education online McKerrow, I., Carney, P. A., Caretta-Weyer, H., Furnari, M., Miller Juve, A. 2020; 25 (1): 1709278

    Abstract

    Background: Medical student wellness, including physical health, emotional health, and levels of perceived stress, appears to decline during training, with students reporting high levels of depression, anxiety, and burnout as early as the first year of medical school. The impact of curricular changes on health and stress remains unclear, and a modified curriculum that compresses training of the foundational sciences and its effect on wellness has not been studied. Oregon Health & Science University School of Medicine has recently instituted a unique competency-based model, which provides an important opportunity to assess the effects of curricular change on student wellness.Objective: Assess the effects of curricular change on student wellness.Design: Medical students at a single institution were administered the SF-8, an 8-item health-related quality of life survey, as well as the Perceived Stress Scale, a 10-item scale that measures the degree to which life situations are appraised as stressful, at baseline (matriculation) and at the end of Year 1, 2 and 3. Individual variables were assessed over time, as well as a trend analysis of summary domain scores over the 4 time periods.Results: Physical, emotional, and overall health were highest at baseline and lowest at the end of Year 1, after which they improved but never again reached baseline levels. Physical health declined less than emotional health. Perceived stress levels did not change over time but remained moderately high. There were no differences in health or perceived stress based on demographic variables.Conclusions: In a competency-based curriculum, physical, emotional and overall health significantly worsened during Year 1 but improved thereafter, while perceived stress remained unchanged. Early in training, stress and poor overall health may be related to concerns about self-efficacy and workload. Although advanced students show improved wellness, concerns remained about emotional difficulties, such as anxiety and irritability, and feeling a lack of control.

    View details for DOI 10.1080/10872981.2019.1709278

    View details for PubMedID 31902315

  • The Flipped Classroom: A Critical Appraisal. The western journal of emergency medicine Kraut, A. S., Omron, R., Caretta-Weyer, H., Jordan, J., Manthey, D., Wolf, S. J., Yarris, L. M., Johnson, S., Kornegay, J. 2019; 20 (3): 527–36

    Abstract

    Introduction: The objective of this study was to review and critically appraise the medical education literature pertaining to a flipped-classroom (FC) education model, and to highlight influential papers that inform our current understanding of the role of the FC in medical education.Methods: A search of the English-language literature querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 296 papers related to the FC using either quantitative, qualitative, or review methods. Two reviewers independently screened each category of publications using previously established exclusion criteria. Eight reviewers then independently scored the remaining 54 publications using either a qualitative, quantitative, or review-paper scoring system. Each scoring system consisted of nine criteria and used parallel metrics that have been previously used in critical appraisals of education research.Results: A total of 54 papers (33 quantitative, four qualitative, and 17 review) on FC met a priori criteria for inclusion and were critically appraised and reviewed. The top 10 highest scoring articles (five quantitative studies, two qualitative studies, and three review papers) are summarized in this article.Conclusion: This installment of the Council of Emergency Medicine Residency Directors (CORD) Academy Critical Appraisal series highlights 10 papers that describe the current state of literature on the flipped classroom, including an analysis of the benefits and drawbacks of an FC approach, practical implications for emergency medicine educators, and next steps for future research.

    View details for DOI 10.5811/westjem.2019.2.40979

    View details for PubMedID 31123556

  • Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education. Cureus Schnapp, B. H., Caretta-Weyer, H. A., Cortez, E., Heinrich, S. A., Kraut, A. S., Lloyd, C. M., Silvester, C., Sorge, R. M., Wain, A., Gottlieb, M. 2019; 11 (4): e4383

    Abstract

    Introduction The Accreditation Council for Graduate Medical Education calls graduated responsibility "one of the core tenets of American graduate medical education." However, there is no clear set of resources for programs to implement a system of progressively increasing responsibilities for trainees. This project aimed to identify a set of high-yield papers on graduated responsibility for junior faculty members. Methods A study group of Academic Life in Emergency Medicine Faculty Incubator participants identified relevant literature on graduated responsibility via a comprehensive literature search and a call to the online medical education community; 59 total papers were identified. The most relevant and applicable were selected by the study group via a three-round modified Delphi process. Results Five key articles for junior faculty interested in implementing more robust graduated responsibility at their residency training program were selected and described here. Summaries of key points, along with considerations for faculty developers and relevance to junior faculty, are presented for each article. Conclusions The articles presented here provide a solid theoretical and practical basis for junior faculty to explore graduated responsibility. The five articles presented here provide the junior faculty with a toolkit to examine and improve their systems for assigning responsibilities in a graded fashion at their own institutions.

    View details for DOI 10.7759/cureus.4383

    View details for PubMedID 31218147

    View details for PubMedCentralID PMC6553674

  • Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education CUREUS Schnapp, B. H., Caretta-Weyer, H. A., Cortez, E., Heinrich, S. A., Kraut, A. S., Lloyd, C. M., Silvester, C., Sorge, R. M., Wain, A., Gottlieb, M. 2019; 11 (4)
  • Design Your Clinical Workplace to Facilitate Competency-Based Education. The western journal of emergency medicine Caretta-Weyer, H. A., Gisondi, M. A. 2019; 20 (4): 651–53

    View details for DOI 10.5811/westjem.2019.4.43216

    View details for PubMedID 31316706

    View details for PubMedCentralID PMC6625682

  • Critical Electrocardiogram Curriculum: Setting the Standard for Flipped-Classroom EKG Instruction. The western journal of emergency medicine Burns, W. P., Hartman, N. D., Weygandt, P. L., Jones, S. C., Caretta-Weyer, H. n., Moore, K. G. 2019; 21 (1): 52–57

    Abstract

    Electrocardiogram (EKG) interpretation is integral to emergency medicine (EM). In 2003 Ginde et al. found 48% of emergency medicine (EM) residency directors supported creating a national EKG curriculum. No formal national curriculum exists, and it is unknown whether residents gain sufficient skill from clinical exposure alone.The authors sought to assess the value of this EKG curriculum, which provides exposure to critical EKG patterns, a framework for EKG interpretation when the diagnosis is not obvious, and implementation guidelines and open access to any interested residency. The Foundations of Emergency Medicine (FoEM) EKG I course launched in January 2016, followed by EKG II in July 2017; they are benchmarked to post-graduate year 1 (PGY) and PGY2 level learners, respectively. Selected topics included 15 published critical EKG diagnoses and 33 selected by the authors. Cases included presenting symptoms, EKGs, and Free Open Access Medical Education (FOAM) links. Full EKG interpretations and question answers were provided.Enrollment during 2017-2018 included 37 EM residencies with 663 learners in EKG I and 22 EM residencies with 438 learners in EKG II. Program leaders and learners were surveyed annually. Leaders indicated that content was appropriate for intended PGY levels. Leaders and learners indicated the curriculum improved the ability of learners to interpret EKGs while working in the emergency department (ED).There is an unmet need for standardization and improvement of EM resident EKG training. Leaders and learners exposed to FoEM EKG courses report improved ability of learners to interpret EKGs in the ED.

    View details for DOI 10.5811/westjem.2019.11.44509

    View details for PubMedID 31913819

  • Transition to Practice: A Novel Life Skills Curriculum for Emergency Medicine Residents. The western journal of emergency medicine Caretta-Weyer, H. 2019; 20 (1): 100–104

    View details for PubMedID 30643610

  • Transition to Practice: A Novel Life Skills Curriculum for Emergency Medicine Residents WESTERN JOURNAL OF EMERGENCY MEDICINE Caretta-Weyer, H. 2019; 20 (1): 100–104
  • Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship. Cureus Quinn, A. n., Chan, T. M., Sampson, C. n., Grossman, C. n., Butts, C. n., Casey, J. n., Caretta-Weyer, H. n., Gottlieb, M. n. 2018; 10 (1): e2021

    Abstract

    Traditionally, scholarship that was recognized for promotion and tenure consisted of clinical research, bench research, and grant funding. Recent trends have allowed for differing approaches to scholarship, including digital publication. As increasing numbers of trainees and faculty turn to online educational resources, it is imperative to critically evaluate these resources. This article summarizes five key papers that address the appraisal of digital scholarship and describes their relevance to junior clinician educators and faculty developers. In May 2017, the Academic Life in Emergency Medicine Faculty Incubator program focused on the topic of digital scholarship, providing and discussing papers relevant to the topic. We augmented this list of papers with further suggestions by guest experts and by an open call via Twitter for other important papers. Through this process, we created a list of 38 papers in total on the topic of evaluating digital scholarship. In order to determine which of these papers best describe how to evaluate digital scholarship, the authorship group assessed the papers using a modified Delphi approach to build consensus. In this paper we present the five most highly rated papers from our process about evaluating digital scholarship. We summarize each paper and discuss its specific relevance to junior faculty members and to faculty developers. These papers provide a framework for assessing the quality of digital scholarship, so that junior faculty can recommend high-quality educational resources to their trainees. These papers help guide educators on how to produce high quality digital scholarship and maximize recognition and credit in respect to receiving promotion and tenure.

    View details for PubMedID 29531874

  • The View From Over Here: A Framework for Multi-Source Feedback. Journal of graduate medical education Caretta-Weyer, H. A., Kraut, A. S., Kornegay, J. G., Yarris, L. M. 2017; 9 (3): 367-368

    View details for DOI 10.4300/JGME-D-17-00200.1

    View details for PubMedID 28638519

    View details for PubMedCentralID PMC5476390

  • Feedback in Medical Education: A Critical Appraisal. AEM education and training Kornegay, J. G., Kraut, A. n., Manthey, D. n., Omron, R. n., Caretta-Weyer, H. n., Kuhn, G. n., Martin, S. n., Yarris, L. M. 2017; 1 (2): 98–109

    Abstract

    The objective was to review and critically appraise the medical education literature pertaining to feedback and highlight influential papers that inform our current understanding of the role of feedback in medical education.A search of the English language literature in querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 327 feedback-related papers using either quantitative (hypothesis-testing or observational investigations of educational interventions), qualitative methods (exploring important phenomena in emergency medicine [EM] education), or review methods.Two reviewers independently screened each category of publications using previously established exclusion criteria. Six reviewers then independently scored the remaining 54 publications using a qualitative, quantitative, or review paper scoring system. Each scoring system consisted of nine criteria and used parallel scoring metrics that have been previously used in critical appraisals of education research.Fifty-four feedback papers (25 quantitative studies, 24 qualitative studies, five review papers) met the a priori criteria for inclusion and were reviewed. Eight quantitative studies, nine qualitative studies, and three review papers were ranked highly by the reviewers and are summarized in this article.This inaugural Council of Emergency Medicine Residency Directors Academy critical appraisal highlights 20 feedback in medical education papers that describe the current state of the feedback literature. A summary of current factors that influence feedback effectiveness is discussed, along with practical implications for EM educators and the next steps for research.

    View details for PubMedID 30051017

    View details for PubMedCentralID PMC6001508

  • Determining breast cancer axillary surgery within the surveillance epidemiology and end results-Medicare database JOURNAL OF SURGICAL ONCOLOGY Schmocker, R. K., Caretta-Weyer, H., Weiss, J. M., Ronk, K., Havlena, J., Loconte, N. K., Decker, M., Smith, M. A., Greenberg, C. C., Neuman, H. B. 2014; 109 (8): 756-759

    Abstract

    Use of sentinel lymph node biopsy (SLNB) is under-reported by cancer registries' "Scope of Regional Lymph Node Surgery" variable. In 2011, the Surveillance Epidemiology and End Results (SEER) Program recommended against its use to determine extent of axillary surgery, leaving a gap in the utilization of claims data for breast cancer research. The objective was to develop an algorithm using SEER registry and claims data to classify extent of axillary surgery for breast cancer.We analyzed data for 24,534 breast cancer patients. CPT codes and number of examined lymph nodes classified the extent of axillary surgery. The final algorithm was validated by comparing the algorithm derived extent of axillary surgery to direct chart review for 100 breast cancer patients treated at our breast center.Using the algorithm, 13% had no axillary surgery, 56% SLNB and 31% axillary lymph node dissection (ALND). SLNB was performed in 77% of node negative patients and ALND in 72% of node positive. In our validation study, concordance between algorithm and direct chart review was 97%.Given recognized inaccuracies in cancer registries' "Scope of Regional Lymph Node Surgery" variable, these findings have high utility for health services researchers studying breast cancer treatment.

    View details for DOI 10.1002/jso.23579

    View details for Web of Science ID 000335369700003

    View details for PubMedID 24643795

    View details for PubMedCentralID PMC4227499

  • Impact of the American College of Surgeons Oncology Group (ACOSOG) Z0011 trial on Clinical Management of the Axilla in Older Breast Cancer Patients: A SEER-Medicare Analysis ANNALS OF SURGICAL ONCOLOGY Caretta-Weyer, H., Greenberg, C. G., Wilke, L. G., Weiss, J., Loconte, N. K., Decker, M., Steffens, N. M., Smith, M. A., Neuman, H. B. 2013; 20 (13): 4145-4152

    Abstract

    American College of Surgeons Oncology Group (ACOSOG) Z0011 demonstrated that eligible breast cancer patients with positive sentinel lymph nodes (SLN) could be spared an axillary lymph node dissection (ALND) without sacrificing survival or local control. Although heralded as a ‘‘practice-changing trial,’’ some argue that the stringent inclusion criteria limit the trial’s clinical significance. The objective was to assess the potential impact of ACOSOG Z0011 on axillary surgical management of Medicare patients and examine current practice patterns.Medicare beneficiaries aged C66 years with nonmetastatic invasive breast cancer diagnosed from 2001 to 2007 were identified from the Surveillance, Epidemiology and End Results-Medicare database (n = 59,431). Eligibility for ACOSOG Z0011 was determined: SLN mapping, tumor\5 cm, no neoadjuvant treatment, breast conservation; number of positive nodes was determined. Actual surgical axillary management for eligible patients was assessed.Twelve percent (6,942/59,431) underwent SLN mapping and were node positive. Overall, 2,637 patients (4.4 % (2,637/59,431) of the total cohort, but 38 % (2,637/6,942) of patients with SLN mapping and positive nodes) met inclusion criteria for ACOSOG Z0011, had 1 or 2 positive lymph nodes, and could have been spared an ALND. Of these 2,637 patients, 46 % received a completion ALND and 54 % received only SLN biopsy.Widespread implementation of ACOSOG Z0011 trial results could potentially spare 38 % of older breast cancer patients who undergo SLN mapping with positive lymph nodes an ALND. However, 54 % of these patients are already managed with SLN biopsy alone, lessening the impact of this trial on clinical practice in older breast cancer patients.

    View details for DOI 10.1245/s10434-013-3193-1

    View details for Web of Science ID 000328256600016

    View details for PubMedID 23959051

    View details for PubMedCentralID PMC3874252

  • Impact of axillary ultrasound and core needle biopsy on the utility of intraoperative frozen section analysis and treatment decision making in women with invasive breast cancer AMERICAN JOURNAL OF SURGERY Caretta-Weyer, H., Sisney, G. A., Beckman, C., Burnside, E. S., Salkowsi, L. R., Strigel, R. M., Wilke, L. G., Neuman, H. B. 2012; 204 (3): 308-314

    Abstract

    Our objective was to evaluate the impact of preoperative axillary ultrasound and core needle biopsy (CNB) on breast cancer treatment decision making. A secondary aim was to evaluate the impact on the utility of intraoperative sentinel lymph node (SLN) frozen section.A review of 84 patients with clinically negative axilla who underwent axillary ultrasound was performed. Sensitivity, specificity, and positive/negative predictive value for axillary ultrasound with CNB was calculated.Thirty-one (37%) had suspicious nodes. Of 27 amenable to CNB, 12 (14%) were malignant, changing treatment plans. The sensitivity of ultrasound and CNB was 54% and specificity 100%; the positive and negative predictive values were 100% and 80%, respectively. In 41 patients with normal ultrasounds who underwent SLN frozen section, 10 (24%) were positive.Preoperative axillary ultrasound impacts treatment decision making in 14%. With a sensitivity of 54%, it is a useful adjunct to, but not replacement for, SLN biopsy. Frozen section remains of utility even after a negative axillary ultrasound.

    View details for DOI 10.1016/j.amjsurg.2011.10.016

    View details for PubMedID 22483606

  • Involvement of Noradrenergic Neurotransmission in the Stress- but not Cocaine-Induced Reinstatement of Extinguished Cocaine-Induced Conditioned Place Preference in Mice: Role for beta-2 Adrenergic Receptors NEUROPSYCHOPHARMACOLOGY Mantsch, J. R., Weyer, A., Vranjkovic, O., Beyer, C. E., Baker, D. A., Caretta, H. 2010; 35 (11): 2165-2178

    Abstract

    The responsiveness of central noradrenergic systems to stressors and cocaine poses norepinephrine as a potential common mechanism through which drug re-exposure and stressful stimuli promote relapse. This study investigated the role of noradrenergic systems in the reinstatement of extinguished cocaine-induced conditioned place preference by cocaine and stress in male C57BL/6 mice. Cocaine- (15 mg/kg, i.p.) induced conditioned place preference was extinguished by repeated exposure to the apparatus in the absence of drug and reestablished by a cocaine challenge (15 mg/kg), exposure to a stressor (6-min forced swim (FS); 20-25°C water), or administration of the α-2 adrenergic receptor (AR) antagonists yohimbine (2 mg/kg, i.p.) or BRL44408 (5, 10 mg/kg, i.p.). To investigate the role of ARs, mice were administered the nonselective β-AR antagonist, propranolol (5, 10 mg/kg, i.p.), the α-1 AR antagonist, prazosin (1, 2 mg/kg, i.p.), or the α-2 AR agonist, clonidine (0.03, 0.3 mg/kg, i.p.) before reinstatement testing. Clonidine, prazosin, and propranolol failed to block cocaine-induced reinstatement. The low (0.03 mg/kg) but not high (0.3 mg/kg) clonidine dose fully blocked FS-induced reinstatement but not reinstatement by yohimbine. Propranolol, but not prazosin, blocked reinstatement by both yohimbine and FS, suggesting the involvement of β-ARs. The β-2 AR antagonist ICI-118551 (1 mg/kg, i.p.), but not the β-1 AR antagonist betaxolol (10 mg/kg, i.p.), also blocked FS-induced reinstatement. These findings suggest that stress-induced reinstatement requires noradrenergic signaling through β-2 ARs and that cocaine-induced reinstatement does not require AR activation, even though stimulation of central noradrenergic neurotransmission is sufficient to reinstate.

    View details for DOI 10.1038/npp.2010.86

    View details for Web of Science ID 000281821200003

    View details for PubMedID 20613718

    View details for PubMedCentralID PMC2939933