Bio


Holly Caretta-Weyer is currently Associate Residency Program Director and Director of Evaluation and Assessment for the Stanford University Emergency Medicine Residency Program as well as EPA/CBME Implementation Lead at the Stanford University School of Medicine. Dr. Caretta-Weyer attended medical school at the University of Wisconsin School of Medicine and Public Health where she graduated Alpha Omega Alpha with Honors in Research. She loved being a Badger so much that she stayed for her Emergency Medicine Residency at the University of Wisconsin where she was also Chief Resident. Dr. Caretta-Weyer then moved to the West Coast where she completed her Medical Education Scholarship Fellowship at Oregon Health & Science University (OHSU) and completed her Masters in Health Professions Education (MHPE) at the University of Illinois-Chicago. She began her PhD at Maastricht University studying residency selection in a competency-based system in March of 2021.

While at OHSU, Dr. Caretta-Weyer worked as a member of the Association of American Medical Colleges (AAMC) Core Entrustable Professional Activities for Entering Residency pilot team and was a founding member of the OHSU undergraduate medical education entrustment committee. She continues to be involved with the national AAMC Core EPA Pilot through her continued collaboration with the OHSU team. Through this process she has gained valuable experience in working to define programmatic assessment, formulate summative entrustment decisions, and more seamlessly bridge the transition from undergraduate to graduate medical education, all of which are key initiatives within medical education.

Dr. Caretta-Weyer is also the PI on a $1.3M AMA Reimagining Residency Grant focused on redesigning assessment across the continuum of emergency medicine training and introducing predictive learning analytics to the process. She is additionally a member of the International Competency-Based Medical Education (ICBME) Collaborators, a group that seeks to further research on CBME around the world.

Dr. Caretta-Weyer's education research interests focus on the implementation of competency-based education and assessment across the continuum of medical education, summative entrustment and promotion decision-making processes, residency selection in a competency-based system, and the development of learner handovers to span key transitions in the educational continuum. When not focusing on her administrative and education research interests, Dr. Caretta-Weyer can be found kayaking, hiking, cycling, playing volleyball, or cheering on her favorite sports teams including the Wisconsin Badgers and Milwaukee Brewers.

Clinical Focus


  • Emergency Medicine

Academic Appointments


Administrative Appointments


  • Associate Residency Program Director, Stanford University, Department of Emergency Medicine (2018 - Present)
  • Director of Evaluation and Assessment, Stanford University, Department of Emergency Medicine (2018 - Present)
  • Education Strategic Plan Lead, Stanford University Department of Emergency Medicine (2019 - Present)
  • EPA Implementation Lead, Stanford University School of Medicine (2018 - Present)
  • Medical Education Line Director, Stanford University, Department of Emergency Medicine (2018 - Present)

Honors & Awards


  • Top Education Research Paper, International Conference on Residency Education (ICRE) (2020)
  • Top 10 Education Research Abstract, Accreditation Council for Graduate Medical Education (ACGME) (2020)
  • Outstanding Educator Award, Stanford University Department of Emergency Medicine (2019)
  • Outstanding Peer Reviewer, Academic Emergency Medicine Education and Training (2019)
  • Academic Instructor of the Year Award, Oregon Health and Science University Department of Emergency Medicine (2018)
  • Resident Advocate Award, Oregon Health and Science University Department of Emergency Medicine (2018)
  • Best Poster Award - Education Research Category, Oregon Health & Science University Symposium on Educational Excellence (2018)
  • Academic Instructor of the Year Award, Oregon Health and Science University Department of Emergency Medicine (2017)
  • Gold Standard Peer Review Award, Western Journal of Emergency Medicine (WestJEM) Education Supplement (2017)
  • The Ripple Award for Excellence in Leadership and Service, University of Wisconsin Department of Emergency Medicine (2016)
  • Award for Excellence in Scholarship, Wisconsin Chapter of the American College of Emergency Physicians (2015)
  • Alpha Omega Alpha Honor Society Induction, University of Wisconsin School of Medicine (2012)

Boards, Advisory Committees, Professional Organizations


  • Editorial Board Member, Academic Emergency Medicine Education and Training (2020 - Present)
  • Director of Grants, ARMED MedEd - SAEM (2020 - Present)
  • Member-at-Large, International Competency-Based Medical Education Collaborators (2019 - Present)
  • Grant Reviewer, Society for Academic Emergency Medicine (2019 - Present)
  • Section Editor, Western Journal of Emergency Medicine - Education Scholarship Section (2018 - Present)

Professional Education


  • PhD Candidate, Maastricht University, Medical Education
  • MHPE, University of Illinois - Chicago, Health Professions Education (2021)
  • Fellowship: Oregon Health and Science University Emergency Medicine Fellowships (2018) OR
  • Board Certification: American Board of Emergency Medicine, Emergency Medicine (2017)
  • Residency: University of Wisconsin Emergency Medicine Residency (2016) WI
  • Medical Education: University of Wisconsin School of Medicine and Public Health (2013) WI

2021-22 Courses


Stanford Advisees


All Publications


  • The Effects of Leadership Curricula With and Without Implicit Bias Training on Graduate Medical Education: A Multicenter Randomized Trial. Academic medicine : journal of the Association of American Medical Colleges Hansen, M., Harrod, T., Bahr, N., Schoonover, A., Adams, K., Kornegay, J., Stenson, A., Ng, V., Plitt, J., Cooper, D., Scott, N., Chinai, S., Johnson, J., Conlon, L. W., Salva, C., Caretta-Weyer, H., Huynh, T., Jones, D., Jorda, K., Lo, J., Mayersak, R., Pare, E., Hughes, K., Ahmed, R., Patel, S., Tsao, S., Wang, E., Ogburn, T., Guise, J. 1800

    Abstract

    PURPOSE: To determine whether a brief leadership curriculum including high-fidelity simulation can improve leadership skills among resident physicians.METHOD: This was a double-blind randomized controlled trial among obstetrics and gynecology (OB/GYN) and emergency medicine (EM) residents across 5 academic medical centers from different geographic areas of the United States, 2015-2017. Participants were assigned to 1 of 3 study arms: the LEADS (Leadership Education Advanced During Simulation) curriculum, a shortened TeamSTEPPS (Team Strategies and Tools to Enhance Performance and Patient Safety) curriculum, or as active controls (no leadership curriculum). Active controls were recruited from a separate site and not randomized in order to limit any unintentional introduction of materials from the leadership curricula. The LEADS curriculum was developed in partnership with the Council on Resident Education in Obstetrics and Gynecology and Council of Residency Directors in Emergency Medicine as a novel way to provide a leadership toolkit. Both LEADS and the abbreviated TeamSTEPPS were designed as six 10-minute interactive web-based modules.The primary outcome of interest was the leadership performance score from the validated Clinical Teamwork Scale instrument measured during standardized high-fidelity simulation scenarios. Secondary outcomes were 9 key components of leadership from the detailed leadership evaluation measured on 5-point Likert scales. Both outcomes were rated by a blinded clinical video reviewer.RESULTS: One hundred and ten OB/GYN and EM residents participated in this 2-year trial. Participants in both LEADS and TeamSTEPPS had statistically significant improvement in leadership scores from "average" to "good" ranges both immediately and at the 6-month follow-up, while controls remained unchanged in the "average" category throughout the study. There were no differences between the LEADS and TeamSTEPPS curricula with respect to the primary outcome.CONCLUSIONS: Residents who participated in a brief structured leadership training intervention had improved leadership skills that were maintained at 6-month follow-up.

    View details for DOI 10.1097/ACM.0000000000004573

    View details for PubMedID 34966032

  • Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study. JMIR medical education Peng, C. R., Schertzer, K. A., Caretta-Weyer, H. A., Sebok-Syer, S. S., Lu, W., Tansomboon, C., Gisondi, M. A. 2021; 7 (4): e32356

    Abstract

    BACKGROUND: The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment.OBJECTIVE: In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform.METHODS: Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis.RESULTS: All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface.CONCLUSIONS: This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

    View details for DOI 10.2196/32356

    View details for PubMedID 34787582

  • Impact of the COVID-19 pandemic on emergency medicine education: Insights from faculty and residents. AEM education and training Weygandt, P. L., Jordan, J., Caretta-Weyer, H., Osborne, A., Grabow Moore, K. 2021; 5 (3): e10603

    Abstract

    Objectives: The COVID-19 pandemic continues to impact health systems across the United States and worldwide in an unprecedented way; however, its influence on frontline medical trainees' educational experiences is unknown. Our objective was to determine the effects of COVID-19 on emergency medicine (EM) training programs and residents.Methods: We performed a mixed-methods cross-sectional survey study of faculty and residents at programs registered with Foundations of Emergency Medicine. Participants completed an online survey consisting of closed and open-ended response items. We reported descriptive statistics for discrete and continuous data. Free-response data were analyzed qualitatively using a thematic approach.Results: Ninety-two percent of faculty (119/129) and 47% (1,965/4,154) of residents responded to our survey. We identified three major themes related to effects on learning: 1) impact on clinical training, 2) impact on didactic education, and 3) impact on the trainee. Nearly all residencies (96%, 111/116) allowed residents to work with patients suspected of having COVID-19, although fewer (83%, 96/115) allowed residents to intubate them. We found that 99% (1918/1928) of residents experienced virtual didactics. Faculty and trainees noted multiple educational challenges and strategies for adaptation. Trainees also expressed concerns about stress and safety.Conclusion: COVID-19 has impacted EM education in many ways including clinical training, didactic education, and trainee emotional state and concentration. Challenges and suggested solutions for learning in the virtual environment were also identified. While the pandemic continues to evolve and impact EM residents in various ways, our results may inform strategies to support medical educators and trainees during pandemics or other periods of significant disruption or crisis.

    View details for DOI 10.1002/aet2.10603

    View details for PubMedID 34141998

  • Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment. Academic medicine : journal of the Association of American Medical Colleges Kinnear, B., Warm, E. J., Caretta-Weyer, H., Holmboe, E. S., Turner, D. A., van der Vleuten, C., Schumacher, D. J. 2021; 96 (7S): S56-S63

    Abstract

    Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

    View details for DOI 10.1097/ACM.0000000000004108

    View details for PubMedID 34183603

  • Entrustable Professional Activities and Entrustment Decision Making: A Development and Research Agenda for the Next Decade. Academic medicine : journal of the Association of American Medical Colleges Ten Cate, O., Balmer, D. F., Caretta-Weyer, H., Hatala, R., Hennus, M. P., West, D. C. 2021; 96 (7S): S96-S104

    Abstract

    To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and development: the micro level of learning and teaching; the meso level of institutions, programs, and specialty domains; and the macro level of regional, national, and international dynamics. Within these levels, the authors categorized their recommendations for research and development. The authors identified 14 discrete themes, each including multiple questions or issues for potential exploration, that range from foundational and conceptual to practical. Much research to date has focused on a variety of issues regarding development and early implementation of EPAs. Future research should focus on large-scale implementation of EPAs to support competency-based medical education (CBME) and on its consequences at the 3 levels. In addition, emerging from the implementation phase, the authors call for rigorous studies focusing on conceptual issues. These issues include the nature of entrustment decisions and their relationship with education and learner progress and the use of EPAs across boundaries of training phases, disciplines and professions, including continuing professional development. International studies evaluating the value of EPAs across countries are another important consideration. Future studies should also remain alert for unintended consequences of the use of EPAs. EPAs were conceptualized to support CBME in its endeavor to improve outcomes of education and patient care, prompting creation of this agenda.

    View details for DOI 10.1097/ACM.0000000000004106

    View details for PubMedID 34183610

  • Missing the mark: Alternative sources of variation in graduate milestone achievement. AEM education and training Caretta-Weyer, H. A., Schumacher, D. J. 2021; 5 (3): e10618

    View details for DOI 10.1002/aet2.10618

    View details for PubMedID 34222752

  • Diagnosing conflict: Conflicting data, interpersonal conflict, and conflicts of interest in clinical competency committees. Medical teacher Chan, T., Oswald, A., Hauer, K. E., Caretta-Weyer, H. A., Nousiainen, M. T., Cheung, W. J., ICBME Collaborators 2021: 1-9

    Abstract

    Clinical competency committees (CCCs) are increasingly used within health professions education as their decisions are thought to be more defensible and fairer than those generated by previous training promotion processes. However, as with most group-based processes, it is inevitable that conflict will arise. In this paper the authors explore three ways conflict may arise within a CCC: (1) conflicting data submissions that are presented to the committee, (2) conflicts between members of the committee, and (3) conflicts of interest between a specific committee member and a trainee. The authors describe each of these conflict situations, dissect out the underlying problems, and explore possible solutions based on the current literature.

    View details for DOI 10.1080/0142159X.2021.1925101

    View details for PubMedID 34182879

  • Capturing outcomes of competency-based medical education: The call and the challenge. Medical teacher Van Melle, E., Hall, A. K., Schumacher, D. J., Kinnear, B., Gruppen, L., Thoma, B., Caretta-Weyer, H., Cooke, L. J., Frank, J. R., ; On behalf of the ICBME Collaborators 2021: 1-7

    Abstract

    There is an urgent need to capture the outcomes of the ongoing global implementation of competency-based medical education (CBME). However, the measurement of downstream outcomes following educational innovations, such as CBME is fraught with challenges stemming from the complexities of medical training, the breadth and variability of inputs, and the difficulties attributing outcomes to specific educational elements. In this article, we present a logic model for CBME to conceptualize an impact pathway relating to CBME and facilitate outcomes evaluation. We further identify six strategies to mitigate the challenges of outcomes measurement: (1) clearly identify the outcome of interest, (2) distinguish between outputs and outcomes, (3) carefully consider attribution versus contribution, (4) connect outcomes to the fidelity and integrity of implementation, (5) pay attention to unanticipated outcomes, and (6) embrace methodological pluralism. Embracing these challenges, we argue that careful and thoughtful evaluation strategies will move us forward in answering the all-important question: Are the desired outcomes of CBME being achieved?

    View details for DOI 10.1080/0142159X.2021.1925640

    View details for PubMedID 34121596

  • Outcomes of competency-based medical education: A taxonomy for shared language. Medical teacher Hall, A. K., Schumacher, D. J., Thoma, B., Caretta-Weyer, H., Kinnear, B., Gruppen, L., Cooke, L. J., Frank, J. R., Van Melle, E., ICBME Collaborators 2021: 1-6

    Abstract

    As the global transformation of postgraduate medical training continues, there are persistent calls for program evaluation efforts to understand the impact and outcomes of competency-based medical education (CBME) implementation. The measurement of a complex educational intervention such as CBME is challenging because of the multifaceted nature of activities and outcomes. What is needed, therefore, is an organizational taxonomy to both conceptualize and categorize multiple outcomes. In this manuscript we propose a taxonomy that builds on preceding works to organize CBME outcomes across three domains: focus (educational, clinical), level (micro, meso, macro), and timeline (training, transition to practice, practice). We also provide examples of how to conceptualize outcomes of educational interventions across medical specialties using this taxonomy. By proposing a shared language for outcomes of CBME, we hope that this taxonomy will help organize ongoing evaluation work and catalyze those seeking to engage in the evaluation effort to help understand the impact and outcomes of CBME.

    View details for DOI 10.1080/0142159X.2021.1925643

    View details for PubMedID 34038673

  • The deliberately developmental organization: A conceptual framework for CBME. Medical teacher Thoma, B., Caretta-Weyer, H., Schumacher, D. J., Warm, E., Hall, A. K., Hamstra, S. J., Cavalcanti, R., Chan, T. M., ICBME Collaborators 2021: 1-9

    Abstract

    Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

    View details for DOI 10.1080/0142159X.2021.1925100

    View details for PubMedID 34033512

  • If we could turn back time: Imagining time-variable, competency-based medical education in the context of COVID-19. Medical teacher Caretta-Weyer, H. A., Chan, T., Bigham, B. L., Kinnear, B., Huwendiek, S., Schumacher, D. J., ICBME Collaborators 2021: 1-6

    Abstract

    The COVID-19 pandemic has exposed a paradox in historical models of medical education: organizations responsible for applying consistent standards for progression have needed to adapt to training environments marked by inconsistency and change. Although some institutions have maintained their traditional requirements, others have accelerated their programs to rush nearly graduated trainees to the front lines. One interpretation of the unplanned shortening of the duration of training programs during a crisis is that standards have been lowered. But it is also possible that these trainees were examined according to the same standards as usual and were judged to have already met them. This paper discusses the impacts of the COVID-19 pandemic on the current workforce, provides an analysis of how competency-based medical education (CBME) in the context of the pandemic might have mitigated wide-scale disruption, and identifies structural barriers to achieving an ideal state. The paper further calls upon universities, health centres, governments, certifying bodies, regulatory authorities, and health care professionals to work collectively on a truly time-variable model of CBME. The pandemic has made clear that time variability in medical education already exists and should be adopted widely and formally. If our systems today had used a framework of outcome competencies, sequenced progression, tailored learning, focused instruction, and programmatic assessment, we may have been even more nimble in changing our systems to care for our patients with COVID-19.

    View details for DOI 10.1080/0142159X.2021.1925641

    View details for PubMedID 34027813

  • On the validity of summative entrustment decisions. Medical teacher Touchie, C., Kinnear, B., Schumacher, D., Caretta-Weyer, H., Hamstra, S. J., Hart, D., Gruppen, L., Ross, S., Warm, E., Ten Cate, O., ICBME Collaborators 2021: 1-8

    Abstract

    Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

    View details for DOI 10.1080/0142159X.2021.1925642

    View details for PubMedID 34020576

  • Clarifying essential terminology in entrustment MEDICAL TEACHER Schumacher, D. J., ten Cate, O., Damodaran, A., Richardson, D., Hamstra, S. J., Ross, S., Hodgson, J., Touchie, C., Molgaard, L., Gofton, W., Carraccio, C., ICBME Collaborators 2021: 1-8

    Abstract

    With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

    View details for DOI 10.1080/0142159X.2021.1924365

    View details for Web of Science ID 000650606100001

    View details for PubMedID 33989100

  • Curated collection for clinician educators: Six key papers on residency recruitment. AEM education and training Schnapp, B. H., Alvarez, A., Bianchi, R., Caretta-Weyer, H., Jewell, C., Kalantari, A., Lee, E., Miller, D., Quinn, A. 2021; 5 (2): e10597

    Abstract

    Introduction: All emergency medicine (EM) residency programs must recruit new medical school graduates each year. The process is often overwhelming, with each program receiving far more applicants than available positions. We searched for evidence-based best practices to guide residency programs in screening, interviewing, and ranking applicants to ensure a high-performing and diverse residency class.Methods: A literature search was conducted on the topic of residency recruitment, utilizing a call on social media as well as multiple databases. After identifying relevant articles, we performed a modified Delphi process in three rounds, utilizing junior educators as well as more senior faculty.Results: We identified 51 relevant articles on the topic of residency recruitment. The Delphi process yielded six articles that were deemed most highly relevant over the three rounds. Transparency with selection criteria, holistic application review, standardized letters of evaluation, and blinding applicant files for interviewers were among noted best practices.Conclusions: Well-supported evidence-based practices exist for residency recruitment, and programs may benefit from understanding which common recruitment practices offer the most value. The articles discussed here provide a foundation for faculty looking to improve their program's recruiting practices.

    View details for DOI 10.1002/aet2.10597

    View details for PubMedID 33969251

  • Development, Implementation, and Evaluation of an Open Access, Level-Specific, Core Content Curriculum for Emergency Medicine Residents. Journal of graduate medical education Grabow Moore, K., Ketterer, A., Wheaton, N., Weygandt, P. L., Caretta-Weyer, H. A., Berberian, J., Jordan, J. 2021; 13 (5): 699-710

    Abstract

    Leaders in graduate medical education must provide robust clinical and didactic experiences to prepare residents for independent practice. Programs traditionally create didactic experiences individually, requiring tremendous resources with variable content exposure and quality.We sought to create and implement a free, open access, learner-centric, level-specific, emergency medicine (EM) residency curriculum.We developed Foundations of Emergency Medicine (FoEM) Foundations I and II courses using Kern's model of curriculum development. Fundamental topics were identified through content guidelines from the American Board of Emergency Medicine. We incorporated learner-centric strategies into 2 flipped classroom, case-based courses targeting postgraduate year (PGY) 1 and PGY-2 residents. The curriculum was made freely available online in 2016. Faculty and resident users were surveyed annually for feedback, which informed iterative refinement of the curriculum.Between 2016 and 2020, registration for FoEM expanded from 2 sites with 36 learners to 154 sites and 4453 learners. In 2019, 98 of 102 (96%) site leaders and 1618 of 2996 (54%) learners completed the evaluative survey. One hundred percent of responding leaders and 93% of learners were "satisfied" or "very satisfied" with FoEM content. Faculty and residents valued FoEM's usability, large volume of content, quality, adaptability, organization, resident-faculty interaction, and resident-as-teacher opportunities. Challenges to implementation included resident attendance, conference structure, technology limitations, and faculty engagement.We developed and implemented a learner-centric, level-specific, national EM curriculum that has been widely adopted in the United States.

    View details for DOI 10.4300/JGME-D-21-00067.1

    View details for PubMedID 34721800

    View details for PubMedCentralID PMC8527934

  • Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic. Medical teacher Schumacher, D. J., Caretta-Weyer, H., Busari, J., Carraccio, C., Damodaran, A., Gruppen, L. D., Hall, A. K., Kinnear, B., Warm, E., Ten Cate, O. 2021: 1-13

    Abstract

    Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

    View details for DOI 10.1080/0142159X.2021.1925098

    View details for PubMedID 34038645

  • Demonstrating Your Work: A Guide to Educators' Portfolios for Graduate Medical Educators. Journal of graduate medical education Mayersak, R. J., Yarris, L. M., Tuttle, R. C., Jones, D. C., Nelson, A. M., Bengtzen, R. R., Kornegay, J. G., Caretta-Weyer, H. A. 2021; 13 (5): 635-639

    View details for DOI 10.4300/JGME-D-21-00375.1

    View details for PubMedID 34721790

    View details for PubMedCentralID PMC8527930

  • Trends in medical students' stress, physical, and emotional health throughout training. Medical education online McKerrow, I., Carney, P. A., Caretta-Weyer, H., Furnari, M., Miller Juve, A. 2020; 25 (1): 1709278

    Abstract

    Background: Medical student wellness, including physical health, emotional health, and levels of perceived stress, appears to decline during training, with students reporting high levels of depression, anxiety, and burnout as early as the first year of medical school. The impact of curricular changes on health and stress remains unclear, and a modified curriculum that compresses training of the foundational sciences and its effect on wellness has not been studied. Oregon Health & Science University School of Medicine has recently instituted a unique competency-based model, which provides an important opportunity to assess the effects of curricular change on student wellness.Objective: Assess the effects of curricular change on student wellness.Design: Medical students at a single institution were administered the SF-8, an 8-item health-related quality of life survey, as well as the Perceived Stress Scale, a 10-item scale that measures the degree to which life situations are appraised as stressful, at baseline (matriculation) and at the end of Year 1, 2 and 3. Individual variables were assessed over time, as well as a trend analysis of summary domain scores over the 4 time periods.Results: Physical, emotional, and overall health were highest at baseline and lowest at the end of Year 1, after which they improved but never again reached baseline levels. Physical health declined less than emotional health. Perceived stress levels did not change over time but remained moderately high. There were no differences in health or perceived stress based on demographic variables.Conclusions: In a competency-based curriculum, physical, emotional and overall health significantly worsened during Year 1 but improved thereafter, while perceived stress remained unchanged. Early in training, stress and poor overall health may be related to concerns about self-efficacy and workload. Although advanced students show improved wellness, concerns remained about emotional difficulties, such as anxiety and irritability, and feeling a lack of control.

    View details for DOI 10.1080/10872981.2019.1709278

    View details for PubMedID 31902315

  • The Flipped Classroom: A Critical Appraisal. The western journal of emergency medicine Kraut, A. S., Omron, R., Caretta-Weyer, H., Jordan, J., Manthey, D., Wolf, S. J., Yarris, L. M., Johnson, S., Kornegay, J. 2019; 20 (3): 527–36

    Abstract

    Introduction: The objective of this study was to review and critically appraise the medical education literature pertaining to a flipped-classroom (FC) education model, and to highlight influential papers that inform our current understanding of the role of the FC in medical education.Methods: A search of the English-language literature querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 296 papers related to the FC using either quantitative, qualitative, or review methods. Two reviewers independently screened each category of publications using previously established exclusion criteria. Eight reviewers then independently scored the remaining 54 publications using either a qualitative, quantitative, or review-paper scoring system. Each scoring system consisted of nine criteria and used parallel metrics that have been previously used in critical appraisals of education research.Results: A total of 54 papers (33 quantitative, four qualitative, and 17 review) on FC met a priori criteria for inclusion and were critically appraised and reviewed. The top 10 highest scoring articles (five quantitative studies, two qualitative studies, and three review papers) are summarized in this article.Conclusion: This installment of the Council of Emergency Medicine Residency Directors (CORD) Academy Critical Appraisal series highlights 10 papers that describe the current state of literature on the flipped classroom, including an analysis of the benefits and drawbacks of an FC approach, practical implications for emergency medicine educators, and next steps for future research.

    View details for DOI 10.5811/westjem.2019.2.40979

    View details for PubMedID 31123556

  • Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education CUREUS Schnapp, B. H., Caretta-Weyer, H. A., Cortez, E., Heinrich, S. A., Kraut, A. S., Lloyd, C. M., Silvester, C., Sorge, R. M., Wain, A., Gottlieb, M. 2019; 11 (4)
  • Design Your Clinical Workplace to Facilitate Competency-Based Education. The western journal of emergency medicine Caretta-Weyer, H. A., Gisondi, M. A. 2019; 20 (4): 651–53

    View details for DOI 10.5811/westjem.2019.4.43216

    View details for PubMedID 31316706

    View details for PubMedCentralID PMC6625682

  • Critical Electrocardiogram Curriculum: Setting the Standard for Flipped-Classroom EKG Instruction. The western journal of emergency medicine Burns, W. P., Hartman, N. D., Weygandt, P. L., Jones, S. C., Caretta-Weyer, H. n., Moore, K. G. 2019; 21 (1): 52–57

    Abstract

    Electrocardiogram (EKG) interpretation is integral to emergency medicine (EM). In 2003 Ginde et al. found 48% of emergency medicine (EM) residency directors supported creating a national EKG curriculum. No formal national curriculum exists, and it is unknown whether residents gain sufficient skill from clinical exposure alone.The authors sought to assess the value of this EKG curriculum, which provides exposure to critical EKG patterns, a framework for EKG interpretation when the diagnosis is not obvious, and implementation guidelines and open access to any interested residency. The Foundations of Emergency Medicine (FoEM) EKG I course launched in January 2016, followed by EKG II in July 2017; they are benchmarked to post-graduate year 1 (PGY) and PGY2 level learners, respectively. Selected topics included 15 published critical EKG diagnoses and 33 selected by the authors. Cases included presenting symptoms, EKGs, and Free Open Access Medical Education (FOAM) links. Full EKG interpretations and question answers were provided.Enrollment during 2017-2018 included 37 EM residencies with 663 learners in EKG I and 22 EM residencies with 438 learners in EKG II. Program leaders and learners were surveyed annually. Leaders indicated that content was appropriate for intended PGY levels. Leaders and learners indicated the curriculum improved the ability of learners to interpret EKGs while working in the emergency department (ED).There is an unmet need for standardization and improvement of EM resident EKG training. Leaders and learners exposed to FoEM EKG courses report improved ability of learners to interpret EKGs in the ED.

    View details for DOI 10.5811/westjem.2019.11.44509

    View details for PubMedID 31913819

  • Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education. Cureus Schnapp, B. H., Caretta-Weyer, H. A., Cortez, E. n., Heinrich, S. A., Kraut, A. S., Lloyd, C. M., Silvester, C. n., Sorge, R. M., Wain, A. n., Gottlieb, M. n. 2019; 11 (4): e4383

    Abstract

    Introduction The Accreditation Council for Graduate Medical Education calls graduated responsibility "one of the core tenets of American graduate medical education." However, there is no clear set of resources for programs to implement a system of progressively increasing responsibilities for trainees. This project aimed to identify a set of high-yield papers on graduated responsibility for junior faculty members. Methods A study group of Academic Life in Emergency Medicine Faculty Incubator participants identified relevant literature on graduated responsibility via a comprehensive literature search and a call to the online medical education community; 59 total papers were identified. The most relevant and applicable were selected by the study group via a three-round modified Delphi process. Results Five key articles for junior faculty interested in implementing more robust graduated responsibility at their residency training program were selected and described here. Summaries of key points, along with considerations for faculty developers and relevance to junior faculty, are presented for each article. Conclusions The articles presented here provide a solid theoretical and practical basis for junior faculty to explore graduated responsibility. The five articles presented here provide the junior faculty with a toolkit to examine and improve their systems for assigning responsibilities in a graded fashion at their own institutions.

    View details for DOI 10.7759/cureus.4383

    View details for PubMedID 31218147

    View details for PubMedCentralID PMC6553674

  • Transition to Practice: A Novel Life Skills Curriculum for Emergency Medicine Residents. The western journal of emergency medicine Caretta-Weyer, H. 2019; 20 (1): 100–104

    View details for PubMedID 30643610

  • Transition to Practice: A Novel Life Skills Curriculum for Emergency Medicine Residents WESTERN JOURNAL OF EMERGENCY MEDICINE Caretta-Weyer, H. 2019; 20 (1): 100–104
  • Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship. Cureus Quinn, A. n., Chan, T. M., Sampson, C. n., Grossman, C. n., Butts, C. n., Casey, J. n., Caretta-Weyer, H. n., Gottlieb, M. n. 2018; 10 (1): e2021

    Abstract

    Traditionally, scholarship that was recognized for promotion and tenure consisted of clinical research, bench research, and grant funding. Recent trends have allowed for differing approaches to scholarship, including digital publication. As increasing numbers of trainees and faculty turn to online educational resources, it is imperative to critically evaluate these resources. This article summarizes five key papers that address the appraisal of digital scholarship and describes their relevance to junior clinician educators and faculty developers. In May 2017, the Academic Life in Emergency Medicine Faculty Incubator program focused on the topic of digital scholarship, providing and discussing papers relevant to the topic. We augmented this list of papers with further suggestions by guest experts and by an open call via Twitter for other important papers. Through this process, we created a list of 38 papers in total on the topic of evaluating digital scholarship. In order to determine which of these papers best describe how to evaluate digital scholarship, the authorship group assessed the papers using a modified Delphi approach to build consensus. In this paper we present the five most highly rated papers from our process about evaluating digital scholarship. We summarize each paper and discuss its specific relevance to junior faculty members and to faculty developers. These papers provide a framework for assessing the quality of digital scholarship, so that junior faculty can recommend high-quality educational resources to their trainees. These papers help guide educators on how to produce high quality digital scholarship and maximize recognition and credit in respect to receiving promotion and tenure.

    View details for PubMedID 29531874

  • The View From Over Here: A Framework for Multi-Source Feedback. Journal of graduate medical education Caretta-Weyer, H. A., Kraut, A. S., Kornegay, J. G., Yarris, L. M. 2017; 9 (3): 367–68

    View details for DOI 10.4300/JGME-D-17-00200.1

    View details for PubMedID 28638519

    View details for PubMedCentralID PMC5476390

  • Feedback in Medical Education: A Critical Appraisal. AEM education and training Kornegay, J. G., Kraut, A. n., Manthey, D. n., Omron, R. n., Caretta-Weyer, H. n., Kuhn, G. n., Martin, S. n., Yarris, L. M. 2017; 1 (2): 98–109

    Abstract

    The objective was to review and critically appraise the medical education literature pertaining to feedback and highlight influential papers that inform our current understanding of the role of feedback in medical education.A search of the English language literature in querying Education Resources Information Center (ERIC), PsychINFO, PubMed, and Scopus identified 327 feedback-related papers using either quantitative (hypothesis-testing or observational investigations of educational interventions), qualitative methods (exploring important phenomena in emergency medicine [EM] education), or review methods.Two reviewers independently screened each category of publications using previously established exclusion criteria. Six reviewers then independently scored the remaining 54 publications using a qualitative, quantitative, or review paper scoring system. Each scoring system consisted of nine criteria and used parallel scoring metrics that have been previously used in critical appraisals of education research.Fifty-four feedback papers (25 quantitative studies, 24 qualitative studies, five review papers) met the a priori criteria for inclusion and were reviewed. Eight quantitative studies, nine qualitative studies, and three review papers were ranked highly by the reviewers and are summarized in this article.This inaugural Council of Emergency Medicine Residency Directors Academy critical appraisal highlights 20 feedback in medical education papers that describe the current state of the feedback literature. A summary of current factors that influence feedback effectiveness is discussed, along with practical implications for EM educators and the next steps for research.

    View details for PubMedID 30051017

    View details for PubMedCentralID PMC6001508

  • Determining breast cancer axillary surgery within the surveillance epidemiology and end results-Medicare database JOURNAL OF SURGICAL ONCOLOGY Schmocker, R. K., Caretta-Weyer, H., Weiss, J. M., Ronk, K., Havlena, J., Loconte, N. K., Decker, M., Smith, M. A., Greenberg, C. C., Neuman, H. B. 2014; 109 (8): 756-759

    Abstract

    Use of sentinel lymph node biopsy (SLNB) is under-reported by cancer registries' "Scope of Regional Lymph Node Surgery" variable. In 2011, the Surveillance Epidemiology and End Results (SEER) Program recommended against its use to determine extent of axillary surgery, leaving a gap in the utilization of claims data for breast cancer research. The objective was to develop an algorithm using SEER registry and claims data to classify extent of axillary surgery for breast cancer.We analyzed data for 24,534 breast cancer patients. CPT codes and number of examined lymph nodes classified the extent of axillary surgery. The final algorithm was validated by comparing the algorithm derived extent of axillary surgery to direct chart review for 100 breast cancer patients treated at our breast center.Using the algorithm, 13% had no axillary surgery, 56% SLNB and 31% axillary lymph node dissection (ALND). SLNB was performed in 77% of node negative patients and ALND in 72% of node positive. In our validation study, concordance between algorithm and direct chart review was 97%.Given recognized inaccuracies in cancer registries' "Scope of Regional Lymph Node Surgery" variable, these findings have high utility for health services researchers studying breast cancer treatment.

    View details for DOI 10.1002/jso.23579

    View details for Web of Science ID 000335369700003

    View details for PubMedID 24643795

    View details for PubMedCentralID PMC4227499

  • Impact of the American College of Surgeons Oncology Group (ACOSOG) Z0011 trial on Clinical Management of the Axilla in Older Breast Cancer Patients: A SEER-Medicare Analysis ANNALS OF SURGICAL ONCOLOGY Caretta-Weyer, H., Greenberg, C. G., Wilke, L. G., Weiss, J., Loconte, N. K., Decker, M., Steffens, N. M., Smith, M. A., Neuman, H. B. 2013; 20 (13): 4145-4152

    Abstract

    American College of Surgeons Oncology Group (ACOSOG) Z0011 demonstrated that eligible breast cancer patients with positive sentinel lymph nodes (SLN) could be spared an axillary lymph node dissection (ALND) without sacrificing survival or local control. Although heralded as a ‘‘practice-changing trial,’’ some argue that the stringent inclusion criteria limit the trial’s clinical significance. The objective was to assess the potential impact of ACOSOG Z0011 on axillary surgical management of Medicare patients and examine current practice patterns.Medicare beneficiaries aged C66 years with nonmetastatic invasive breast cancer diagnosed from 2001 to 2007 were identified from the Surveillance, Epidemiology and End Results-Medicare database (n = 59,431). Eligibility for ACOSOG Z0011 was determined: SLN mapping, tumor\5 cm, no neoadjuvant treatment, breast conservation; number of positive nodes was determined. Actual surgical axillary management for eligible patients was assessed.Twelve percent (6,942/59,431) underwent SLN mapping and were node positive. Overall, 2,637 patients (4.4 % (2,637/59,431) of the total cohort, but 38 % (2,637/6,942) of patients with SLN mapping and positive nodes) met inclusion criteria for ACOSOG Z0011, had 1 or 2 positive lymph nodes, and could have been spared an ALND. Of these 2,637 patients, 46 % received a completion ALND and 54 % received only SLN biopsy.Widespread implementation of ACOSOG Z0011 trial results could potentially spare 38 % of older breast cancer patients who undergo SLN mapping with positive lymph nodes an ALND. However, 54 % of these patients are already managed with SLN biopsy alone, lessening the impact of this trial on clinical practice in older breast cancer patients.

    View details for DOI 10.1245/s10434-013-3193-1

    View details for Web of Science ID 000328256600016

    View details for PubMedID 23959051

    View details for PubMedCentralID PMC3874252

  • Impact of axillary ultrasound and core needle biopsy on the utility of intraoperative frozen section analysis and treatment decision making in women with invasive breast cancer AMERICAN JOURNAL OF SURGERY Caretta-Weyer, H., Sisney, G. A., Beckman, C., Burnside, E. S., Salkowsi, L. R., Strigel, R. M., Wilke, L. G., Neuman, H. B. 2012; 204 (3): 308-314

    Abstract

    Our objective was to evaluate the impact of preoperative axillary ultrasound and core needle biopsy (CNB) on breast cancer treatment decision making. A secondary aim was to evaluate the impact on the utility of intraoperative sentinel lymph node (SLN) frozen section.A review of 84 patients with clinically negative axilla who underwent axillary ultrasound was performed. Sensitivity, specificity, and positive/negative predictive value for axillary ultrasound with CNB was calculated.Thirty-one (37%) had suspicious nodes. Of 27 amenable to CNB, 12 (14%) were malignant, changing treatment plans. The sensitivity of ultrasound and CNB was 54% and specificity 100%; the positive and negative predictive values were 100% and 80%, respectively. In 41 patients with normal ultrasounds who underwent SLN frozen section, 10 (24%) were positive.Preoperative axillary ultrasound impacts treatment decision making in 14%. With a sensitivity of 54%, it is a useful adjunct to, but not replacement for, SLN biopsy. Frozen section remains of utility even after a negative axillary ultrasound.

    View details for DOI 10.1016/j.amjsurg.2011.10.016

    View details for PubMedID 22483606

  • Involvement of Noradrenergic Neurotransmission in the Stress- but not Cocaine-Induced Reinstatement of Extinguished Cocaine-Induced Conditioned Place Preference in Mice: Role for beta-2 Adrenergic Receptors NEUROPSYCHOPHARMACOLOGY Mantsch, J. R., Weyer, A., Vranjkovic, O., Beyer, C. E., Baker, D. A., Caretta, H. 2010; 35 (11): 2165-2178

    Abstract

    The responsiveness of central noradrenergic systems to stressors and cocaine poses norepinephrine as a potential common mechanism through which drug re-exposure and stressful stimuli promote relapse. This study investigated the role of noradrenergic systems in the reinstatement of extinguished cocaine-induced conditioned place preference by cocaine and stress in male C57BL/6 mice. Cocaine- (15 mg/kg, i.p.) induced conditioned place preference was extinguished by repeated exposure to the apparatus in the absence of drug and reestablished by a cocaine challenge (15 mg/kg), exposure to a stressor (6-min forced swim (FS); 20-25°C water), or administration of the α-2 adrenergic receptor (AR) antagonists yohimbine (2 mg/kg, i.p.) or BRL44408 (5, 10 mg/kg, i.p.). To investigate the role of ARs, mice were administered the nonselective β-AR antagonist, propranolol (5, 10 mg/kg, i.p.), the α-1 AR antagonist, prazosin (1, 2 mg/kg, i.p.), or the α-2 AR agonist, clonidine (0.03, 0.3 mg/kg, i.p.) before reinstatement testing. Clonidine, prazosin, and propranolol failed to block cocaine-induced reinstatement. The low (0.03 mg/kg) but not high (0.3 mg/kg) clonidine dose fully blocked FS-induced reinstatement but not reinstatement by yohimbine. Propranolol, but not prazosin, blocked reinstatement by both yohimbine and FS, suggesting the involvement of β-ARs. The β-2 AR antagonist ICI-118551 (1 mg/kg, i.p.), but not the β-1 AR antagonist betaxolol (10 mg/kg, i.p.), also blocked FS-induced reinstatement. These findings suggest that stress-induced reinstatement requires noradrenergic signaling through β-2 ARs and that cocaine-induced reinstatement does not require AR activation, even though stimulation of central noradrenergic neurotransmission is sufficient to reinstate.

    View details for DOI 10.1038/npp.2010.86

    View details for Web of Science ID 000281821200003

    View details for PubMedID 20613718

    View details for PubMedCentralID PMC2939933