Cara A. Liebert, MD, FACS

All Publications

• Correlation of Performance on the ENTRUST Assessment Platform With Other Variables in Competency-Based Surgical Education. Journal of surgical education Wong, L. Y., Lam, N., Son, Y. A., Eddington, H., Arnow, K. D., Tsai, J., Anand, A., Peralta, F. C., Shields, S., Melcer, E. F., Lin, D. T., Liebert, C. A. 2024; 82 (1): 103293

  Abstract

  With the implementation of American Board of Surgery (ABS) Entrustable Professional Activities (EPAs), there is continued need for objective, evidence-based assessment tools to augment existing microassessments and inform readiness for entrustment. The ENTRUST Assessment Platform is an online virtual-patient simulation platform to assess trainees' surgical decision-making competence across preoperative, intraoperative, and postoperative phases of care. This study collects additional validity evidence for the ENTRUST platform in its relationship to other established variables in competency-based surgical education.This is a prospective analysis of surgical resident performance on the ENTRUST Right Lower Quadrant (RLQ) pain/Appendicitis EPA Assessment. ENTRUST scores were analyzed by PGY-level and correlations with Accreditation Council for Graduate Medical Education (ACGME) Case Logs, ACGME Surgery Milestones, and ABS In-Service Training Examination (ABSITE) scores were evaluated. Bivariate analyses were performed using Spearman rank correlations.This study was conducted at a tertiary academic center (Stanford University, Palo Alto, CA) in a proctored exam setting.Thirty-two PGY-1 though PGY-5 general surgery residents completed the ENTRUST RLQ Pain/Appendicitis EPA Assessment containing four case scenarios which were iteratively developed and scored by expert consensus and aligned with ABS EPA definitions.ENTRUST grand total score was positively correlated with PGY-level (rho = 0.57, p = 0.001), ACGME appendectomy case log volume (rho = 0.55, p = 0.002), and ABSITE raw score (rho = 0.66, p = 0.0004). ENTRUST performance was significantly correlated with all eighteen ACGME Surgery Milestones (rho = 0.43 to rho = 0.54, all p≤0.01), with the strongest correlation seen for PC1 (Patient Evaluation and Decision Making) (rho = 0.54, p = 0.006).Performance on ENTRUST was significantly correlated with established variables in surgical training, including ACGME Appendectomy Case Logs, ABSITE, and ACGME Surgery Milestones. This study strengthens existing validity evidence for the ENTRUST Assessment Platform as an objective assessment of clinical decision-making. ENTRUST is an assessment tool which can augment microassessments and support competency-based medical education.

  View details for DOI 10.1016/j.jsurg.2024.09.012

  View details for PubMedID 39532037

• Resident-Applicant Buddy Program Increases Applicant Interest and Program Transparency. Journal of surgical education Shearer, J., Ngongoni, R. F., Yelorda, K., Nobuhara, C., Lin, D. T., Gahagan, J., Dua, M., Spain, D. A., Liebert, C. A. 2024; 81 (11): 1792-1797

  Abstract

  Resident-Applicant Buddy Programs (RABPs) are a new initiative designed to improve resident recruitment. This study aims to evaluate the impact and perceived value of RABPs and to identify areas for improvement for future recruitment cycles.Anonymous online survey study of RABP participants with mixed-methods approach to evaluate participants' experience and perceived impact of the program. The survey queried demographics, Likert responses, and open-ended responses. Qualitative thematic analysis of open-ended responses was performed with inductive coding in an iterative fashion by 2 raters.This study was conducted at a general surgery residency program at a tertiary academic institution during 2022-2023 recruitment cycle.Of 125 RABP participants (n = 39 residents and n = 86 interviewed applicants), surveys from n = 45 participants (n = 19 residents, 66%; n = 26 applicants, 30%) were completed and analyzed.Applicants were predominantly female (65%) and first-generation physicians (69%). Buddy pairings were 65% gender concordant and 48% race/ethnicity concordant. Many applicants (60%) participated in RABPs at other institutions. Buddies connected for a mean (SD) of 52 (28) minutes. Majority of applicants agreed the program decreased stress/apprehension about interviewing (70%, 4.0 [1.1]), helped understand resident life at the program (91%, 4.3 [1.0]), and increased desire to match in the program (65%, 4.0 [1.1]). Residents agreed they enjoyed participation (89%, 4.5 [0.7]), the program should be continued (100%, 4.8 [0.4]), and desired to participate again (100%, 4.8 [0.4]). Thematic analysis revealed applicants valued the program as an approachable source of information, illumination of program culture, aid in interview preparation, and connection between applicant and program. Applicants appreciated the intentionality of the program to create a RABP.RABP decreased applicants' stress, improved understanding of resident life, and for the majority, increased desire to match at the program. Resident engagement and desire for ongoing participation in the RABP was high. Overall, RABPs can increase applicant interest and program transparency.

  View details for DOI 10.1016/j.jsurg.2024.08.010

  View details for PubMedID 39321695

• Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis. Journal of surgical education Lee, M. C., Melcer, E. F., Merrell, S. B., Wong, L. Y., Shields, S., Eddington, H., Trickey, A. W., Tsai, J., Korndorffer, J. R., Lin, D. T., Liebert, C. A. 2023

  Abstract

  As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings.This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework.This study was conducted at an academic institution in a proctored exam setting.The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment.The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST.ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

  View details for DOI 10.1016/j.jsurg.2023.09.001

  View details for PubMedID 37821350

• Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa. Journal of the American College of Surgeons Liebert, C. A., Melcer, E. F., Eddington, H., Trickey, A., Shields, S., Lee, M., Korndorffer, J. R., Bekele, A., Wren, S. M., Lin, D. T. 2023

  Abstract

  To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence.In partnership with COSECSA, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n=110) completed the traditional 11-station oral OSCE, followed by three ENTRUST cases, authored to query similar clinical content of three corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Exam outcome using independent sample t-tests. Correlation of ENTRUST scores to MCS Exam Percentage and OSCE Station Scores were calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance.ENTRUST performance was significantly higher in examinees who passed the MCS Exam compared to those who failed (p<0.001). ENTRUST score was positively correlated with MCS Exam Percentage (p<0.001) and combined OSCE Station Scores (p<0.001). On multivariate analysis, there was a strong association between MCS Exam Percentage and ENTRUST Grand Total Score (p<0.001), Simulation Total Score (p=0.018), and Question Total Score (p<0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST.This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.

  View details for DOI 10.1097/XCS.0000000000000740

  View details for PubMedID 37144790

• ENTRUST: A Serious Game-Based Virtual Patient Platform to Assess Entrustable Professional Activities in Graduate Medical Education. Journal of graduate medical education Lin, D. T., Melcer, E. F., Keehl, O., Eddington, H., Trickey, A. W., Tsai, J., Camacho, F., Merrell, S. B., Korndorffer, J., Liebert, C. A. 2023; 15 (2): 228-236

  Abstract

  As entrustable professional activities (EPAs) are implemented in graduate medical education, there is a great need for tools to efficiently and objectively evaluate clinical competence. Readiness for entrustment in surgery requires not only assessment of technical ability, but also the critical skill of clinical decision-making.We report the development of ENTRUST, a serious game-based, virtual patient case creation and simulation platform to assess trainees' decision-making competence. A case scenario and corresponding scoring algorithm for the Inguinal Hernia EPA was iteratively developed and aligned with the description and essential functions outlined by the American Board of Surgery. In this study we report preliminary feasibility data and validity evidence.In January 2021, the case scenario was deployed and piloted on ENTRUST with 19 participants of varying surgical expertise levels to demonstrate proof of concept and initial validity evidence. Total score, preoperative sub-score, and intraoperative sub-score were analyzed by training level and years of medical experience using Spearman rank correlations. Participants completed a Likert scale user acceptance survey (1=strongly agree to 7=strongly disagree).Median total score and intraoperative mode sub-score were higher with each progressive level of training (rho=0.79, P<.001 and rho=0.69, P=.001, respectively). There were significant correlations between performance and years of medical experience for total score (rho=0.82, P<.001) and intraoperative sub-scores (rho=0.70, P<.001). Participants reported high levels of platform engagement (mean 2.06) and ease of use (mean 1.88).Our study demonstrates feasibility and early validity evidence for ENTRUST as an assessment platform for clinical decision-making.

  View details for DOI 10.4300/JGME-D-22-00518.1

  View details for PubMedID 37139206

  View details for PubMedCentralID PMC10150817

• Robotic General Surgery Trends in the Veterans Health Administration, Community Practice, and Academic Centers From 2013 to 2021. JAMA surgery Mills, J., Liebert, C., Wren, S. M., Pratt, J. S., Earley, M., Eisenberg, D. 2023

  Abstract

  This cross-sectional study compares trends in use of robotic surgery for general surgical procedures among the Veterans Health Administration (VHA), community practice, and academic health centers from 2013 to 2021.

  View details for DOI 10.1001/jamasurg.2022.7728

  View details for PubMedID 36790771

• Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA). Journal of surgical education Liebert, C. A., Melcer, E. F., Keehl, O., Eddington, H., Trickey, A. W., Lee, M., Tsai, J., Camacho, F., Merrell, S. B., Korndorffer, J. R., Lin, D. T. 2022

  Abstract

  OBJECTIVE: As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence.DESIGN: This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses.SETTING: This study was conducted at an academic institution in a proctored exam setting.PARTICIPANTS: Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment.RESULTS: Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56).CONCLUSIONS: Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

  View details for DOI 10.1016/j.jsurg.2022.07.008

  View details for PubMedID 35909070

• ENTRUST: A Serious Game-Based Virtual Patient Platform to Assess Entrustable Professional Activities in Surgical Education Liebert, C. A., Lin, D. T., Melcer, E. F., Keehl, O. G., Trickey, A., Eddington, H., Merrell, S., Korndorffer, J. R. ELSEVIER SCIENCE INC. 2021: S224

  View details for Web of Science ID 000718303100423

• Perception Is Not Reality: Examining the Quality of Endof-Rotation Evaluations Anand, A., Jensen, R., Liebert, C., Korndorffer, J. R. LIPPINCOTT WILLIAMS & WILKINS. 2023: S445

  View details for Web of Science ID 001094086301372

• Using AI and computer vision to analyze technical proficiency in robotic surgery. Surgical endoscopy Yang, J. H., Goodman, E. D., Dawes, A. J., Gahagan, J. V., Esquivel, M. M., Liebert, C. A., Kin, C., Yeung, S., Gurland, B. H. 2022

  Abstract

  BACKGROUND: Intraoperative skills assessment is time-consuming and subjective; an efficient and objective computer vision-based approach for feedback is desired. In this work, we aim to design and validate an interpretable automated method to evaluate technical proficiency using colorectal robotic surgery videos with artificial intelligence.METHODS: 92 curated clips of peritoneal closure were characterized by both board-certified surgeons and a computer vision AI algorithm to compare the measures of surgical skill. For human ratings, six surgeons graded clips according to the GEARS assessment tool; for AI assessment, deep learning computer vision algorithms for surgical tool detection and tracking were developed and implemented.RESULTS: For the GEARS category of efficiency, we observe a positive correlation between human expert ratings of technical efficiency and AI-determined total tool movement (r=-0.72). Additionally, we show that more proficient surgeons perform closure with significantly less tool movement compared to less proficient surgeons (p<0.001). For the GEARS category of bimanual dexterity, a positive correlation between expert ratings of bimanual dexterity and the AI model's calculated measure of bimanual movement based on simultaneous tool movement (r=0.48) was also observed. On average, we also find that higher skill clips have significantly more simultaneous movement in both hands compared to lower skill clips (p<0.001).CONCLUSIONS: In this study, measurements of technical proficiency extracted from AI algorithms are shown to correlate with those given by expert surgeons. Although we target measurements of efficiency and bimanual dexterity, this work suggests that artificial intelligence through computer vision holds promise for efficiently standardizing grading of surgical technique, which may help in surgical skills training.

  View details for DOI 10.1007/s00464-022-09781-y

  View details for PubMedID 36536082

• Complete Telehealth for Multidisciplinary Preoperative Workup Does Not Delay Time to Metabolic and Bariatric Surgery: a Pilot Study. Obesity surgery Mills, J., Liebert, C., Pratt, J., Earley, M., Eisenberg, D. 2022

  Abstract

  PURPOSE: The COVID-19 pandemic accelerated implementation of telehealth throughout the US healthcare system. At our institution, we converted a fully integrated multidisciplinary bariatric clinic from face-to-face visits to entirely telehealth video/telephone visits. We hypothesized telehealth would increase the number of provider/patient encounters and therefore delay time to surgery.METHODS: This is a retrospective review of consecutive patients who underwent total telehealth preoperative workup. Demographics, comorbidities, and surgical characteristics were compared to the same number of consecutive patients who underwent a face-to-face approach 12months prior, using a Wilcoxon test for continuous variables and chi-square or Fisher's exact test for categorical variables. Differences between time and surgery were compared using inverse probability of treatment-weighted estimates and number of preoperative visits using Poisson regression with distance to hospital as a confounder. Noninferiority margin for time to surgery was set to 60days, and the number of visits was set to 2 visits.RESULTS: Between March of 2020 and December of 2021, 36 patients had total telehealth workup, and were compared to 36 patients in the traditional group. Age, sex, body mass index, and comorbidities did not differ between groups. The average number of days to surgery was 121.1days shorter in the telehealth group (90% bootstrap CI [-160.4,-81.8]). Estimated shift in the total number of visits was additional .76 visits in the traditional group (90% CI [.64, .91).CONCLUSIONS: The total telehealth approach to preoperative bariatric multidisciplinary workup did not delay surgery and decreased number of total outpatient visits and time to surgery.

  View details for DOI 10.1007/s11695-022-06233-3

  View details for PubMedID 36114438

• Preventive Health Screening in Veterans Undergoing Bariatric Surgery. American journal of preventive medicine Stoltz, D. J., Liebert, C. A., Seib, C. D., Bruun, A., Arnow, K. D., Barreto, N. B., Pratt, J. S., Eisenberg, D. 2022

  Abstract

  INTRODUCTION: Individuals with obesity are vulnerable to low rates of preventive health screening. Veterans with obesity seeking bariatric surgery are also hypothesized to have gaps in preventive health screening. Evaluation in a multidisciplinary bariatric surgery clinic is a point of interaction with the healthcare system that could facilitate improvements in screening.METHODS: This is a retrospective cohort study of 381 consecutive patients undergoing bariatric surgery at a Veterans Affairs Hospital from January 2010 to October 2021. Age- and sex-appropriate health screening rates were determined at initial referral to a multidisciplinary bariatric surgery clinic and at the time of surgery. Rates of guideline concordance at both time points were compared using McNemar's test. Univariate and multivariate analyses were performed to identify the risk factors for nonconcordance.RESULTS: Concordance with all recommended screening was low at initial referral and significantly improved by time of surgery (39.1%‒63.8%; p<0.001). Screening rates significantly improved for HIV (p<0.001), cervical cancer (p=0.03), and colon cancer (p<0.001). Increases in BMI (p=0.005) and the number of indicated screening tests (p=0.029) were associated with reduced odds of concordance at initial referral. Smoking history (p=0.012) and increasing distance to the nearest Veterans Affairs Medical Center (p=0.039) were associated with reduced odds of change from nonconcordance at initial referral to concordance at the time of surgery.CONCLUSIONS: Rates of preventive health screening in Veterans with obesity are low. A multidisciplinary bariatric surgery clinic is an opportunity to improve preventive health screening in Veterans referred for bariatric surgery.

  View details for DOI 10.1016/j.amepre.2022.06.014

  View details for PubMedID 36100538

• Recurrent bilateral cutaneous squamous cell carcinoma arising within pretibial hypertrophic lichen planus with metastasis while on cemiplimab Journal of the American Academy of Dermatology Case Reports Leeolou, M., Burgren, N., Lee, C., Momeni, A., Pinto, H., Johannet, P., Nord, K., Chang, A. S. 2022

  View details for DOI 10.1016/j.jdcr.2022.08.046

• A protected time policy to improve dental health among resident physicians JOURNAL OF THE AMERICAN DENTAL ASSOCIATION Jun, T., Liebert, C., Esquivel, M., Cox, J., Trockel, M., Katznelson, L. 2019; 150 (5): 362-+

  Abstract

  Resident physicians underuse preventive dental health services. The authors assessed the barriers to and need for oral health care among residents and piloted a program to enhance dental health care among house staff.Participants from 5 residency programs received 2 hours of protected time during business hours for visits to a nearby dental office. The authors surveyed participating residents before and after the visits about barriers to seeking oral health care and their experiences with the program. The authors recorded dental findings for each participant.A total of 35 of 243 eligible residents (14.4%) participated in the study; 71.4% reported delaying or skipping preventive dental examinations during residency. Lack of time and norms and peer perceptions were important barriers; 28.6% of residents had dental findings requiring further management.Residents neglect preventive oral health care because of work obligations. More than one-quarter of residents had clinically significant dental findings. Providing protected time addressed common barriers and was well received.Resident physicians have unmet oral health care needs. Collaborations between residency programs and dental practices to provide protected time for residents to seek oral health care could address common barriers to care.

  View details for DOI 10.1016/j.adaj.2018.12.016

  View details for Web of Science ID 000465431400024

  View details for PubMedID 31029211

• Authors' response. Journal of the American Dental Association (1939) Jun, T. W., Liebert, C. A., Esquivel, M. n., Cox, J. A., Trockel, M. n., Katznelson, L. n. 2019; 150 (7): 568–69

  View details for DOI 10.1016/j.adaj.2019.05.010

  View details for PubMedID 31248480

• Social Belonging as a Predictor of Surgical Resident Well-being and Attrition. Journal of surgical education Salles, A., Wright, R. C., Milam, L., Panni, R. Z., Liebert, C. A., Lau, J. N., Lin, D. T., Mueller, C. M. 2018

  Abstract

  OBJECTIVE: In light of the predicted shortage of surgeons, attrition from surgical residency is a significant problem. Prior data have shown that those who are happier are more productive, and those who are less well have higher rates of absenteeism. This study sought to identify the role of social belonging and its relationship to well-being and risk of attrition.DESIGN: Surgical residents were invited to participate in an online survey containing measures of social belonging (a 10-item scale adapted from previous studies), well-being (the Dupuy Psychological General Well-Being Scale, Beck Depression Inventory Short Form, and Maslach Burnout Inventory), and risk of attrition (indicated by frequency of thoughts of leaving the program).SETTING: We surveyed residents at 2 tertiary care centers, Stanford Health Care (2010, 2011, and 2015) and Washington University in St. Louis (2017).PARTICIPANTS: Categorical general surgery residents, designated preliminary residents going into 7 surgical subspecialties, and nondesignated preliminary residents were included.RESULTS: One hundred sixty-nine residents responded to the survey for a response rate of 66%. Belonging was positively correlated with general psychological well-being (r = 0.56, p < 0.0001) and negatively correlated with depression (r = -0.57, p < 0.0001), emotional exhaustion (r = -0.58, p < 0.0001), and depersonalization (r = -0.36, p < 0.0001). Further, belonging was negatively correlated with frequency of thoughts of leaving residency (r = -0.45, p < 0.0001). In regression analysis controlling for demographic variables, belonging was a significant positive predictor of psychological well-being (B = 0.95, t = 8.18, p < 0.0001) and a significant negative predictor of thoughts of leaving (B = -1.04, t = -5.44, p < 0.0001).CONCLUSIONS: Social belonging has a significant positive correlation with well-being and negative correlation with thoughts of leaving surgical training. Lack of social belonging appears to be a significant predictor of risk of attrition in surgical residency. Efforts to enhance social belonging may protect against resident attrition.ACGME COMPETENCY: Interpersonal and Communication Skills.

  View details for DOI 10.1016/j.jsurg.2018.08.022

  View details for PubMedID 30243929

• Perceived Value of a Program to Promote Surgical Resident Well-being. Journal of surgical education Salles, A., Liebert, C. A., Esquivel, M., Greco, R. S., Henry, R., Mueller, C. 2017

  Abstract

  The demands of surgical residency are intense and threaten not only trainees' physical wellness, but also risk depression, burnout, and suicide. Our residency program implemented a multifaceted Balance in Life program that is designed to improve residents' well-being. The purpose of this study was to evaluate the program utilization and perceived value by residents.Residents (n = 56, 76% response rate) were invited to participate in a voluntary survey from December 2013 to February 2014 regarding utilization, barriers to use, and perceived value of 6 program components (refrigerator, After Hours Guide, psychological counseling sessions, Resident Mentorship Program, Class Representative System, and social events). They were also asked questions about psychological well-being, burnout, grit, and sleep and exercise habits before and after implementation of the program.The most valued components of the program were the refrigerator (mean = 4.61) and the psychological counseling sessions (mean = 3.58), followed by social events (mean = 3.48), the Resident Mentorship Program (mean = 2.79), the Class Representative System (mean = 2.62), and the After Hours Guide (mean = 2.10). When residents were asked how they would allocate $100 among the different programs, the majority was allocated to the refrigerator ($54.31), social events ($26.43), and counseling sessions ($24.06). There was no change in psychological well-being or burnout after the program. Residents had higher levels of grit (β = 0.26, p < 0.01) and exercised (β = 1.02, p < 0.001) and slept (β = 1.17, p < 0.0001) more after the program was implemented.This study demonstrated that a multifaceted program to improve the well-being of trainees is feasible, highly valued, and positively perceived by the residents. Further research is needed to quantify the effectiveness and longitudinal impact such a program has on resident depression, burnout, and other psychological factors.

  View details for DOI 10.1016/j.jsurg.2017.04.006

  View details for PubMedID 28457875

• Prevalence and predictors of depression among general surgery residents. American journal of surgery Lin, D. T., Liebert, C. A., Esquivel, M. M., Tran, J., Lau, J. N., Greco, R. S., Mueller, C. M., Salles, A. 2017; 213 (2): 313-317

  Abstract

  Recent resident suicides have highlighted the need to address depression among medical trainees. This study sought to identify the prevalence and predictors of depression among surgical residents.Surgical residents at a single institution were surveyed. Depression and personal traits were assessed using validated measures; participant demographics were also obtained.73 residents completed the survey (response rate 63%). 36% met criteria for at least mild depression, of which 20% met criteria for moderate to severe depression. In multivariate linear regression analyses controlling for demographic factors, trait emotional intelligence alone was a significant inverse predictor of depression (β = -0.60, p < 0.001).Depression is prevalent among general surgery residents. Identifying protective factors and at-risk populations may allow for effective initiatives to be developed to address depression, and optimize the mental health of trainees.The aim of this study is to identify the prevalence and predictors of depression among surgical trainees. Over one third of respondents met criteria for at least mild depression, of which 20% met criteria for moderate to severe depression. Among demographic and personal trait variables, emotional intelligence emerged as a significant inverse predictor of depression.

  View details for DOI 10.1016/j.amjsurg.2016.10.017

  View details for PubMedID 28017297

• Grit as a predictor of risk of attrition in surgical residency. American journal of surgery Salles, A., Lin, D., Liebert, C., Esquivel, M., Lau, J. N., Greco, R. S., Mueller, C. 2017; 213 (2): 288-291

  Abstract

  Grit, a measure of perseverance, has been shown to predict resident well-being. In this study we assess the relationship between grit and attrition.We collected survey data from residents in a single institution over two consecutive years. All residents in general surgery were invited to participate (N = 115). Grit and psychological well-being were assessed using validated measures. Risk of attrition was measured using survey items.73 residents participated (63% response rate). Grit was positively correlated with general psychological well-being (r = 0.30, p < 0.05) and inversely correlated with depression (r = -0.25, p < 0.05) and risk of attrition (r = -0.37, p < 0.01). In regression analyses, grit was positively predictive of well-being (B = 0.77, t = 2.96, p < 0.01) and negatively predictive of depression (B = -0.28 t = -2.74, p < 0.01) and attrition (B = -0.99, t = -2.53, p < 0.05).Attrition is a costly and disruptive problem in residency. Grit is a quick, reliable measure which appears to be predictive of attrition risk in this single-institution study.

  View details for DOI 10.1016/j.amjsurg.2016.10.012

  View details for PubMedID 27932088

• A Mixed-Methods Analysis of a Novel Mistreatment Program for the Surgery Core Clerkship. Academic medicine Lau, J. N., Mazer, L. M., Liebert, C. A., Merrell, S. B., Lin, D. T., Harris, I. 2017

  Abstract

  To review mistreatment reports from before and after implementation of a mistreatment program, and student ratings of and qualitative responses to the program to evaluate the short-term impact on students.In January 2014, a video- and discussion-based mistreatment program was implemented for the surgery clerkship at the Stanford University School of Medicine. The program aims to help students establish expectations for the learning environment; create a shared and personal definition of mistreatment; and promote advocacy and empowerment to address mistreatment. Counts and types of mistreatment were compared from a year before (January-December 2013) and two years after (January 2014-December 2015) implementation. Students' end-of-clerkship ratings and responses to open-ended questions were analyzed.From March 2014-December 2015, 141/164 (86%) students completed ratings, and all 47 (100%) students enrolled from January-August 2014 provided qualitative program evaluations. Most students rated the initial (108/141 [77%]) and final (120/141 [85%]) sessions as excellent or outstanding. In the qualitative analysis, students valued that the program helped establish expectations; allowed for sharing experiences; provided formal resources; and provided a supportive environment. Students felt the learning environment and culture were improved and reported increased interest in surgery. There were 14 mistreatment reports the year before the program, 9 in the program's first year, and 4 in the second year.The authors found a rotation-specific mistreatment program, focused on creating shared understanding about mistreatment, was well received among surgery clerkship students, and the number of mistreatment reports decreased each year following implementation.

  View details for DOI 10.1097/ACM.0000000000001575

  View details for PubMedID 28121657

• A Narrative Review and Novel Framework for Application of Team-Based Learning in Graduate Medical Education. Journal of graduate medical education Poeppelman, R. S., Liebert, C. A., Vegas, D. B., Germann, C. A., Volerman, A. 2016; 8 (4): 510-517

  Abstract

  Team-based learning (TBL) promotes problem solving and teamwork, and has been applied as an instructional method in undergraduate medical education with purported benefits. Although TBL curricula have been implemented for residents, no published systematic reviews or guidelines exist for the development and use of TBL in graduate medical education (GME).To review TBL curricula in GME, identify gaps in the literature, and synthesize a framework to guide the development of TBL curricula at the GME level.We searched PubMed, MEDLINE, and ERIC databases from 1990 to 2014 for relevant articles. References were reviewed to identify additional studies. The inclusion criteria were peer-reviewed publications in English that described TBL curriculum implementation in GME. Data were systematically abstracted and reviewed for consensus. Based on included publications, a 4-element framework-system, residents, significance, and scaffolding-was developed to serve as a step-wise guide to planning a TBL curriculum in GME.Nine publications describing 7 unique TBL curricula in residency met inclusion criteria. Outcomes included feasibility, satisfaction, clinical behavior, teamwork, and knowledge application.TBL appears feasible in the GME environment, with learner reactions ranging from positive to neutral. Gaps in the literature occur within each of the 4 elements of the suggested framework, including: system, faculty preparation time and minimum length of effective TBL sessions; residents, impact of team heterogeneity and inconsistent attendance; significance, comparison to other instructional methods and outcomes measuring knowledge retention, knowledge application, and skill development; and scaffolding, factors that influence the completion of preparatory work.

  View details for PubMedID 27777660

  View details for PubMedCentralID PMC5058582

• Belonging, Well-being, and Attrition in General Surgery Salles, A., Lin, D. T., Liebert, C. A., Esquivel, M., Mueller, C. ELSEVIER SCIENCE INC. 2016: E40–E41

  View details for DOI 10.1016/j.jamcollsurg.2016.08.106

  View details for Web of Science ID 000395825100087

• Student perceptions of a simulation-based flipped classroom for the surgery clerkship: A mixed-methods study. Surgery Liebert, C. A., Mazer, L., Bereknyei Merrell, S., Lin, D. T., Lau, J. N. 2016; 160 (3): 591-598

  Abstract

  The flipped classroom, a blended learning paradigm that uses pre-session online videos reinforced with interactive sessions, has been proposed as an alternative to traditional lectures. This article investigates medical students' perceptions of a simulation-based, flipped classroom for the surgery clerkship and suggests best practices for implementation in this setting.A prospective cohort of students (n = 89), who were enrolled in the surgery clerkship during a 1-year period, was taught via a simulation-based, flipped classroom approach. Students completed an anonymous, end-of-clerkship survey regarding their perceptions of the curriculum. Quantitative analysis of Likert responses and qualitative analysis of narrative responses were performed.Students' perceptions of the curriculum were positive, with 90% rating it excellent or outstanding. The majority reported the curriculum should be continued (95%) and applied to other clerkships (84%). The component received most favorably by the students was the simulation-based skill sessions. Students rated the effectiveness of the Khan Academy-style videos the highest compared with other video formats (P < .001). Qualitative analysis identified 21 subthemes in 4 domains: general positive feedback, educational content, learning environment, and specific benefits to medical students. The students reported that the learning environment fostered accountability and self-directed learning. Specific perceived benefits included preparation for the clinical rotation and the National Board of Medical Examiners shelf exam, decreased class time, socialization with peers, and faculty interaction.Medical students' perceptions of a simulation-based, flipped classroom in the surgery clerkship were overwhelmingly positive. The flipped classroom approach can be applied successfully in a surgery clerkship setting and may offer additional benefits compared with traditional lecture-based curricula.

  View details for DOI 10.1016/j.surg.2016.03.034

  View details for PubMedID 27262534

• Novel Use of Google Glass for Procedural Wireless Vital Sign Monitoring. Surgical innovation Liebert, C. A., Zayed, M. A., Aalami, O., Tran, J., Lau, J. N. 2016; 23 (4): 366-373

  Abstract

  This study investigates the feasibility and potential utility of head-mounted displays for real-time wireless vital sign monitoring during surgical procedures.In this randomized controlled pilot study, surgery residents (n = 14) performed simulated bedside procedures with traditional vital sign monitors and were randomized to addition of vital sign streaming to Google Glass. Time to recognition of preprogrammed vital sign deterioration and frequency of traditional monitor use was recorded. User feedback was collected by electronic survey.The experimental group spent 90% less time looking away from the procedural field to view traditional monitors during bronchoscopy (P = .003), and recognized critical desaturation 8.8 seconds earlier; the experimental group spent 71% (P = .01) less time looking away from the procedural field during thoracostomy, and recognized hypotension 10.5 seconds earlier. Trends toward earlier recognition of deterioration did not reach statistical significance. The majority of participants agreed that Google Glass increases situational awareness (64%), is helpful in monitoring vitals (86%), is easy to use (93%), and has potential to improve patient safety (85%).In this early feasibility study, use of streaming to Google Glass significantly decreased time looking away from procedural fields and resulted in a nonsignificant trend toward earlier recognition of vital sign deterioration. Vital sign streaming with Google Glass or similar platforms is feasible and may enhance procedural situational awareness.

  View details for DOI 10.1177/1553350616630142

  View details for PubMedID 26848138

• Emotional Intelligence as a Predictor of Resident Well-Being. Journal of the American College of Surgeons Lin, D. T., Liebert, C. A., Tran, J., Lau, J. N., Salles, A. 2016; 223 (2): 352-358

  Abstract

  There is increasing recognition that physician wellness is critical; it not only benefits the provider, but also influences quality and patient care outcomes. Despite this, resident physicians suffer from a high rate of burnout and personal distress. Individuals with higher emotional intelligence (EI) are thought to perceive, process, and regulate emotions more effectively, which can lead to enhanced well-being and less emotional disturbance. This study sought to understand the relationship between EI and wellness among surgical residents.Residents in a single general surgery residency program were surveyed on a voluntary basis. Emotional intelligence was measured using the Trait Emotional Intelligence Questionnaire-Short Form. Resident wellness was assessed with the Dupuy Psychological General Well-Being Index, Maslach Burnout Inventory, and Beck Depression Inventory-Short Form. Emotional intelligence and wellness parameters were correlated using Pearson coefficients. Multivariate analysis was performed to identify factors predictive of well-being.Seventy-three residents participated in the survey (response rate 63%). Emotional intelligence scores correlated positively with psychological well-being (r = 0.74; p < 0.001) and inversely with depression (r = -0.69, p < 0.001) and 2 burnout parameters, emotional exhaustion (r = -0.69; p < 0.001) and depersonalization (r = -0.59; p < 0.001). In regression analyses controlling for demographic factors such as sex, age, and relationship status, EI was strongly predictive of well-being (β = 0.76; p < 0.001), emotional exhaustion (β = -0.63; p < 0.001), depersonalization (β = -0.48; p = 0.002), and depression (β = -0.60; p < 0.001).Emotional intelligence is a strong predictor of resident well-being. Prospectively measuring EI can identify those who are most likely to thrive in surgical residency. Interventions to increase EI can be effective at optimizing the wellness of residents.

  View details for DOI 10.1016/j.jamcollsurg.2016.04.044

  View details for PubMedID 27182037

• Effectiveness of the Surgery Core Clerkship Flipped Classroom: a prospective cohort trial AMERICAN JOURNAL OF SURGERY Liebert, C. A., Lin, D. T., Mazer, L. M., Bereknyei, S., Lau, J. N. 2016; 211 (2): 451-U214

  Abstract

  The flipped classroom has been proposed as an alternative curricular approach to traditional didactic lectures but has not been previously applied to a surgery clerkship.A 1-year prospective cohort of students (n = 89) enrolled in the surgery clerkship was taught using a flipped classroom approach. A historical cohort of students (n = 92) taught with a traditional lecture curriculum was used for comparison. Pretest and post-test performance, end-of-clerkship surveys, and National Board of Medical Examiners (NBME) scores were analyzed to assess effectiveness.Mean pretest and post-test scores increased across all modules (P < .001). There was no difference between mean NBME examination score in the prospective and historical cohorts (74.75 vs 75.74, P = .28). Mean ratings of career interest in surgery increased after curriculum completion (4.75 to 6.50, P < .001), with 90% reporting that the flipped classroom contributed to this increase.Implementation of a flipped classroom in the surgery clerkship is feasible and results in high learner satisfaction, effective knowledge acquisition, and increased career interest in surgery with noninferior NBME performance.

  View details for DOI 10.1016/j.amjsurg.2015.10.004

  View details for Web of Science ID 000368344800023

• Belonging as a Measure of Risk of Attrition Salles, A., Liebert, C. A., Lin, D. T. ELSEVIER SCIENCE INC. 2015: S49

  View details for DOI 10.1016/j.jamcollsurg.2015.07.103

  View details for Web of Science ID 000361119700084

• Emotional Intelligence as a Predictor of Resident Wellness Lin, D. T., Liebert, C. A., Lau, J. N., Salles, A. ELSEVIER SCIENCE INC. 2015: S52

  View details for DOI 10.1016/j.jamcollsurg.2015.07.110

  View details for Web of Science ID 000361119700091

• Promoting Balance in the Lives of Resident Physicians: A Call to Action. JAMA surgery Salles, A., Liebert, C. A., Greco, R. S. 2015; 150 (7): 607-608

  View details for DOI 10.1001/jamasurg.2015.0257

  View details for PubMedID 25992632

• Validity evidence for Surgical Improvement of Clinical Knowledge Ops: a novel gaming platform to assess surgical decision making AMERICAN JOURNAL OF SURGERY Lin, D. T., Park, J., Liebert, C. A., Lau, J. N. 2015; 209 (1): 79-85

  Abstract

  Current surgical education curricula focus mainly on the acquisition of technical skill rather than clinical and operative judgment. SICKO (Surgical Improvement of Clinical Knowledge Ops) is a novel gaming platform developed to address this critical need. A pilot study was performed to collect validity evidence for SICKO as an assessment for surgical decision making.Forty-nine subjects stratified into 4 levels of expertise were recruited to play SICKO. Later, players were surveyed regarding the realism of the gaming platform as well as the clinical competencies required of them while playing SICKO.Each group of increasing expertise outperformed the less experienced groups. Mean total game scores for the novice, junior resident, senior resident, and expert groups were 5,461, 8,519, 11,404, and 13,913, respectively (P = .001). Survey results revealed high scores for realism and content.SICKO holds the potential to be not only an engaging and immersive educational tool, but also a valid assessment in the armamentarium of surgical educators.

  View details for DOI 10.1016/j.amjsurg.2014.08.033

  View details for PubMedID 25454955