Melissa Chaehyun Lee

All Publications

• Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis. Journal of surgical education Lee, M. C., Melcer, E. F., Merrell, S. B., Wong, L. Y., Shields, S., Eddington, H., Trickey, A. W., Tsai, J., Korndorffer, J. R., Lin, D. T., Liebert, C. A. 2023

  Abstract

  As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings.This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework.This study was conducted at an academic institution in a proctored exam setting.The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment.The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST.ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

  View details for DOI 10.1016/j.jsurg.2023.09.001

  View details for PubMedID 37821350

• Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa. Journal of the American College of Surgeons Liebert, C. A., Melcer, E. F., Eddington, H., Trickey, A., Shields, S., Lee, M., Korndorffer, J. R., Bekele, A., Wren, S. M., Lin, D. T. 2023

  Abstract

  To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence.In partnership with COSECSA, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n=110) completed the traditional 11-station oral OSCE, followed by three ENTRUST cases, authored to query similar clinical content of three corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Exam outcome using independent sample t-tests. Correlation of ENTRUST scores to MCS Exam Percentage and OSCE Station Scores were calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance.ENTRUST performance was significantly higher in examinees who passed the MCS Exam compared to those who failed (p<0.001). ENTRUST score was positively correlated with MCS Exam Percentage (p<0.001) and combined OSCE Station Scores (p<0.001). On multivariate analysis, there was a strong association between MCS Exam Percentage and ENTRUST Grand Total Score (p<0.001), Simulation Total Score (p=0.018), and Question Total Score (p<0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST.This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.

  View details for DOI 10.1097/XCS.0000000000000740

  View details for PubMedID 37144790

• Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA). Journal of surgical education Liebert, C. A., Melcer, E. F., Keehl, O., Eddington, H., Trickey, A. W., Lee, M., Tsai, J., Camacho, F., Merrell, S. B., Korndorffer, J. R., Lin, D. T. 2022

  Abstract

  OBJECTIVE: As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence.DESIGN: This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses.SETTING: This study was conducted at an academic institution in a proctored exam setting.PARTICIPANTS: Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment.RESULTS: Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56).CONCLUSIONS: Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

  View details for DOI 10.1016/j.jsurg.2022.07.008

  View details for PubMedID 35909070

• How much is enough? Exploring the dose-response relationship between cash transfers and surgical utilization in a resource-poor setting. PloS one Strader, C., Ashby, J., Vervoort, D., Ebrahimi, A., Agbortoko, S., Lee, M., Reiner, N., Zeme, M., Shrime, M. G. 2020; 15 (5): e0232761

  Abstract

  Cash transfers are a common intervention to incentivize salutary behavior in resource-constrained settings. Many cash transfer studies do not, however, account for the effect of the size of the cash transfer in design or analysis. A randomized, controlled trial of a cash-transfer intervention is planned to incentivize appropriate surgical utilization in Guinea. The aim of the current study is to determine the size of that cash transfer so as to maximize compliance while minimizing cost.Data were collected from nine coastal Guinean hospitals on their surgical capabilities and the cost of receiving surgery. These data were combined with publicly available data about the general Guinean population to create an agent-based model predicting surgical utilization. The model was validated to the available literature on surgical utilization. Cash transfer sizes from 0 to 1,000,000 Guinean francs were evaluated, with surgical compliance as the primary outcome.Compliance with scheduled surgery increases as the size of a cash transfer increases. This increase is asymptotic, with a leveling in utilization occurring when the cash transfer pays for all the costs associated with surgical care. Below that cash transfer size, no other optima are found. Once a cash transfer completely covers the costs of surgery, other barriers to care such as distance and hospital quality dominate.Cash transfers to incentivize health-promoting behavior appear to be dose-dependent. Maximal impact is likely only to occur when full patient costs are eliminated. These findings should be incorporated in the design of future cash transfer studies.

  View details for DOI 10.1371/journal.pone.0232761

  View details for PubMedID 32407327

  View details for PubMedCentralID PMC7224483