Carla Pugh is Professor of Surgery at Stanford University School of Medicine. She is also the Director of the Technology Enabled Clinical Improvement (T.E.C.I.) Center. Her clinical area of expertise is Acute Care Surgery. Dr. Pugh obtained her undergraduate degree at U.C. Berkeley in Neurobiology and her medical degree at Howard University School of Medicine. Upon completion of her surgical training at Howard University Hospital, she went to Stanford University and obtained a PhD in Education. She is the first surgeon in the United States to obtain a PhD in Education. Her goal is to use technology to change the face of medical and surgical education.
Her research involves the use of simulation and advanced engineering technologies to develop new approaches for assessing and defining competency in clinical procedural skills. Dr. Pugh holds three patents on the use of sensor and data acquisition technology to measure and characterize hands-on clinical skills. Currently, over two hundred medical and nursing schools are using one of her sensor enabled training tools for their students and trainees. Her work has received numerous awards from medical and engineering organizations. In 2011 Dr. Pugh received the Presidential Early Career Award for Scientists and Engineers from President Barak Obama at the White House. She is considered to be a lead, international expert on the use of sensors and motion tracking technology for performance measurement. In 2014 she was invited to give a TEDMED talk on the potential uses of technology to transform how we measure clinical skills in medicine. In April 2018, Dr. Pugh was inducted into the American Institute for Medical and Biological Engineering.
- General Surgery
Fellowship:University of Michigan Medical School (2009) MI
PhD, Stanford University Graduate School of Education, Education & Technology (2001)
Board Certification: General Surgery, American Board of Surgery (1999)
Residency:Howard University Hospital General Surgery Residency (1997) DC
Medical Education:Howard University College of Medicine (1992) DC
Current Research and Scholarly Interests
The Technology Enabled Clinical Improvement (T.E.C.I.) Center is a multidisciplinary team of researchers dedicated to the design and implementation of advanced engineering technologies that facilitate data acquisition relating to clinical performance.
The T.E.C.I. team has had great success in quantifying physicians’ clinical experiences using sensor, video, and motion tracking technologies. This work has resulted in an information rich database that enables empirical evaluation of clinical excellence and medical decision making.
By leveraging highly specific and objective clinical performance metrics, the T.E.C.I. Center is harnessing the unique opportunity to support peer to peer data sharing and clinical collaborations that can transform the clinical workflow and ultimately benefit healthcare providers.
The T.E.C.I. Center aims to transform human health and welfare through advances in data science and personalized, technology-based performance metrics for healthcare providers.
Shortcut assessment: Can residents' operative performance be determined in the first five minutes of an operative task?
2018; 163 (6): 1207–12
BACKGROUND: The aim was to validate the potential use of a single, early procedure, operative task as a predictive metric for overall performance. The authors hypothesized that a shortcut psychomotor assessment would be as informative as a total procedural psychomotor assessment when evaluating laparoscopic ventral hernia repair performance on a simulator.METHODS: Using electromagnetic sensors, hand motion data were collected from 38 surgery residents during a simulated laparoscopic ventral hernia repair procedure. Three time-based phases of the procedure were defined: Early Phase (start time through completion of first anchoring suture), Mid Phase (start time through completion of second anchoring suture), and Total Operative Time. Correlations were calculated comparing time and motion metrics for each phase with the final laparoscopic ventral hernia repair score.RESULTS: Analyses revealed that execution time and motion, for the first anchoring suture, predicted procedural outcomes. Greater execution times and path lengths correlated to lesser laparoscopic ventral hernia repair scores (r = -0.56, P = .0008 and r = -0.51, P = .0025, respectively). Greater bimanual dexterity measures correlated to Greater LVH repair scores (r = + 0.47, P = .0058).CONCLUSIONS: This study provides validity evidence for use of a single, early operative task as a shortcut assessment to predict resident performance during a simulated laparoscopic ventral hernia repair procedure. With the continued development and decreasing costs of motion technology, faculty should be well-versed in the use of motion metrics for performance measurements. The results strongly support the use of dexterity and economy of motion (path length + execution time) metrics as early predictors of operative performance.
View details for PubMedID 29728259
Use of sensors to quantify procedural idle time: Validity evidence for a new mastery metric.
BACKGROUND: Quantification of mastery is the first step in using objective metrics for teaching. We hypothesized that during orotracheal intubation, top tier performers have less idle time compared to lower tier performers.METHODS: At the Anesthesiology 2018 Annual Meeting, 82 participants intubated a normal airway simulator and a burnt airway simulator. The movements of the participant's laryngoscope were quantified using electromagnetic motion sensors. Top tier performers were defined as participants who intubated both simulators successfully in less than the median time for each simulator. Idle time was defined as the duration of time when the laryngoscope was not moving.RESULTS: Top performers showed less Idle Time when intubating the normal airway compared to lower tier performers (14.5 ± 9.8 seconds vs 34.0 ± 52.0 seconds, respectively P < .01). Likewise, top performers showed less Idle Time when intubating the burnt airway compared to lower tier performers (18.6 ± 15.2 seconds vs 63.4 ± 59.11 seconds; P < .01). Comparing performance on the burnt airway to the normal airway, there was a difference for lower tier performers (63.4 ± 59.1 seconds vs 34.0 ± 52.0 seconds; P < .01) but not for top tier performers (18.6 ± 15.2 seconds vs 14.5 ± 9.8 seconds; P= .07).CONCLUSION: Similar to our previous findings with other procedures, Idle Time was shown to have known group validity evidence when comparing top performers with lower tier performers. Further, Idle Time was correlated with procedure difficulty in our prior work. We observed statistically significant differences in Idle Times for lower tier performers when comparing the normal airway to the burnt airway but not for top tier performers. Our findings support the continued exploration of Idle Time for development of objective assessment and curricula.
View details for DOI 10.1016/j.surg.2019.09.016
View details for PubMedID 31708084
- In Search of Characterizing Surgical Skill JOURNAL OF SURGICAL EDUCATION 2019; 76 (5): 1348–63
Screening surgical residents' laparoscopic skills using virtual realitytasks: Who needs more time in the sim lab?
BACKGROUND: This study investigated the possibility of using virtual reality perceptual-motor tasks as a screening tool for laparoscopic ability. We hypothesized that perceptual-motor skills assessed using virtual reality will correlate with the quality of simulated laparoscopic ventral hernia repair.MATERIALS AND METHODS: Surgical residents (N= 37), performed 2 virtual reality perceptual-motor tasks: (1) force matching and (2) target tracking. Participants also performed a laparoscopic ventral hernia repair on a simulator and final product quality score, and endoscopic visualization errors were calculated. Correlational analysis was performed to assess the relationship between performance on virtual reality tasks and laparoscopic ventral hernia repair.RESULTS: Residents with poor performance on force matching in virtual reality-"peak deflection" (r= -0.34, P < .05) and "summation distance" (r= -0.36, P < .05)-had lower final product quality scores. Likewise, poor performance in virtual reality-based target tracking-"path length" (r= -0.49, P < .05) and "maximum distance" (r= -0.37, P < .05)-correlated with a lower final product quality score.CONCLUSION: Our findings support the notion that virtual reality could be used as a screening tool for perceptual-motor skill. Trainees identified as having poor perceptual-motor skill can benefit from focused curricula, allowing them to hone personal areas of weakness and maximize technical skill.
View details for DOI 10.1016/j.surg.2019.04.013
View details for PubMedID 31229312
- Teaching practicing surgeons what not to do: An analysis of instruction fluidity during a simulation-based continuing medical education course MOSBY-ELSEVIER. 2019: 1082–87
- Advanced Volumetric 3-Dimensional Visualization of Surgical Anatomy-Are We There Yet? JAMA surgery 2019
In Search of Characterizing Surgical Skill.
Journal of surgical education
OBJECTIVE: This paper provides a literature review and detailed discussion of surgical skill terminology. Culminating in a novel model that proposes a set of unique definitions, this review is designed to facilitate shared understanding to study and develop metrics quantifying surgical skill.DESIGN: Objective surgical skill analysis depends on consistent definitions and shared understanding of terms like performance, expertise, experience, aptitude, ability, competency, and proficiency.STRUCTURE: Each term is discussed in turn, drawing from existing literature and colloquial uses.IMPLICATIONS: A new model of definitions is proposed to cement a common and consistent lexicon for future skills analysis, and to quantitatively describe a surgeon's performance throughout their career.
View details for PubMedID 30890315
Teaching practicing surgeons what not to do: An analysis of instruction fluidity during a simulation-based continuing medical education course.
BACKGROUND: Interest is growing in simulation-based continuing medical education courses for practicing surgeons. However, little research has explored the instruction employed during these courses. This study examines instruction practices used during an annual simulation-based continuing medical education course.METHODS: Audio-video data were collected from surgeon instructors (n= 12) who taught a simulated laparoscopic hernia repair continuing medical education course across 2 years. Surgeon learners (n= 58) were grouped by their self-reported laparoscopic and hernia repair experience. Instructors' transcribed dialogue was automatically coded for 5 types of responses to the following questions: anecdotes, confirming, correcting, guidance, and what not to do. Differences in these responses were measured against the progress of the simulations and across learners with different experience levels. Postcourse interviews with instructors were conducted for additional qualitative validation.RESULTS: Performing t tests of instructor responses revealed that they were significantly more likely to answer in forms coded as anecdotes when responding to relative experts and in forms coded as what not to do when responding to novices. Linear regressions of each code against normalized progressions of each simulation revealed a significant relationship between progression through a simulation and frequency of the what not to do code for less-experienced learners. Postcourse interviews revealed that instructors continuously assess participants throughout a session and modify their teaching strategies.CONCLUSION: Instructors significantly modified the focus of their teaching as a function both of their learners' self-reported experience levels, their assessment of learner needs, and learner progression through the training sessions.
View details for PubMedID 30876670
- Electronic health records, physician workflows and system change: defining a pathway to better healthcare ANNALS OF TRANSLATIONAL MEDICINE 2019; 7
Electronic health records, physician workflows and system change: defining a pathway to better healthcare.
Annals of translational medicine
2019; 7 (Suppl 1): S27
View details for PubMedID 31032307
Combining metrics from clinical simulators and sensorimotor tasks can reveal the training background of surgeons.
IEEE transactions on bio-medical engineering
Skill assessment in surgery traditionally has relied on expert observation and qualitative scoring. Our novel study design demonstrates how analysis of performance in sensorimotor tasks and bench-top surgical simulators can provide inferences about the technical proficiency as well as the training history of surgeons.Our unique study design examined metrics for basic sensorimotor tasks in a virtual reality interface as well as motion metrics in clinical scenario simulations. As indicators of training level, we considered survey responses from surgery residents, including the number of years post-graduation (PGY, four levels), research years (RY, three levels), and clinical years (CY, three levels). Next, we performed a linear discriminant analysis with cross-validation (90% training, 10% testing) to relate the training levels to the selected metrics.Using combined metrics from all stations, we found greater than chance predictions for each survey category, with an overall accuracy of 43.4±2.9% for identifying the level for post-graduate years, 79.1±1.0% accuracy for research training years, and 64.2±1.0% for clinical training years. Our main finding was that combining metrics from all stations resulted in more accurate predictions than using only sensorimotor or clinical scenario tasks. In addition, our analysis indicates that metrics related to the ability to cope with changes in the task environment were the most important predictors of training level.These results suggest that each simulator type provided crucial information for evaluating surgical proficiency. The methods developed in this study could improve evaluations of a surgeon's clinical proficiency as well as training potential in terms of basic sensorimotor ability.
View details for DOI 10.1109/TBME.2019.2892342
View details for PubMedID 30629489
Dynamic Visual Feedback During Junctional Tourniquet Training.
The Journal of surgical research
2019; 233: 444–52
BACKGROUND: This project involved the development and evaluation of a new visual bleeding feedback (VBF) system for tourniquet training. We hypothesized that dynamic VBF during junctional tourniquet training would be helpful and well received by trainees.MATERIALS AND METHODS: We designed the VBF to simulate femoral bleeding. Medical students (n=15) and emergency medical service (EMS) members (n=4) were randomized in a single-blind, crossover study to the VBF or without feedback groups. Poststudy surveys assessing VBF usefulness and recommendations were conducted along with participants' reported confidence using a 7-point Likert scale. Data from the different groups were compared using Wilcoxon signed-rank and rank-sum tests.RESULTS: Participants rated the helpfulness of the VBF highly (6.53/7.00) and indicated they were very likely to recommend the VBF simulator to others (6.80/7.00). Pre- and post-VBF confidence were not statistically different (P=0.59). Likewise, tourniquet application times for VBF and without feedback before crossover were not statistically different (P=0.63). Although participant confidence did not change significantly from beginning to end of the study (P=0.46), application time was significantly reduced (P=0.001).CONCLUSIONS: New tourniquet learners liked our VBF prototype and found it useful. Although confidence did not change over the course of the study for any group, application times improved. Future studies using outcomes of this study will allow us to continue VBF development as well as incorporate other quantitative measures of task performance to elucidate VBF's true benefit and help trainees achieve mastery in junctional tourniquet skills.
View details for PubMedID 30502284
Can VR Be Used to Track Skills Decay During the Research Years?
The Journal of surgical research
Time away from surgical practice can lead to skills decay. Research residents are thought to be prone to skills decay, given their limited experience and reduced exposure to clinical activities during their research training years. This study takes a cross-sectional approach to assess differences in residents' skills at the beginning and end of their research years using virtual reality. We hypothesized that research residents will have measurable decay in psychomotor skills when evaluated using virtual reality.Surgical residents (n = 28) were divided into two groups; the first group was just beginning their research time (clinical residents: n = 19) and the second group (research residents: n = 9) had just finished at least 2 y of research. All participants were asked to perform a target-tracking task using a haptic device, and their performance was compared using Welch's t-test.Research residents showed a higher level of "tracking error" (1.69 ± 0.44 cm versus 1.40 ± 0.19 cm; P = 0.04) and a similar level of "path length" (62.5 ± 10.5 cm versus 62.1 ± 5.2 cm; P = 0.92) when compared with clinical residents.The increased "tracking error" among residents at the end of their research time suggests fine psychomotor skills decay in residents who spend time away from clinical duties during laboratory time. This decay demonstrates the need for research residents to regularly participate in clinical activities, simulation, or assessments to minimize and monitor skills decay while away from clinical practice. Additional longitudinal studies may help better map learning and decay curves for residents who spend time away from clinical practice.
View details for DOI 10.1016/j.jss.2019.10.030
View details for PubMedID 31776024
What do you want to know? Operative experience predicts the type of questions practicing surgeons ask during a CME laparoscopic hernia repair course.
American journal of surgery
BACKGROUND: Given their variegated backgrounds, surgeons taking continuing medical education (CME) courses possess different learning needs. This study examines the relationship between surgeons' levels of experience and the questions they asked in a simulation-based CME course.METHODS: We analyzed transcribed audio-video data collected from surgeons participating in a simulated laparoscopic hernia repair CME course and identified four types of questions learners posed to their instructors. Linear regressions compared how often these questions were asked versus self-reported operative experience.RESULTS: Both Requesting Guidance and Requesting Confirmation were inversely proportional to experience, whereas Asking About a Specific Case was directly proportional to experience. Requesting Instructor Preference exhibited no significant correlation with experience.CONCLUSION: Practicing surgeons with relatively less experience tend to ask for confirmation and guidance, whereas those with greater experience tend to focus on specific hypothetical scenarios. This data can be used to tailor instruction based on learners' self-reported experience level.
View details for PubMedID 30527925
Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy.
American journal of surgery
INTRODUCTION: The objective of this study was to determine whether decision-based procedural mapping demonstrates differences in attendings versus residents.METHODS: Attendings and residents were interviewed about operative decision-making in laparoscopic cholecystectomy (LC) using a cognitive task analysis framework. Interviews were converted into procedural maps. Operative steps, patient factors, and surgeon factors noted by attendings and residents were compared. Two scoring methods were used to compare map structures of attendings versus residents.RESULTS: Six attendings and six residents were interviewed. There were no significant differences in the number of patient or surgeon factors identified. Attendings had significantly more operative steps (29.67 ± 1.9 vs. 23.3 ± 1.9, p = 0.04) and crosslinks (3.2 ± 0.5 vs. 1 ± 0.4, p = 0.005) in their maps and a higher total score (90.2 ± 8.4 vs. 63.2 ± 3.8, p = 0.015) than residents.CONCLUSION: LC procedural map scoring for attendings and residents demonstrated significant differences in structural complexity and may provide a useful framework for assessing decision making.
View details for PubMedID 30470551
Faculty perceptions of resident skills decay during dedicated research fellowships
AMERICAN JOURNAL OF SURGERY
2018; 215 (2): 336–40
Residents engaging in dedicated research experiences may return to clinical training with less surgical skill. The study aims were 1) to evaluate faculty perceptions of residents skills decay during dedicated research fellowships, and 2) to compare faculty and resident perceptions of residents skills decay.Faculty and residents were surveyed on resident research practices and perceptions of resident skills decay.Faculty thought residents returning from research demonstrate less technical skill (Median = 4; 5-point Likert scale, 1 = Strongly disagree, 5 = Strongly agree), demonstrate less confidence (Median = 4), and require more instruction (Median = 4). Both faculty and residents perceived the largest skill reduction in complex procedures, technical surgical skills, and knowledge of procedure steps (p < 0.05).While dedicated research experiences provide valuable academic experience, there is a cost to clinical skills retention and confidence specifically in the areas of complex operative procedures and technical surgical skills.
View details for DOI 10.1016/j.amjsurg.2017.11.018
View details for Web of Science ID 000425193700025
View details for PubMedID 29169821
A structured, extended training program to facilitate adoption of new techniques for practicing surgeons
SPRINGER. 2018: 217–24
Laparoscopic inguinal hernia repair has been shown to have significant benefits when compared to open inguinal hernia repair, yet remains underutilized in the United States. The traditional model of short, hands-on, cognitive courses to enhance the adoption of new techniques fails to lead to significant levels of practice implementation for most surgeons. We hypothesized that a comprehensive program would facilitate the adoption of laparoscopic inguinal hernia repair (TEP) for practicing surgeons.A team of experts in simulation, coaching, and hernia care created a comprehensive training program to facilitate the adoption of TEP. Three surgeons who routinely performed open inguinal hernia repair with greater than 50 cases annually were recruited to participate in the program. Coaches were selected based on their procedural expertise and underwent formal training in surgical coaching. Participants were required to evaluate all aspects of the educational program and were surveyed out to one year following completion of the program to assess for sustained adoption of TEP.All three participants successfully completed the first three steps of the seven-step program. Two participants completed the full course, while the third dropped out of the program due to time constraints and low case volume. Participant surgeons rated Orientation (4.7/5), GlovesOn training (5/5), and Preceptored Cases (5/5) as highly important training activities that contributed to advancing their knowledge and technical performance of the TEP procedure. At one year, both participants were performing TEPs for "most of their cases" and were confident in their ability to perform the procedure. The total cost of the program including all travel, personal coaching, and simulation was $8638.60 per participant.Our comprehensive educational program led to full and sustained adoption of TEP for those who completed the course. Time constraints, travel costs, and case volume are major considerations for successful completion; however, the program is feasible, acceptable, and affordable.
View details for DOI 10.1007/s00464-017-5662-2
View details for Web of Science ID 000422854700025
View details for PubMedID 28643054
- Residents' response to bleeding during a simulated robotic surgery task JOURNAL OF SURGICAL RESEARCH 2017; 220: 385–90
- A Holistic Model of Surgical Expertise and Competency ANNALS OF SURGERY 2017; 265 (2): 268–69
- Sensor technology in assessments of clinical skill. The New England journal of medicine 2015; 372 (8): 784–86
Characterizing Touch Using Pressure Data and Auto Regressive Models
IEEE. 2014: 1839–42
Palpation plays a critical role in medical physical exams. Despite the wide range of exams, there are several reproducible and subconscious sets of maneuvers that are common to examination by palpation. Previous studies by our group demonstrated the use of manikins and pressure sensors for measuring and quantifying how physicians palpate during different physical exams. In this study we develop mathematical models that describe some of these common maneuvers. Dynamic pressure data was measured using a simplified testbed and different autoregressive models were used to describe the motion of interest. The frequency, direction and type of motion used were identified from the models. We believe these models can a provide better understanding of how humans explore objects in general and more specifically give insights to understand medical physical exams.
View details for Web of Science ID 000350044701204
View details for PubMedID 25570335
View details for PubMedCentralID PMC4288476
Intra-operative decision making: More than meets the eye
JOURNAL OF BIOMEDICAL INFORMATICS
2011; 44 (3): 486–96
Operating room teams consist of team members with diverse training backgrounds. In addition to differences in training, each team member has unique and complex decision making paths. As such, team members may function in the same environment largely unaware of their team members' perspectives. The goal of our work was to use a theory-based approach to better understand the complexity of knowledge-based intra-operative decision making. Cognitive task analysis methods were used to extract the knowledge, thought processes, goal structures and critical decisions that provide the foundation for surgical task performance. A triangulated and iterative approach is presented.
View details for DOI 10.1016/j.jbi.2010.01.001
View details for Web of Science ID 000291768200014
View details for PubMedID 20096376
Development and validation of assessment measures for a newly developed physical examination simulator
JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION
2002; 9 (5): 448-460
Define, extract and evaluate potential performance indicators from computer-generated data collected during simulated clinical female pelvic examinations.Qualitative and quantitative study analyzing computer generated simulator data and written clinical assessments collected from medical students who performed physical examinations on three clinically different pelvic simulators.Introduction to patient care course at a major United States medical school.Seventy-three pre-clinical medical students performed 219 simulated pelvic examinations and generated 219 written clinical assessments.Cronbach's alpha for the newly defined performance indicators, Pearson's correlation of performance indicators with scored written clinical assessments of simulator findings.Four novel performance indicators were defined: time to perform a complete examination, number of critical areas touched during the exam, the maximum pressure used, and the frequency at which these areas were touched. The reliability coefficients (alpha) were time = 0.7240, critical areas = 0.6329, maximum pressure = 0.7701, and frequency = 0.5011. Of the four indicators, three correlated positively and significantly with the written clinical assessment scores: critical areas, p < 0.01; frequency, p < 0.05; and maximum pressure, p < 0.05.This study demonstrates a novel method of analyzing raw numerical data generated from a newly developed patient simulator; deriving performance indicators from computer generated simulator data; and assessing validity of those indicators by comparing them with written assessment scores. Results show the new assessment measures provide an objective, reliable, and valid method of assessing students' physical examination techniques on the pelvic exam simulator.
View details for DOI 10.1197/jamia.M1107
View details for Web of Science ID 000178205000003
View details for PubMedID 12223497
View details for PubMedCentralID PMC346632
Qualitative and quantitative analysis of pressure sensor data acquired by the E-Pelvis simulator during simulated pelvic examinations.
Studies in health technology and informatics
2002; 85: 376–79
View details for PubMedID 15458117
- Visual representations of physical abilities: Reverse haptic technology? 10th Annual Medicine Meets Virtual Reality Conference I O S PRESS. 2002: 380–381