Utilizing a novel tool for the comprehensive assessment of resident operative performance.

A mechanism for more effective and comprehensive assessment of surgical residents' performance in the operating room (OR) is needed, especially in light of the new requirements issued by the American Board of Surgery. Furthermore, there is an increased awareness that assessments need to be more meaningful by including not only procedure-specific and general technical skills, but also nontechnical skills (NOTECHS), such as teamwork and communication skills. Our aims were to develop a methodology and create a tool that comprehensively assesses residents' operative performance. A procedure-specific technical skill assessment for laparoscopic colon resections was created through use of task analysis. Components of previously validated tools were added to broaden the assessment to include general technical skills and NOTECHS. Our instrument was then piloted in the OR to measure face and content validity through an iterative process with faculty evaluators. Once the tool was finalized, postgraduate 3 (PG3) and PG5 residents on a 2-month long rotation were assessed by 1 of 4 colorectal surgeons immediately after completing a case together. Construct validity was measured by evaluating the difference in scores between PG3 and PG5 residents' performance as well as the change in scores over the course of the rotation. Sixty-three assessments were performed. All evaluations were completed within 48 hours of the operation. There was a statistically significant difference between the PG3 and PG5 scores on procedure-specific performance, general technical skills, NOTECHS, and overall performance. Over the course of the rotation, a statistically significant improvement was found in residents' scores on the procedure-specific portion of the assessment but not on the general surgical skills or NOTECHS. This is a feasible, valid, and reliable assessment tool for the comprehensive evaluation of resident performance in the OR. We plan to use this tool to assess resident operative skill development and to improve direct resident feedback.

Glarner CE ，McDonald RJ ，Smith AB ，Leverson GE ，Peyre S ，Pugh CM ，Greenberg CC ，Greenberg JA ，Foley EF ... -  《-》

Assessing residents' operative skills for external ventricular drain placement and shunt surgery in pediatric neurosurgery.

《-》

Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery.

Hadley C ，Lam SK ，Briceño V ，Luerssen TG ，Jea A ... -  《-》

Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship.

Preparatory training for new trainees beginning residency has been used by a variety of programs across the country. To improve the clinical orientation process for our new postgraduate year (PGY)-1 residents, we developed an intensive preparatory training curriculum inclusive of cognitive and procedural skills, training activities considered essential for early PGY-1 clinical management. We define our surgical PGY-1 Boot Camp as preparatory simulation-based training implemented at the onset of internship for introduction of skills necessary for basic surgical patient problem assessment and management. This orientation process includes exposure to simulated patient care encounters and technical skills training essential to new resident education. We report educational results of 4 successive years of Boot Camp training. Results were analyzed to determine if performance evidenced at onset of training was predictive of later educational outcomes. Learners were PGY-1 residents, in both categorical and preliminary positions, at our medium-sized surgical residency program. Over a 4-year period, from July 2007 to July 2010, all 30 PGY-1 residents starting surgical residency at our institution underwent specific preparatory didactic and skills training over a 9-week period. This consisted of mandatory weekly 1-hour and 3-hour sessions in the Simulation Center, representing a 4-fold increase in time in simulation laboratory training compared with the remainder of the year. Training occurred in 8 procedural skills areas (instrument use, knot-tying, suturing, laparoscopic skills, airway management, cardiopulmonary resuscitation, central venous catheter, and chest tube insertion) and in simulated patient care (shock, surgical emergencies, and respiratory, cardiac, and trauma management) using a variety of high- and low-tech simulation platforms. Faculty and senior residents served as instructors. All educational activities were structured to include preparatory materials, pretraining briefing sessions, and immediate in-training or post-training review and debriefing. Baseline cognitive skills were assessed with written tests on basic patient management. Post-Boot Camp tests similarly evaluated cognitive skills. Technical skills were assessed using a variety of task-specific instruments, and expressed as a mean score for all activities for each resident. All measurements were expressed as percent (%) best possible score. Cognitive and technical performance in Boot Camp was compared with subsequent clinical and core curriculum evaluations including weekly quiz scores, annual American Board of Surgery In-Training Examination (ABSITE) scores, program in-training evaluations (New Innovations, Uniontown, Ohio), and operative assessment instrument scores (OP-Rate, Baystate Medical Center, Springfield, Massachusetts) for the remainder of the PGY-1 year. Performance data were available for 30 PGY-1 residents over 4 years. Baseline cognitive skills were lower for the first year of Boot Camp as compared with subsequent years (71 ± 13, 83 ± 9, 84 ± 11, and 86 ± 6, respectively; p = 0.028, analysis of variance; ANOVA). Performance improved between pretests and final testing (81 ± 11 vs 89 ± 7; p < 0.001 paired t test). There was statistically significant correlation between Boot Camp final cognitive test results and American Board of Surgery In-Training Examination scores (p = 0.01; n = 22), but not quite significant for weekly curriculum quiz scores (p = 0.055; n = 22) and New Innovations cognitive assessments (p = 0.09; n = 25). Statistically significant correlation was also noted between Boot Camp mean overall skills and New Innovations technical skills assessments (p = 0.002; n = 25) and OP-Rate assessments (p = 0.01; n = 12). Individual simulation-based Boot Camp performance scores for cognitive and procedural skills assessments in PGY-1 residents correlate with subjective and objective clinical performance evaluations. This concurrent correlation with multiple traditional evaluation methods used to express competency in our residency program supports the use of Boot Camp performance measures as needs assessment tools as well as adjuncts to cumulative resident evaluation data.

Fernandez GL ，Page DW ，Coe NP ，Lee PC ，Patterson LA ，Skylizard L ，St Louis M ，Amaral MH ，Wait RB ，Seymour NE ... -  《-》

Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment.

The breadth of technical skills included in general surgery training continues to expand. The current competency-based training model requires assessment tools to measure acquisition, learning, and mastery of technical skill longitudinally in a reliable and valid manner. This study describes a novel skills assessment tool, the Omni, which evaluates performance in a broad range of skills over time. The 5 Omni tasks, consisting of open bowel anastomosis, knot tying, laparoscopic clover pattern cut, robotic needle drive, and endoscopic bubble pop, were developed by general surgery faculty. Component performance metrics assessed speed, accuracy, and quality, which were scaled into an overall score ranging from 0 to 10 for each task. For each task, ANOVAs with Scheffé's post hoc comparisons and Pearson's chi-squared tests compared performance between 6 resident cohorts (clinical years (CY1-5) and research fellows (RF)). Paired samples t-tests evaluated changes in performance across academic years. Cronbach's alpha coefficient determined the internal consistency of the Omni as an overall assessment. The Omni was developed by the Department of Surgery at Duke University. Annual assessment and this research study took place in the Surgical Education and Activities Lab. All active general surgery residents in 2 consecutive academic years spanning 2015 to 2017. A total of 62 general surgery residents completed the Omni and 39 (67.2%) of those residents completed the assessment in 2 consecutive years. Based on data from all residents' first assessment, statistically significant differences (p < 0.05) were observed among CY cohorts for bowel anastomosis, robotic, and laparoscopic task metrics. By pair-wise comparisons, mean bowel anastomosis scores distinguished CY1 from CY3-5 and CY2 from CY5. Mean robotic scores distinguished CY1 from RF, and mean laparoscopic scores distinguished CY1 from RF, CY3, and CY5 in addition to CY2 from CY3. Mean scores in performance on the knot tying and endoscopic tasks were not significantly different. Statistically significant improvement in mean scores was observed for all tasks from year 1 to year 2 (all p < 0.02). The internal consistency analysis revealed an alpha coefficient of 0.656. The Omni is a novel composite assessment tool for surgical technical skill that utilizes objective measures and scoring algorithms to evaluate performance. In this pilot study, 3 tasks demonstrated discriminative ability of performance by CY, and all 5 tasks demonstrated construct validity by showing longitudinal improvement in performance. Additionally, the Omni has adequate internal consistency for a formative assessment. These results suggest the Omni holds promise for the evaluation of resident technical skill and early identification of outliers requiring intervention.

Cox ML ，Risucci DA ，Gilmore BF ，Nag UP ，Turner MC ，Sprinkle SR ，Migaly J ，Sudan R ... -  《-》