Association Between American Board of Surgery In-Training Examination Scores and Resident Performance.

Ray JJ ，Sznol JA ，Teisch LF ，Meizoso JP ，Allen CJ ，Namias N ，Pizano LR ，Sleeman D ，Spector SA ，Schulman CI ... -  《-》

Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship.

Preparatory training for new trainees beginning residency has been used by a variety of programs across the country. To improve the clinical orientation process for our new postgraduate year (PGY)-1 residents, we developed an intensive preparatory training curriculum inclusive of cognitive and procedural skills, training activities considered essential for early PGY-1 clinical management. We define our surgical PGY-1 Boot Camp as preparatory simulation-based training implemented at the onset of internship for introduction of skills necessary for basic surgical patient problem assessment and management. This orientation process includes exposure to simulated patient care encounters and technical skills training essential to new resident education. We report educational results of 4 successive years of Boot Camp training. Results were analyzed to determine if performance evidenced at onset of training was predictive of later educational outcomes. Learners were PGY-1 residents, in both categorical and preliminary positions, at our medium-sized surgical residency program. Over a 4-year period, from July 2007 to July 2010, all 30 PGY-1 residents starting surgical residency at our institution underwent specific preparatory didactic and skills training over a 9-week period. This consisted of mandatory weekly 1-hour and 3-hour sessions in the Simulation Center, representing a 4-fold increase in time in simulation laboratory training compared with the remainder of the year. Training occurred in 8 procedural skills areas (instrument use, knot-tying, suturing, laparoscopic skills, airway management, cardiopulmonary resuscitation, central venous catheter, and chest tube insertion) and in simulated patient care (shock, surgical emergencies, and respiratory, cardiac, and trauma management) using a variety of high- and low-tech simulation platforms. Faculty and senior residents served as instructors. All educational activities were structured to include preparatory materials, pretraining briefing sessions, and immediate in-training or post-training review and debriefing. Baseline cognitive skills were assessed with written tests on basic patient management. Post-Boot Camp tests similarly evaluated cognitive skills. Technical skills were assessed using a variety of task-specific instruments, and expressed as a mean score for all activities for each resident. All measurements were expressed as percent (%) best possible score. Cognitive and technical performance in Boot Camp was compared with subsequent clinical and core curriculum evaluations including weekly quiz scores, annual American Board of Surgery In-Training Examination (ABSITE) scores, program in-training evaluations (New Innovations, Uniontown, Ohio), and operative assessment instrument scores (OP-Rate, Baystate Medical Center, Springfield, Massachusetts) for the remainder of the PGY-1 year. Performance data were available for 30 PGY-1 residents over 4 years. Baseline cognitive skills were lower for the first year of Boot Camp as compared with subsequent years (71 ± 13, 83 ± 9, 84 ± 11, and 86 ± 6, respectively; p = 0.028, analysis of variance; ANOVA). Performance improved between pretests and final testing (81 ± 11 vs 89 ± 7; p < 0.001 paired t test). There was statistically significant correlation between Boot Camp final cognitive test results and American Board of Surgery In-Training Examination scores (p = 0.01; n = 22), but not quite significant for weekly curriculum quiz scores (p = 0.055; n = 22) and New Innovations cognitive assessments (p = 0.09; n = 25). Statistically significant correlation was also noted between Boot Camp mean overall skills and New Innovations technical skills assessments (p = 0.002; n = 25) and OP-Rate assessments (p = 0.01; n = 12). Individual simulation-based Boot Camp performance scores for cognitive and procedural skills assessments in PGY-1 residents correlate with subjective and objective clinical performance evaluations. This concurrent correlation with multiple traditional evaluation methods used to express competency in our residency program supports the use of Boot Camp performance measures as needs assessment tools as well as adjuncts to cumulative resident evaluation data.

Fernandez GL ，Page DW ，Coe NP ，Lee PC ，Patterson LA ，Skylizard L ，St Louis M ，Amaral MH ，Wait RB ，Seymour NE ... -  《-》

Effect of January vacations and prior night call status on resident ABSITE performance.

To determine if vacations in January or on-call status have an effect on American Board of Surgery In-Training Examination (ABSITE) scores. Retrospective review of the performance of general surgery residents on ABSITE. Data collected included ABSITE scores, United States Medical Licensing Examination Step 2 scores, January vacation schedules, and call schedules. ABSITE performance was examined for correlation with vacation or call schedules. Student t test was used for statistical analysis, with a p value of less than 0.05 considered significant. General surgery residency program at the Louisiana State University Health Sciences Center-Shreveport, a university hospital-based program with 5 categorical residents per year. Postgraduate year (PGY) 1 through 5 general surgery categorical residents from 2006 to 2012. A total of 170 ABSITE scores from 55 residents were reviewed. The mean score when vacation was taken was 48.6 as compared with 36.3 when no vacation was taken (p = 0.02). Residents who took a January vacation at least once in their residency had a mean score of 42.8 as compared with 37.7 of those who did not (p = 0.43). The mean United States Medical Licensing Examination Step 2 score of residents who took a January vacation at least once in their residency was 218 as compared with 217 for their peers (p = 0.78). Among residents who took January vacations, the mean score in the years they took vacation was 49.4 as compared with 35.4 in the years they did not (p = 0.02). Prior night call status had no effect on the examination scores (44.2 vs 38.6, p = 0.30). Mean ABSITE scores were higher for residents who took a January vacation before the examination, despite no apparent difference in baseline test-taking ability. Among residents who took January vacations, mean scores were higher in the years they took vacation than in other years. On-call status did not have an effect on ABSITE performance. Vacation schedules in January can have a significant effect on ABSITE scores.

Sugar JG ，Chu QD ，Cole PA ，Li BD ，Kim RH ... -  《-》

Program Factors That Influence American Board of Surgery In-Training Examination Performance: A Multi-Institutional Study.

To determine the influence of program strategies, such as program directors' (PD) attitudes about the American Board of Surgery In-Training Examination (ABSITE) and approach to ABSITE preparation, on residents' ABSITE performance. A 17-item questionnaire was sent to PDs at surgical residency programs. The questions were designed to elicit information regarding the educational curriculum, remediation protocols, and opinions relating to the ABSITE. Main outcome measure was categorical resident ABSITE percentile scores from the January 2014 examination. Statistical analysis was performed using the Student t-test, analysis of variance, and linear regression as appropriate. The study was carried out at general surgery residency programs across the country. In total, 15 general surgery residency PDs participated in the study. The PD response rate was 100%. All 460 resident ABSITE scores from the 15 programs were obtained. In total, 10 programs (67%) identified as university affiliated, 4 programs (27%) as independent academic, and 1 program (7%) as hybrid. The mean number of residents per program was 30.7 (range: 15-57). In total, 14 PDs (93%) indicated that an ABSITE review curriculum was in place and 13 PDs (87%) indicated they had a remediation protocol for residents with low ABSITE scores (with differing thresholds of <30th, <35th, and <40th percentile). The median overall ABSITE score for all residents was 61st percentile (interquartile range = 39.5). The mean ABSITE score for each program ranged from 39th to 75th percentile. Program factors associated with higher ABSITE scores included tracking resident reading throughout the year (median 63rd percentile with tracking vs 59th percentile without, p = 0.040) and the type of remediation (by PD: 77th percentile, by PD and faculty: 57th percentile, faculty only: 64th percentile, with Surgical Education and Self-Assessment Program (SESAP): 63rd percentile, outside review course: 43rd percentile; p < 0.001). Programs with a remediation protocol trended toward higher ABSITE scores compared with programs without remediation protocols (median 61st percentile vs 53rd percentile, p = 0.098). Factors not significantly associated with ABSITE performance included number of structured educational hours per week and frequency of ABSITE review sessions. Program factors appear to significantly influence ABSITE performance. Programs where the PD was actively involved in remediation mentorship and the tracking of resident reading achieved higher ABSITE percentile scores on the January 2014 examination. Counterintuitively, residents from programs with a lower ABSITE threshold for remediation performed better on the examination.

Kim JJ ，Gifford ED ，Moazzez A ，Sidwell RA ，Reeves ME ，Hartranft TH ，Inaba K ，Jarman BT ，Are C ，Galante JM ，Amersi F ，Smith BR ，Melcher ML ，Nelson MT ，Donahue T ，Jacobsen G ，Arnell TD ，Lee S ，Neville A ，de Virgilio C ... -  《-》

Predicting academic performance in surgical training.

During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards. Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of ≥70% correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments. A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B = -0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents. Behavioral, motivational, and acumen characteristics can be useful to identify residents "at risk" for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.

Yost MJ ，Gardner J ，Bell RM ，Fann SA ，Lisk JR ，TriMetrix and Success Research Group ，Cheadle WG ，Goldman MH ，Rawn S ，Weigelt JA ，Termuhlen PM ，Woods RJ ，Endean ED ，Kimbrough J ，Hulme M ... -  《-》