Outcome measures for emergency medicine residency graduates: do measures of academic and clinical performance during residency training correlate with American Board of Emergency Medicine test performance?

Emergency medicine (EM) residency programs are increasingly asked to have measurable outcomes of residents' performance. Successful completion of the written and oral American Board of Emergency Medicine (ABEM) examinations is one key outcome. In the clinical practice of EM, emergency physicians (EPs) are often measured by their clinical productivity (patients per hour). This study explored the correlation between these measures of academic and clinical performance and hypothesized that clinical productivity would have a positive association with ABEM performance. A prospective written survey was sent to all EPs completing training at an established Midwest 3-year EM residency program between 1994 and 2005 (53,000 annual visits in 1994 to 65,000 annual visits in 2005). Physicians self-reported their national ABEM written and oral board scores in a blinded fashion. Simulated oral board scores and senior written in-training examination scores were also recorded. Postgraduate Year 3 (PGY3) clinical productivity was calculated as annual patient encounters divided by hours worked. Correlations among these variables were assessed by Pearson's correlation coefficient, with p < 0.05 being considered statistically significant. Multiple regression analysis was performed for ABEM oral and written examination scores. Fifty-six of 85 residents responded to the initial survey. There was no significant correlation between clinical productivity and ABEM scores, either written (r = -0.021, p = 0.881) or oral (r = -0.02, p = 0.879). There was also no significant correlation between productivity and simulated oral board scores (r = 0.065, p = 0.639) of PGY3 in-training scores (r = 0.078, p = 0.57). As previously reported, there were positive and significant correlations between PGY3 in-service scores and ABEM written examination scores (r = 0.60, p < 0.0001), as well as ABEM oral and written examination scores (r = 0.51, p < 0.0001). Multiple regression analysis revealed only the PGY3 in-training examination was a significant predictor of the ABEM oral and written scores (p < 0.001). PGY3 resident clinical productivity, when measured as patients per hour, correlated poorly with academic performance when measured by written and oral ABEM scores. The PGY3 in-training examination was predictive of the ABEM written and oral examination scores.

Frederick RC ，Hafner JW ，Schaefer TJ ，Aldag JC ... -  《-》

Boot cAMP: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship.

Preparatory training for new trainees beginning residency has been used by a variety of programs across the country. To improve the clinical orientation process for our new postgraduate year (PGY)-1 residents, we developed an intensive preparatory training curriculum inclusive of cognitive and procedural skills, training activities considered essential for early PGY-1 clinical management. We define our surgical PGY-1 Boot Camp as preparatory simulation-based training implemented at the onset of internship for introduction of skills necessary for basic surgical patient problem assessment and management. This orientation process includes exposure to simulated patient care encounters and technical skills training essential to new resident education. We report educational results of 4 successive years of Boot Camp training. Results were analyzed to determine if performance evidenced at onset of training was predictive of later educational outcomes. Learners were PGY-1 residents, in both categorical and preliminary positions, at our medium-sized surgical residency program. Over a 4-year period, from July 2007 to July 2010, all 30 PGY-1 residents starting surgical residency at our institution underwent specific preparatory didactic and skills training over a 9-week period. This consisted of mandatory weekly 1-hour and 3-hour sessions in the Simulation Center, representing a 4-fold increase in time in simulation laboratory training compared with the remainder of the year. Training occurred in 8 procedural skills areas (instrument use, knot-tying, suturing, laparoscopic skills, airway management, cardiopulmonary resuscitation, central venous catheter, and chest tube insertion) and in simulated patient care (shock, surgical emergencies, and respiratory, cardiac, and trauma management) using a variety of high- and low-tech simulation platforms. Faculty and senior residents served as instructors. All educational activities were structured to include preparatory materials, pretraining briefing sessions, and immediate in-training or post-training review and debriefing. Baseline cognitive skills were assessed with written tests on basic patient management. Post-Boot Camp tests similarly evaluated cognitive skills. Technical skills were assessed using a variety of task-specific instruments, and expressed as a mean score for all activities for each resident. All measurements were expressed as percent (%) best possible score. Cognitive and technical performance in Boot Camp was compared with subsequent clinical and core curriculum evaluations including weekly quiz scores, annual American Board of Surgery In-Training Examination (ABSITE) scores, program in-training evaluations (New Innovations, Uniontown, Ohio), and operative assessment instrument scores (OP-Rate, Baystate Medical Center, Springfield, Massachusetts) for the remainder of the PGY-1 year. Performance data were available for 30 PGY-1 residents over 4 years. Baseline cognitive skills were lower for the first year of Boot Camp as compared with subsequent years (71 ± 13, 83 ± 9, 84 ± 11, and 86 ± 6, respectively; p = 0.028, analysis of variance; ANOVA). Performance improved between pretests and final testing (81 ± 11 vs 89 ± 7; p < 0.001 paired t test). There was statistically significant correlation between Boot Camp final cognitive test results and American Board of Surgery In-Training Examination scores (p = 0.01; n = 22), but not quite significant for weekly curriculum quiz scores (p = 0.055; n = 22) and New Innovations cognitive assessments (p = 0.09; n = 25). Statistically significant correlation was also noted between Boot Camp mean overall skills and New Innovations technical skills assessments (p = 0.002; n = 25) and OP-Rate assessments (p = 0.01; n = 12). Individual simulation-based Boot Camp performance scores for cognitive and procedural skills assessments in PGY-1 residents correlate with subjective and objective clinical performance evaluations. This concurrent correlation with multiple traditional evaluation methods used to express competency in our residency program supports the use of Boot Camp performance measures as needs assessment tools as well as adjuncts to cumulative resident evaluation data.

Fernandez GL ，Page DW ，Coe NP ，Lee PC ，Patterson LA ，Skylizard L ，St Louis M ，Amaral MH ，Wait RB ，Seymour NE ... -  《-》

Do United States Medical Licensing Examination (USMLE) scores predict in-training test performance for emergency medicine residents?

Residency selection committees commonly utilize USMLE scores as criteria to screen residency applicants. The purpose of this study is to evaluate the relationship between United States Medical Licensing Examination (USMLE) and American Board of Emergency Medicine (ABEM) in-training examination scores (ITEs). In an Accreditation Council for Graduate Medical Education-accredited emergency medicine residency program, data were collected for this retrospective cohort study for the classes of 2002-2006. USMLE Step 1 and 2 scores and the ABEM ITEs were recorded for each post-graduate year (PGY) within the aforementioned time frame. Step 1 and 2 scores were compared to consecutive PGY ABEM ITEs to evaluate for an association. There were 51 USMLE Step 1 and 39 Step 2 scores available for comparison with 153 ABEM ITEs. The mean USMLE Step 1 and Step 2 scores were 228.9 (range 197-252) and 228.4 (range 168-259), respectively. The mean in-training percentiles for the PGY 1, 2, and 3 years were 40.4, 68.3, and 81.7, respectively. The R-squared values for the Step 1 scores compared to the PGY 1, 2, and 3 years' ITEs were 0.25, 0.18, and 0.16, respectively. The R-squared values for Step 2 scores as compared to the ABEM ITEs for the PGY 1, 2, and 3 years were 0.43, 0.44, and 0.38, respectively. Residents who scored below 200 on either USMLE Step 1 or Step 2 had significantly lower mean ABEM ITEs than residents who scored above 200 (p < 0.05) and were 10-fold more likely than residents who scored above 220 to score below the 70th percentile in their PGY3 ABEM ITE. USMLE Step 1 scores are mildly correlated and Step 2 scores are moderately correlated with ABEM ITEs. Scoring below 200 on either test is associated with significantly lower ABEM ITEs.

Thundiyil JG ，Modica RF ，Silvestri S ，Papa L ... -  《JOURNAL OF EMERGENCY MEDICINE》

Conference attendance does not correlate with emergency medicine residency in-training examination scores.

The residency review committee for emergency medicine (EM) requires residents to have greater than 70% attendance of educational conferences during residency training, but it is unknown whether attendance improves clinical competence or scores on the American Board of Emergency Medicine (ABEM) in-training examination (ITE). This study examined the relationship between conference attendance and ITE scores. The hypothesis was that greater attendance would correlate to a higher examination score. This was a multi-center retrospective cohort study using conference attendance data and examination results from residents in four large county EM residency training programs. Longitudinal multi-level models, adjusting for training site, U.S. Medical Licensing Examination (USMLE) Step 1 score, and sex were used to explore the relationship between conference attendance and in-training examination scores according to year of training. Each year of training was studied, as well as the overall effect of mean attendance as it related to examination score. Four training sites reported data on 405 residents during 2002 to 2008; 386 residents had sufficient data to analyze. In the multi-level longitudinal models, attendance at conference was not a significant predictor of in-training percentile score (coefficient = 0.005, 95% confidence interval [CI] = -0.053 to 0.063, p = 0.87). Score on the USMLE Step 1 examination was a strong predictor of ITE score (coefficient = 0.186, 95% CI = 0.155 to 0.217; p < 0.001), as was female sex (coefficient = 2.117, 95% CI = 0.987 to 3.25; p < 0.001). Greater conference attendance does not correlate with performance on an individual's ITE scores. Conference attendance may represent an important part of EM residency training but perhaps not of ITE performance.

Gene Hern H Jr ，Wills C ，Alter H ，Bowman SH ，Katz E ，Shayne P ，Vahidnia F ... -  《-》

Correlation of the National Board of Medical Examiners Emergency Medicine Advanced Clinical Examination Given in July to Intern American Board of Emergency Medicine in-training Examination Scores: A Predictor of Performance?

Hiller K ，Franzen D ，Heitz C ，Emery M ，Poznanski S ... -  《-》