A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones: Clinical Competency Committee vs. Resident Self-Assessment.

The Accreditation Council for Graduate Medical Education requires accredited residency programs to implement competency-based assessments of medical trainees based upon nationally established Milestones. Clinical competency committees (CCC) are required to prepare biannual reports using the Milestones and ensure reporting to the Accreditation Council for Graduate Medical Education. Previous research demonstrated a strong correlation between CCC and resident scores on the Milestones at 1 institution. We sought to evaluate a national sampling of general surgery residency programs and hypothesized that CCC and resident assessments are similar. Details regarding the makeup and process of each CCC were obtained. Major disparities were defined as an absolute mean difference of ≥0.5 on the 4-point scale. A negative assessment disparity indicated that the residents evaluated themselves at a lower level than did the CCC. Statistical analysis included Wilcoxon rank sum and Sign tests. CCCs and categorical general surgery residents from 15 residency programs completed the Milestones document independently during the spring of 2016. Overall, 334 residents were included; 44 (13%) and 43 (13%) residents scored themselves ≥0.5 points higher and lower than the CCC, respectively. Female residents scored themselves a mean of 0.08 points lower, and male residents scored themselves a mean of 0.03 points higher than the CCC. Median assessment differences for postgraduate year (PGY) 1-5 were 0.03 (range: -0.94 to 1.28), -0.11 (range: -1.22 to 1.22), -0.08 (range: -1.28 to 0.81), 0.02 (range: -0.91 to 1.00), and -0.19 (range: -1.16 to 0.50), respectively. Residents in university vs. independent programs had higher rates of negative assessment differences in medical knowledge (15% vs. 6%; P = 0.015), patient care (17% vs. 5%; P = 0.002), professionalism (23% vs. 14%; P = 0.013), and system-based practice (18% vs. 9%; P = 0.031) competencies. Major assessment disparities by sex or PGY were similar among individual competencies. Surgery residents in this national cohort demonstrated self-awareness when compared to assessments by their respective CCCs. This was independent of program type, sex, or level of training. PGY 5 residents, female residents, and those from university programs consistently rated themselves lower than the CCC, but these were not major disparities and the significance of this is unclear.

Watson RS ，Borgert AJ ，O Heron CT ，Kallies KJ ，Sidwell RA ，Mellinger JD ，Joshi AR ，Galante JM ，Chambers LW ，Morris JB ，Josloff RK ，Melcher ML ，Fuhrman GM ，Terhune KP ，Chang L ，Ferguson EM ，Auyang ED ，Patel KR ，Jarman BT ... -  《-》

Do Attending Surgeons and Residents See Eye To Eye? An Evaluation of the Accreditation Council For Graduate Medical Education Milestones in General Surgery Residency.

The Accreditation Council for Graduate Medical Education requires accredited general surgery residencies to implement competency-based developmental outcomes in resident evaluations. Overall, 16 milestones are evaluated by a clinical competency committee (CCC). The milestones span 8 domains of surgical practice and 6 Accreditation Council for Graduate Medical Education clinical competencies. The highest level suggests preparedness for independent practice. Our objective was to compare self-assessments and committee evaluations within the milestone framework. All residents underwent semiannual evaluations from 2013 to 2015. Residents independently completed a self-assessment using the milestones. The CCC completed the milestones document using resident evaluations and consensus opinion of committee members. Assessment differences were calculated for each evaluation. A negative value indicated that the residents evaluated themselves at a lower level than the committee. Major assessment disparities were defined as >0.5 on a 4-point scale. An independent academic medical center. General surgery residents. Overall, 20 residents participated; 7 were female. In total, 5 (7%) evaluations had a mean overall assessment difference >0.5, whereas 6 (8%) had a difference <-0.5. Residents evaluated themselves lower than the committee with a median assessment difference of -0.06 [-0.25 to 0.16] (p = 0.041). Evaluations were similar across surgical domains. Negative self-evaluations were more common for medical knowledge (-0.25 [-0.25 to 0.25], p = 0.025). Female residents had 2% positive and 13% negative major assessment disparity rates versus 10% positive and 9% negative rates among male residents. Postgraduate year III residents had 12% positive and 4% negative major disparity rates; all other years had higher negative than positive rates. Surgery residents within our program demonstrated adequate self-awareness, with most self-evaluations falling within a half level of the CCC report. This self-awareness was consistent across surgical domains and most clinical competencies. Residents perceived a lower level of medical knowledge than the CCC. Subgroup analysis revealed interesting trends in the effects of sex, postgraduate year level, and academic year timing, which will take additional study to fully delineate.

Lyle B ，Borgert AJ ，Kallies KJ ，Jarman BT ... -  《-》

Using the ACGME Milestones for Resident Self-Evaluation and Faculty Engagement.

Since July 2014 General Surgery residency programs have been required to use the Accreditation Council for Graduate Medical Education milestones twice annually to assess the progress of their trainees. We felt this change was a great opportunity to use this new evaluation tool for resident self-assessment and to furthermore engage the faculty in the educational efforts of the program. We piloted the milestones with postgraduate year (PGY) II and IV residents during the 2013/2014 academic year to get faculty and residents acquainted with the instrument. In July 2014, we implemented the same protocol for all residents. Residents meet with their advisers quarterly. Two of these meetings are used for milestones assessment. The residents perform an independent self-evaluation and the adviser grades them independently. They discuss the evaluations focusing mainly on areas of greatest disagreement. The faculty member then presents the resident to the clinical competency committee (CCC) and the committee decides on the final scores and submits them to the Accreditation Council for Graduate Medical Education website. We stored all records anonymously in a MySQL database. We used Anova with Tukey post hoc analysis to evaluate differences between groups. We used intraclass correlation coefficients and Krippendorff's α to assess interrater reliability. We analyzed evaluations for 44 residents. We created scale scores across all Likert items for each evaluation. We compared score differences by PGY level and raters (self, adviser, and CCC). We found highly significant increases of scores between most PGY levels (p < 0.05). There were no significant score differences per PGY level between the raters. The interrater reliability for the total score and 6 competency domains was very high (ICC: 0.87-0.98 and α: 0.84-0.97). Even though this milestone evaluation process added additional work for residents and faculty we had very good participation (93.9% by residents and 92.9% by faculty) and feedback was generally positive. Even though implementation of the milestones has added additional work for general surgery residency programs, it has also opened opportunities to furthermore engage the residents in reflection and self-evaluation and to create additional venues for faculty to get involved with the educational process within the residency program. Using the adviser as the initial rater seems to correlate closely with the final CCC assessment. Self-evaluation by the resident is a requirement by the RRC and the milestones seem to be a good instrument to use for this purpose. Our early assessment suggests the milestones provide a useful instrument to track trainee progression through their residency.

Meier AH ，Gruessner A ，Cooney RN 《-》

Ability of Ophthalmology Residents to Self-Assess Their Performance Through Established Milestones.

Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores. Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores. Wilmer Eye Institute, Johns Hopkins Hospital. Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016. Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms. Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.

Srikumaran D ，Tian J ，Ramulu P ，Boland MV ，Woreta F ，Wang KM ，Mahoney N ... -  《-》

Do You See What I See? A Comparison of CCC and Self-Assigned Milestones Across Military Medical Specialties.

The Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to complete competency-based assessments of medical trainees based on nationally established Milestones. Previous research demonstrates a strong correlation between CCC and resident scores on the Milestones in surgery, but little is known if this is true between specialties. In this study, we investigated a variety of specialties and sought to determine what factors affect self-assessment of milestones. In addition, a post-hoc analysis was completed on the COVID-19 pandemic effects on self-evaluation. This is an IRB approved observational study on prospectively collected self-evaluation milestone data that is used within each ACGME program's Clinical Competency Committees. Medical trainees within the San Antonio Uniformed Services Health Education Consortium were approached for possible participation in this study with permission from program directors. There was no significant difference between self-assessments and CCC-assessments based on self-identified gender or residency type (surgical versus nonsurgical) for any milestone domain. Within the postgraduate year (PGY) groups, the PGY5 and PGY6 tended to rate themselves higher than CCC. Chiefs (Internal Medicine PGY2/3, and General Surgery PGY5/6) tended to be more accurate in scoring themselves than the interns (PGY1) within the milestone of Interpersonal Skills and Communication (chiefs 0.5 vs. interns 0.62, p = 0.03). On post hoc analysis of self-rating, during the first wave of the COVID 19 pandemic, Post-Covid residents were more likely to underrate themselves in Systems-Based Practice compared to the Pre-Covid cohort (-0.49 vs 0.10; p = 0.007) and more likely to rate themselves higher in Professionalism (-0.54 vs. -0.10, p = 0.012). Unique to this study and our institution, there was no gender difference found in self vs CCC evaluations. With the change in learning environment from COVID, there was also a change in ability for some learners to self-assess accurately. As medical educators, we should understand the importance of both encouraging learners to practice self-assessment as well as give feedback to trainees on their progress. We also need to educate our faculty on the use of milestones for assessment to create a true gold standard in the CCC.

Haischer-Rollo G ，Geringer JL ，Thomas K ，Hale D ... -  《-》