Case Reporting, Competence, and Confidence: A Discrepancy in the Numbers.

The Accreditation Council for Graduate Medical Education (ACGME) continues to play an integral role in accreditation of surgical programs. The institution of case logs to demonstrate competency of graduating residents is a key component of evaluation. This study compared the number of vascular cases a surgical resident has completed according to the ACGME operative log to their operative proficiency, quality of anastomosis, operative experience, and confidence in both a simulation and operative setting. General surgery residents ranging from PGY 1 to 5 participated in a simulation laboratory in which they completed an end-to-side vascular anastomosis. Each participant was given a weighted score based on technical proficiency and anastomosis quality using a previously validated Global Rating Scale (Duran et al, 2014). These scores were correlated to the General Surgery Milestones. Participants completed preoperative and postoperative surveys assessing resident operative experience using the 4-level Zwisch scale (DaRosa et al., 2013), confidence with vascular procedures and confidence performing simulated anastomoses. Confidence was assessed on a scale from 1 to 9 (not confident to extremely confident). Case logs were recorded for each participant. An IRB approved questionnaire was distributed to assess preoperative and postoperative roles of both the resident physician and faculty, with a defined goal. Univariate and multivariate analysis was performed. Twenty-one general surgery residents were evaluated in the simulation laboratory and 8 residents were assessed intraoperatively. The residents were evenly distributed throughout clinical years. Groups of residents were divided into quartiles based upon the number of vascular cases recorded in the ACGME database. No correlation was found between number of cases, Milestones score and the weighted score (p = 0.94). No statistical significance was found between confidence and quality of anastomosis (p = 0.1). Resident operative experience per the Zwisch scale was categorized most commonly as "Smart Help" by both the trainee and attending surgeon, despite mean resident confidence ratings of 6.67 (± 1.61) with vascular procedures. ACGME case logs, which are utilized to assess readiness for completion of general surgery residency, may not be indicative of a resident's operative competency and technical proficiency. Confidence is not correlated with technical ability. Faculty and resident insight as to their role in a procedure differ, as faculty feel that they are providing less help than the resident perceives. Careful examination of resident operative technique is the best measure of competency.

Shah D ，Haisch CE ，Noland SL 《-》

Evaluating Coding Accuracy in General Surgery Residents' Accreditation Council for Graduate Medical Education Procedural Case Logs.

The Accreditation Council for Graduate Medical Education (ACGME) case log captures resident operative experience based on Current Procedural Terminology (CPT) codes and is used to track operative experience during residency. With increasing emphasis on resident operative experiences, coding is more important than ever. It has been shown in other surgical specialties at similar institutions that the residents' ACGME case log may not accurately reflect their operative experience. What barriers may influence this remains unclear. As the only objective measure of resident operative experience, an accurate case log is paramount in representing one's operative experience. This study aims to determine the accuracy of procedural coding by general surgical residents at a single institution. Data were collected from 2 consecutive graduating classes of surgical residents' ACGME case logs from 2008 to 2014. A total of 5799 entries from 7 residents were collected. The CPT codes entered by residents were compared to departmental billing records submitted by the attending surgeon for each procedure. Assigned CPT codes by institutional American Academy of Professional Coders certified abstract coders were considered the "gold standard." A total of 4356 (75.12%) of 5799 entries were identified in billing records. Excel 2010 and SAS 9.3 were used for analysis. In the event of multiple codes for the same patient, any match between resident codes and billing record codes was considered a "correct" entry. A 4-question survey was distributed to all current general surgical residents at our institution for feedback on coding habits, limitations to accurate coding, and opinions on ACGME case log representation of their operative experience. All 7 residents had a low percentage of correctly entered CPT codes. The overall accuracy proportion for all residents was 52.82% (range: 43.32%-60.07%). Only 1 resident showed significant improvement in accuracy during his/her training (p = 0.0043). The survey response rate was 100%. Survey results indicated that inability to find the precise code within the ACGME search interface and unfamiliarity with available CPT codes were by far the most common perceived barriers to accuracy. Survey results also indicated that most residents (74%) believe that they code accurately most of the time and agree that their case log would accurately represent their operative experience (66.6%). This is the first study to evaluate correctness of residents' ACGME case logs in general surgery. The degree of inaccuracy found here necessitates further investigation into the etiology of these discrepancies. Instruction on coding practices should also benefit the residents after graduation. Optimizing communication among attendings and residents, improving ACGME coding search interface, and implementing consistent coding practices could improve accuracy giving a more realistic view of residents' operative experience.

Balla F ，Garwe T ，Motghare P ，Stamile T ，Kim J ，Mahnken H ，Lees J ... -  《-》

Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.

The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption. We aimed to demonstrate that this is a reliable, valid, and feasible method of measuring resident operative autonomy. Before implementation, general surgery residents and faculty underwent frame-of-reference training to the Zwisch scale. Immediately after any operation in which a resident participated, the system automatically sent a text message prompting the attending physician to rate the resident's level of operative autonomy according to the 4-level Zwisch scale. Of these procedures, 8 were videotaped and independently rated by 2 additional surgeons. The Zwisch ratings of the 3 raters were compared using an intraclass correlation coefficient. Videotaped procedures were also scored using 2 alternative operating room (OR) performance assessment instruments (Operative Performance Rating System and Ottawa Surgical Competency OR Evaluation), against which the item correlations were calculated. Between December 2012 and June 2013, 27 faculty used the smartphone system to complete 1490 operative performance assessments on 31 residents. During this period, faculty completed evaluations for 92% of all operations performed with general surgery residents. The Zwisch scores were shown to correlate with postgraduate year (PGY) levels based on sequential pairwise chi-squared tests: PGY 1 vs PGY 2 (χ(2) = 106.9, df = 3, p < 0.001); PGY 2 vs PGY 3 (χ(2) = 22.2, df = 3, p < 0.001); and PGY 3 vs PGY 4 (χ(2) = 56.4, df = 3, p < 0.001). Comparison of PGY 4 to PGY 5 scores were not significantly different (χ(2) = 4.5, df = 3, p = 0.21). For the 8 operations reviewed for interrater reliability, the intraclass correlation coefficient was 0.90 (95% CI: 0.72-0.98, p < 0.01). Correlation of Procedural Autonomy and Supervisions System ratings with both Operative Performance Rating System items (each r > 0.90, all p's < 0.01) and Ottawa Surgical Competency OR Evaluation items (each r > 0.86, all p's < 0.01) was high. The Zwisch scale can be used to make reliable and valid measurements of faculty guidance and resident autonomy. Our data also suggest that Zwisch ratings may be used to infer resident operative performance. Deployed on an automated smartphone-based system, it can be used to feasibly record evaluations for most operations performed by residents. This information can be used to council individual residents, modify programmatic curricula, and potentially inform national training guidelines.

George BC ，Teitelbaum EN ，Meyerson SL ，Schuller MC ，DaRosa DA ，Petrusa ER ，Petito LC ，Fryer JP ... -  《-》

Characterizing the Relationship Between Surgical Resident and Faculty Perceptions of Autonomy in the Operating Room.

Characterize the concordance among faculty and resident perceptions of surgical case complexity, resident technical performance, and autonomy in a diverse sample of general surgery procedures using case-specific evaluations. A prospective study was conducted in which a faculty surgeon and surgical resident independently completed a postoperative assessment examining case complexity, resident operative performance (Milestone assessment) and autonomy (Zwisch model). Pearson correlation coefficients (r) reaching statistical significance (p < 0.05) were further classified as moderate (r ≥ 0.40), strong (r ≥ 0.60), or very strong (r ≥ 0.80). This study was conducted in the General Surgery Residency Program at an academic tertiary care facility (Geisinger Medical Center, Danville, PA). Participants included 6 faculty surgeons, in addition to 5 postgraduate year (PGY) 1, 6 midlevel (PGY 2-3), and 4 chief (PGY 4-5) residents. In total, 75 surgical cases were analyzed. Midlevel residents accounted for the highest number of cases (35, 46.6%). Overall, faculty and resident perceptions of case complexity demonstrated a strong correlation (r = 0.76, p < 0.0001). Technical performance scores were also strongly correlated (r = 0.66, p < 0.0001), whereas perceptions of autonomy demonstrated a moderate correlation (r = 0.56, p < 0.0001). Subgroup analysis revealed very strong correlations among faculty perceptions of case complexity and the perceptions of PGY 1 (r = 0.80, p < 0.0001) and chief residents (r = 0.82, p < 0.0001). All other intergroup correlations were strong with 2 notable exceptions as follows: midlevel and chief residents failed to correlate with faculty perceptions of autonomy and operative performance, respectively. General surgery residents generally demonstrated high correlations with faculty perceptions of case complexity, technical performance, and operative autonomy. This generalized accord supports the use of the Milestone and Zwisch assessments in residency programs. However, discordance among perceptions of midlevel resident autonomy and chief resident operative performance suggests that these trainees may need more direct communication from the faculty.

Young KA ，Lane SM ，Widger JE ，Neuhaus NM ，Dove JT ，Fluck M ，Hunsinger MA ，Blansfield JA ，Shabahang MM ... -  《-》

An Analysis of Operative Experiences of Junior General Surgical Residents and Correlation With the SCORE Curriculum.

Junior surgical resident education at academic institutions is traditionally focused to preoperative and postoperative patient management. Our objective was to investigate the number and type of surgical procedures performed by junior general surgery residents in comparison with the American Board of Surgery requirements and the Surgical Council on Resident Education (SCORE) curriculum. This was a retrospective study using the Accreditation Counsel for Graduate Medical Education (ACGME) operative case logs of junior surgical residents. The Ohio State University Wexner Medical Center, Columbus, OH; a tertiary academic medical center. We performed, an institutional review board approved, retrospective review of logged surgical cases from general surgical residents during postgraduate year (PGY) 1 and 2 from 2009 to 2015 at an academic medical center. Summary case logs were accessed from the ACGME. Procedures were extracted from the SCORE curriculum and correlated to corresponding ACGME defined procedures for total cases, major cases, and endoscopy. Minor cases and patient care cases were excluded as they were not clearly defined on the category report. SCORE procedures were excluded if there was not a corresponding ACGME procedure on the summary report. SCORE procedures and ACGME procedures were combined with each other if there was overlap with correlation. Statistics were performed on individual and total resident data. One-sample student's t-test was used to compare total number of cases logged with the 250 case log ABS requirement and to compare the total major cases and endoscopy performed with those represented on SCORE. Overall, 26 residents completed both PGY-1 and 2 years from 2009 to 2015, and remained at the same institution for case logs to be accessed during the study period. A total of 21 residents (80.76%) completed 250 cases or more after their first 2 years of residency. Across all years, the mean case log was 349 cases (p = 0.20), and was statistically more than than 250 cases in 3 of the 5 class years. Junior residents completed a total mean of 312 major and endoscopy cases (89%) at the end of 2 years, which was statistically higher than the 75% testing hypothesis across all years (p < 0.01). Of major and endoscopy cases performed in total by the completion of the PGY-2 year, a mean of 275 cases (88%) were included in the SCORE curriculum. Using one-sample t-test, SCORE procedures represented more than 85% of the major and endoscopy cases logged (p < 0.01). Of all major and endoscopy cases logged that correlate to a SCORE procedure, 95% were found to be "core" and 5% were "advanced." Our study demonstrates that junior surgical residents meet the 250 case log requirement put forth by the ABS, and most major procedures and endoscopy performed correspond with the core cases of the SCORE curriculum at our institution. This study aid in the confirmation of the SCORE curriculum for junior residents, and those procedures which should be designated as core.

Strosberg DS ，Quinn KM ，Abdel-Misih SR ，Harzman AE ... -  《-》