Evaluating Coding Accuracy in General Surgery Residents' Accreditation Council for Graduate Medical Education Procedural Case Logs.

The Accreditation Council for Graduate Medical Education (ACGME) case log captures resident operative experience based on Current Procedural Terminology (CPT) codes and is used to track operative experience during residency. With increasing emphasis on resident operative experiences, coding is more important than ever. It has been shown in other surgical specialties at similar institutions that the residents' ACGME case log may not accurately reflect their operative experience. What barriers may influence this remains unclear. As the only objective measure of resident operative experience, an accurate case log is paramount in representing one's operative experience. This study aims to determine the accuracy of procedural coding by general surgical residents at a single institution. Data were collected from 2 consecutive graduating classes of surgical residents' ACGME case logs from 2008 to 2014. A total of 5799 entries from 7 residents were collected. The CPT codes entered by residents were compared to departmental billing records submitted by the attending surgeon for each procedure. Assigned CPT codes by institutional American Academy of Professional Coders certified abstract coders were considered the "gold standard." A total of 4356 (75.12%) of 5799 entries were identified in billing records. Excel 2010 and SAS 9.3 were used for analysis. In the event of multiple codes for the same patient, any match between resident codes and billing record codes was considered a "correct" entry. A 4-question survey was distributed to all current general surgical residents at our institution for feedback on coding habits, limitations to accurate coding, and opinions on ACGME case log representation of their operative experience. All 7 residents had a low percentage of correctly entered CPT codes. The overall accuracy proportion for all residents was 52.82% (range: 43.32%-60.07%). Only 1 resident showed significant improvement in accuracy during his/her training (p = 0.0043). The survey response rate was 100%. Survey results indicated that inability to find the precise code within the ACGME search interface and unfamiliarity with available CPT codes were by far the most common perceived barriers to accuracy. Survey results also indicated that most residents (74%) believe that they code accurately most of the time and agree that their case log would accurately represent their operative experience (66.6%). This is the first study to evaluate correctness of residents' ACGME case logs in general surgery. The degree of inaccuracy found here necessitates further investigation into the etiology of these discrepancies. Instruction on coding practices should also benefit the residents after graduation. Optimizing communication among attendings and residents, improving ACGME coding search interface, and implementing consistent coding practices could improve accuracy giving a more realistic view of residents' operative experience.

Balla F ，Garwe T ，Motghare P ，Stamile T ，Kim J ，Mahnken H ，Lees J ... -  《-》

ACGME Operative Case Log Accuracy Varies Among Surgical Programs.

This study evaluates the accuracy of reported the Accreditation Council for Graduate Medical Education (ACGME) operative case logs from graduated residents compared to institutional operating room electronic records (ORER). We hope this will help guide review committees and institutions develop complete, accurate resident case logs. This is a retrospective, cross-sectional study of general surgery (GS), neurosurgery (NS), and orthopedic surgery (OS) resident physicians. ACGME and ORER cases from 2009 to 2010 were analyzed and each case and current procedural terminology (CPT) code directly compared (ORER vs. ACGME). Single academic tertiary-care medical center (University of Kentucky, Lexington, KY). Eleven thousand nine hundred and twenty-three cases for 46 residents among the 3 residency programs were analyzed. There was an overall logging accuracy of 72% for ORER cases reflected in the ACGME case logs. OS residents had a higher rate of logging accuracy (OS 91%, GS 69%, NS 58%, chi-square p = 0.014) and mean annual number of cases compared to the other 2 programs (OS 452, GS 183, NS 237, ANOVA p = 0.001). NS residents had higher accuracy of CPT codes than post-graduate years 2 to 5 in other programs (p < 0.017). There was a strong positive correlation between the number of cases completed per resident and case logging accuracy, (rho = 0.769, p < 0.001) consistent for NS and GS, but not OS. This study shows only 72% of a residents' operative experience is captured in the ACGME case log across 3 surgical programs. There is significant variability among surgical programs and among post-graduate year cohorts regarding case log and CPT code accuracy. There is a strong correlation with the total number of cases performed and increasing case log accuracy. Low case log accuracy may reflect individual resident behavior instead of program operative exposure. Further studies are needed to determine if ORER may serve as a more complete assessment of the operative experience of a resident and program.

Collins C ，Dudas L ，Johnson M ，Davenport D ，Bernard A ，Beck S ，Muchow R ，Pittman T ，Talley C ... -  《-》

The ACGME Case Log System May Not Accurately Represent Operative Experience Among General Surgery Interns.

To assess if the Accreditation Council for Graduate Medical Education (ACGME) case log system accurately captures operative experience of our postgraduate year 1 (PGY-1) residents. ACGME case log information was retrospectively obtained for 5 cohorts of PGY-1 residents (2011-2015) and compared to the number of operative cases captured by an institutional automated operative case report system, Surgical Access Utility System (SAUS). SAUS automatically captures all surgical team members who are listed in the operative dictation for a given case, including interns. A paired t-test analysis was used to compare number of cases coded between the 2 systems. Academic, tertiary care referral center with a large general surgery training program. PGY-1 general surgery trainees (interns) from the years 2011-2015. Forty-nine PGY-1 general surgery residents were identified over a 5-year period. Mean operative case volume per intern, per year, captured by the automated SAUS was 176.5 ± 28.1 (SD) compared to 126.3 ± 58.0 ACGME cases logged (mean difference = 50.2 cases, p < 0.001). ACGME case log data may not accurately reflect the actual operative experience of our PGY-1 residents. If such data holds true for other general surgery training programs, the true impact of duty hour regulations on operative volume may be unclear when using the ACGME case log data. This current standard approach for using ACGME case logs as a representation of operative experience requires further scrutiny and potential revision to more accurately determine operative experience for accreditation purposes.

Naik ND ，Abbott EF ，Aho JM ，Pandian TK ，Thiels CA ，Heller SF ，Farley DR ... -  《-》

General Surgery Resident Case Logs: Do They Accurately Reflect Resident Experience?

Operative experience during residency lays the foundation for independent practice and additional specialty training following general surgery residency. The aim of this study was to examine operative experience of general surgery residents and detail the results of an intervention aimed at improving resident record keeping in the Accreditation Council for Graduate Medical Education (ACGME) case log system to better reflect their experience. Residents were asked to characterize variances in recorded operative experience identified through an audit of operative logs. Based on the results of the audit, an intervention was designed to prompt timely record keeping by residents. The intervention included education and discussion of survey audit results, weekly presentation of graphs detailing operative experience, and possible missed cases in the ACGME logs and addition of a first assistant column in morbidity and mortality (M&M) logs. The audit of case logs identified discrepancies in 24.2% of the 636 cases examined. Chief residents were significantly more accurate (95.9%) in recording operative experience in ACGME case logs, whereas 50.3% of junior resident case logs contained variances. Residents characterized discrepancies as "forgot to log" (9.6%), "staff did the case" (5.2%), "another resident did more of the case" (3.6%), "other" (3.6%), a "more advanced resident was present for the case" (1.6%), "not present for case" (0.6%), and "left for consult" (0.3%). Over the 4-week intervention period, residents logged between 72.7% and 94.0% of cases. A month following the intervention period, we observed a 13.3% increase in recorded cases compared with the intervention period. Review of first assistant case logging following inclusion of a "first assistant" column in M&M logs demonstrated a 70.5% increase in first assistant cases logged into the ACGME system compared with the same time period a year ago. Based on our results, we found that weekly displays of cases improved resident record keeping in the ACGME case log system, especially by junior residents. We believe that the addition of first assistant column on M&M lists, periodic audits reviewed at conferences, and semiannual evaluations will help junior residents more accurately report their experience during training.

Nygaard RM ，Daly SR ，Van Camp JM 《-》

Case Reporting, Competence, and Confidence: A Discrepancy in the Numbers.

The Accreditation Council for Graduate Medical Education (ACGME) continues to play an integral role in accreditation of surgical programs. The institution of case logs to demonstrate competency of graduating residents is a key component of evaluation. This study compared the number of vascular cases a surgical resident has completed according to the ACGME operative log to their operative proficiency, quality of anastomosis, operative experience, and confidence in both a simulation and operative setting. General surgery residents ranging from PGY 1 to 5 participated in a simulation laboratory in which they completed an end-to-side vascular anastomosis. Each participant was given a weighted score based on technical proficiency and anastomosis quality using a previously validated Global Rating Scale (Duran et al, 2014). These scores were correlated to the General Surgery Milestones. Participants completed preoperative and postoperative surveys assessing resident operative experience using the 4-level Zwisch scale (DaRosa et al., 2013), confidence with vascular procedures and confidence performing simulated anastomoses. Confidence was assessed on a scale from 1 to 9 (not confident to extremely confident). Case logs were recorded for each participant. An IRB approved questionnaire was distributed to assess preoperative and postoperative roles of both the resident physician and faculty, with a defined goal. Univariate and multivariate analysis was performed. Twenty-one general surgery residents were evaluated in the simulation laboratory and 8 residents were assessed intraoperatively. The residents were evenly distributed throughout clinical years. Groups of residents were divided into quartiles based upon the number of vascular cases recorded in the ACGME database. No correlation was found between number of cases, Milestones score and the weighted score (p = 0.94). No statistical significance was found between confidence and quality of anastomosis (p = 0.1). Resident operative experience per the Zwisch scale was categorized most commonly as "Smart Help" by both the trainee and attending surgeon, despite mean resident confidence ratings of 6.67 (± 1.61) with vascular procedures. ACGME case logs, which are utilized to assess readiness for completion of general surgery residency, may not be indicative of a resident's operative competency and technical proficiency. Confidence is not correlated with technical ability. Faculty and resident insight as to their role in a procedure differ, as faculty feel that they are providing less help than the resident perceives. Careful examination of resident operative technique is the best measure of competency.

Shah D ，Haisch CE ，Noland SL 《-》