How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review.

Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation. In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation? PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race. Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race. Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident. We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

Burkhart RJ ，Lavu MS ，Hecht Ii CJ ，Ina JG ，Gillespie RJ ，Liu RW ... -  《-》

More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.

The American Orthopaedic Association (AOA) released the standardized letter of recommendation (SLOR) form to provide standardized information to evaluators of orthopaedic residency applicants. The SLOR associates numerical data to an applicant's letter of recommendation. However, it remains unclear whether the new letter form effectively distinguishes among orthopaedic applicants, for whom letters are perceived to suffer from "grade inflation." In addition, it is unknown whether letters from more experienced faculty members differ in important ways from those written by less experienced faculty. (1) What proportion of SLOR recipients were rated in the top 10th percentile and top one-third of the applicant pool? (2) Did letters from program leaders (program directors and department chairs) demonstrate lower aggregate SLOR scores compared with letters written by other faculty members? (3) Did letters from away rotation program leaders demonstrate lower aggregate SLOR scores compared with letters written by faculty at the applicant's home institution? This was a retrospective, single institution study examining 559 applications from the 2018 orthopaedic match. Inclusion criteria were all applications submitted to this residency. Exclusion criteria included all letters without an associated SLOR. In all, 1852 letters were received; of these, 26% (476) were excluded, and 74% (1376) were analyzed for SLOR data. We excluded 12% (169 of 1376) of letters that did not include a final summative score. Program leaders were defined as orthopaedic chairs and program directors. Away rotation letters were defined as letters written by faculty during an applicant's away rotation. Our study questions were answered accounting for each subcategory on the SLOR (scale 1-10) and the final ranking (scale 1-5) to form an aggregated score from the SLOR form for each letter. All SLOR questions were included in the creation of these scores. Correlations between program leaders and other faculty letter writers were assessed using a chi-square test. We considered a 1-point difference on 5-point scales to be a clinically important difference and a 2-point difference on 10-point scales to be clinically important. We found that 36% (437 of 1207) of the letters we reviewed indicated the candidate was in the top 10th percentile of all applicants evaluated, and 51% (619 of 1207) of the letters we reviewed indicated the candidate was in the top one-third of all applicants evaluated. We found no clinically important difference between program leaders and other faculty members in terms of summative scores on the SLOR (1.9 ± 0.7 versus 1.7 ± 0.7, mean difference -0.2 [95% CI -0.3 to 0.1]; p < 0.001). We also found no clinically important difference between home program letter writers and away program letter writers in terms of the mean summative scores (1.9 ± 0.7 versus 1.7 ± 0.7, mean difference 0.2; p < 0.001). In light of these discoveries, programs should examine the data obtained from SLOR forms carefully. SLOR scores skew very positively, which may benefit weaker applicants and harm stronger applicants. Program leaders give summative scores that do not differ substantially from junior faculty, suggesting there is no important difference in grade inflation between these faculty types, and as such, there is no strong need to adjust scores by faculty level. Likewise, away rotation letter writers' summative scores were not substantially different from those of home institution letters writers, indicating that there is no need to adjust scores between these groups either. Based on these findings, we should interpret letters with the understanding that overall there is substantial grade inflation. However, while weight used to be given to letters written by senior faculty members and those obtained on away rotations, we should now examine them equally, rather than trying to adjust them for overly high or low scores. Level III, therapeutic study.

Pacana MJ ，Thier ZT ，Jackson JB 3rd ，Koon DE Jr ，Grabowski G ... -  《-》

Are There Gender-based Differences in Language in Letters of Recommendation to an Orthopaedic Surgery Residency Program?

Kobayashi AN ，Sterling RS ，Tackett SA ，Chee BW ，Laporte DM ，Humbyrd CJ ... -  《-》

What Are the Trends in Research Publication Misrepresentation Among Orthopaedic Residency and Fellowship Applicants From 1996 to 2019? A Systematic Review.

Because research experience is increasingly important in ranking orthopaedic residency and fellowship applicants, determining the accuracy of candidates reporting their scholarly activity is essential. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies. In this systematic review, we asked: (1) What percentage of research publications are misrepresented among orthopaedic residency and fellowship applicants? (2) What percentage of applications contain one or more example of academic misrepresentation? (3) Is research misrepresentation associated with any individual applicant characteristics? (4) What is the publication status of articles listed by applicants as having been submitted to journals? A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. PubMed, EBSCOhost, Medline, and Google Scholar electronic databases were searched on March 10, 2022, to identify all studies that evaluated research misrepresentation in orthopaedic residency and fellowship applications between January 1, 1995, and March 1, 2022. Articles were included if full-text articles in English were available and the study reported on research misrepresentation among orthopaedic residency or fellowship applicants. Studies investigating nonorthopaedic publications, systematic reviews, case studies, duplicate studies among databases, and gray literature were excluded. Two reviewers independently evaluated the quality of included studies using the Methodological Index for Nonrandomized Studies (MINORS) tool. This is a validated assessment tool that grades noncomparative studies from 0 to 16 and studies with control groups from 0 to 24, based on eight criteria related to study design, outcomes assessed, and follow-up. All included articles were noncomparative studies, so the maximum score here was 16, with higher scores indicating better study quality. The mean MINORS score was 13 ± 1 in the studies we included. The final analysis included 10 studies with 5119 applicants. Eight studies evaluated orthopaedic residency applicants and two evaluated fellowship applicants. The applicant classes ranged from 1996 to 2019. Research misrepresentation was defined among studies as nonauthorship of an existing article, claimed authorship of a nonexistent article, or incorrect listing of authorship order for an existing article. Each study's findings and definition of research misrepresentation were considered to allow for a discussion of overall trends. The percentage of misrepresentation was further broken down by the misrepresentation type. Applicant characteristics and destination of submitted articles were also evaluated. Given the potential overlap between applicants among the studies, no pooled analysis was conducted, and results are presented as a narrative summary. The percentage of overall publication misrepresentation was estimated to range between 1% (13 of 1100) and 21% (27 of 131), with more-recent studies reporting a lower proportion of overall articles misrepresented. Most studies we found claimed that authorship of a nonexistent article was the most common type of misrepresentation. Nonauthorship of an existing article and incorrect authorship order were less common. The percentage of applications with at least one misrepresentation was approximately 20% between 1998 and 2017. Most studies found no applicant characteristics, such as match outcomes, demographic markers, or academic records, that were consistently associated with a higher odds of the candidate misrepresenting his or her research credentials. Finally, approximately half of the articles listed as submitted to journals went on to publication, with one-third going to a different journal with a lower Impact Factor. Our systematic review found that the percentage of overall publication misrepresentations among orthopaedic residency and fellowship applicants has generally been low over the past 20 years. However, approximately one-fifth of applications had at least one research misrepresentation, with 2% having multiple misrepresentations on reported publications. There were no consistent applicant characteristics associated with higher odds of research misrepresentation. Additionally, most of the articles listed as submitted to journals for publication were ultimately published. Although the decrease in overall publication misrepresentation is encouraging, our finding that one-fifth of applicants have research misrepresentation is a cause for concern. In light of a continually evolving application process, orthopaedic residency and fellowship programs must ensure there is integrity related to information that is self-reported by applicants. These findings also serve to encourage faculty members involved in the application screening and decision process to limit biases related to applicant demographics perceived to be associated with a high odds of misrepresentation. Furthermore, governing agencies and program leadership should evaluate methods of verifying unpublished work and provide opportunities for applicants to give publication updates throughout the application cycle.

Burkhart RJ ，Hecht CJ 2nd ，Karimi AH ，Acuña AJ ，Kamath AF ... -  《-》

Use of Standardized Letters of Recommendation for Orthopaedic Surgery Residency Applications: A Single-Institution Retrospective Review.

Samade R ，Balch Samora J ，Scharschmidt TJ ，Goyal KS ... -  《-》