Are There Gender-based Differences in Language in Letters of Recommendation to an Orthopaedic Surgery Residency Program?

Kobayashi AN ，Sterling RS ，Tackett SA ，Chee BW ，Laporte DM ，Humbyrd CJ ... -  《-》

How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review.

Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation. In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation? PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race. Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race. Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident. We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

Burkhart RJ ，Lavu MS ，Hecht Ii CJ ，Ina JG ，Gillespie RJ ，Liu RW ... -  《-》

Don't Judge a Letter by its Title: Linguistic Analysis of Letters of Recommendation by Author's Academic Rank.

This study analyzed the linguistic differences in letters of recommendation (LORs) for general surgery residency applicants written by authors of various academic ranks. Given that many general surgery residency programs require a LOR from the Chair of surgery, this study also examined whether LORs written by the Chair demonstrate linguistic differences to support this practice. A single institution, retrospective review analyzed LORs from two application cycles of general surgery residency applicants who were selected for interview at a large academic institution. Word count (WC) and linguistic characteristics of LORs were analyzed with a previously developed institution-specific dictionary using the Linguistic Inquiry and Word Count software (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). WC and linguistic characteristics of LORs reported as frequencies of terms within twenty-four categories were examined based on the letter authors' academic rank. Further examination compared LORs written by a Chair of surgery with those written by non-Chairs. A single large, Midwestern academic general surgery residency program PARTICIPANTS: Four hundred and sixty-five letters of recommendation received during two interview cycles were included for analysis. A total of 465 LORs written by assistant (n = 82), associate (n = 94), and full professors (n = 289) were included in the study. No statistically significant difference was noted in the WC of LORs based on the letter writers' academic ranks (p = 0.95). Assistant professors utilized grindstone, communal, and technical skill terms with higher frequencies compared to associate professors and full professors. LORs written by assistant professors demonstrated the highest authentic variable score followed by associate professors then full professors (4.94, 3.92, 3.28, p < 0.01). LORs written by Chairs (n = 128) had lower authentic variable scores compared to LORs written by non-Chairs (n = 337; 2.71 vs. 3.91, p = 0.001). Only 50 (39%) LORs written by Chairs indicated working directly with the applicant, and sub-group analysis demonstrated a higher authentic variable score in this group compared with LORs written by Chairs who did not indicate having worked directly with the applicant (3.51 vs. 2.5, p = 0.01). Linguistic analysis of LORs for general surgery residency applicants demonstrated minor yet statistically significant differences based on the author's academic rank. If applicants can obtain linguistically similar LORs from surgeons of any academic rank, but less authentic LORs from writers with higher academic ranks, these LORs may be less valuable for the residency programs when evaluating applicants. Based on the subgroup analysis, less than 40% of Chair LORs indicated that the Chair worked directly with the applicant, calling into question the utility of the Chair LORs as meaningful evaluation of applicants. Further study to compare LORs of applicants selected and not selected for interview may add additional insight into linguistic differences in LORs written by authors of different academic ranks.

Han AY ，French JC ，Tu C ，Obiri-Yeboah D ，Lipman JM ，Prabhu AS ... -  《-》

Do gender and racial differences exist in letters of recommendation for obstetrics and gynecology residency applicants?

Studies in several fields demonstrate gender and racial differences in descriptions of applicants in letters of recommendation. However, gender and racial biases in obstetrics and gynecology residency letters of recommendation are understudied. This study aimed to determine whether there are differences in the letters of recommendation written for medical students applying for obstetrics and gynecology residency based on applicant gender and underrepresented in medicine status. We analyzed all letters of recommendation submitted to a single obstetrics and gynecology residency program during the 2019-2020 application cycle using a computerized text analysis software to evaluate the presence of 25 word categories. Multivariable regression was used to compare differences in the frequency of word categories by gender and underrepresented in medicine status. Concurrently, we performed a qualitative content analysis of letters from a random sample of applicants balanced for gender and underrepresented in medicine status and explored emerging themes until thematic saturation was reached. We analyzed 3060 letters written for 834 applicants, 721 (87%) of whom were women and 198 (24%) of whom were underrepresented in medicine. Men authored 1605 letters (53%). There was no difference in word categories used in letters written for men and women applicants on multivariable analysis. Similarly, there was no word category difference between letters written for applicants who were underrepresented in medicine compared with White and Asian applicants. However, women letter writers used more communal (relationship-oriented) words compared with men letter writers (P<.001). Moreover, on the qualitative analysis of letters from a random sample of 30 applicants, comments about personality traits were frequent in all letters. Comments on surgical skills, work ethic, and leadership were found more in letters for White and Asian applicants, and phrases that were doubt raisers were found more in letters written for underrepresented medicine applicants. On linguistic analysis, letters of recommendation written for obstetrics and gynecology residency applicants were similar overall compared with applicant race and gender.

Brown O ，Mou T ，Lim SI ，Jones S ，Sade S ，Kwasny MJ ，Mueller MG ，Kenton K ... -  《-》

Gender and Letters of Recommendation: A Linguistic Comparison of the Impact of Gender on General Surgery Residency Applicants(✰).

We investigated letters of recommendation for general surgery residency applicants to determine if any gender-based disparities exist. A dictionary of over 400 terms describing applicants and 24 unique categories into which these terms were classified was created. Word count and language comparisons were performed using linguistic analysis software to assess for differences in applicant characterization, letter length, and writing style between male and female applicants and letter writers. A large, Midwest, academic general surgery residency program. Five hundred and fifty-nine letters of recommendation received during the 2015 and 2016 interview cycles were selected for analysis. Average word count was approximately equal for male and female applicants (503 vs 508, respectively). Female writers wrote longer letters (mean word count 545.5 vs 497.1, p = 0.028). "Standout" terms were more likely to be used to describe female applicants. Otherwise no statistically significant differences in applicant characterization were discovered. Letters of recommendation for general surgery are written using similar descriptive terms and lengths for male and female applicants. This suggests that there is no specific gender disadvantage with regard to letters of recommendation when applying for general surgery residency.

French JC ，Zolin SJ ，Lampert E ，Aiello A ，Bencsath KP ，Ritter KA ，Strong AT ，Lipman JM ，Valente MA ，Prabhu AS ... -  《-》