Novel Measures Reveal Subtle Gender Bias in Academic Job Recommendations

Linguistic analysis of 2,206 letters of recommendation compared gender bias in two disciplines differing in women's representation: experimental particle physics (EPP, 60% female). Standard lexical measures (e.g., communal, agentic, and standout) did not show bias against women in either discipline. On the contrary, in letters about women, female physicists used more positive-affect words, while male physicists used fewer negative-affect words as well as more references to hard work/effort. Neither discipline showed gender differences in the rank of letter writers, but social scientists wrote longer letters about women and wrote more often for candidates of their own gender. However, standard lexical measures assess only overt expressions of bias and may miss more subtle gendered language. We therefore developed a novel, open-ended measure of gendered usages. In striking contrast to conventional measures, our open-ended analysis uncovered troubling gender disparities. Positive references (e.g., to talent, innovation, creativity, etc.) appeared more often in letters about men in EPP and about women in social (although female EPP candidates were more likely to be characterized as brilliant). Two of the largest gender disparities were for references to physicist in EPP and science in social science. These terms were used in more letters for men in both disciplines, possibly indicating unconscious gender stereotypes, even in a majority-female discipline. We conclude that future studies of linguistic bias should include open-ended measures, and that policies to correct gender imbalances in physics should raise awareness of subtleties of potential bias in letters of recommendation.