Romy van der Lee and Naomi Ellemers (PNAS December 22, 2015)

With interest we read the response of Volker and Steenbeek (1) to van der Lee and Ellemers (2), reporting gender disparities in grant funding success for three cohorts (2010–2013) of early-career (Veni) researchers in the Netherlands Organization for Scientific Research (NWO). Volker and Steenbeek (1) argue that there is no evidence of gender bias because the overall gender effect “borders on significance” (P = 0.045) and would be prone to Simpson’s paradox (3). They present additional data suggesting that overall gender effects may disappear when correcting for alpha inflation or other control variables. The analyses presented by Volker and Steenbeek (1) pertain to other cohorts (2006–2013) and grant schemes and do not address the different evaluation phases or criteria we examined. Instead, they compare overall awarding rates for different subject areas within the social sciences, which were not coded in the data we had access to.

The aim of our research (2) was to examine potential explanations for the overall effect suggesting an uneven distribution in the awarding rates of men and women. Because awarding rates differ strongly between disciplines, we tested for gender disparities within each of the scientific disciplines we could distinguish in our data. Correcting for discipline reduces the effect of applicant gender, so that it is no longer significant across disciplines, which seems to be in line with Simpson’s paradox. However, when taking into account both scientific discipline and applicant gender, we find a significant interaction between them [Wald(8) = 17.574, P = 0.025] as well as a significant main effect of gender. This justifies our examination of gender disparities in awarding rates per discipline. Results revealed (table S1 in ref. 2) that the awarding rates of women were significantly lower than those of men in the disciplines with a high proportion of female applicants (and overall relatively low success rates). This gender difference within scientific disciplines cannot be explained from Simpson’s paradox. Simpson’s paradox also cannot account for the observation that in every step of the review procedure women are less likely than men to be prioritized. Nor does it explain why the differences in awarding decisions can only be traced to gender differences in “quality of researcher” ratings, because male and female applicants received equal ratings for the quality of their proposals and knowledge utilization. Our conclusion is based on these multiple indicators of gender disparities revealed in the grant review procedure, as well as on the observation that language use in instructional and evaluation materials favors male over female applicants.

In response to our report, NWO has announced its intent to invest in evidence-based policies to optimize the quality of their grant review procedures. Thus, no public money will be spent on changes in evaluative procedures unless these have been demonstrated to contribute to a more inclusive academic climate that provides equal opportunities for all scientists.

See http://www.pnas.org/content/112/51/E7038.full