Title: Multi-institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark
Authors: Sandra L. Petersen, Evelyn S. Erenrich, Dovev L. Levine, Jim Vigoreaux, Krista Gile
First author institution: University of Massachusetts Amherst
Journal: PLOS One (10) e0206570
As today’s graduate students become tomorrow’s professors and leaders of their fields, who these graduate students are has long term consequences. Yet of the students who enroll in a STEM PhD program, only 59% of them actually earn their PhD, meaning the current admissions practices and support structures in these programs are not adequate. While doctoral training has changed over the years from classroom learning and individual projects to problem-based learning and collaborative, interdisciplinary research, the doctoral admissions process remains largely unchanged. As discussed in a previous post, the GRE is still a important part of the admissions process, even though it does not measure skills such as problem solving and creativity, or personality traits needed to be successful in graduate school. Prior work has found that GRE scores and degree completion in life sciences and physical sciences are weakly correlated or not correlated at all. Further, a study of biomedical graduate students found no differences in GRE scores of those who completed their doctoral program and those who did not. Given the large role the GRE plays in determining who gets admitted to a PhD program, one would expect the GRE to be predictive of degree completion. Today’s paper extends this work to a broad range of STEM fields and asks whether there may be gender differences in how well the GRE can predict degree completion.
The researchers collected data from 1805 students (42.5% women) who were accepted to a STEM PhD program (biological sciences, physical sciences, chemical sciences, computer and information sciences, engineering, geosciences, or mathematical sciences) at one of the four state flagship research universities included in the study between 2000 and 2005. The researchers used data from this time period to ensure that all the students in the sample would have had sufficient time to complete their PhD program. Specifically, the researchers collected the verbal and quantitative GRE scores from each student, the name of their doctoral program, the year they enrolled, the year they completed or withdrew from the program, their gender, and their race/ethnicity. Since the data came from between 2000-2005, the GRE scores used the pre-2011 scoring system where the minimum score was 200 and the top score is 800.
So what did they find? First, they found that on average, men and women perform equally well on the verbal portion of the GRE but men significantly outperformed women on the quantitative portion at all four of the institutions studied (figure 1).
However, there was no statistically significant difference in the completion rate, the time to degree, the number of years enrolled in the program before withdrawing, and the first year withdrawal rate between men and women. On average, both men and women completed their degrees in just under 6 years and spent 3 years in their program before withdrawing if they did not complete their degree.
Next the researchers used logistic regression to investigate any interactions between gender, institutions, and GRE scores. Only gender and quantitative GRE scores showed a significant interaction. When the researchers compared women who completed their doctoral program and those who did not, they found no statistical difference in their quantitative GRE scores. However, when the researchers compared men who completed their doctoral program with men who did not, they found that men who completed their programs had lower quantitative GRE scores than those who did not complete their programs. No, that isn’t a typo. Men who did not complete their program scored nearly 25 points higher on the quantitative GRE than men who did complete their program.
To understand this counterintuitive finding, the researchers split the GRE scores into quartiles. Yet when they did this, the finding only became more counterintuitive: of the men who enrolled in graduate school, men who scored in the 4th quartile (meaning they had the lowest 25% of quantitative GRE scores among enrolled graduate students) completed their degrees at a rate that significantly differed from men who were in any of the other quartiles. Interestingly, men in the 4th quartile completed their degree at a rate 20% higher than men who scored in the 1st quartile (and hence had the highest quantitative GRE scores), 74% to 56.2% (figure 2). However, this trend was not present among the women or when using verbal GRE scores instead of quantitative GRE scores.
Next, the researchers decided to look at only engineering doctoral programs to see if the same trends were present. Perhaps looking at a program that is mathematics intensive would eliminate the trend if the trend were a result of more PhD completing men being in a less mathematics intense doctoral program (and hence assumed to have lower quantitative GRE scores) than in mathematics intense programs. But the trend didn’t disappear when looking at engineering doctoral programs. Men who completed an engineering doctoral program still scored about 20 points lower than men who didn’t complete the engineering doctoral program they had been enrolled in while the trend was still not present for women. When using the quartile analysis again, men in the 4th quartile still had a higher completion rate than any of the other quartiles (figure 3).
While the results of this study are surprising, the negative association between quantitative GRE scores and degree completion has been observed before with life sciences doctoral students and in a study of biology, chemistry, and psychology students; however, none of these studies separated by gender and it wasn’t clear that the difference was statistically significant. Other studies have found that quantitative GRE scores did not differ between students who completed their doctoral program and those who didn’t, but these studies did not compare men and women separately.
So what can we take from this paper? First, the GRE does not appear to be predictive of completing a STEM PhD. Students who complete a doctoral program do no better than students who do not complete a doctoral program and in the case of the quantitative GRE, men who score higher are less likely to complete their degree than men who score lower. In addition to evidence that the GRE is biased against women and racial minorities, the results of this study further question the role of the GRE in the graduate admissions process.
Figures used under Creative Commons Attribution 4.0 International License.
I am a physics graduate student at Michigan State University and the founder of PERbites. I’m interested in applying machine learning to analyze educational datasets and how students use computation in physics courses.