Authors: Daniel L. Belavy, Patrick J. Owen, Patricia M. Livingston
First author’s institution: Deakin University
Journal: PLoS One 15(8) e0236327 (2020) [open access]
It’s graduate admissions season so departments across the United States will soon be looking over applications to select the next round of graduate students for their programs. Typically, applicants submit test scores (this year is an exception due to testing centers being closed), their undergraduate transcript, recommendation letters from faculty or other advisors, and written statements. Admissions committees then use this information to select some applicants for their program.
Which applicants are admitted often depends on the priorities of the department, such as funding, “fit”, or diversity. Regardless of the department’s priorities, there is often an assumption that departments are looking for the “best” applicants who will do well in their programs. After all, nearly 1 in 3 research publications are from doctoral students and publications play a large role in university rankings and hence, reputation.
Today’s paper out of an Australian university suggests that we may be looking at admissions incorrectly. If we want successful students, we should be looking at the research environment they will be working in rather than just the applicant’s credentials.
To reach their conclusion, the researchers collected 324 PhD applications over 4 years to a health program at an Australian university. The applications included information about the applicant’s undergraduate education, previous research experience, publications, and whether their potential advisor’s research focus was in one of the university’s priority areas.
The review panel then scored applicants on these measures as well as whether the applicant’s proposed research aligned with university’s goals and the experience of their applicant’s potential supervisory team. Experience was based on the prior PhD completions in the group, student progress, grants won, and publications.
Four years after the data collection ended, the researchers linked publication info to each applicant who had enrolled in the program (198 or 61% of the applicants who applied). This included the number of publications, the number of citations, and the impact factor of the journals these publications were in. The researchers also included information about whether the applicant completed their program, was still enrolled, or withdrew.
From this data, the researchers then performed statistical tests and regression to determine which variables had an effect on publications and degree completion.
Results: publications, citations, and impact factors
When looking at their results, the researchers found that having an advisor in one of the university’s priority areas, having an excellent supervisory team (as assessed by the review panel), earning a scholarship, and academic merit were associated with more publications and those publications appearing in journals with higher impact factors (which is often assumed to mean higher quality journals). In contrast, undergraduate performance, previous research training, and research topic were not associated with more publications or publishing in journals with higher impact factors.
In terms of citations and citations per publication, the results were similar. The only difference is that academic merit was not associated with more citations or citations per publication.
Results: PhD completion
As this article’s title suggests, factors not related to the student’s prior performance had more of an impact on PhD completion. Specifically, students were two times more likely to withdraw from their program when their supervisory team did not receive the highest rating from the review panel.
Conclusions & Implications
This study was relatively small in size and only looked at one university so the conclusions may not generalize. Nevertheless, the results suggest that research environments are a bigger predictor of student outcomes than students’ previous academic outcomes or research training. Similar results such as the finding that the supervisory team directly influences the career trajectory and number of citations for trainees and employment chances of PhD graduates are affected more by department standing than individual achievements have been found before, suggesting that research environments playing an outsized role in outcomes is not unreasonable.
Second, like many PhD studies, this study has a range restriction problem in that the sample is restricted to only those who were admitted and enrolled and hence, the outcome should be taken in the appropriate context. That is, among those who were admitted to this PhD program, the research environment played a larger role in outcomes than individual academic achievement or research training. The authors did address this caveat in their discussion, noting that once the minimum academic ability for admission is met, any additional ability does not influence whether the applicant completes their degree.
For departments who will be selecting their next class over the next few weeks, really think about how your faculty’s lab culture and environment may influence who will be successful in your program. Do your students really need to have high test scores to succeed in Professor X’s lab and will any amount of preparation help a student who chooses to work with Professor Y? After all, most graduate students who do not complete their programs do so for non-academic reasons (e.g. lack of funding, department culture, inadequate advising).
I am a postdoc in education data science at the University of Michigan and the founder of PERbites. I’m interested in applying data science techniques to analyze educational datasets and improve higher education for all students