Title: A comparison of student evaluations of instruction vs students’ conceptual learning gains
Authors: LaDoris J. Lee, Manju E. Connolly, Melissa H. Dancy, Charles R. Henderson, and Warren M. Christensen
First author’s institution: Northeastern Illinois University
Journal: American Journal of Physics 86 (7) 2018 (Closed Access)
If you have ever taken a university course in the United States, you have probably been asked to fill out a course evaluation form, often called Student Evaluations of Instruction (SEI). An example of an SEI form can be found here. These SEIs are used by many American universities to make decisions about faculty retention and promotion. While these evaluations may be useful for obtaining feedback about the course, it is unclear whether these evaluations serve as good measures of arguably the most important outcome of the class, student learning. Previous research has shown positive correlations between SEI scores and student learning but the study’s results have been questioned due to the use of small sample sizes. Other studies have shown no correlation at all. A problem with these types of studies in general however is how student learning is measured. Typically, student learning is measured by final exam performance but final exams are not standardized across institutions or even the same courses at an institution. The goal of today’s paper is to see if a correlation exists between SEI scores and student learning when that learning is measured by a conceptual inventory, which is a test that has been extensively researched and validated. As there is a somewhat common belief that instructors may be able to achieve better reviews by giving students better grades, the researchers also wanted to see if course grades are correlated with SEI scores.
To get their data for this study, the researchers recruited participants from the New Faculty Workshop who were interested in implementing active learning or interactive engagement techniques from the workshop and would be teaching an introductory physics course over the next five semesters. The researchers selected 15 faculty members for this study.
Of these fifteen instructors, 10 taught introductory mechanics courses (motion, forces, energy) while 9 taught introductory electricity and magnetism (some instructors taught both types of classes). At the end of each semester, the participants were asked to provide the average pre- and post-test scores on the conceptual inventory they selected to use (FCI or FMCE for mechanics, BEMA or CSEM for electricity and magnetism), student final grades, and their SEIs. From the conceptual inventories, the researchers calculated the normalized gain for the class as a whole. Since SEIs are not standard across institutions, the researchers selected the item that best represented the students’ overall rating of the instructor. Examples are shown in figure 1.
Since most of the questions on the SEIs from the various institutions used a 5 point scale (strongly disagree to strongly agree), the researchers adjusted all responses to questions not on a 5 point scale to a 5 point scale. Since this did not occur for many questions, the researchers claimed this had a minimal effect on the results. Finally, the researchers used a weighted average to calculate the average final grade of the students in each course.
So what did they find? Since the FCI was used by all but one of the mechanics instructors while the BEMA and CSEM were used about equally among the electricity and magnetism instructors, the researchers focused their analysis around the FCI to avoid low number statistics since the normalized gains on different conceptual inventories are not comparable. This resulted in 17 courses being used in the analysis. The researchers found the correlation between the normalized gain and the SEI score on the 5 point scale to be R2=0.01544, which means there is no correlation (R2<0.25 is considered weak). The researchers also found no correlation between the SEI scores and the CSEM or BEMA scores, however only 6 and 4 classes were analyzed. Nevertheless, these results support no correlation between student learning as measured by a conceptual inventory and SEI ratings.
Next, the researchers looked at the correlation between the course grades and the SEI scores. Since all the instructors submitted grade information, 37 courses were included in the analysis. The researchers found the correlation between final course grades and SEI scores to be R2=0.00158 meaning there was no correlation between SEI scores and course grades.
As the authors note, the results have some limitations. First, the SEI is not standard across the participants but as similar questions are asked, they believed them to be comparable. Second, the results are taken at the course level rather than the student level. Perhaps students who learn more as measured by the normalized gains tend to give better SEI scores than students with lower normalized gains do or students who earn better grades do give higher SEI scores. Since researchers only had data on the class level, they could not investigate these possibilities. Additionally, as we discussed in a previous bite, the normalized gain itself can be a biased measure. Finally, conceptual learning is only one positive outcome so high SEI scores may be correlated with other positive outcomes such as improved student motivation, improved student problem solving, persistence in the student’s major, etc. Thus, this study cannot make any claims about the usefulness of SEIs in general.
So what can we takeaway from this study? First, Student Evaluations of Instruction (SEI) are not correlated with student learning as measured by the FCI when measured on the class level, suggesting that SEIs may not be very useful in assessing arguably the most important outcome of a class, student learning. Second, final grades are not correlated with SEI scores. Since this is a correlation rather than a causation, the authors cannot dispel the claim that giving students higher grades will result in better SEI ratings, only that it may not be the case.
Heading image courtesy of https://cft.vanderbilt.edu/courseevals/ used under CC BY-ND 4.0 license.
I am a postdoc in education data science at the University of Michigan and the founder of PERbites. I’m interested in applying data science techniques to analyze educational datasets and improve higher education for all students