Title: Value added or misattributed? A multi-institution study on the educational benefit of labs for reinforcing physics content
Authors: N. G. Holmes, Jack Olsen, James L. Thomas, Carl E. Wieman
First author’s institution: Cornell University
Journal: Physical Review Physics Education Research 13 010129 (2017)
Laboratory experiments are a standard part of both high school and university physics courses across the world. While giving students experience with laboratory equipment and experimentation, it is often assumed that labs reinforce course content beyond what is already provided in lecture, recitation, and homework. However, that assumption has remained just that, an assumption. The goal of today’s paper is determine if the claim is true: do physics laboratories in fact reinforce course content?
Early in the twentieth century, physics experiments were used as opportunities for students to inductively discover physical phenomena. However, as enrollment in physics labs increased and course requirements became more rigid, lab activities became more structured and now often present students with a well-defined experiment protocol. These labs take a step-by-step approach where students are often tasked with verifying or confirming an already known result (for example, verifying that the acceleration due to gravity is 9.8 m/s2 by dropping objects or by using a pendulum). By ensuring that students get the right answer, these types of labs are assumed to help students develop more conceptual physics knowledge. Even in reformed lab courses where students are often given more open-ended tasks, some type of conceptual gain is still expected by doing the experiments.
To see if there were learning gains as a result of doing lab activities, the researchers collected exam scores from 9 courses at 3 different institutions. The courses included both algebra- and calculus-based physics courses as well as introductory courses primarily taken by engineers and introductory courses primarily taken by health science majors. The universities were diverse in both their geographical location and the diversity of their students. At each of these universities, the physics laboratory was offered as a separate, optional course.
Since students with higher levels of incoming physics knowledge tended to take the laboratories, the researchers couldn’t directly compare the students who enrolled in the optional labs to those who didn’t. Instead, the researchers used the final exam scores of each student and separated the exam into questions related to lab material and questions not related to lab material. If labs were reinforcing content covered in other areas of the course, then students who took the labs should do better on questions related to lab material than questions not related to lab material while students not in the labs should show no difference. The researchers then averaged the difference in performance on lab-related questions and non-lab-related questions for students who enrolled in the lab and for students who weren’t enrolled in the lab. The researchers then calculated what they called the “mean lab benefit” which was the difference in the average differences between lab and non-lab questions for students who enrolled and didn’t enroll in the lab courses. If labs do reinforce course content, the mean lab benefit will be positive while if labs do not reinforce course content, the mean lab benefit will be zero.
So what did they find? For each of the nine courses, the researchers found the mean lab benefit was not statistically different from zero, meaning the labs weren’t providing any benefit beyond what was already being provided in lecture, recitation, and homework. The results for individual courses are shown in figure 1.
As lab activities often focus around concepts rather than quantitative problem solving, the researchers worried the mean lab benefits of zero may be a result of quantitative problems on the final exam. To see if this was the case, the researchers recalculated the mean lab benefit using only the conceptual problems on the exams. But even with using only conceptual problems, the mean lab benefit was still not statistically different than zero in any of the courses (figure 2).
Finally, the researchers worried that they were looking on too long of a time scale. Maybe the lab activities only provide benefits in the short term and those benefits had worn off by the final exam. Instead of using the final exams in each course, the researchers performed the same analysis on each of the midterm exams in the course, looking first at all of the questions on the exams and then only using the conceptual questions. While some of the exams showed a positive mean lab benefit (figure 3) most of the exams showed mean lab benefits consistent with zero and in some cases, negative mean lab benefits! Thus even when looking on short time scales, there didn’t appear to be any consistent benefits of enrolling in the lab course.
While the results may seem surprising at first, the researchers suggest that the lack of any additional “knowledge” gained from the lab activities may be a result of the structure of the activities themselves. The researchers suggest that students often have to follow a specific set of instructions and data analysis procedures while learning to use unfamiliar lab equipment, leaving little time for the students to focus on the physics concepts. The researchers suggest including some degree of open-endedness in lab activities and focusing on the process rather than the result may allow students to focus more on the relevant concepts rather than getting the “right” answer.
So what can we take away from this paper? The key takeaway is that lab activities seem to be redundant: whether measured on midterm exams or final exams, all questions or just conceptual questions, the researchers were unable to find a consistent benefit of taking the lab course compared to just the lecture course. This result does not imply that lab activities are not without merit however, as lab activities can still teach important experimental skills. But if the goal of the lab is to reinforce physics knowledge, it seems that the students’ time would be better spent elsewhere.
Figures used under Creative Commons Attribution 4.0 International license.
I am a postdoc in education data science at the University of Michigan and the founder of PERbites. I’m interested in applying data science techniques to analyze educational datasets and improve higher education for all students