Longitudinal Study of Online Statics Homework as a Method to Improve Learning

Abstract

Background – Score improvements and retention are important metrics in studying the efficacy of any learning intervention. Research studies have shown a positive correlation between time spent completing online homework assignments and student performance on final exams. Yet these studies are not only rare in the field of engineering; they are generally not longitudinal in nature.

Method –This study design was three-fold. Firstly, it employed a quasi-experimental study using an online homework system, MasteringEngineering, in the statics course. Students were compared in two statics courses: one course prepared by written homework and another course prepared by online homework. Students received the same final exam, making direct comparisons feasible. Secondly, data was gathered from a traditional mechanics of material course where no online homework system was employed. Again, the same final exam was given for direct comparison. Thirdly, an item analysis test was run using CATS, Concept Assessment Tool for Statics, courtesy of Dr. Paul S. Steif from Carnegie Mellon University.

Results – Students who used the online system showed an improvement of 0.7 (± 0.2) in effect size on the final exam when compared to written homework. Students who used the online system scored 79% (SD=8%; N=69) on average on the final exam. In comparison, students prepared via written homework scored 70% (SD=16%; N=64) on average on the same final exam. These results held for the subsequent mechanics course where students previously prepared via online statics homework scored 79% (SD=8% N=66), and students prepared by print homework scored 63% (SD=18%; N=79) on the same final. Furthermore, the mechanics course was taught by an independent instructor from the statics course, removing potential bias. All results were statistically significant. Independent of the historical analysis, another test was run on students in the online homework statics course. All were given pre / post tests to measure understanding on nine concepts, with friction being the only topic yielding an inadequate learning gain (difficulty change of friction 0.04 ±0.28). In conclusion, the online homework intervention showed an improvement of 0.7 effect size when all other elements in the course remained unchanged.