Using data-mining to enable early interventions in introductory engineering courses
Researchers have long sought to understand the effect of study habits on academic achievement. Study habit, skill, and attitude inventories may be useful for predicting academic performance. However, researchers have questioned the reliability of such methods as they rely on students' self-reports of study habits. As a remedy, we are developing methods to use mobile technology and data mining to directly assess students' ordinary learning activities.
The goal of this project is to develop techniques to directly assess student study strategies and to provide interventions suitable for success in engineering courses. We are accomplishing these goals by using smartpens and tablet computers to instrument ordinary learning activities in undergraduate engineering courses.
In this project, we are using smartpens and tablet computers to instrument ordinary learning activities in undergraduate engineering courses. The smartpens are used to record students' written work. They serve the same function as a traditional pen, but also record the work as time-stamped pen strokes. The tablet computers are used with an instrumented document reader we developed to measure how and when students read instructional materials. This combination of technologies provides an unprecedented view of the learning process and enables microscopic formative assessment. We use data mining techniques to examine the correlation between these learning activities and academic achievement, thus distinguishing between ineffective and effective learning strategies. We will then develop interventions designed to overcome the ineffective strategies and foster the effective ones.
Managing privacy issues and student concerns for privacy are critical for the success of this project. All personally identifying information is removed from the data. Even though the data is anonymous, the instructor does not review the data (other than deliverables) until after the term has ended and grades have been assigned. In this way, data about student study habits will never influence grades or even be linked to specific students.
We have completed studies in several introductory engineering courses and found that the students were not sufficiently engaged in learning. For example, we found that many students completed their homework just before it was due, and that this behavior strongly correlated with low grades in the course. Additionally, we found the most of the students did not read the textbook. We are currently developing and evaluating a system designed to overcome these ineffective strategies. The system provides students with feedback on their study efforts, including time spent on homework, time spent reading, and effort on note taking in lecture.
We found a strong correlation between time spent on homework, as measured with smartpens, and course grade (r = 0.44, p < 0.001). We also found that nearly all students over-reported their study time, and there was no correlation between self-reported study time and grade. These results confirm what other researchers have long suspected: students' self-reports of study effort are often unreliable. We also found a surprising lack of student engagement. For example, most students waited until the last minute to do their homework. In one course, 74% of the homework activity, measured by the number of pen strokes written, occurred within 24 hours of the due date. Furthermore, there was a strong negative correlation between doing homework at the last minute and course grade (r = -0.38, p < 0.001). We also found that students did little reading. On average, students spent only 3.4 hours reading the text over the entire quarter.
The project is providing training in research and technical communication to seven graduate and three undergraduate students. We have given six presentations of our work and published two journal articles. A third journal article is under review and three more are in preparation. Our studies have provided important insights about the study habits and engagement of students in introductory engineering courses at UCR, which is one of the most diverse major research institutions in the US, and has been designated a Hispanic Serving Institution. Many of the students are first generation college-attending. Understanding the learning processes of these students will increase student success and create increased opportunities for students from traditionally underrepresented groups to succeed in an STEM careers.
Kevin Rawson, Thomas Stahovich, and Richard Mayer, 'Homework and Achievement: Using Smartpen Technology to Find the Connection', to appear, Journal of Education Psychology, 2016.
Hanlung Lin and Thomas Stahovich, 'Enabling data mining of handwritten coursework', to appear, Computers & Graphics, 2016.
Thomas Stahovich and Hanlung Lin, 'Using lexical properties of handwritten equations to estimate the correctness of students' solutions to engineering problems', SUBMITTED, Educational Technology Research and Development, Sep 2015.