Data driven decision-making in STEM departments: A field study of faculty engagement in continuous improvement systems for teaching
The use of data to inform decisions is a defining characteristic of current educational policy. Data driven decision-making (DDDM) is viewed as a corrective to decision-making based on anecdote or intuition and as a key aspect of educational reform. While not yet at the level of policy as in K-12 education, an impetus for the use of data to support policy and practice in higher education is mounting. However, little is known about how postsecondary instructors think about and use data in their teaching. In this paper we report findings from a study about how data are used by STEM faculty as part of their instructional decision-making.
We report findings from interviews with 56 faculty. Research questions were: (1) What types of data and other information are used by faculty to prepare for their courses? (2) What are defining characteristics regarding faculty data use practices? (3) What, if any, patterns exist across these data use practices?
This study took place at three large, public research universities. Purposive sampling procedures were utilized. Data included in-depth interviews that utilized the Critical Decision Method and classroom observations conducted with the Teaching Dimensions Observation Protocol. Interview transcripts were analyzed using an inductive approach to qualitative data analysis. Next, to examine the degree to which the data types exhibited dissimilarity or similarity in the aggregate, we used two exploratory analysis techniques, Wardﾒs method of hierarchical cluster analysis and multi-dimensional scaling. Finally, using these results, we returned to the raw data to identify how individuals utilized these group-based routines in their own unique instructional situations.
We found that five distinct types of data practices exist and that individual and institutional motivation to engage in continuous improvement is of utmost importance to effective DDDM. Some faculty described formal, statistical analyses of numeric data as part of a continuous improvement process that, for some, represents the primary feature of effective DDDM. But there are many other instances where the use of data is less clearly aligned with these traditional expectations of DDDM systems. These indicate that different forms of DDDM may be utilized in practice, each playing an important role in how instructors continually improved their teaching practices. We argue that the field adopt a broad perspective of DDDM that extends beyond the use of large numeric datasets and sophisticated algorithms to include these other forms of data and information.
The five repertoires of data use may operate as a heuristic for diagnosing and understanding how faculty within a given institution think about and use data. With this knowledge in hand, educational leaders can identify �leverage points� to be supported or altered to facilitate optimal interactions and processes for teachers within their organization. Above all, while the underlying motivation of DDDM is to replace non-numeric sources of information such as anecdotes for a more systematic and automated approach to educational decision-making, it is important to recognize that, in practice, other sources of data play an important and useful role in teaching practice.
Hora, M. T. (2015). Toward a Descriptive Science of Teaching: How the TDOP Illuminates the Multidimensional Nature of Active Learning in Postsecondary Classrooms. Science Education.
Bouwma-Gearhart, J. & Collins, J. (2015). What We Know About Data-Driven Decision Making In Higher Education: Informing Educational Policy And Practice. In Proceedings of the 19th International Academic Conference, p. 89-131. Edited by J. Rotschedl & K. Cermakova. International Institute of Social and Economic Sciences (IISES). Prague, Czech Republic.
Matthew T. Hora, Jana Bouwma-Gearhart, and Hyoung Joon Park (2014). Exploring Data-Driven Decision-Making in the Field: How Faculty Use Data and Other Forms of Information to Guide Instructional Decision-making (WCER Working Paper No. 2014-3). Wisconsin Center for Education Research, University of Wisconsin-Madison.