Measuring Progress toward Systemic Change in Undergraduate STEM Teaching and Learning

Project No.
PI Name
Tobin Smith
Association of American Universities

Abstract 1

Measuring Progress toward Systemic Change in Undergraduate STEM Teaching and Learning

Presentation Type
Toby Smith, AAU Vice President of Policy, Co-Principal Investigator; James S. Fairweather, Ph.D. Co-Principal Investigator; Emily R. Miller, Ph.D., Director of the AAU STEM Initiative; Josh Trapani, Ph.D., Associate Vice President for Research and Policy Analysis, Association of American Universities; Linda L. Slakey, Ph.D., Senior Advisor for STEM Education, Association of American Universities; Hannah Poulson, Policy Associate, Association of American Universities.


Currently there are not good metrics to measure effective teaching and learning at the individual faculty/course level, departmental, and institutional level. This effort is focused on providing better measures of undergraduate teaching quality by developing a more comprehensive set of metrics, mapped to the existing AAU 'Framework for Systemic Change in Undergraduate STEM Teaching and Learning' which specifically focuses improving pedagogy, enhancing scaffolding to support improved teaching and achieving real cultural change at the institutional level.


As part of AAUメs NSF モWidening Implementation & Demonstration of Evidence-Based Reformsヤ (WIDER) grant AAU developed a set of baseline measures that project sites, and other institutions, may use to better understand the current status of teaching and learning. Starting this fall AAU will identify a more comprehensive set of questions, data sources, and analytical tools, possibly in the form of a matrix, that will provide a comprehensive and customizable way to measure progress along the specific elements identified in the Framework for Systemic Change in Undergraduate STEM Teaching and Learning and at multiple levels within the university.


AAU will specifically be trying to identify key questions and corresponding data sources, indicators, and analytical tools that align with the existing AAU Framework for Systemic Change in Undergraduate STEM Teaching and Learning. To help us with this process, we will be requesting feedback from all 62 AAU member campuses. We will then work to develop a second document to accompany the Framework which can be used to help campuses to evaluate their progress towards achieving meaningful STEM reform at multiple levels within the university. This document is intended to provide a comprehensive and customizable way to measure progress along the set of key institutional elements identified in AAUメs Framework for Systemic Change in Undergraduate STEM Teaching and Learning. Measuring Progress will provide a set of questions that can be used at multiple levels within the university to assess the improvement to undergraduate STEM teaching and learning. Associated with questions are data sources and analytical tools available to answer these questions.


The overall objective of the AAU STEM Initiative is to influence the culture of STEM departments at AAU universities so that faculty members are encouraged to use teaching practices proven by research to be effective in engaging students in STEM education and in helping students learn. In order to effectively accomplish this goal, universities, departments and individual faculty members will need much better measures by which to evaluate their progress in improving the quality of undergraduate teaching. This effort is specifically focused on providing such measures. The aim is for continuous improvement. The use of measures and indicators are to advance conversations and inform decision-making about reforms to undergraduate STEM education among faculty members and institutional leaders. We believe that this will compliment other efforts currently be undertaken by various university consortia (e.g. UC Davis 'Tools for Evidence-based Action (T.E.A.) community, Committee on Institutional Cooperation Student Learning Analytics project) and with a new consensus study being undertaking by the National Academie � Developing Indicators for Undergraduate STEM Education � to produce a framework and set of indicators that can be used to measure the quality of undergraduate STEM education at universities and colleges.

Broader Impacts

Developing new metrics that can be used to assess and evaluate the quality of undergraduate teaching will help our institutions and their faculty members to move towards broadly implementing teaching practices which are known to be more effective at enhancing student learning. We hope this project will help to benefit students by enhancing their learning in STEM classes and as a result improving overall students success and retention rates in STEM fields.

Unexpected Challenges

While there has been some progress made in starting to develop data and tools to measure changes in the classroom itself, we face a significant challenges in developing new metrics and data to analyze change at the departmental and institutional levels.


Searching for Better Approaches: Effective Evaluation of Teaching and Learning in STEM, Research Corporation for Science Advancement, Tucson, AZ, 2015, Available at:

Bradforth, Stephen E., Emily R. Miller, William R. Dichtel, Adam K. Leibovich, Andrew L. Feig, James D. Martin, Karen S. Bjorkman, Zachary D. Schultz, and Tobin L. Smith, モUniversity learning: Improve undergraduate science educationヤ Nature, July 15, 2015, 282-284. Available at:!/menu/main/topColumns/topLeftColumn/pdf/523282a.pdf

Trapani, J., Fairweather, J., Paulson, K. (2015). 'The Roles of Data in Promoting Institutional Commitment to Undergraduate STEM Reform:The AAU STEM Initiative Experienceヤ In Weaver, G.C., Burgess, W. D., Childress, A.L., Slakey, L (Eds.), Transforming Instituions: Undergraduate STEM education for the 21st century. West Lafayette, IN: Purdue University Press.

Association of American Universities. (n.d.), AAU Framework for Systemic Change in Undergraduate STEM Teaching and Learning. Available at:

Project Page