Our research is based on an innovative approach that integrates computational thinking and creative thinking in CS1 to improve student learning performance. Referencing Epsteinメs Generativity Theory, we designed and deployed a suite of creative thinking exercises with linkages to concepts in computer science and computational thinking, with the premise that students can leverage their creative thinking skills to モunlockヤ their understanding of computational thinking. In this paper, we focus on our studies on differential impacts of the exercises on different student populations. In our first study, we found that there was a linear モdosage effectヤ where completion of each additional exercise increased retention of course content. The impacts on course grades, however, were more nuanced. CS majors had a consistent increase for each exercise, while non-majors benefited more from completing at least three exercises. It was also important for freshmen to complete all four exercises. We did find differences between women and men but cannot draw conclusions. In our second study, we confirmed our prior findings that the computational creativity exercises positively impact student learning of computational thinking and CS knowledge and skills. Furthermore, we analyzed the students' learning profiles using the Implicit Theory of Intelligence and investigated how majors and non-majors differed in their learning behaviors.

Project No.
1431874
PI Name
Leen-Kiat Soh
Institution
University of Nebraska



Abstract 1

Our research is based on an innovative approach that integrates computational thinking and creative thinking in CS1 to improve student learning performance. Referencing Epsteinメs Generativity Theory, we designed and deployed a suite of creative thinking exercises with linkages to concepts in computer science and computational thinking, with the premise that students can leverage their creative thinking skills to モunlockヤ their understanding of computational thinking. In this paper, we focus on our studies on differential impacts of the exercises on different student populations. In our first study, we found that there was a linear モdosage effectヤ where completion of each additional exercise increased retention of course content. The impacts on course grades, however, were more nuanced. CS majors had a consistent increase for each exercise, while non-majors benefited more from completing at least three exercises. It was also important for freshmen to complete all four exercises. We did find differences between women and men but cannot draw conclusions. In our second study, we confirmed our prior findings that the computational creativity exercises positively impact student learning of computational thinking and CS knowledge and skills. Furthermore, we analyzed the students' learning profiles using the Implicit Theory of Intelligence and investigated how majors and non-majors differed in their learning behaviors.

Presentation Type
Paper
Team
Leen-Kiat Soh, University of Nebraska Duane Shell, University of Nebraska Elizabeth Ingraham, University of Nebraska Brian Moore, University of Nebraska Steve Ramsay, University of Nebraska


Need

(a) This project is important because of three key reasons. First, increasingly more CS courses service non-majors from different disciplines and combining CS and creativity can help students apply CS concepts and techniques to their disciplines. Second, CS still severely lacks participation from female students and other underrepresented groups and further investigations are warranted to confirm or inform the impact of creativity in attracting such students. Third, strategically integrating computational thinking and creative thinking into CS curriculum is non-trivial and requires rigorous research. Our project responds to a gap in research as there still lacks understanding on what computational creativity exercises are effective, how to incorporate them into courses, and how to support them large scale.

(b) The project benefits practitioners, researchers, and students in CS education in particular (and also STEM education in general). And here, we include both CS majors and non-majors who take CS courses. First, incorporating creative thinking into CS courses has been shown to improve student learning and performance. Second, creative activities can be more engaging than their conventional counterparts and prove to be important in attracting and retaining women and underrepresented groups to CS and serving non-CS majors, thus having the potential to broaden participation in computing. Third, creative thinking skills are fundamental to problem solving, and students equipped with these skills can better discover and innovate in todayメs world.

Goals

(a) Specific aims are to produce a final suite of validated, high quality CCEs and a Computational Creativity undergraduate course, and to conduct rigorous research to understand for whom and under what conditions the CCEs are most efficacious, why the CCEs are effective by studying studentsメ collaborative interactions and learning processes, and how the CCEs impact studentsメ enrollment and retention in CS and STEM courses.

(b) Key activities: (1) development of computatioal creativity exercises, (2) deployment of computational creativity exercises in seven courses including developing grading rubrics, automation, and revision of exercises to fit course topics, (3) data collection for research for both control and treatment courses since August 2014, (4) creation of a brand new computational creativity course to be offered in Spring 2016, (5) data analysis and modeling of session logs tracked and recorded of students interacting with online platform, (6) analyses of student learning profiles using Implicit Theory of Intelligence, (7) promotion of the exercises at conferences and technical meetings, and (8) outreach activities to illustrate the use of computational creativity exercises and benefits, including talks to faculty in Discipline-Based Educational Research and K-12 teachers.

Approach

(a) Specific aims are to:
Aim 1. Produce a final suite of validated, high quality Computational Creativity Exercises and a Computational Creativity undergraduate course for broad dissemination to other post-secondary educational institutions and organize a workshop to share and document instructional experiences, lessons learned, and student pedagogy teaching with these exercises as part of the suite.
Aim 2. Investigate and understand for whom, and under what conditions, Computational Creativity Exercises are most efficacious by conducting systematic studies to test how variations in delivery and utilization of the Computational Creativity Exercises (timing, supplement or entire course, introductory vs advanced courses, CS and non-CS courses, and STEM and non-STEM courses) affect exercise efficacy and examine how different student characteristics (prior knowledge, motivation, ability, strategic self-regulation, demographics) impact exercise effectiveness.
Aim 3. Investigate and understand why the Computational Creativity Exercises are effective by conducting systematic studies to test how studentsメ collaborative interactions during exercises impact exercise effectiveness, how the Unified Learning Model (ᄃ2.2) learning processes of attention, repetition, and connection occur during the exercises and lead to learning and achievement, and how studentsメ reactions to the exercises impact their motivation and engagement.
Aim 4. Investigate and understand how the Computational Creativity Exercises impact studentsメ enrollment and retention in CS and STEM courses by examining exercise impacts on subsequent enrollment in more CS and STEM courses, subsequent enrollment of women and underrepresented minorities in CS and STEM courses and majors, and retention of current CS and STEM majors.
(b) Our interpretation of creative thinking is based on Epstein's creative competency theory; our computational thinking is based on a collection of articles, including J. Wing's CACM article; our Implicit Theory of Intelligence is based on C. Dweck's work; our learning and teaching theory is based on the Unified Learning Model of D. Shell's. Further, our project uses design-based research and a Year 3 professional development workshop to further develop the CCEs and to evaluate their efficacy which in turn will maximize the dissemination and implementation of this proven effective instructional strategy. Our course materials will be disseminated to other post-secondary institutions, thereby advancing student understanding of CS, computational thinking, and creative application of CS knowledge. Research findings on 1) the conditions influencing effectiveness will inform future implementation by identifying best practices for delivery and utilization, 2) student characteristics will inform classroom practices to motivate student engagement with and maximize student learning from the CCEs, 3) why the exercises are effective will advance understanding of how CCEs facilitate student collaborative interaction, learning of computational thinking/CS content, and creative competency, and 4) retention and enrollment will advance understanding of how CCEs can broaden CS and STEM participation and increase retention.
(c) Our interdisciplinary team consists of faculty and researchers from computer science, arts, music, digital humanities, and education psychology. Leen-Kiat Soh (PI), Associate Professor of Computer Science and Engineering and Duane Shell (Co-PI), Research Professor of Educational Psychology co-direct the project. Co-PI Elizabeth Ingraham is an Associate Professor of Art & Art History. Co-PI Brian Moore is an Associate Professor of Music. Co-PI Stephen Ramsay is an Associate Professor of English. Several undergraduate and graduate students in Computer Science, Arts, and Educational Psychology are also participating in this project. Several instructors are also involved as adopters of our computational creativity exercises in their classrooms.

Outcomes

We have just gone through almost a year of work with our project.

(a & b) Here are our outcomes/deliverables for the past year:

Under Aim 1: We revised and extended our original suite of five Computational Creativity exercises to now eleven. We have piloted some of these exercises in Spring 2015. We have created the Computational Creativity Course and gotten university approval to offer in Spring 2016 as 'CSCE/ARTP 270: Create! Compute! Compete!'. We have also created an introductory informatics course 'CSCE100 Introduction to Informatics' that will be offered as part of the new UNL Interdisciplinary Informatics Minor that was started during IC2Think and completed during IUSE. We have conducted a min-workshop for the three CSCE faculty in addition to co-PI Ingraham who will be implementing the CCE's in their classes in Fall 2015. We are in the process of tailoring the CCE's to each of the specific implementation courses. Further, we have continued to revise and improve the delivery platform called the Written Agora. The Written Agora is an online collaborative environment allowing students to collaborative write wikis, discuss, and debate. We are improving its features to better support student navigation and access to contents, as well as improving its tracking mechanism to better track student activities or interactions with the system. These will allow us to collect more detailed data about student online and collaborative behaviors.
2. Under Aim 2: We have collected two semesters of control data from targeted CSCE and non-CSCE courses for use in comparison studies with implementation classes. In Fall 2014, we collected data in ARTP189, CSCE155A, CSCE155E, CSCE155N, CSCE230, CSCE322, CSCE428/828, CSCE478/878, and RAIKE183H. We had a total of 512 (428 men; 88 women) agree to participate and complete pre-course assessments. Of these 359 (292 men; 67 women) completed mid-semester assessments and 310 (250 men; 60 women) completed post-course assessments. Pairwise retention was 54% (52% men; 65% women). In Spring 2015, we collected data in ARTS398, CSCE155A, CSCE155E, CSCE155N, CSCE155T, CSCE310, CSCE310H, CSCE322, CSCE423/823, CSCE471/871, CSCE475, and MUSC398. We had a total of 458 (359 men; 99 women) agree to participate and complete pre-course assessments. Of these 324 (214 men; 74 women) completed mid-semester assessments and 283(214 men; 69 women) completed post-course assessments. Pairwise retention was 54% (51% men; 64% women). Retention for surveys is generally consistent with what is achieved in longitudinal studies. Most individual classes have retention that is high enough (above 60%) for use as controls for future studies. CSCE155N and CSCE155E both had retention issues, but comparision data for these classes is available from the prior IC2Think grant studies. CSCE230, CSCE310, and CSCE428/828 all had retention rates between 40% and 50% and likely will need a future control data collection when not being used for implementation. Overall, control group data collection appears to have been adequate for use in future studies under AIM 2. Additionally, we have pilot tested shortened versions of the motivation and strategic self-regulation instruments that we previously used in IC2Think. The shortened versions have excellent psychometics and estimated reliability. The short versions are important for conserving class time for in-class administration and for getting students to complete them.
3. Under Aim 3: Using data previously collected during IC2Think and Written Agora studies, we are conducting data analysis using the session logs (real-time data collected of student online interactions with the Written Agora platform, a collaborative web-based environment for online wiki writing and discussions), survey data collected in the beginning, during, and at the end of the semester, and grades to conduct student-level analyses and essay-level analyses, taking into account student learning profiles and behaviors. We have merged the datasets and are now deriving visuals as well as analyses. These analyses are pilot tests for the more refined analyses proposed for the CCE Written Agora data to be collected in the next two years. We have identified limitations in the old Written Agora platform for fully capturing the ULM learning processes of attention, repetition, and connection. As a result, we have modified the pagination of the Written Agora to provide more precise metrics for estimating these learning processes.
4. Under Aim 4: Our control semester data collection described previously under AIM 2 has provided us with a large and solid comparison sample for examining the impact of the CCE implementation classes and the computational creativity course CSCE/ARTP 270: Create! Compute! Compete! on student retention and enrollment. Although only about half of the students completed all surveys, the pre-survey students consenting to grades, which allow for examining withdrawal rates in classes, and to enrollment tracking are very high, about 90% of the total enrollment. We have data on both students current and intentions to major in CSCE that we can compare. We also have high responses from our non-STEM courses in Art and Music to allow us to track whether these students subsequently enroll in STEM or CS courses. The numbers of women in both semesters of control data is high enough to allow for meaningful tracking of enrollment comparisons. Other than women, minority enrollment in the target classes is too low to do meaningful comparisons. We will be particularly interested in whether student STEM and CS enrollment is impacted by the CSCE100 Introduction to Informatics and ARTP/CSCE270 Create! Compute! Compete! classes.

In a study to be presented at ICER'15 and published in the proceedings we found that:
1. Regardless of student profile, incremental intelligence theories decreased from the beginning until the end of the semester while entity intelligence theories increased from the beginning until the end of the semester. Even though incremental theories decreased and entity theories increased throughout the course of the semester, incremental theories were still more strongly endorsed by CS1 students across the semester
2. Change in implicit intelligence theory across the semester differed for different introductory CS1 coursse. Decreases in incremental intelligence theory were associated with higher standardized course grades and increases in entity intelligence theory were associated with higher grades for honors students.
3. CS students inclinations towards entity theories increased across the semester. CS educators have several options to help combat against this trend: use effort versus intelligence feedback or praise; teach students that deep learning takes extended time; encourage students to persevere in the face of obstacles and set mastery goals.
In a study of Fall 2014 control class data that has been submitted to the 2016 American Educational Research Association Conference we found that:
1. Although students' entering the CSCE155 and RAIK183H courses had generally high motivation and postitive attitudes toward the course, these entering motivations did not predict achievement except for the RAIKE183H class where prediction was very high.
2. Motivational variables were associated with grades and retention in unanticipated ways that were inconsistent with previous findings for these variables at the end of the course.
3. Setting goals to learn the course material for personal growth and development emerged as the only consistent predictor of grades and retention, consistent with the theoretical formulations of the Unified Learning Model which is the theoretical foundation for this project.
4. The lack of association for students' entering personal motivations and course achievment indicates that students' motivations are malable as a function of their in-class experiences, which means that the CCE's and other instructional aspects can impact how students are motivated in the course.

(c) Next steps:
1. Under Aim 1: We will continue to pilot, evaluate, and revise our Computational Creativity exercises. We will collect feedback from instructors after each semester of deployment. We held a mini workshop for instructors July 23, 2015, and are working with them to incorporate exercises into their classes. For Fall 2015, we will implement CCE's in the following classes: ARTP189H, CSCE155N, CSCE230, CSCE310, CSCE322, and CSCE428. We will implement in additional classes in Spring 2016 following revisions made based on the Fall 2015 implementations, with those classes to-be-determined later. We will also offer the two new courses in the upcoming AY (CSCE100 in Fall 2015, and CSCE/ARTP 270: Create! Compute! Compete! in Spring 2016). We will also finalize our design and implementation of the Written Agora platform for delivery of the exercises, and will provide maintenance and support for its usage. We anticipate making the final major revisions if any to the computational creativity exercises during Summer 2016 so that the production versions suitable for dissemination are ready for testing in 2016-17. We will be specifically looking at implementation issues that emerge as we move from introductory classes (CSCE155N), where we have implemented CCE's previously, to more advanced classes (CSCE230, CSCE310, CSCE322, and CSCE428/828) and to non-CSCE classes (ARTP189H).
2. Under Aim 2: We will continue to collect the battery of pre-, mid-, and post-surveys that were collected in pilot courses in 2014-15. We also will continue to obtain students' grades and track their enrollments. During Fall 2015, we will be conducting analyses of pilot course data to examine research questions related to student motivation and self-regulation in courses for publication and presentation. Beginning in Spring 2016, we will be using data from the first wave of implementation courses to research the impacts of the CCE's on students' motivation and strategic self-regulation and how CCE's impact their learning. We will sprecifically be examining how differences in students' reported motivation and strategic self-regulation exercise effectiveness. We will be collecting embedded survey data from the CCE's which will allow examination of students motivation within the exercises themselves. We also will be examining differential effects of the CCEs across the implementation classes which represent a spectrum of course level (freshmen to senior/grad) and background knowledge (introductory or non-CS to advanced).
3. Under Aim 3: We will conduct the first formal studies using the data from the revised and enhanced Written Agora session capture logs. These studies will employ data mining and analytics to model the Unified Learning Model processes of attention, repetition, and connection and then use these data to look at associations between students' real time learning processes and their self-reported motivation and engagement. We will be specifically focusing on how students' real time actions within the CCEs differ as a function of motivated strategic self-regulation profiles that are identifiable from survey data. We will begin our first data mining of creative competencies (broadening, surrounding, capturing), initial Semantic Network Analysis, and initial Social Network Analysis using the Written Agora data.
4. Under Aim 4: We will conduct the first comparison studies of the effects of CCEs' or the CSCE100 Introduction to Informatics and ARTP/CSCE270 Create! Compute! Compete! courses on course retention and subsequent STEM and CS enrollments for women and minority students.

Broader Impacts

Addressing societal problems in health, education, and business increasingly requires data intensive solutions with skills in big data and informatics becoming critical. Computational thinking is foundational for these skills and necessary for 21st century careers. The improved computational creativity exercises (CCE) and computational creativity course (CCC) will help advance STEM learning not only for CS students, but also for students in all disciplines who will need to creatively apply computational thinking to engage with the growing body of big data informatics and analytics. We will expand deployment of the exercises within the university undergraduate curriculum and through dissemination to other post-secondary institutions. The CCC can be delivered via distance education, facilitating expansion of computational and creative thinking and core CS content to extended audiences, including informal and business/industry settings. During Year 3, we will develop formal dissemination plans with partner institutions for deployment of the CCEs and CCC. These partners will be invited to the Year 3 Workshop to foster adoption. K-12 versions of six exercises have been implemented and deployed on Googleメs Exploring Computational Thinking. We will use study findings to update and enhance these existing K-12 exercises and also to expand them to include the entire suite. CCEs can help increase the number of students pursuing STEM education at the post-secondary level and STEM careers by introducing K-12 students to important computational thinking knowledge and skills in a creative non-technical format. The integration of creative thinking with computational thinking in the exercises may be helpful in broadening the participation of women and other underrepresented groups in CS and STEM.
In addition to providing information about the CCEs themselves, research findings will advance general scientific understanding of the conditions influencing real world classroom implementation of curricular and instructional practices and how student characteristics including social interaction, motivation, affect, self-regulation, and demographics influence instructional effectiveness. Research findings also will advance scientific understanding of the underlying learning processes involved in creative thinking and computational thinking and how these interact to impact learning of CS content, and advancement of creative competency. These results will advance both educational theory and practice and be of interest to a broad range of scholars and researchers. We will prepare articles and presentations for dissemination to scientific and professional outlets in the wider educational practice and policy communities (K-16) and to outlets informing the general public about the findings.

Unexpected Challenges

1. Scaling up the deployment, to support a large number of students in terms of grading and providing assistance. We have tried an online platform to support student collaboration; we have also developed grading rubrics to simplify grading; we have also devised group grading to speed up the actual grading process.
2. Getting instructors to truly incorporate our computational creativity exercises meaningfully into their courses. We have tried drafting 'lightbulbs' as inspirations for instructors; meeting with them; providing feedback of how their students have been doing during the semester and suggestions for improvement.

Citations

Soh, L.-K., D. F. Shell, E. Ingraham, S. Ramsay, and Brian Moore (2015). Viewpoint: Improving Learning and Achievement in Introductory Computer Science through Computational Creativity, Communications of the ACM, 58(8):33-35.

Muto-Nelson, K. G., D. F. Shell, J. Husman, E. Fishman, and L.-K. Soh (2015). Student Approaches to Learning in a Foundational Engineering Course: A Motivational and Self-Regulated Learning Profiles Perspective, Journal of Engineering Education, 104(1):74-100.

Shell, D. and L.-K. Soh (2013). Profiles of Motivated Self-Regulation in College Computer Science Courses: Differences in Major versus Required Non-Major Courses, Journal of Science Education and Technology, 22(6):899-913.

Flanigan, A., M. Peteranetz, D. F. Shell, and L.-K. Soh (2015). Exploring Changes in Computer Science Studentsメ Implicit Theories of Intelligence Across the Semester, in Proceedings of the International Computing Education Research (ICERメ2015), Omaha, NE, August 9-13, pp. 161-168.

Patterson Hazley, M., D. F. Shell, L.-K. Soh, L. D. Miller, V. Chiriacescu, and E. Ingraham (2014). Changes in Student Goal Orientation across the Semester in Undergraduate Computer Science Courses, in Proceedings of the Frontiers in Education (FIEメ2014), Madrid, Spain, October 22-25, pp. 2278-2284.

Shell, D. F., M. Patterson Hazley, L.-K. Soh, L. D. Miller, V. Chiriacescu, and E. Ingraham (2014). Improving Learning of Computational Thinking Using Computational Creativity Exercises in a College CS1 Computer Science Course for Engineers, in Proceedings of the Frontiers in Education (FIEメ2014), Madrid, Spain, October 22-25, pp. 3029-3036.

Miller, L. D., L.-K. Soh, V. Chiriacescu, E. Ingraham, D. F. Shell, and M. Patterson Hazley* (2014). Integrating Computational and Creative Thinking to Improve Learning and Performance in CS1, Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSEメ2014), March 5-8, Atlanta, GA, pp. 475-480.

Shell, D. F., M. Patterson Hazley, L.-K. Soh, E. Ingraham, and S. Ramsay (2013). Association of Studentsメ Creativity, Motivation, and Self-Regulation with Learning and Achievement in College Computer Science Courses, in Proceedings of the Frontiers in Education Conference (FIEメ2013), Oklahoma City, OK, October 23-26, pp. 1637-1643.

Miller, L. D., L.-K. Soh, V. Chiriacescu, E. Ingraham, D. F. Shell, S. Ramsay, and M. Patterson Hazley (2013). Improving Learning of Computational Thinking Using Creative Competency Exercises in College CS-1 Computer Science Courses, in Proceedings of the Frontiers in Education Conference (FIEメ2013), Oklahoma City, OK, October 23-26, pp. 1426-1432.