NCVPS Blended Learning STEM Courses: Initial Formative Assessment, Part I

Executive Summary

The Consortium for Educational Research and Evaluation–North Carolina is evaluating North Carolina’s use of Race to the Top (RttT) funds to develop a series of STEM-based courses to be delivered to underserved students through the state’s Virtual Public School (NCVPS) via a blended-learning model. The evaluation’s goals are to assess the extent to which this initiative contributes to: (a) the enrollment of underserved students targeted by the initiative; (b) the success of those students in the STEM courses offered; and (c) an increase in the availability of effective STEM teaching to students in high-need schools.

Purpose and Structure of the Report

This report—the first part of a two-part report on the first year of implementation—presents baseline measures from the first semester of implementation (August-December, 2012), to be used in future evaluations of the NCVPS blended learning STEM courses and to provide formative feedback to NCVPS in support of the growth and development of this initiative. The second part of this report (Fall 2013) will provide an update on implementation progress, based on additional data and observations from the second semester of implementation.

The report begins with an overview of the implementation of the initiative to date, followed by a review of current courses and details about the first cohort of participating students. The report then provides initial reviews of those courses from three different perspectives: the quality of the subject-matter content, the degree to which the Grand Challenges of Engineering have been incorporated, and the degree to which the courses reflect best practices in online pedagogy. These reviews are followed by analyses of initial feedback about the courses from participating teachers and students, as well as of observations made by the Evaluation Team during the first semester of course implementation. Finally, the report shares early evidence related to the effectiveness of the courses in the area of developing capacity among on-site (face-to-face) teachers.

This report provides a formative review of preliminary results for a still-developing initiative in order to inform ongoing initiative improvements; it is not intended to serve as a statement about the anticipated quality of the final form of this initiative.

The NCVPS Blended Learning STEM Course Initiative

For this initiative, blended learning refers to a course that is taught by a local teacher in a traditional setting with the aid of a virtual co-teacher and the support of online materials. The overarching goal for the initiative is to increase the number of highly-qualified STEM teachers in low-income rural areas and low-performing urban schools by pairing current face-to-face STEM teachers in target schools with online STEM mentor co-teachers. Over the course of the initiative, NCVPS will pilot eight blended-learning STEM courses, beginning with the three courses first offered in Fall 2012. Each blended learning course consists of a sequence of project-based learning units, or units that focus student energies on solving challenging and complex problems that incorporate concepts from the curriculum of the course. Each course also is designed to align with one of the National Academy of Engineering’s Grand Challenges of Engineering.^[1]

Initial Observations and Findings

Capacity

• NCVPS and three partnering Local Education Agencies (LEAs) enrolled 147 students in three blended learning STEM-focused courses in Fall 2012 (Earth and Environmental Science, Integrated Math I, and Forensics).
• Participants’ demographic data indicate that, collectively, the courses enrolled students from groups traditionally underrepresented in STEM fields (i.e., females and minorities).
• Most participants were 9th graders (72.8%). Fifty-seven percent (n=84) of the participants also were enrolled in one or more non-blended NCVPS courses, but only 2 percent (n=3) of the participants were enrolled in more than one of the RttT-funded blended courses.

Course Quality

The quality of each course was assessed by a team of reviewers with collective expertise in online learning, engineering, and the specific content of the courses. Reviewers noted course strengths and also provided recommendations for improvement.

• Course Content:
  • Forensics: Reviewers noted multiple strengths, praising the richness and depth of the project work, sub-task assignments designed to help students understand concepts and improve critical thinking skills, technology integration, and focus on methodological problems in the field.
  • Integrated Math I: This course exhibited similar strengths, but reviewers also noted that elements of the class appeared to be incomplete and that some Common Core content did not appear to be addressed. Reviewers recommended improvement in the organization of units and expansion of the guidance provided for students and teachers.
  • Earth and Environmental Science: The content review suggested that this course was not as strong as the other two. Reviewers cited incomplete coverage of important Essential Standards content, minimal coordination of materials, and the generally thin design of many of the projects and assignments.
• Grand Challenges Integration: Each course is informed by the Grand Challenges, and each includes projects that address a Grand Challenge; however, reviewers noted that courses may need to dedicate more direct attention to the engineering aspect of STEM and to clarifying connections between project content, activities, and the associated Grand Challenges.
• General Pedagogy, Online-Aware Pedagogy, and Project-Based Learning Components:
  • Orientation: Some orienting information on technology tools and available technical assistance is provided in each course, along with course content orientation and remediation, though not to the same degree across courses. While each course includes some guidance for teachers, that guidance often could be more robust.
  • Guidance: Forensics includes the most information about overarching standards and clarifying objectives; reviewers recommended that Earth and Environmental Science and Integrated Math I provide similar overall objectives. Reviewers also noted that all three courses would benefit from more student training and guidance related to their participation in the courses.
  • Student-Centered and Project-Based Teaching and Learning: Courses employ a mix of common and course-specific procedures. Course designers incorporated multiple elements to support student-content interactivity. Provision of clear guidance for managing group work and teacher-student team interactions could strengthen all three courses.
  • Leveraging of Technology and the Online Medium: Course designers incorporated multiple media elements across all courses, and teachers have integrated these online resources into their daily routines. Over the course of the semester, student use of technology for higher-order thinking exercises appeared to increase slightly.
• Student and Teacher Participation in the Courses
  • Teacher engagement in course development and delivery. Though face-to-face teachers did not take part in initial course development, early-semester technological problems required them to become more heavily involved in the delivery of their courses than anticipated, which led to more collaborative relationships with their online partners. Face-to-face teachers felt empowered to identify areas of the course that needed adjustment, and they became comfortable making those adjustments as needed.
  • Teacher-student and student-student interactions. Students and teachers took advantage of the additional communications tools provided by the courses. Some students appreciated the presence of a second, online teacher; other students were less appreciative, but that reaction appeared to be related to the degree to which the online teacher was integrated into the classroom experience. Several students also shared positive reactions to the increased group work, and many teachers pushed students toward collaborative or self-directed approaches to learning.

Program Effectiveness

• Impact on Teacher Capacity-Building: Several of the face-to-face teachers indicated that the relationships they developed with their online teaching counterparts were very beneficial. Their professional growth was manifested in at least three ways: a growing awareness of the importance of teacher-to-teacher communication; an appreciation of the importance of collaboration and of how that collaboration can aid their development as teachers; and, in a few cases, their emerging development as mentors for other teachers at their schools. However, most face-to-face teachers believed that the initiative would benefit from more extensive initial preparation for the many new teaching demands inherent in a blended approach.
• Impact on Student Learning Processes: While it is too early in the implementation to address student outcome questions meaningfully, there is at least one early leading indicator of a possible impact on student academic achievement: improved time management skills. There is also, however, considerable ground still to cover in the area of student self-direction.

Conclusions and Formative Recommendations^[2]

Summary of Early-Implementation Strengths

1. Courses are reaching the intended audiences. Each course is over-represented by minority students and female students.
2. Forensics exhibits multiple strengths. The Forensics course is the most developed of the three pilot courses, in terms of content, attention to Grand Challenges, and application of online teaching and learning strategies and devices.
3. Integration of technology tools and online resources appears to be growing. Observers noted more frequent use of technology and access of online materials as their Fall 2012 site visits progressed. Students commented on the helpfulness of having these resources available.
4. Project-based learning seems to be establishing roots. While the move to project-based learning represents a major change for teachers and students, there is early evidence that the project-based learning approach is beginning to work well for most teachers and for some (but not yet all) students.
5. Co-teaching relationships are strong and constructive. Communications between face-to-face teachers and their online co-teachers has become very strong. Online teachers were especially critical in helping the face-to-face teachers keep pace during the first weeks.
6. Face-to-face teachers are beginning to take on roles as mentor teachers. While this is not a required or targeted component of the initiative, it appears to be happening in some locations even without formal support for this effort.

Formative Recommendations

1. Provide additional pre-course support and guidance for teachers. Dedicate more resources to providing support and guidance during the weeks leading up to and through the start of each semester, especially for teachers who are new to the blended learning environment. In particular, consider providing guidance related to role-definition for face-to-face and online teachers, more opportunities to shape and interact with course material before school starts, strategies for planning the opening weeks of the course and for managing student interactions with technology, and deeper preparation for managing a project-based learning classroom. Moving some of the related professional development currently provided online to mandatory summer face-to-face sessions may be one strategy for addressing this need.
2. Provide additional support and guidance for students. It may be helpful to provide blended course students with some training early in the course on how to make the most of their new technology environments.
3. Restructure iPad integration. Among other things, some critical aspects of the courses (e.g., assignment documents and media applications) were not compatible with the initiative-provided iPads. Also, some students suggested that the novelty of the iPads was a major distraction.
4. Find ways to clearly define the roles of and increase the involvement of online teachers. Consider emphasizing the teacher mentoring role, which appeared to be the strongest aspect of the online teachers’ presence. In addition, since students reported feeling that the online teacher did not play a role in their learning, investigate ways for online teachers to engage with students more directly during the school day.
5. Continue to improve course content. Of the three initial courses, Forensics appears to be the strongest; it should be used as a model for strengthening the other two and for guiding the development of the next set of courses. In particular, individual projects should be reviewed to ensure proper alignment with course standards, Grand Challenges, and teacher and student needs and expectations (per the more thorough reviews included in this report). NCVPS staff already have identified the Spring 2013 course development period as a time for addressing course content improvements.
6. Consider developing relationships with others working on similar blended learning initiatives. The RttT Instructional Design Team at NCDPI—which has been refining a collaborative development approach for blended-learning professional development online—is one possibility, as are the North Carolina School of Science and Mathematics (which also offers online STEM courses) and NCVPS’s own Occupational Course of Study blended learning team. In addition, refining the course development and delivery process to include various initiative stakeholders will help ensure courses are better tailored to instructor and student needs, as well as the context in which they are implemented.

Recommendations for Improving Evaluation Implementation

7. Work with the Evaluation Team to improve data collection. In particular, the Evaluation Team’s work will benefit from more direct involvement in the administration and collection of the early-experience and end-of-experience surveys.

¹ The Grand Challenges of Engineering are a set of 21st-century challenges identified by members of the National Academy of Engineering and other groups worldwide to serve as a framework for focusing engineering efforts at all levels of education and innovation: http://www.engineeringchallenges.org/
² Note: A response from North Carolina Virtual Public School to both the Initial Observations and Findings and the Conclusions and Formative Recommendations sections is included in Appendix F of the main report.

View Resource

Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

April 1, 2013

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina