NCVPS Blended Learning STEM Courses: Final Report

Executive Summary

This report completes the Consortium for Educational Research and Evaluation–North Carolina’s evaluation of North Carolina’s use of Race to the Top (RttT) funds to develop a series of STEM-based courses to be delivered to underserved students through the state’s Virtual Public School (NCVPS) via a blended-learning model. The evaluation’s goals have been to assess the extent to which this initiative contributed to: (a) the enrollment of underserved students targeted by the initiative; (b) the success of those students in the STEM courses offered; and (c) an increase in the availability of effective STEM teaching to students in high-need schools.

Purpose and Structure of the Report

This report—the final summative evaluation report for the initiative—addresses the evaluation goals by providing updated assessments of program capacity, course quality, and program effectiveness. These assessments are supported by data collected during the first three semesters of implementation (Fall 2012, Spring 2013, and Fall 2013).

The NCVPS Blended Learning STEM Course Initiative

For this initiative, blended learning refers to a course that is taught by a local teacher in a traditional setting with the aid of a virtual co-teacher and the support of online materials. The state’s overarching goal for the initiative has been to increase the number of highly-qualified STEM teachers in low-income rural areas and low-performing urban schools by pairing current classroom STEM teachers in target schools with online STEM mentor co-teachers. To date, NCVPS has piloted five blended-learning STEM courses (three courses first offered in school year 2012-13, one first offered in Fall 2013, and one more first offered in Spring 2014) and is in the process of developing two more for Fall 2014. Each blended learning course consists of project-based learning units that focus students on solving challenging and complex problems that incorporate concepts from the curriculum of the course. Each course also is designed to align with one or more of the National Academy of Engineering’s Grand Challenges of Engineering.^[1]

Strengths of the Second Year of Implementation

Signs of overall initiative improvement emerged during the second year of implementation:

Teachers who remained in the program across two academic years indicated that they were more comfortable with the program in Year Two, and that they encountered fewer programmatic barriers to success.
In addition, there were signs of increased capacity among participating teachers, especially in the areas of instruction-related technology and pedagogy; some indicated that their participation impacted their performance in their traditional classes as well.
The very low student-to-teacher ratio helped students by providing more opportunities for meaningful contact with their teachers than would have been possible in larger classes.

Summative Findings

Capacity

Expansion of mathematics and science offerings: NCVPS has expanded its mathematics and science offerings for both required and optional courses. In 2012-13, NCVPS offered three new blended-learning STEM courses (Earth and Environmental Science, Mathematics I, and Forensics) in three LEAs and four schools. Two more courses were added in the 2013-14 school year (Mathematics II and Biotechnology and Agriscience). However, the initiative’s success has been limited in three ways: (1) NCVPS was not able to develop as many courses as either originally planned or as planned in revised Scopes of Work; (2) courses have yet to be offered beyond the three pilot LEAs; and (3) since the majority of the courses offered to date have been required courses, the initiative has met only limited success in broadening STEM offerings in participating schools.
Enrollment of underserved populations: NCVPS has reached nearly 400 students in its pilot LEAs, and the program enrolled mostly freshmen students from groups traditionally underrepresented in STEM fields (i.e., females and minorities).
Cost-effectiveness of courses: Data are not yet available to determine cost-effectiveness. That said, until NCVPS is able to reduce the number of ongoing course revisions, significantly expand the number of LEAs with access to the courses, and increase the teacher-student ratio (especially given the high costs of providing two teachers per section and personal student devices), the initiative as enacted under RttT is unlikely to be cost-effective in the long run. NCVPS’s stated plans to offer some variations of the courses at cost to additional LEAs in future semesters will address one of these concern, as will plans for non-mobile and non-blended versions of some courses.

Course Quality

Rigor and relevance: Previous evaluation reports noted that teachers, students, and independent reviewers all expressed concerns about course quality, both in terms of the rigor of the courses and their incomplete alignment to state standards. Integration of both the Grand Challenges and project-based learning techniques was inconsistent.
Degree to which new courses take advantage of their e-format: Initially, participants noted frequent technological problems with the course website and the integration of iPads, and, while there have been improvements since the beginning of the initiative, most of the blended courses continue to under-utilize the e-format, and much work remains to achieve an optimal combination of functioning, reliable, and integrated technology resources. In particular, the iPads continue to suffer from ongoing technical difficulties and remain a distraction for some students. The new Mathematics II course represents a promising development, however, as it appears to have integrated online tools effectively.
Overall, strong communication between online and face-to-face teachers has led to significant contributions in course e-content, and as a result the online portion of the courses remains a prominent component.
Effects of blended-course structure on student-teacher interactions: On most measures, student interactions with face-to-face teachers appear to have been positively impacted by the blended-course structure, though to a large extent these interactions are a product of smaller class sizes and not blended learning, per se. In addition, much work remains to integrate the online teachers fully into the complete course experience and improve the quantity and quality of student-online teacher interactions.
Local capacity-building via involvement of the face-to-face teachers: Face-to-face teacher roles varied across course topics and teacher partner pairs. Face-to-face teachers have been involved in course development, but almost always via ad hoc, localized re-design of already-developed courses. Face-to-face teachers appear to have been meaningfully involved in instructional delivery, with most providing at least half of all instruction and collaborating with online co-teachers to plan that instruction.
Impact of blended learning on student engagement: Opportunities for meaningful student engagement were moderate to high across all courses. The primary contributor to this engagement was the structure of the courses and qualities inherent in the courses themselves. Reaching that level of engagement was not instantaneous or guaranteed, however, and required adjustment periods for teachers and students alike.
Student evaluations of the course experience: Student evaluations of the original three blended courses have improved over time, with students highlighting in particular the smaller class sizes and certain aspects of the project-based learning approach.
Face-to-face and online teacher quality relative to teacher quality in traditional courses in comparison schools: Administrative data necessary to answer this evaluation question were not available in time for inclusion in this report.

Program Effectiveness—Preliminary Impacts on Students and Teachers

Success of participating students: Formal test data were not available to assess quantitatively whether participating students grew academically, but analyses of student focus group and teacher interview data suggest that many participating students developed useful academic skills. Previous evaluation reports noted that, although several students struggled with self-direction, many improved their time management skills and appeared to have expanded their future educational aspirations. Teachers commented that they began to notice some early indicators of success in their students. Longer-term data related to student persistence in STEM-related courses and in staying on-track to graduate also are not yet available, but student focus group and survey responses indicated that many participating students gained both confidence and interest in STEM coursework.
Connections between course participation and teacher capacity-building: Capacity-building among teachers—both participating teachers and their non-participating STEM colleagues, with whom they shared resources and strategies—was more evident in the second year of the program than in the first. Previous evaluation reports highlighted that participating classroom teachers gained comfort employing student-centered learning. In addition, some classroom teachers began servings as mentors for other teachers in their schools and have increased their and their peers’ use of technology in the classroom. Capacity-building continues to be a challenge for teachers new to the initiative, who often are overwhelmed by the challenges of converting to blended learning. Formal professional development provided by the initiative continues to be underutilized in these capacity-building efforts.

Overall Conclusions

Although this final report is summative in nature, in light of likely continuation of the initiative after RttT, to strengthen the program, the Team suggests:

Improving existing courses to address ongoing concerns about content, design, and delivery;
Reducing the number and complexity of program features (e.g., integration of Grand Challenges, use of iPads, integration of project-based learning, etc.);
Better integrating professional development;
Engaging participating teachers earlier and involving them more in planning and design;
Providing balanced coverage for all aspects of STEM; and
Formalizing a participant feedback loop.

Despite these remaining issues, this initiative does appear to have provided some real benefits—albeit to this point largely unquantifiable—to participating teachers and students, as well as to non-participating teachers in their schools. Because several ongoing problems identified and detailed throughout the evaluation’s three reports have prevented the initiative from reaching its full potential, the Team recommends that initiative directors adjust the current approach to planning by transitioning from a single-semester outlook (e.g., student and teacher success in individual courses) to a focus on longer-term objectives. Next steps might include designing methods for supporting phased engagement of face-to-face teachers (e.g., first helping them grow comfortable with blended teaching generally before challenging them to teach blended classes outside of their core areas of expertise). Working toward a longer-term vision should help NCVPS increase the likelihood of achieving greater success in reaching its ambitious goals for participating teachers and students alike.

¹ The Grand Challenges of Engineering are a set of 21st-century challenges identified by members of the National Academy of Engineering and other groups worldwide to serve as a framework for focusing engineering efforts at all levels of education and innovation: http://www.engineeringchallenges.org/

View Resource

Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

August 1, 2014

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina