Skip to main content
William & Ida Friday Institute for Educational Innovation

Race to the Top STEM Affinity Network – Final Report

Executive Summary

Overview of the Evaluation

This final report of the Race to the Top (RttT) STEM evaluation presents outcomes of the initiative at the end of the original funding period (2010-14) and discusses implementation and sustainability of the program activities. It also presents recommendations with regard to improving effectiveness of and continuation of the program.

The evaluation was guided by the following three research questions:

  1. What were the short-term outcomes for students and staff in the 20 schools in the STEM network at the end of the funding period?
  2. To what extent have the four key elements of the network of STEM anchor and affinity schools (network structure, professional development, curriculum, and partnerships) been implemented as intended?
  3. What mechanisms and funding were put in place for the sustainability and scaling-up of the model, or of its most successful elements?

Findings

I. Short-Term Outcomes

Short-term demographics, achievement, and course-taking outcomes.

  • In 2012-13, RttT-funded STEM schools were similar to all other high schools in the state in terms of gender representation, the availability of advanced STEM courses (with the exception of advanced chemistry), and student achievement. Relative to all other high schools, a higher proportion of RttT-funded STEM schools hosted high-poverty student populations, and a smaller proportion hosted student populations with moderate levels of poverty.
  • Between 2010-11 and 2012-13, the analyses revealed the following changes:
    • The number of low-poverty STEM schools increased from 7% to 26% and the number of high-poverty STEM schools decreased from 53% to 37%, partially due to the addition of new STEM schools.
    • The percent of advanced-level Algebra II courses (compared to all math courses) offered in STEM schools increased from 18% to 46%, the percent of advanced-level biology courses (compared to all biology courses) offered in STEM schools increased from 27% to 61%, and the percent of advanced-level chemistry courses (compared to all chemistry courses) offered in STEM schools decreased from 51% to 45%.

Short-term outcomes in RttT STEM network schools: Evidence from staff and student surveys. This section compares student and staff responses of surveys administered between October and December of 2013 with surveys administered earlier (between May and December of 2012).

  • In general, students and teachers reported relatively high ratings of most measures in both survey administrations (students’ perseverance, confidence in ability to learn STEM subjects, and attitudes towards school, learning, and STEM; teachers’ use of rigorous, relevant, and collaborative instructional strategies).
  • Staff and students reported a more STEM-focused environment on the second survey administration than on the first.
  • Overall, teachers’ reported use of targeted instructional practices was higher on the second survey than on the first survey.
  • In most cases, in both survey administrations, the levels of implementation of short-term outcomes reported by teachers and students were higher in anchor schools, small new schools, and STEM academies, and they were lower in comprehensive schools. However, positive changes in outcomes between the first and second surveys were seen most frequently in comprehensive high schools.

Because surveys were administered only one year apart, they reflect perceptions of impacts during a short, initial period of the project. In addition, the samples for the first and second survey administrations were different; therefore, differences between survey administrations could be affected by differences in samples. Consequently, results should be considered as merely suggestive of possible trends.

Short-term outcomes in anchor schools: Evidence from the qualitative data. This report describes data from fall 2013 and notes changes between spring 2012 to fall 2013.

  • Staff in all anchor schools report that they need more time and support—such as via professional development and paid summer employment—in order to become model STEM schools.
  • According to staff and as judged by independent observations, there were notable improvements in instruction in anchor school classrooms (such as increases in frequency of project-based learning and content-focused discussions). This result is somewhat supported by teacher and student survey reports (e.g., increase in rigor of instruction, according to anchor school staff).
  • All four anchor schools have developed strong partnerships with industry and institutions of higher education (IHE) that provide substantial benefits to staff and students.

II. Implementation

  • Surveys indicate that the frequency of face-to-face and online networking among staff remained at the same level during the two years analyzed in this report. Networking happened mostly during professional development events and often did not continue outside of these events.
  • North Carolina New Schools (NC New Schools) fulfilled its obligations for instructional coaching, significantly exceeded its obligations for leadership coaching, and fulfilled obligations for out-of-school professional development days by the end of the project.
  • Compared to early survey data, by 2013, more staff in comprehensive schools attended different forms of professional development and coaching (an 11 percentage point increase in participation in workshops and a 21 percentage point increase in participation in coaching); however, participation rates in anchor and small schools did not change.
  • More than two-thirds of staff in all schools found both professional development and coaching somewhat or very helpful, with 86% to 89% of staff, respectively, in anchor schools reporting that these offerings were helpful.
  • Sixteen integrated STEM courses were completed by the North Carolina School of Science and Mathematics (NCSSM) on time and were digitally delivered to the North Carolina Department of Public Instruction (NCDPI). The courses will be hosted on NCDPI’s Home Base platform.
  • More than a quarter of teachers from the 20 STEM schools participated in the development of project-based curriculum units. The most notable gains in participation occurred in comprehensive schools.
  • According to staff surveys, as a result of NC New Schools’ partnership-building work, participation of school staff in collaborations with business and community partners significantly increased in all types of schools.

III. Sustainability

  • Overall, the RttT STEM funds helped to move forward STEM education in the state by helping to create a state-wide strategic plan and by funding the development and support of STEM programming in 20 STEM-themed schools. After the end of RttT funding, NCDPI will use its own funds to continue its STEM school recognition program and implement the NCSSM-designed courses, while private funds will be used to support the STEM portal.
  • As judged by the Evaluation Team and according to anchor school and NC New Schools staff, anchor schools need additional time and support to become STEM school models and sustain the changes. All four anchor schools will continue to receive professional development and coaching support from NC New Schools, funded by various sources.
  • After the end of RttT funding, support for the remaining 16 STEM affinity schools will be either reduced or completely eliminated, at least temporarily. Many of these schools and their Local Education Agencies (LEAs) expressed interest in continuing to receive support from NC New Schools. These LEAs and NC New Schools are looking for funding opportunities to continue support.
  • The work of the Industry Innovation Councils (IICs) continues to be sponsored by business partners, and as such, will be sustainable beyond the RttT grant period.
  • Lessons learned by NC New Schools during the project will be applied to improve services NC New Schools already provides to more than one hundred innovative schools in North Carolina, as well as to scale up STEM learning across the state. For example, starting this year, NC New Schools will expand their STEM offerings to all partner schools and will take a more direct approach to engaging LEA staff in different ways.

Recommendations

RttT STEM Initiative as a Whole

Continuation of the initiative. While it is not yet clear whether the initiative has contributed to desired longer-term outcomes (e.g., improved student performance in STEM subjects; increased numbers of students majoring in STEM subjects in college or entering STEM careers), it has directly or indirectly impacted STEM education statewide in a number of ways. Products developed under the initiative (STEM curricula and STEM web portal) can be used by all educators in the state, there has been an increased focus statewide on STEM education, and there is evidence of desired short-term outcomes in the RttT-supported STEM schools (schoolwide STEM focus, student participation in STEM activities, etc.). Based on these impacts to date, the Evaluation Team considers implementation of the RttT STEM initiative in North Carolina to have been a success and recommends continuing this work and providing financial support for its continuation.

Short-Term Outcomes of the Program

  1. Continue to monitor outcomes. The implementation team (NCDPI and NC New Schools) should continue to monitor outcomes at all levels—student, teacher, leadership, school, and network—in participating schools and LEAs. Additional data can provide valuable information about professional development impact, areas that need additional targeted interventions, and the speed with which desired changes for schools and students takes place. Such data can inform planning for new interventions and maximize their effectiveness.
  2. Conduct an impact evaluation. An independent evaluation of short- and longer-term impacts of the STEM initiative on students in schools that continue to receive professional development and coaching supports after RttT will strengthen understanding of program outcomes.

Implementation Activities

  1. Expand planning for future implementations. Various areas of the project experienced challenges in the beginning due to insufficient communication among stakeholders and insufficient advanced planning. The Evaluation Team recommends devoting planning and preparation time in the beginning of each future project to research-based implementation steps (e.g., Meyers, Durlak, & Wandersman, 2012).
  2. Expand peer school visits and cross-school collaboration. According to participants, networking and collaboration outside of schools has been the weakest implementation area throughout the project—especially in the area of online networking. Although participants enjoyed networking at professional development events, it rarely continued outside of these events. Two venues emerged as promising for greater facilitation of networking: peer schools visits and collaboration on common tasks among staff from schools within an LEA. As teachers find both of these forms of collaboration very valuable, the Evaluation Team recommends that initiative leads continue to use them as much as possible.
  3. Customize professional development. Professional development and coaching was planned as a fixed number of days per year per school, but the effectiveness of professional development and coaching depends on many school-level variables (e.g., school size, staff readiness for innovation, in-school mechanisms of sharing new knowledge and skills with all staff). NC New Schools should estimate the amount of professional development needed on a school-by-school basis, based on these and other factors, and communicate these estimates to prospective partner schools or LEAs ahead of time.

Sustainability

  1. Plan for sustainability. In future projects, plan for the transition to sustainability in the beginning of the project and articulate this plan in an agreement with participating schools and LEAs. These plans should take into account possible turnover of teachers and principals who receive professional development and other services.
  2. Develop LEA ability to contribute to sustainability. Engage LEAs by building their capacity to support specific initiatives and by involving them in planning for sustainability.
  3. Expand reach. Provide services to multiple schools in an LEA or across neighboring LEAs to increase involvement, capacity, networking, and collaboration within an LEA or region, and to improve the likelihood of realizing a culture of innovation across the whole LEA.
  4. Encourage more partnerships. Facilitate partnerships between schools and LEAs and local industries and institutions of higher education, and involve those partners in efforts to support sustainability.
  5. Target model schools. Provide logistical, professional development, and coaching supports to schools designated by the state as STEM schools to help them become model sites and share their best practices with other state (and even out-of-state) schools.

View Resource

Full Report PDF Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

December 1, 2014

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina