Third Annual Race to the Top Professional Development Evaluation Report

Executive Summary

The North Carolina Race to the Top (RttT) Professional Development Initiative is an expansive and multi-faceted effort to increase student achievement by updating the knowledge and skills of the state’s public education workforce, which includes about 100,000 teachers and 2,400 principals. This initiative is driven by a host of substantial policy changes, including: adoption of new Common Core State Standards and North Carolina Essential Standards; implementation of new standardized state assessments; increased use of data to inform classroom and school decisions; rapid changes in the technologies and digital resources available for teaching and learning; new teacher and administrator evaluation processes; an increased emphasis on formative assessment to inform instructional decisions; and a heightened emphasis on improving college and career readiness among all student groups.

The human resources challenge of the initiative—to provide the state’s educators with professional development that will enable them to extend their knowledge, improve professional practices, and, ultimately, increase student achievement overall and close achievement gaps among student groups—is formidable. The challenge is increased by the limited timeframe of the grant, the diversity of the State (large metropolitan to small rural local education agencies [LEAs]), the number of new teachers entering the workforce each year, the fiscal constraints across the State that are compounded in economically distressed regions, and expectations for sustainability of the statewide professional development infrastructure after RttT.

The RttT Evaluation Team’s charge is to: (a) document these professional development activities and collect data about teachers’ and administrators’ participation in and satisfaction with statewide, regional, and local professional development activities; as well as to (b) collect data about the impact of these activities on educators’ professional practices. In addition, the Evaluation Team collects information (through LEA and school field visits, teacher and administrator surveys, educator evaluation data, and student achievement data) to examine whether there has been an increase in the education workforce’s capacity to deliver effective instruction.

Four general questions guide the overall evaluation effort, with a number of more specific questions embedded in each one (see Appendix A of the full report for more details about the evaluation plan and the full set of questions). These overall questions are organized to reflect the general sequence in which they can be addressed over the four years of RttT:

State Strategies: To what extent did the State implement and support the proposed RttT professional development efforts? (Statewide Efforts)
Short-Term Outcomes: What were the direct outcomes of state-level RttT professional development efforts? (Statewide Efforts)
Intermediate Outcomes: To what extent did RttT professional development efforts successfully update the North Carolina education workforce? (Local Impact)
Impacts on Student Performance: To what extent are gains in student performance outcomes associated with RttT professional development? (Local Impact)

The RttT Evaluation Team has been documenting the implementation and impact of the State’s RttT professional development efforts since the beginning of the grant.^[1] This Year 3 report documents the current status and direct outcomes of the State’s RttT professional development for the 2012-13 school year (Evaluation Question 1, State Strategies, and Question 2, Short-Term Outcomes). It also provides detailed information concerning near- and longer-term impacts of statewide face-to-face and online RttT professional development efforts at the local level (Evaluation Question 3, Intermediate Outcomes). The final evaluation report, to be completed in July 2014, will focus on local impacts of the RttT professional development initiative (Evaluation Question 3, Intermediate Outcomes) including impacts on students (Evaluation Question 4, Impacts on Student Performance), to the extent possible.

In addition to the statewide professional development initiative, the overall RttT plan includes professional development activities housed under other RttT-funded initiatives. These activities are designed to address specific groups of educators: principals; educators in the lowest-achieving schools; educators in selected STEM schools; online teachers of NCVPS STEM courses; and new teachers entering low-performing schools. The evaluations of these activities are included in other reports.

Summary of Findings

The data collected and shared in this report indicate that, during the 2012-13 school year, the North Carolina Department of Public Instruction (NCDPI) implemented and supported face-to-face and online professional development efforts as outlined in the RttT proposal. In addition, regional and local leadership teams implemented a wide variety of professional development strategies intended to help local educators transition to the new state standards, new assessments, and new educator evaluation systems. That is, the grant requirements were met. This evaluation report highlights a number of findings that merit consideration during the final year of RttT-supported professional development:

Evaluation Question 1. State Strategies: To what extent did the State implement and support the proposed RttT professional development efforts? (Statewide Efforts)

Overall, participants reported that the professional development activities were well-designed and -implemented and provided valuable learning opportunities. The 2012 Summer Institutes were two-day face-to-face institutes conducted in six locations and attended by 2,541 educators who were members of local professional development leadership teams assembled by each LEA and charter school. NCDPI and Regional Education Service Alliances collaborated to provide 1,479 hours of training to 7,326 participants across all eight State Board of Education regions. NCDPI staff also provided approximately 906 hours of locally-requested professional development sessions above and beyond the sessions outlined in the Annual Professional Development Cycle. Between June 2012 and June 2013, 18,259 educators were enrolled in at least one Phase II online module and 15,935 of those educators received a certificate of completion for at least one module. Finally, during the 2012-13 school year, NCDPI offered approximately 100 webinars addressing a range of RttT-related topics.
North Carolina educators were generally satisfied with the quality of both the face-to-face and online professional development events. While the data revealed that the quality of the face-to-face sessions (with between 87% and 95% of participants agreeing or strongly agreeing that various sessions were of high quality) was moderately higher than that of the online professional development offerings (with between 62% and 88% of participants agreeing or strongly agreeing), the online professional development activities served different purposes and reached thousands of participants beyond those reached in the face-to-face sessions.

Evaluation Question 2. Short-Term Outcomes: What were the direct outcomes of state-level RttT professional development efforts? (Statewide Efforts)

Increased understanding and application of RttT priorities. NCDPI’s state-level RttT professional development efforts led to an increase in participants’ knowledge and understanding of several RttT priorities, including transition to the new standards and the new accountability models (with between 81% and 97% of participants agreeing or strongly agreeing that their understanding of various priorities increased). This result was consistent across both face-to-face and online offerings. Most participants reported (with between 75% and 92% of participants agreeing or strongly agreeing) that the professional development in which they participated helped them understand how to apply their new knowledge and skills within their roles in their LEAs.
Some differences in understanding and application across LEAs of different sizes. Leaders from small- and medium-sized LEAs (with between 70% and 95% agreeing or strongly agreeing) tended to report more consistently that they developed increased understanding and application of new skills from RttT-related professional development activities and resources than did education leaders from larger LEAs (with a wider range—between 56% and 90%—agreeing or strongly agreeing).

Evaluation Question 3. Intermediate Outcomes: To what extent did RttT professional development efforts successfully update the North Carolina education workforce? (Local Impact)

Increased cross-LEA collaboration. Region-based and size-based cross-LEA collaborations to share professional development and resources have grown significantly and are playing important roles in increasing professional development opportunities and capacity. Smaller LEAs with fewer resources tended to engage in more cross-LEA or region-based collaborations in their efforts to provide professional development than did larger LEAs.
Continued reliance on school-level PLCs. Many LEAs used a blended professional development delivery approach that combined face-to-face sessions, online modules, and professional learning community (PLC) activities. LEA staff reported that PLCs served as the primary opportunity for teachers to work together on organizing professional development resources by content or grade level and on creating instructional materials that align with new curriculum standards.
Teachers less positive about readiness than leaders, and also less positive over time. Approximately 25% fewer teacher respondents than LEA and school leader respondents (ranging from 10% to 34% fewer) reported agreement with statements about local professional development capacity and quality, teacher knowledge, and instructional practices. Results also show a general decline over time in teachers’ assessments of their and their colleagues’ readiness regarding the State’s new curriculum standards, data-based decision making, and the revised teacher evaluation process.
No evidence yet of gain in either self-reported or observed frequency of instructional strategies aligned with new standards. Across content areas, between the 2011-12 and 2012-13 school years, teachers reported a decrease in the frequency of learning activities related to new standards. Similarly, classroom visits revealed that, across content areas, there was a decline from 2011-12 to 2012-13 in observers’ ratings of the quality of teacher-student interactions that are reflective of specific instructional practices outlined in the new standards.
Though not a result of the professional development provided, interviews conducted for this component of the evaluation also revealed evidence of decreased school morale. When asked about cultural shifts in their schools, educators discussed their attempts to cope with the increased expectations brought on by several shifts in state education policy (e.g., new standards and assessments, new educator evaluation procedures, etc.) and the resulting increased time pressures. The interviews suggested that this general decrease in school morale may be due to feelings of being overwhelmed as a result of adjusting to these changes alongside feelings of being underappreciated amid other non-RttT education challenges (e.g., public perceptions of public schools and ongoing state-level discussions about compensation, employment policies, etc.).

Overall, the data provide a clear and important message for State leadership. The State is fulfilling the requirements of the RttT professional development plan well, with participants generally rating the activities as directly addressing their professional development needs and as providing valuable professional learning opportunities. In addition, some progress has been made on developing local and regional professional development capacity, but further progress is needed and is expected to be addressed in the final year of the RttT grant. However, as the State’s educators become more knowledgeable about the rapid and significant changes in curriculum standards, student assessments, data systems, educator evaluations, technologies, and overall expectations, they are recognizing that they have a long way to go to integrate all these changes into their day-to-day professional practices. Therefore, they are reporting a heightened awareness of how much more professional development and support they will need in order to be successful in responding to the changing expectations and improving their students’ achievement. The effort to increase student achievement by updating the knowledge and skills of the State’s entire public education workforce has begun well, but will need to be continued after the RttT grant ends.

¹ Executive summaries of the formative evaluations from the Year 1 and Year 2 reports are provided in Appendix B of the full report.

View Resource

Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

February 1, 2014

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina