Outcomes and Impacts of North Carolina’s Initiative to Turn Around the Lowest-Achieving Schools
Executive Summary
Through its Race to the Top (RttT)-funded initiative to Turn Around the Lowest-Achieving Schools (TALAS), North Carolina has carried out an effort to transform low-performing schools that is more ambitious than those of all other states that received RttT funding. The District and School Transformation (DST) Division of the North Carolina Department of Public Instruction (NCDPI) continued its work on the transformation of North Carolina’s 118 lowest-achieving schools through the 2014-15 school year.[1] DST also worked with 12 of the state’s lowest-performing school districts to support and sustain transformation implementation.
DST services began with a Comprehensive Needs Assessment in each TALAS school, which served as the basis for the School Improvement Plan. The implementation of each plan was supported through leadership coaching, instructional coaching, and district-level coaching in the 12 districts that received direct DST services. DST also provided professional development for school leaders, and other educators received professional development offered through other NCDPI interventions and RttT initiatives.
Outcomes
From 2009-10 to 2013-14, 75% of TALAS schools increased their graduation rates more than the average increase in similar comparison schools. In terms of student proficiency on achievement tests during the transition to assessments based on the Common Core State Standards, 60% of TALAS schools outperformed the average change in the comparison schools. TALAS schools and comparison schools registered similar gains in school growth as measured by value-added scores (provided through the Education Value-Added Assessment System [EVAAS]) during the study period.
This final TALAS evaluation report focuses on the impact of school transformation on students, teachers, and schools. Throughout this summary and the report, we present the effect estimates in standardized units, known as standard deviation units (sdu). This allows for the effect sizes to be directly comparable to each other and to prior studies. For example, in class-size experiments, the effect of reducing classes from 25 or more students to about 15 students was 0.22 sdu.
Overall Impact
Throughout the three years of full program implementation, TALAS raised school proficiency rates by an average of 0.18 sdu. The estimates of the effects on schools closest to the fifth percentile of performance (the percentile used as the cutoff for identifying TALAS schools) were not significant, which seems to indicate that the positive effects were concentrated on the lower of the lowest-performing schools in the state. TALAS was estimated to have had positive effects on graduation rates as well, but these effects were not statistically significant—most likely due to the small number of high schools (17) participating in TALAS.[2] When examining effects by level of schooling, all but one of the effects were positive, and most were substantial, but only the effect on teacher value-added at the middle school level was statistically significant—again, likely due to the small number of schools at each level of analysis.
Immediacy of the Impacts
According to an Institute of Education Sciences report (Herman et al., 2008), one of the keys to successful school turnaround efforts is “quick wins”—visible improvements early in the turnaround process, which result in immediate increases in student outcomes. These initial changes can set the tone for transformation by creating educator buy-in and by establishing a climate for long-term change (p. 22). When compared with other low-performing schools in 2011-12 (the first full year of the implementation of the intervention), TALAS schools increased school-wide achievement growth as measured by EVAAS by a significant 0.34 sdu. When data from the second year were added, this trend continued into 2012-13 at 0.26 sdu. School-value-added increases were positive and sizeable but no longer significant in 2013-14, perhaps due to the fact that the gains for 2013-14 were measured on top of the gains posted during the first two years of TALAS.
Improvement in student proficiency in TALAS schools took slightly longer to achieve but did improve by a significant 0.16 and 0.18 sdu through 2012-13 and 2013-14, respectively. Estimates of the effects of TALAS on graduation rates were uniformly positive and increased as more years of data were added (0.15 to 0.27 to 0.29 sdu), but as in the previous analyses of graduation rates, the effects were not statistically significant, likely due to the limited number of high schools (17) in TALAS.
Subject-Matter Impacts
Averaging across all TALAS schools, proficiency in mathematics and science improved more than in comparison schools (0.21 sdu in both subjects). Consistent with the TALAS emphasis on literacy, English language arts (ELA) gains were positive and significant in elementary schools and middle schools where reading was directly assessed each year; gains on high school English tests also were positive, though not significant. In addition, proficiency gains in elementary science (0.23 sdu) and middle school mathematics (0.37 sdu) were larger in TALAS schools than in the comparison schools.
Teacher Turnover
Throughout the TALAS intervention, teacher turnover was higher in TALAS schools than in the comparison schools, although the difference was not statistically significant. It appears that the lower levels of teacher retention in the TALAS schools may have suppressed some positive effects of transformation.
District-Level Transformation
The 12 districts with the lowest proficiency rates received additional support services, which the Evaluation Team evaluated separately. The district-level transformation produced statistically significant effects on school-wide growth in student achievement in TALAS schools (0.44), while the gains were positive but not statistically significant in TALAS schools in districts that did not receive district transformation services. This finding may indicate the value of district coaching for increasing attention to and support for increasing student achievement growth in the lowest-performing districts—perhaps through leadership and teacher recruitment and placement, providing resources focused on student performance, or creating a structure for discipline and safety.
Sustainability
Sustainability of the effects of TALAS is particularly important as RttT funds run out. The Evaluation Team examined the differential effects on schools that participated in the state’s first transformation efforts (2006 to 2010) as well as the TALAS transformation. First, the gains in proficiency were larger and statistically significant in the schools that participated in TALAS but not in the state’s prior turnaround program (2006-2010), which speaks to the immediacy of positive effects from the TALAS intervention. In addition, gains in both graduation rates (0.69 sdu) and school achievement growth (0.37 sdu) as measured by EVAAS were large and statistically significant in schools that both participated in the prior turnaround initiative and received TALAS services, which may indicate that the turnaround program supports sustained in both initiatives (e.g., coaching and professional development) are needed to maintain positive effects. The finding that graduation rates decreased (-0.60 sdu) in the schools that participated in the prior turnaround program but not TALAS raises concerns about the ability of the lowest-achieving high schools to sustain positive effects without continued support from DST.
Summary and Conclusion
The findings clearly indicate that North Carolina’s lowest-achieving schools in 2009-10 improved their performance during the four years of TALAS. In addition, DST efforts to emphasize literacy have paid dividends in increased proficiency on reading and Language Arts tests in elementary and middle schools when compared to other low-performing schools. Further, it does not appear that these gains in literacy came at the expense of other subjects, since both elementary science and middle school mathematics proficiencies also increased in TALAS schools more than in the comparison schools.
Many TALAS high schools made large gains in their graduation rates. Comparisons to other low-performing high schools indicated that gains were larger in TALAS schools, but the effects were seldom statistically significant. While this may be attributable to the limited number of high schools in TALAS (17) and in the comparison schools (18), it also may be that the effects of TALAS are difficult to distinguish from the nearly ten-percentage-point increase in the statewide graduation rate during the RttT period.
It appears that the effects of TALAS are larger when district-level coaching and support are included with school leadership and instructional coaching. The schools that participated in both the earlier transformation program and TALAS registered the largest gains during the RttT funding period. This may indicate that services sustained over a longer time contribute to greater growth. The fact that TALAS school-wide student growth began to improve in the first year of TALAS and was sustained throughout the duration of the program may indicate that the Comprehensive Needs Assessments and School Improvement Plans that were developed in the first year of TALAS were more effective in producing immediate school-wide student achievement growth and sustaining the growth throughout the study period than in the first round of school transformation in North Carolina. However, if all students are to receive an adequate education, the conditions in North Carolina schools, turnover in the educator workforce, and the variable capacity of school districts to foster and maintain satisfactory levels of student proficiency and achievement may mandate that the state find resources and continue to intervene to transform low-performing schools on an ongoing basis for the foreseeable future.
1Eleven of the original 118 schools have closed since the beginning of the initiative.
2In addition to the TALAS schools that were closed, the outcomes for one high school were not reported in each year of the study period and it was omitted from the analysis. Small sample sizes result in reduced power to detect statistically significant effects.
View Resource
Full Report PDF Projects
This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.
Published
September 1, 2015
Resource Type
Report
Published By
Consortium for Educational Research and Evaluation–North Carolina