Evaluation of District and School Transformation School-Level Coaching and Professional Development Activities
Executive Summary
North Carolina’s Race to the Top (RttT)-funded initiative to Turn Around the Lowest Achieving Schools (TALAS) is one of the most ambitious school turnaround efforts undertaken across the United States, including other states supported through the federal RttT Fund. In 2012-13, the District and School Transformation (DST) Division of the North Carolina Department of Public Instruction (NCDPI) continued to work on the transformation of North Carolina’s 118 lowest-achieving schools (11 of which have closed since the beginning of the initiative) and also work with 12 districts to support and sustain the transformation implementation. Also in 2012-13, North Carolina began administering new, more rigorous assessments based on the Common Core State Standards and fewer students were found to be proficient on assessments of these higher standards than on previous state assessments. As a result, schools across the state had lower Performance Composites. Even so, comparing changes in Performance Composites from 2009-10 to 2012-13, the schools targeted by DST experienced smaller declines than did other schools at their grade level. On average, under the more rigorous standards, DST elementary schools declined 7.9 points less than did other elementary schools; DST middle schools declined 4.4 points less than did other middle schools; and DST high school changes were about the same as those of other high schools in North Carolina.
We focused much of our attention in this third evaluation report on leadership coaching and how it may affect the leadership and organizational characteristics of DST target schools. The focus on coaching drew on DST’s theory of action, as well as on conversations with RttT and DST leadership at NCDPI. In addition, a recent federal report noted that coaching was the primary support for transforming the DST target schools (USED 2014). This report relied on survey responses by principals and teachers in a random sample of North Carolina public schools gathered by the evaluation team each spring from 2011 through 2013. Throughout the report, the information provided on leadership and organizational characteristics reflects the experiences and perceptions of teachers and the information provided on coaching reflects the experiences and perceptions of principals.
Changes in Leadership and Organizational Characteristics
Overall, leadership and organizational characteristics of all North Carolina public schools, as perceived by teachers, have changed very little from spring 2011 to spring 2013. The largest changes, which amounted to about 1/10th of a point on a seven-point scale, indicated very slight and statistically non-significant declines in classroom management, ratings of principals’ instructional leadership, and teachers’ use of higher-level instructional practices. By spring of 2013, DST target schools registered higher levels on two of nineteen dimensions of leadership and organizational conditions—teacher knowledge-sharing and use of formative assessment—than did the comparison schools that were similar on these measures in 2011. DST target schools improved at a slower rate than did the comparison schools on two other measures of leadership and organizational characteristics: teacher-leader respect and team orientation; it seems reasonable, however, that the lack of improvement on these two measures may be associated with the effects of being designated a turnaround school, or with staff responses to the types of changes (such as increased accountability) that occur in turnaround schools.
Leadership Coaching
Principals of DST schools agreed that their DST School Transformation Coaches had helped them do a better job. When comparing DST principal responses with responses of principals of comparison schools (for whom coaches were defined as someone who “has provided you with deliberate, sustained assistance designed to help you learn or figure out how to improve your school”), DST coaching was rated higher in terms of improving shared leadership and order. The two sets of principals reported no differences in the effectiveness of the coaching they received for improving teaching and assessment practices or for improving teacher efficacy and responsibility. Principals of DST schools either with less experience, or who were rated as having lower skill levels, or both, rated their coaches as more effective than more experienced and higher-skilled principals. This finding suggests that principals at this level of experience found the presence of DST coaches to be more beneficial than did more experienced, higher-skilled principals, and that perhaps this information might be useful for developing a targeting strategy when resources become more limited.
Principals in both the DST target schools and the comparison schools reported that they had been working with their current coaches between four and ten months. Principals of DST schools reported meeting with their coaches a little more often than once every two weeks, while principals of comparison schools reported meeting about once a month with the individuals from whom they received assistance and support. Principals in DST target schools reported that their coaches were significantly more likely to suggest actionable approaches or solutions to the challenges and problems they faced than were reported for the coaches in non-DST schools. Other coaching strategies (such as providing effective feedback and modeling effective behaviors) were reported to be about the same in both DST and comparison schools. All responses from principals in the comparison schools were based on their views of individuals who they personally identified as fulfilling the functions of a coach; as such, their responses were not about a specific source or formal set of coaches.
Across DST and comparison schools, more effective coaching as perceived by principals was associated with a positive and significant difference in four of nineteen dimensions of leadership and organizational conditions that were measured via surveys of teachers: alignment of professional development, program coherence, teacher-teacher trust, and data-driven instruction. This finding suggests that supporting principals through coaching or mentoring can and does make a difference in some important aspects of their schools’ working conditions, but that both the principal coaching presumably provided by some districts or through other sources such as School Improvement Grants as well as coaching provided by DST produced positive effects.
Effects of DST on Leadership and Organizational Conditions
While, overall, teachers’ responses about leadership and organizational characteristics of schools did not change significantly, we found that teachers’ ratings of teacher-leader respect and team orientation within the school went down in DST schools. These changes cannot be attributed to any single cause—such as having been labeled as one of the lowest-performing schools, or any changes in the school as a result of turnaround. This finding could be used to increase awareness of and support for team-building and the development of positive relationship between teachers and leaders.
DST Professional Development
In addition to evaluating DST coaching, we assessed the professional development that DST provided to school leaders. DST provided five professional development experiences that drew between 96 and 198 of the leaders from target schools and districts. The sessions were geared to the specific needs of turnaround schools and, overall, all of the sessions were highly rated by participants. In addition, of the 15 segments of the professional development sessions observed by the evaluation team, the eight characteristics of quality professional development occurred in between 67 and 100 percent of the segments. All occurrences of the eight characteristics of quality were rated “good,” and between 64 and 100 percent of the professional development segments had “a lot” of each high-quality characteristic.
Next Steps for the Evaluation
In the final phase of the evaluation of the RttT DST initiative, the evaluation team will begin to assess the overall impacts of the initiative on student achievement, teacher effectiveness, teacher retention, and other measures. In addition, we will examine, to the extent possible, how DST has influenced improvements in performance to help guide future turnaround efforts.
View Resource
Full Report PDF Projects
This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.
Published
October 1, 2014
Resource Type
Report
Published By
Consortium for Educational Research and Evaluation–North Carolina