Skip to main content
William & Ida Friday Institute for Educational Innovation

Distinguished Leadership in Practice (DLP): Third Annual RttT Evaluation Report— A Final Summary

Executive Summary

Providing high-quality, accessible professional development to all teachers and principals is a critical component of the professional development plan funded by North Carolina’s federal Race to the Top (RttT) grant. One key professional development program funded through RttT is the Distinguished Leadership in Practice (DLP) program. Designed for practicing principals, DLP is aligned to the performance evaluation standards adopted by the State Board of Education for North Carolina’s school leaders (i.e., the North Carolina Standards for School Executives). The DLP program is provided by the North Carolina Principals and Assistant Principals’ Association (NCPAPA) in partnership with North Carolina Department of Public Instruction (NCDPI).

Overview of North Carolina RttT DLP Activities

The DLP initiative employs a non-traditional professional development model. Participants examine the meaning and application of school leadership through a problem-based approach delivered via a series of face-to-face, regional, cohort-based sessions, supplemented by online activities. Throughout the year-long experience, practicing North Carolina principals are coached using a continuous improvement model. Participating principals are provided with models of exemplary school leadership, which allows them to study the behaviors, attitudes, and competencies that define a distinguished school leader. The DLP experience is built around six components:

  • Component One: Strategic Leadership for High-Performing Schools
  • Component Two: Maximizing Human Resources for Goal Accomplishment
  • Component Three: Building a Collaborative Culture with Distributed Leadership
  • Component Four: Improving Teaching and Learning for High-Performing Schools
  • Component Five: Creating a Strong Student and External Stakeholder Focus
  • Component Six: Leading Change to Drive Continuous Improvement

Overview of North Carolina RttT DLP Evaluation Activities

North Carolina’s RttT proposal included a commitment to independent evaluations of each initiative. Over the course of the evaluation, the RttT Evaluation Team documented the DLP activities and collected data about participation in, satisfaction with, and the impact of DLP professional development activities through surveys and focus groups with DLP participants and facilitators, as well as analysis of longitudinal education data on principals. The purpose of this evaluation is to provide detailed information about the implementation and impact of this professional development effort that targets practicing principals. This evaluation study is one part of a larger effort to evaluate the implementation and impact of North Carolina’s RttT professional development initiatives in order to determine if the initiatives as implemented have led to the intended outcomes with respect to school leader practice, the culture and climate of achievement at those leaders’ schools, and, potentially, teacher and student performance.

The questions for the DLP evaluation fall into seven categories and are aligned with the overarching evaluation questions for RttT professional development:

  1. Program Description: How is the DLP initiative operationalized and implemented?
  2. Participation: To what extent does DLP reach the intended participants?
  3. Program Quality: To what extent does the DLP program meet standards of high-quality professional development?
  4. Short-Term Outcomes: To what extent did participants acquire intended knowledge and skills as a result of their participation in DLP?
  5. Intermediate Outcomes: What was the impact of DLP on participants’ practice?
  6. Long-Term Outcomes: What was the impact of the principals’ participation in DLP on their schools’ culture/climate of achievement?
  7. Distal Outcome: To what extent are gains in student performance outcomes associated with principals’ participation in DLP?

The first two evaluation reports in this series provide very detailed information about evaluation questions I through IV; therefore, although this report addresses all evaluation questions, the emphasis is on new data that address evaluation questions II, IV, V, and VI. Minimal elapsed time since program participation continues to hamper the capacity to address the longer-term outcomes in evaluation questions VI and VII fully. This report focuses on DLP Cohorts 1 and 2 but also provides enrollment and participation findings for Cohorts 3 and 4.

Evaluation Findings

  1. Program Description: The DLP program is a cohort-based, experiential program that is delivered over a one-year period using a blended model of face-to-face sessions supplemented by online sessions. Its professional development model allows participants to critically examine the meaning and application of school leadership through a problem-based, real-world approach. Sessions are facilitated by 14 highly-qualified individuals who are former or current principals. Overall, DLP consists of approximately 60 hours of face-to-face work and 190 hours of online work, for a total of 250 hours of professional development. Based on estimated expenditures from the 2013-14 year, the program is estimated to cost approximately $2,289 per participant and $3,094 per completer going forward.
  2. Participation: DLP enrolled 634 principals from 2011-2012 through 2014-2015, for an average of 159 principals per year. This participation level met the target of serving 150 principals annually. Participants from 2011-2012 through 2014-2015 came from 100 of the state’s 115 Local Education Agencies (LEAs). Many participating principals already high-achieving before they enrolled in this professional development program. Data from participants’ applications indicate that participants come from a variety of backgrounds and school contexts, and that they are fairly representative of principals across the state. DLP is a rigorous program that requires a considerable time commitment from participants. About 20 percent of initial participants do not complete the program.
  3. Program Quality: Considerable evidence in the former two DLP evaluation reports indicates that the DLP program has been of high quality. Specifically, it aligned with RttT priorities, met principals’ professional development needs, and provided high-quality face-to-face and online sessions. Overall, the program was very highly regarded by each of the first two cohorts of participants. Nearly all of the participants agreed or strongly agreed at the conclusion of the program that it had a clear purpose, was relevant, and was of high quality. Among former participants in the first two cohorts, about three-fourths of participants now consider the components focusing on building a collaborative culture, improving teaching and learning, and leading change to have been the most valuable.
  4. Short-Term Outcomes: Responses to the post-component surveys were very positive overall, with results suggesting that participants learned the intended knowledge and skills over the course of each component. At the program’s conclusion, and in the first and second years that followed, participants were asked to reflect on whether their participation in DLP had given them a better understanding of the knowledge and skills the program was designed to provide. Results were overwhelmingly positive across all eight survey items, with at least 80 percent of respondents (and often many more) agreeing or strongly agreeing with each item.
  5. Intermediate Outcomes: Positive survey responses from both DLP participants and their personnel—including teachers, instructional support staff, and assistant principals—indicated that DLP principals had started to implement the new knowledge and skills they learned in the program. Not only were responses favorable, but the personnel survey responses also matched the responses of their own principals to a high degree across survey items. Participant follow-up interviews, consistent with survey data, revealed that principals reported increased confidence in instructional leadership and cultural leadership through relationship development. However, no clearly discernable, statistically significant patterns emerged for changes in principals’ North Carolina Standards for School Executives evaluation ratings before and after DLP participation. This finding might be attributable to the non-sensitivity of the administrator rating scale.
  6. Long-Term Outcomes: Survey results provide a clear consensus among both teachers and principals that the school- and teacher-level focus on student achievement has increased since principals participated in DLP. Despite these largely favorable survey responses, principals were hesitant in their qualitative responses to attribute changes directly to the program because of competing factors and other concurrent state policy changes.
  7. Distal Outcomes: As detailed above and in previous reports, DLP sets up principals to have the capacity to influence student achievement, vis-à-vis their influence on the school and their staff. However, it is still too early to establish any direct, causal link, due to current data limitations, the limited time elapsed since the principals participated in the program, and the implementation of multiple state policy changes at the same time as this training program. For these reasons, analyses using administrative data to identify DLP principals’ impacts on students are not included in this report.

Recommendations

As this report details, the data clearly show that the DLP team has designed and implemented a very high-quality program that meets the professional development needs of the participating school leaders. These principals are building intended knowledge and skills, improving their practices, and improving the culture of achievement in their schools. This level of quality, which builds upon lessons learned from previous cohorts, reflects the DLP team’s commitment to continuous improvement processes. Some of the data in this report will help inform those processes as the DLP team continues to refine an already-strong program. Areas that the data suggest might be considered in future program improvements are summarized here.

  • Provide graduate course credit. The program should consider offering course credit toward advanced degrees given the amount and depth of work involved. Three-hour university courses generally require 150 hours of student work, and this program requires a 250-hour commitment from participants. Tailoring the training for course credit also will require rigorous assessments.
  • Consider more sustainable funding models. Some of the expenses for future participation might need to be covered by principals’ LEAs. Currently, LEAs incur no direct costs for principals participating in DLP. NCPAPA should consider whether LEAs can provide support for travel-related expenses and/or a subsidized registration fee.
  • Emphasize realistic solutions to challenges of implementing school reform practices. Despite participants’ consistently high praise for the program, some indicate that the DLP teachings are too idealistic for the actual conditions on the ground. Using past DLP participants to speak directly about the challenges involved in implementing the recommended changes would help provide a more balanced perspective. Also consider ways to ensure material is applicable for principals who will move into new schools or into new leadership positions at the LEA level.
  • Further differentiate and customize learning activities. Differentiation and customization could be further supported. Early in the program, facilitators could use data from applications or surveys to determine whether participants have any specific learning or scheduling needs to be addressed. Feedback from participants suggested differentiation of activities based on school level, urbanicity, and size, and also tailored to their professional growth plan.
  • Incorporate additional learning assessments. Targeted learning activities and assignments— incorporated into the modules and designed to measure participants’ changes in knowledge, skills, and behaviors—could help bolster the claims that this is a high-quality program with positive impacts on participants. Program developers and facilitators should consider a pre- and post-test of leadership knowledge and skills, or a portfolio of work, built over the course of the DLP year, as evidence of application of new knowledge and skills and of changes in practice.

View Resource

Full Report PDF Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

December 1, 2014

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina