Skip to main content
William & Ida Friday Institute for Educational Innovation

Distinguished Leadership in Practice (DLP): Second Annual Evaluation Report

Executive Summary

Providing high-quality, accessible professional development to all teachers and principals is a critical component of the professional development plan funded by North Carolina’s federal Race to the Top (RttT) grant. One key professional development program funded through RttT is the Distinguished Leadership in Practice (DLP) program. Designed for all practicing principals, DLP is aligned to the performance evaluation standards adopted by the State Board of Education for North Carolina’s school leaders (i.e., the North Carolina Standards for School Executives). The DLP program is provided by the North Carolina Principals and Assistant Principals’ Association (NCPAPA) in partnership with North Carolina Department of Public Instruction (NCDPI).

Overview of North Carolina RttT DLP Activities

The DLP initiative employs a non-traditional professional development model. Participants examine the meaning and application of school leadership through a problem-based approach delivered via a series of face-to-face, regional, cohort-based sessions, supplemented by online activities. Throughout the year-long experience, practicing North Carolina principals are coached using a continuous improvement model. Participating principals are provided with models of exemplary school leadership, which allows them to study the behaviors, attitudes, and competencies that define a distinguished school leader. The DLP experience is built around six components:

  • Component One: Strategic Leadership for High Performing Schools
  • Component Two: Maximizing Human Resources for Goal Accomplishment
  • Component Three: Building a Collaborative Culture with Distributed Leadership
  • Component Four: Improving Teaching and Learning for High Performing Schools
  • Component Five: Creating a Strong Student and External Stakeholder Focus
  • Component Six: Leading Change to Drive Continuous Improvement

Overview of North Carolina RttT DLP Evaluation Activities

North Carolina’s RttT proposal included a commitment to independent evaluations of each initiative. Over the course of the evaluation, the RttT Evaluation Team will document the DLP activities and collect data about participation in, satisfaction with, and the impact of DLP professional development activities through surveys and focus groups with DLP participants and facilitators, as well as analysis of longitudinal education data on students, teachers, leaders, and schools. The purpose of this evaluation is to provide detailed information about the implementation and impact of this professional development effort that targets practicing principals. This evaluation study is one part of a larger effort to evaluate the implementation and impact of North Carolina’s RttT professional development initiatives in order to determine if the initiatives as implemented have led to the intended outcomes with respect to school leader practice, the culture and climate of achievement at those leaders’ schools, and, potentially, teacher and student performance.

The questions for the DLP evaluation fall into seven categories and are aligned with the overarching evaluation questions for RttT professional development.

  1. Program Description: How is the DLP initiative operationalized and implemented?
  2. Participation: To what extent does DLP reach the intended participants?
  3. Program Quality: To what extent does the DLP program meet standards of high-quality professional development?
  4. Short-Term Outcomes: To what extent did participants acquire intended knowledge and skills as a result of their participation in DLP?
  5. Intermediate Outcomes: What was the impact of DLP on participants’ practice?
  6. Long-Term Outcomes: What was the impact of the principals’ participation in DLP on their schools’ culture/climate?
  7. Distal Outcomes: To what extent are gains in student performance outcomes associated with principals’ participation in DLP?

The first annual DLP evaluation report, submitted in May 2012, provided baseline data to answer evaluation questions related to program description, participation, program quality, and short-term outcomes, and it also provided some initial information related to intermediate outcomes. This report more fully addresses questions I through IV (program description, participation, program quality, and short-term outcomes), and it also provides additional information related to questions V through VII (intermediate, long-term, and distal outcomes). This second annual report focuses on the third cohort of the DLP program (April 2012 through March 2013). In addition, the report includes a one-year follow-up of the previous year’s participants (Cohort 2). A more thorough investigation of the long-term and distal outcomes will be the focus of the final evaluation report.

Evaluation Findings

As detailed in this report, the data clearly show that the DLP team has designed and implemented a very high-quality program that aligns to national professional development standards and meets the professional development needs of the participating school leaders. Participants reported that they are building intended knowledge and skills, positively impacting school leaders’ practice, and improving the culture in their schools. This level of quality, building upon lessons learned from previous cohorts, reflects the DLP team’s commitment to continuous improvement processes.

  1. Program Description: The DLP program employs a non-traditional professional development model that allows participants to critically examine the meaning and application of school leadership through a problem-based, real-world approach. This cohort-based, experiential program is delivered over a one-year period using a blended model of face-to-face sessions and online sessions. Sessions are facilitated by 14 highly-qualified individuals who are former or current principals. Overall, DLP consists of approximately 60 hours of face-to-face work and 190 hours of online work, for a total of 250 hours of professional development. Based on actual expenditures from the 2011-12 year, totaling $395,394, the program is estimated to cost $2,368 per participant (n=167).[1]
  2. Participation: This year, DLP sessions were conducted in three regions (Central, East, and West). The program began with 167 principals participating across the regions, 135 of whom completed all six components. This participation level met the target of serving 150 principals annually. Data from participants’ applications indicate that participants come from a variety of backgrounds and school contexts, and that they are fairly representative of principals across the state.
  3. Program Quality: The DLP program components most closely align with the RttT focus on updating the education workforce, in that DLP’s goal is to help principals progress professionally, as measured by the North Carolina Standards for School Executives. Most participants (92%) agreed or strongly agreed that both the face-to-face sessions and the DLP program as a whole were of high quality overall; a high percentage (84%) of participants also agreed or strongly agreed that the online sessions were of high quality. Nearly all survey respondents agreed or strongly agreed that the face-to-face sessions were relevant to their professional development needs (97% at post-face-to-face, 95% at year-end) and provided them with useful resources (96% at post-face-to-face, 95% at year-end). Also, nearly all of the participants (99%) agreed or strongly agreed that the face-to-face sessions were led by effective facilitators. Observational data provided converging evidence of the overall quality of the DLP program. Participant feedback suggests that some participants enjoyed the face-to-face sessions more than the online sessions. Regional comparisons revealed a pattern whereby participants in the West tended to be less satisfied than participants in the Central or Eastern regions. Given that the curriculum was consistent across regions and that facilitators rotated across regions, the source of these regional differences is unlikely to be programmatic and more likely to be associated with the participants themselves and related group dynamics.
  4. Short-Term Outcomes: Overall, results were very favorable; with at least 80% of respondents agreeing or strongly agreeing that they developed a better understanding of the learning objectives through their participation in DLP. For nearly all of the learning objectives presented in the surveys, at least 90% of the respondents agreed or strongly agreed that they developed a better understanding through their participation in DLP.
  5. Intermediate Outcomes: Based on self-report ratings from DLP Cohort 2, nearly half (47%) of those who had room for improvement (rated as Developing, Proficient, or Accomplished, but not Distinguished) increased their leadership level over the course of their year in DLP. Data from the one-year follow-up survey revealed 99% of principal respondents have applied what they learned about how students learn effectively and how to manage change effectively. Results from an analysis of administrative data suggest that principals in DLP Cohort 2 demonstrated similar changes in leadership over the course of their year in DLP, as did other principals in the state. Likewise, DLP Cohort 2 completers and those who withdrew from the program showed similar growth.
  6. Long-Term Outcomes: Eighty-eight percent of DLP Cohort 2 participants strongly agreed or agreed that they had noticed improvements in their schools’ culture since participating in the DLP program. Moreover, program completers were significantly more likely than withdrawals to indicate noticing such improvements (90% vs. 67%).
  7. Distal Outcomes: About three-quarters of the DLP Cohort 2 principals (n=95) strongly agreed or agreed that they had noticed improvements in student achievement since participating in DLP. Moreover, 78% of program completers (n=88) reported noticing improvements in student performance since participating in DLP, compared to only 56% of those who withdrew (n=9) from the program.

Recommendations

Some of the data in this report will help inform those processes as the DLP team continues to refine the already strong program. Areas that the data suggest might be considered in future program improvements are summarized here.

  • Provide Graduate Course Credit – Some of last year’s participants felt the program should offer course credit towards advanced degrees given the amount and depth of work involved. DLP staff could explore collaborations with Colleges of Education about the possibility of providing graduate course credit for completion of DLP.
  • Further Differentiate and Customize Learning Activities – Differentiation and customization could be further supported through the use of a pre-DLP survey and findings from this report. Such data could inform facilitators if participants have any specific learning or scheduling needs to be addressed. For example, some members of a focus group suggested including content on special topics, such as Professional Learning Communities, providing developmental feedback to staff, and using marketing strategies for creating a positive school image. Feedback from participants suggested differentiation of activities based on school level and size and tailored to their professional growth plan.
  • Adjust the Time, Timing and Number of Some Activities – A majority of participants indicated they would have preferred to spend less time in online sessions and large minority would have preferred spending more time in face-to-face sessions. Some participants suggested better alignment of the DLP conversations, assignments, and programming with the school year; having fewer assignments (i.e., streamlining), giving more time to complete assignments, and giving more advanced notice (i.e., a syllabus), especially for assignments requiring interaction with colleagues and students.
  • Continue to Provide Opportunities for Participant Leadership – Participants could be assigned to lead group discussions or give formal presentations on short segments of material or about their areas of expertise. Small groups of participants also could present to each other after working on a collaborative problem-solving project in face-to-face or online sessions.
  • Increase Time for Collaboration and Networking – Program developers could consider integrating even more activities that require teamwork to complete during face-to-face sessions and during online sessions. Mentoring partnerships could create opportunities for collaboration and networking. Several of the participants suggested that DLP should have follow-up sessions with their cohort to facilitate on-going collaboration with fellow alumni after the program.
  • Continue to Improve Online Sessions – Although the online tools used to support instruction were appropriate to the activities, they were primarily limited to the use of asynchronous discussion forums and static web pages to share content. Tools that can be integrated include, but are not limited to: wikis, video-making tools, audio editing tools, data visualization tools, simulations, synchronous interaction platforms, blogs, survey tools, and mind mapping tools. Also, participants suggested that DLP staff should consider a) providing additional technical support for existing tools and b) clearly communicating up front to participants that requirements for the online sessions account for over three-quarters of the time commitment.

Limitations

Findings on participant outcomes for this report are almost entirely derived from participant self-report survey data. While North Carolina Educator Evaluation System ratings for participating principals also were used, these administrative records were matched at a rate of only 75% to the sample. In addition, there was minimal variability in Evaluation System ratings across the population of North Carolina principals, further limiting the ability to detect meaningful changes.

Next Steps for the DLP Evaluation

The final annual report, scheduled for release in Fall 2014, will be summative in nature. It will seek to identify the longer-term and distal outcomes of DLP Cohort 2 participants (2011-12) using a mixed-methods approach, and will include additional data sources to better triangulate self-reported findings. The evaluation will identify the impact of the principals’ participation in DLP on their schools’ culture/climate of achievement; and, also will address preliminary student achievement impacts.

Also, three general patterns emerged from the data this year that warrant further attention in the final report: first, participants in the West tended to be less satisfied with the program than were participants in the Central or Eastern regions; second, some participants had less favorable impressions of their DLP experience at year-end than they did earlier in the program; and third, the online sessions were consistently rated lower than the face-to-face sessions. The Evaluation Team will work closely with the DLP team to consider survey items or administration techniques that could allow for investigation and explanation of these findings.

1 Marginal cost; does not include original planning and design costs for DLP program.

View Resource

Full Report PDF Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

November 1, 2013

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina