RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
Executive Summary
The North Carolina Race to the Top (RttT) professional development plan is an expansive and multi-faceted effort to increase student achievement by updating the knowledge and skills of the state’s entire K-12 public education workforce. This initiative is driven by a host of recent changes, including: adoption of new Common Core State Standards and North Carolina Essential Standards; increased use of data to inform classroom and school decisions; rapid changes in the technologies and digital resources available for teaching and learning; new teacher and administrator evaluation processes; and an increased emphasis on formative assessment to inform instructional decisions.
The human resources challenge of the initiative—to provide the state’s 100,000 teachers and 2,400 principals with professional development that will enable them to extend their knowledge, improve professional practices, and, ultimately, increase student achievement overall and close achievement gaps among student groups—is formidable. The timeframe (the four-year period of the grant), diversity of the State (from large metropolitan local education agencies [LEAs] to small, rural, and resource-limited LEAs, many of which continue to struggle under the weight of fiscal constraints), and expectations (to create a statewide professional development infrastructure that can be sustained after RttT funding ends) only increase that challenge. The RttT professional development evaluation is being conducted in full recognition of these circumstances, as well as of the deep commitment of the members of the RttT Professional Development Implementation Team. The intent of the evaluation is to provide data-driven information that can support reflection about and improvement of this effort.
Four general questions guide the evaluation:
- State Strategies: To what extent did the state implement and support proposed RttT professional development efforts?
- Short-Term Outcomes: What were direct outcomes of State-level RttT professional development efforts?
- Intermediate Outcomes: To what extent did RttT professional development efforts successfully update the NC education workforce?
- Impacts on Student Performance: To what extent are gains in student performance outcomes associated with RttT professional development?
The Evaluation Team is providing this second annual assessment of progress in three separate but related reports. This report—the Local Outcomes Baseline Study— provides a baseline (first-year) assessment of the State’s progress toward updating the education workforce in North Carolina (Evaluation Question 3). For this report, the Evaluation Team (a) collected and analyzed relevant data from all 115 LEAs, and (b) identified a purposeful sample of 27 schools to examine more deeply the extent to which LEA and school staff increased capacity to provide high-quality professional development. The other two reports address (1) the state’s progress in delivering face-to-face professional development statewide, and (2) implementation and impact of the State’s Online Professional Development.
Summary of Major Baseline Findings
To address Evaluation Question 3, the Team examined the extent to which participation in the state’s professional development efforts impacted (a) local capacity to provide high-quality professional development, (b) shifts in local organizational conditions to support RttT priorities, and (c) changes to instructional practice. Results from these baseline analyses will be used in future reports to gauge progress toward updating the education workforce statewide.
Evaluation Question 3.A.: To what extent did leader participants improve practices for support of organizational change, including capacity to provide high-quality local PD aligned to RttT priorities?
Planning and Accessing Professional Development.
For most schools, focus group and interview data reveal that school-level professional development was informed by what was being offered at the state level and typically was designed around resources available at the LEA level. In line with the state-endorsed “train the trainer” model, LEA-level Professional Development Coordinators indicated that they were primarily responsible for identifying and disseminating the most relevant professional development resources and making them available to principals and teachers in their LEAs. Most LEA Coordinators mentioned using tools like wikis, website postings, and staff development newsletters and calendars to condense professional development resources and make them more easily accessible to teachers and principals. However, despite the multiple methods of dissemination described by LEA leaders, data indicate that some principals and teachers still remained uninformed about NCDPI-support professional development opportunities.
Implementing Professional Development.
Most Professional Development Coordinators agreed that their LEA leadership had the capacity to implement and plan professional development, but fewer believed that their LEA leaders used data to inform their professional development decisions. Some LEAs called on teams of school-level leaders to develop implementation strategies. In most schools, professional development training was implemented through Professional Learning Communities (PLCs), and in some LEAs, training opportunities were extended beyond the school year.
Alignment with RttT Priorities.
LEA Professional Development Coordinators statewide and teachers in the representative school sample held different perceptions about whether professional development helped teachers build their knowledge and skills related to some RttT priorities, such as understanding revised state standards (91% of Coordinators but only 64% of teachers agreed that it helped) and deepening their content knowledge (81% and 58%, respectively). For other priority areas, the perceptions of both groups were more closely aligned, but were typically low (e.g., only 51% and 52%, respectively, agreed that teachers built knowledge and skills related to revised state assessments).
In addition, while 92% of Coordinators reported that LEA-provided RttT professional development was aligned with and built upon existing professional development initiatives, only 64% of the coordinators agreed that their LEAs utilized data on staff technology proficiency when planning RttT professional development. Furthermore, only 60% agreed that their LEAs have plans for how to use the NCDPI Professional Development Leads in each region for support in their delivery of professional development. Overall, these differing perceptions suggest a lack of consistency among LEA PD coordinators in their efforts to align professional development with RttT priorities at the local level.
Quality of Local Professional Development.
As they reflected on the academic year, about 70% of teachers surveyed agreed or strongly agreed that their local professional development experiences were both sustained and coherent, though fewer believed that they had time to reflect on and experiment with what they learned—a sentiment that was echoed in focus group responses. Teachers generally gave lower ratings for their experiences with LEA-provided professional development than did LEA Professional Development Coordinators, but interview and focus group data highlighted principals’ and teachers’ positive experiences with local coaches and specialists, as well as their positive impressions of the value of PLCs.
Evaluation Question 3.B.: How did school culture/organizational conditions change to support RttT priorities?
Changes in Conditions related to Transition to the New Standards
Most LEA Professional Development Coordinators agreed that their LEAs have strategies in place for communicating about availability of, collaborating on, and integrating state resources into professional development related to revised state standards, but preparation for integrating other aspects of the curriculum with those standards varies. Results from the RttT Omnibus Survey showed that nearly all teachers felt strongly about their own preparation for the transition to Common Core but were less sure about whether school leaders or other teachers in their school really understood how to implement the new standards effectively.
Changes in Conditions related to Data-based Decision Making and the Revised Teacher Evaluation Process
Between 2010-11 and 2011-12, there were slight decreases in teacher perceptions of their schools’ use of data for decision-making, as well as in their perceptions of the new teacher evaluation process, but both changes may be the result of a sizeable increase in response rates. The Evaluation Team will monitor annual survey results to determine whether these patterns persist.
Evaluation Question 3.C.: To what extent did teachers improve classroom practice?
Data gathered for this section provide baseline information about current learning activities for comparison to similar data in subsequent years of RttT. Overall, Math teachers most frequently reported daily instructional time on instructional strategies related to new standards, followed by English Language Arts (ELA), Science, and Social Studies teachers. Across each content area:
- ELA: The highest percentage of teachers estimated that students spent time almost daily on listening skills (76%) and on general vocabulary (66%).
- Math: The highest percentage of teachers reported that on almost a daily basis, students made sense of problems (80%), persevered in solving problems (72%), and used appropriate tools strategically (67%).
- Science: Teachers reported that their students practiced participating in hands-on activities (32%), completed activities with a real-world context (24%), and used tools to gather data (19%).
- Social Studies: Students spent time almost daily on recognizing and appreciating contributions of diverse cultural groups (30%), demonstrating chronological thinking (29%), and analyzing cause-and-effect relationships (28%).
On average, observed classrooms in the sample schools received midrange ratings across the five dimensions of the Evaluation Team’s standard classroom observation protocol[1] that are most directly aligned with implementation of the new Common Core and Essential Standards (Regard for Student Perspectives, Quality of Feedback, Instruction Dialogue, Analysis and Problem Solving, and Content Understanding). Across content areas, teachers sometimes scaffolded discussion in the classroom and made attempts to integrate student feedback. ELA classrooms were scored slightly higher on average than were other core courses in the frequency and quality of examples of three of the dimensions of effective teaching (Regard for Student Perspectives, Quality of Feedback, and Instruction Dialogue). Social Studies classrooms received the lowest average ratings for observed frequency and quality of the other two dimensions (Analysis and Problem Solving and Content Understanding). Finally, teacher surveys indicated that while some formative assessment practices were common across classrooms, others were employed infrequently. These observation data mirror results from teachers’ self-report surveys about frequency of effective, standards-based teaching practices.
Summary of Results
Overall, the baseline results presented in this study indicate that schools and LEAs have taken some initial steps toward developing a process for professional development that supports RttT priorities, including transition to the new state standards, implementation of formative and summative assessments, use of data to support instruction, and effective utilization of the new North Carolina Educator Evaluation System. These aggregated results provide a cursory baseline assessment of Year 1 progress toward achieving and sustaining local capacity to provide high-quality professional development, supporting shifts in local organizational conditions to support RttT priorities, and encouraging changes in instructional practice.
1 Classroom Assessment Scoring System, or CLASS (http://www.teachstone.org/about-the-class/). In all, there are 12 CLASS domains.
View Resource
Full Report PDF Projects
This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.
Published
March 1, 2013
Resource Type
Report
Published By
Consortium for Educational Research and Evaluation–North Carolina