Skip to main content
William & Ida Friday Institute for Educational Innovation

Golden LEAF STEM Initiative Evaluation: Baseline Report

Executive Summary

Student success in the core content areas of science, technology, engineering, and mathematics (STEM) is essential for the development of an American workforce that can compete in the global economy. In response to this critical need states across the country, including North Carolina, have developed K–12 initiatives designed to inspire and prepare the next generation of scientists, mathematicians, and engineers.

In North Carolina, the Golden LEAF Foundation (Golden LEAF) is a leader in the effort to promote and sustain high quality STEM education in public schools. A key component of the Golden LEAF grants program provides strategic funding for innovative K-12 education projects. In 2010 the Foundation launched a STEM Initiative to support “successful models that increase STEM education for students in grades four through nine in rural, economically distressed and/or tobacco-dependent counties of North Carolina.”

The Golden LEAF STEM Initiative evaluation team has been charged with completing a formative and summative evaluation and acting as a resource for the participating grantees who will be conducting some evaluation of their own. The two primary objectives of the Golden LEAF STEM Initiative evaluation are to:

  • Provide information about the quality of implementation and extent to which the Golden LEAF STEM Initiative achieved its stated goals, and
  • Provide resources and support for grantees to increase capacity of school and district staff to conduct program evaluation.

This baseline report is divided into the following two sections, organized around the two primary objectives of the evaluation, and a series of appendices.

  1. Evaluation of the Initiative – Describes the evaluation of the Golden LEAF STEM Initiative. This section includes a description of the participating schools, districts, and their program strategies. It also contains a description of the evaluation activities and early results, including the formative themes that have emerged from the evaluation work thus far.
  2. Evaluation Capacity-Building – Reviews the capacity-building work that has taken place in Year 1 and provides a summary of the next phases of the evaluation.

The appendices contain descriptions of the Golden LEAF STEM Initiative grants, the Golden LEAF STEM Implementation Rubric with aggregate results, meeting agendas, interview and focus group protocols, and grantee logic models.

I. Evaluation of the Initiative

In order to evaluate the overall effectiveness of the Golden LEAF STEM Initiative in achieving its goal of improving STEM education outcomes for 4th through 9th graders in rural areas in North Carolina, quantitative and qualitative data is being collected from multiple sources in three separate time periods, October 2011 through April 2012, October 2012 through April 2013, and November 2013 through February 2014. The results from these three periods of data collection will be synthesized and compared in order to answer the four primary evaluation questions:

To what degree or in what ways were the Golden LEAF STEM Initiative grantees

  1. Faithful in implementing their STEM program’s criteria and goals?
  2. Effective in changing student STEM attitudes?
  3. Effective in changing student STEM learning?
  4. Effective in changing teachers’ instructional practices?

Description of the Golden LEAF STEM Initiative Grantees

The 14 Golden LEAF STEM Initiative grantees are similar in their broad characteristics and goals, but vary in both size and strategy. Golden LEAF STEM Initiative participants include 1,192 teachers and 31,889 students from 116 elementary schools, 88 middle schools, and 18 high schools across the state.

The 14 initiative grantees not only share their rural geography and economy, but they also share several strategies and goals for improving STEM teaching and learning in their school districts.

  • Common strategies focus on science kits, Professional Learning Communities, Project Lead the Way courses, science and math-focused professional development, technology purchases, after-school and/or summer activities, and business and/or higher education partnerships.
  • Common goals include improved performance on EOC/EOGs, increased enrollment in advanced STEM courses, increased enrollment and/or performance in Algebra I, improved teacher STEM content knowledge and pedagogical content knowledge, increased enrollment in STEM pathways, increased student interest in STEM, and increased non-traditional student participation and performance in STEM.

An administrative dataset from the 2009-2010 school year was used to create a baseline summary of the Golden LEAF STEM Initiative grantee schools as a group and all other North Carolina public schools as a state comparison. Results show that grantee schools have lower minority populations and also have higher poverty rates compared to all other schools in the state. These data also reveal that in general, the 222 schools participating in the GLF STEM Initiative are faring the same or slightly better than the state average across most indicators. Some examples include:

  • Grantee schools are also more likely to be in rural areas.
  • Grantee schools are slightly instructionally advantaged – teachers are more likely to have an advanced degree, be fully state licensed, and have more experience; and schools, as a whole, have lower teacher turnover rates.
  • More grantee schools met expected and high growth and made AYP than the rest of the state.
  • Grantee schools perform slightly better than students in other schools in the state in math and science (in 5th and 8th grade EOG tests) and Algebra I (in 8th and 9th grade EOC tests).
  • Students in grantee schools take Algebra I and Biology at similar rates to their non-grantee peers, both as advanced track (Algebra I in 8th grade and Biology in 9th grade) and on-time track (Algebra I in 9th grade and Biology in 10th grade).
  • Students in grantee schools are slightly more likely than students in non-grantee schools to use technology (calculators, computers, or other machines), discuss real-world applications, work in groups, and read in both math and science classes.

This data summary from the 2009-2010 school year establishes a baseline with which to compare change over time in STEM learning and teachers’ instructional practices.

Evaluation Activities

The following instrument development and data collection activities have taken place since the evaluation study began in earnest in fall 2011:

  • Development of pilot teacher attitudes towards STEM surveys
  • Development of pilot middle and high school student attitudes towards STEM surveys
  • Development of pilot upper elementary student attitudes towards STEM survey
  • Development of pilot upper elementary student attitudes towards STEM survey
  • Project coordinator interviews
  • Teacher and student surveys
  • Site visits to participating schools, including classroom observations and teacher focus groups
  • Project teams completed the Golden LEAF STEM Implementation Rubric

Evaluation Results

Project coordinators for the 14 grants used the pilot rubric to assess their program’s depth of implementation according to each of the North Carolina Department of Public Instruction’s STEM Attributes. The coordinators were encouraged to work with their grant’s leadership team to identify where on the implementation continuum they believed their program to be operating for each relevant key element. Results from the pilot administration of the rubric show multiple trends in implementation across grantees. Analysis of data from the rubric revealed that many of the Golden LEAF STEM Initiative grantees self-report earliest success in implementing key elements: Research & Development, A2; Teachers Collaboratively Develop Assessments, A4; Students Work in Teams, B1; Communicate STEM Program Plan, B3; and Credit Completion Availability, C2. Components that will seem to require more focus in Year 2 include: Frequency of PBL, A1; Frequency of STEM Integration, A1; Students & STEM Professionals, A2; Teachers Interact with STEM Industries, B1; and Information Sharing, C1.

Formative findings and observations based on data collected up to the writing of this report have been synthesized. These results have been grouped into six, broad categories:

  • Curriculum, instruction, and pedagogy focused on integrating STEM content, new STEM curricula, Common Core and Essential State Standards, hands-on teaching and learning;
  • School schedules, resources, and technology focused on cost of STEM resources, after-school activities, virtual tools, school schedules, and availability of technology;
  • Professional learning, and collaboration focused on organic professional learning communities, teacher collaboration, time to implement in the classroom, and inquiry-based teaching;
  • School and district leadership focused on STEM education awareness, school system transitions, and project vision;
  • Stakeholder engagement focused on accessing STEM professionals and businesses, community support, and key stakeholder support; and,
  • Program evaluation data collection focused on formative data collection and data that is hard to quantify.

These themes attempt to capture some successes and challenges grantees have experienced during the initial implementation of their STEM initiatives. The evaluation team will use these preliminary findings to inform data collection and analysis and to plan evaluation capacity-building activities in year two.

II. Evaluation Capacity Building

The second of the two objectives of the Golden LEAF STEM Initiative evaluation is to provide technical assistance to increase the capacity of schools and districts for data-informed decision-making. The capacity-building work is focused on achieving two main goals, by supporting each of the grantees to (1) develop and apply knowledge about education program evaluation; and (2) collect, interpret, and use formative data to improve their STEM programs.

In order to accomplish these goals the evaluation team has carried-out several activities thus far: hosting annual face-to-face institutes, holding semi-annual webinars, maintaining a Golden LEAF STEM Initiative evaluation wiki, providing access to online student and teacher attitudes towards STEM surveys and results, developing a STEM program implementation rubric, providing on-going access to evaluation team members as evaluation resources, and outreach efforts for the purpose of engaging national and state education leaders in discussions about the on-going evaluation and capacity-building work for the Golden LEAF STEM Initiative.

Discussion

This report provides measures of the current education landscape for the schools and districts participating in the Golden LEAF STEM Initiative. These measures will serve as a baseline against which to estimate specific impacts of Golden LEAF-sponsored STEM activities at project end. It also provides formative results from qualitative data collection activities completed thus far, results which may be used promptly to inform program decisions at the grantee- or initiative-level. In addition, the report outlines and describes the evaluation team’s evaluation capacity-building activities. The discussion includes study limitations and an overview of next steps for evaluation deliverables, activities, and events.

The evaluation is being conducted by the Consortium for Educational Research and Evaluation–North Carolina (CERE–NC), a partnership of the SERVE Center at the University of North Carolina at Greensboro, the Carolina Institute for Public Policy at the University of North Carolina at Chapel Hill, and the Friday Institute for Educational Innovation at North Carolina State University. CERE–NC looks forward to continuing its investigation of the impacts of Golden LEAF-supported initiatives on STEM outcomes in North Carolina schools.

View Resource

Full Report PDF Full Report PDF

Projects

Evaluation of Race to the Top

This evaluation was designed to provide formative feedback for program improvement and determine impact on the target goals of each initiative and on overall state-level outcome goals.

Published

April 1, 2012

Resource Type

Report

Published By

Consortium for Educational Research and Evaluation–North Carolina