Maximizing the Impact of STEM Outreach through Data-Driven Decision-Making (MISO)

The purpose of this NSF-funded I3 project was to creatively provide evaluation capacity-building tools and strategies to NC State’s K-12 STEM outreach programs, particularly those funded by NSF, to help ensure the breadth and depth of the future U.S. STEM workforce. North Carolina State University had over 120 STEM outreach projects funded between 2003 and 2009. Among these are many funded by NSF, including the ITEST, MSP, Noyce, GSE, AGEP, CCLI, GK-12, and ERC programs. The vision for MISO is an integrated institutional structure that will allow pre-college programs to think and act strategically to meet the goals below. While the first two goals are new endeavors made possible through the proposed project, all four goals are shared among all participating K-12 STEM outreach programs:

Provide an innovative network of support and communications among University-based outreach project directors and educational evaluation experts, creating a learning community to promote sharing of best practices and innovation that will deepen the impact of NC State’s pre-college STEM programs on students’ future academic and career choices.
Develop and demonstrate a system of data-driven planning and analysis guided by best practices to facilitate longitudinal assessment of participant outcomes through development of a common STEM Outreach Evaluation Protocol as well as a database integrating records of NC State K-12 outreach participants with NC Department of Public Instruction records and university enrollment.
Use the longitudinal assessment of outreach participant outcomes within the university-wide outreach learning community to support more seamless transitions across critical educational junctures.
Broaden participation among underrepresented groups in pre-college STEM outreach activities through integrated recruitment and support strategies.

Evaluation Tools

One important outcome of the project was the development of the S-STEM and T-STEM instruments. The S-STEM instrument emerged out of the project team’s conclusion that there did not exist a well validated, reliable instrument for measuring student attitudes (i.e., self-efficacy and goal orientation) towards STEM subjects, STEM career pathways, and 21st century skills—key outcomes for critical STEM education and workforce development programs. The T-STEM similarly measured teacher’s perceptions of their ability to teach STEM subjects and 21st century skills. While a few survey instruments measure related attitudinal dimensions, they do not systematically gather this data across all of the STEM subject areas. This data was central to the work of the MISO project and other projects being undertaken at the Friday Institute and thus spurred our efforts in this endeavor.

In our work reporting on the development and validation of these instruments, we have received an enthusiastic response from the STEM education research and evaluation community, witnessed in part by the large number of requests we’ve had in the short time it has been publicly posted. To date, the T-STEM and S-STEM surveys have been requested for use by more than 1,000 potential users. The instruments have been reported on at major conferences (e.g., NARST, AERA, ASEE, AEA) and with NSF umbrella communities such as CADRE and the MSP Knowledge Management and Dissemination (KMD) group. Within the state of North Carolina, they have already been used by both state and district wide STEM education initiatives. We are continuing to use these instruments in our own research and evaluation; and in continuing to develop, validate, and disseminating these instruments.

Please feel free to contact us with any questions. While this project is no longer active, the project team will do our best to respond and answer your queries.

Funders

National Science Foundation

Teams

\ Program Evaluation and Education Research (PEER) Group STEM Cyberlearning

Selected Resources

Survey Instrument – Student Attitudes Toward STEM (S-STEM) Survey: Development and Psychometric Properties

The S-STEM survey invites students to give information about their attitudes toward science, technology, engineering, and mathematics subjects, post-secondary pathways, and career interests. Two versions have been developed: one for 4-5th graders (the “Upper Elementary S-STEM”), and one for 6-12th graders (the “Middle/High School S-STEM”).

Survey Instrument – Student Attitudes Toward STEM (S-STEM) Survey: Tips for Using Your Data

These 5 steps are a useful process by which individuals/teams can organize efforts to learn about the impact of their program on participating students.