The Selection and Evaluation of Advanced Teachers

To help identify an initial set of attributes and measures of Advanced Teachers, the research team will conduct a rapid systematic review of the literature and interview educators and stakeholders with expertise in designing and implementing Advanced Teaching Role programs. The purpose of these activities serving as the foundation of the proposed research project is twofold: 1) to identify commonly agreed upon attributes and measures of Advanced Teachers; and 2) to understand the relationship between the criteria used in selecting Advanced Teachers and their anticipated effect.

Data Collection

The research team will conduct semi-structured interviews with education stakeholders who have extensive experience with Advanced Teaching Role programs, including:

• district leaders at each of the 18 PSUs currently receiving funding from NCDPI,
• a sample of principals and teachers from schools with ATR programs, and
• ATR leadership at the state and national level.

The research team will also conduct a rapid review of the academic literature and program documentation maintained by PSUs and their ATR partners. The research team is also interested in identifying previously validated instruments that measure the characteristics of Advanced Teachers for Phase III of this study.

Through expert interviews and a review of the existing literature, the research team will gain a statewide perspective on the Advanced Teaching Roles program from those steeped in its implementation and from a national and international perspective from researchers who have studied this approach. 

Data Analysis

The research team will conduct qualitative analyses of interview data, ATR job descriptions and the existing literature. The team will employ a constant comparative method, which involves an ongoing process of simultaneous data collection and analysis, where each newly collected item of data is compared with existing data to refine and integrate categories (Glaser & Strauss, 2017). This method is well suited for informing subsequent analyses for this study, as it is characterized by its iterative nature, its grounding in the data, and its utility in theory generation (Boeije, 2002; Charmaz, 2006). This method will continuously compare and contrast data from interviews and prior research.  

To better understand the complexities and interdependencies that characterize effective teachers serving in advanced teaching roles, Phase I will also leverage Epistemic Network Analysis (ENA) to analyze coded interview data further. ENA has emerged as a novel and promising ethnographic method for identifying, quantifying, and visualizing connections among elements in coded data, such as text-based transcripts of stakeholder interviews (Shaffer & Ruis, 2017; Fischer et al., 2023). ENA was originally designed to address challenges in learning

The purpose of this phase is to examine the extent to which current selection criteria are associated with outcomes for students and teachers identified in Phase I. The research team will leverage administrative data  to conduct descriptive and regression analyses using both statistical and machine learning methods.

Data Collection

Building upon data from interviews and document reviews from Phase I, the research team will also develop and administer questionnaires for school leaders, targeting the sources of data and measures they use to evaluate and select candidates for Advanced Teaching Roles. The questionnaire will ask school and district leaders to identify what criteria they used to select current Advanced Teachers in their schools. School and district administrators may choose multiple selection criteria, and where possible, we will connect existing administrative data  with current Advanced Teacher.

Our goal will then be to correlate the specific selection criteria used with outcomes once these teachers take on Advanced Roles. Specific outcomes measures available in NCDPI administrative data including student achievement and turnover among the colleagues receiving support from the Advanced Teacher (if any). Thus, a primary contribution of Phase II will be our collection of data on what selection criteria are currently available to (and used by) school and PSU administrators.

Data Analysis

Having identified a set of criteria and measures that school, district, and state leaders believe are critical for Advanced Teachers, we will conduct exploratory data analyses and modeling techniques using data that is currently available to the research team and used by school and PSU administrators. These analyses will help  identify trends, patterns, and anomalies in the data that might be useful for hypothesis generation and predictive modeling; detect and handle missing data, outliers, and errors in the dataset; and guide the choice of appropriate statistical models and techniques for further analysis. 

Guided by exploratory analyses, the research team will then select a set of statistical and machine learning models to assess the extent to which currently used selection criteria correlate and predict outcomes such as student achievement, teacher retention, and perceived school working conditions. The use of both statistical and machine learning models in this mixed-methods study is intended to help strike a balance between the explainability of findings for non technical audiences and the predictive power of these models for potential use in advanced teacher selection processes. These models will provide critical information on how well existing selection criteria predict future impact as an advanced teacher.

Phase III will focus on uncovering characteristics of effective Advanced Teachers that are currently not used by PSUs, including data not currently available to PSUs, or may be available but is not used by administrators in North Carolina. The main purpose of this phase is to elucidate whether and to what extent new measures can provide better information to guide future selection and evaluation of Advanced Teachers.

Data Collection

In addition to examining NCDPI administrative data that is NOT used by PSUs but could potentially be made available for selection and evaluation, the research team will also collect novel data on current Advanced Teachers using a sample of ATR schools. We will develop the instruments used to collect these data based on findings from the literature review and expert interviews in Phase I. Examples of these novel measures could include questionnaires given to principals and teachers who are supported by Advanced Teachers. Before collecting these data on the full sub-sample of participating ATR schools, we will start with a small pilot collection as a preliminary validation of the instruments used to collect these measures. Using cognitive interviews with pilot participants, we will test the wording of the instruments for content clarity. 

After testing the instruments via the pilot, we will collect these novel measures of effective Advanced Teachers using a sample of ATR schools. The project timeline precludes collecting novel data for all ATR schools statewide, so we will focus on a sample of ATR schools that provides adequate statistical power while also maintaining feasibility in data collection. If possible, we will also collect these data on Advanced Teachers and then correlate their characteristics with their effects on student achievement and teacher retention. Importantly, collecting these novel data on current Advanced Teachers will contribute to a broader system for measuring effective teaching (i.e., measures of effective teaching that go beyond student test scores).

Data Analysis

Data analysis will begin with measurement models that allow us to test the reliability and validity of the novel measures we developed. A main contribution of these measurement models is to support instruments that NCDPI, PSUs, and schools may use to collect new data on the characteristics of Advanced Teachers. There is also a wide range of future applications of these instruments, particularly as ways to measure the effectiveness of classroom teachers and not just Advanced Teachers. 

Following validation of these instruments, the research team will use the collected data to examine how and which characteristics of Advanced Teachers (as measured by the instruments we develop) correlate with effects on student achievement and teacher turnover. Using multivariate regression, we can examine whether and which characteristics correlate with effectiveness as an Advanced Teacher. These analyses will also provide evidence that will illuminate whether the measures we develop are distinct from the measures currently used by administrators, as identified in Phase II. To the extent possible, we will also estimate quasi-experimental models (e.g., teacher, teacher-by-year, and student fixed effects) to control for important confounding variables, such as school leadership, which in turn allows us to better isolate the effect of the new measures we collect on Advanced Teachers’ effectiveness. 

Phase III will also focus on  incorporating these new measures into machine learning models from Phase II and the creation of new variables from existing data with the aim of improving the accuracy of these models and identifying new predictors of desired program outcomes such as student achievement and teacher retention. To help refine these models, the research team will focus on the process of feature engineering (i.e., creating new variables from existing variables within a dataset that goes above and beyond simply recoding or rescaling variables) informed by the literature review and expert interviews in Phase I, as well as analyses in Phase II. Beyond the use in improving the accuracy of prediction models, feature engineering can be valuable for creating more informative and interpretable teacher variables that could be used by administrators to inform selection and evaluation criteria.

This study is conducted on behalf of the North Carolina Department of Public Instruction and is a joint research partnership between the Friday Institute for Educational Innovation, the College of Education at NC State University, and Basis Policy Research. Dr. Shaun Kellogg is the Principal Investigator and point of contact for this study.

This research is funded by The North Carolina Department of Public Instruction. This means that the sponsor is paying the research team for completing the research. The researchers do not, however, have a direct financial interest with the sponsor or in the final results of the study. If you would like more information, please contact Dr. Thomas Tomberlin, the project sponsor, at thomas.tomberlin@dpi.nc.gov.

This mixed-methods study addresses this need for school, district, and state ATR leadership to better understand criteria used for the selection and evaluation of Advanced Teacahers, as well as policies and processes used for the evaluation of Advanced Teachers. By systematically examining the criteria districts use to select and evaluate Advanced Teachers, this study aims to document and share selection and evaluation criteria and processes used for ATR; explore the relationship between selection criteria and desired program outcomes; develop and validate new measures for the selection and evaluation of Advanced Teachers; explore the practical and ethical implications of new and existing measures, models, and processes; and provide data-informed recommendations and resources for identifying, selecting, and evaluating Advanced Teachers.

This study is intended for North Carolina educators in ATR grant-funded districts that are directly involved in ATR programs and include:

• Advanced Teachers who receive salary supplements to support classroom teachers or teach an increased number of students
• Classroom Teachers directly supported by, or on a team lead by, an Advanced Teacher
• School Principals responsible for the selection and evaluation of Advanced Teachers
• District Administrators responsible for the identification and recruitment of Advanced Teachers
• State Leaders responsible for shaping ATR policy and practice in North Carolina and beyond

Educators will be asked to participate in one or more of the following activities depending on their role within the ATR program:

• Interviews. District administrators from grant-funded districts, as well as a strategic sampling of school principals and teachers will participate in a 30-60 minute interview to discuss the selection and evaluation of Advanced Teachers. Interviews will take place online via Zoom and be recorded to ensure the responses to interview questions are accurately documented.
• Surveys. School and district administrators will be asked to complete a brief survey about the selection criteria they currently use to select current Advanced Teachers in their schools.The survey will take approximately 10-15 minutes to complete and will be completed online.
• Instrument Development. A sampling of school principals and teachers supported by Advanced Teachers will be asked to review and complete a pilot instrument that includes novel measures NOT used by PSUs but could potentially be made available for selection and evaluation. A small number of principles and teachers (~5) will also participate in cognitive interviews to test the wording of the instruments for content clarity. Cognitive interviews will last about one hour. We will then also pilot the instrument in some ATR schools throughout the state.
• Stakeholder Convening. Each ATR district lead will be invited to bring a small cohort of 4-5 school and district staff members to participate in a 1 ½ day ATR researcher and practitioner convening at the Friday Institute for Educational Innovation in Raleigh, NC. During the convening, educators will engage with the research team and ATR practitioners from across the state through presentations, discussion panels, breakout groups, data dives, and more.

No. Your participation in this study is not a requirement of your employment, and your participation or lack thereof, will not affect your job. Participation in this study is voluntary and educators who choose to participate can stop at any time and for any reason by contacting the principal researcher, Dr. Shaun Kellogg at sbkellog@ncsu.edu and (919) 513-8563. If you choose to withdraw your consent and to stop participating in this research, you can expect that the researcher(s) will redact your data from their data set, securely destroy your data, and prevent future uses of your data for research purposes wherever possible.

Yes. Educators will receive between $25 and $100 dollars in the form of a gift card for their participation in interviews, questionnaires, and/or instrument development. The total amount will depend on the level of participation and time. In addition, educators participating in the convening will receive a stipend of $500 to help cover the costs of travel. We recognize that this amount does not fully compensate educators for their valuable time and extensive expertise, but we hope this offer demonstrates our appreciation and serves as a token of gratitude for their contributions to the project.

While there are no direct and immediate benefits to participation in this research, we anticipate that individuals, schools, and districts involved may benefit from this study in the following ways:

• Improvement Opportunities: The study’s findings aim to equip school and district leaders with evidence-based insights to refine and enhance the implementation of the ATR program, potentially leading to more effective teaching and learning environments.
• Programmatic Clarity: Participants will help ATR practitioners gain clarity on selection and evaluation practices and criteria designed to enhance teacher performance and student outcomes.
• Contribution to Educational Research: Participants will contribute to important educational research, helping to shape future policies and practices that affect their profession in North Carolina and beyond.
• Network Building: Engaging with researchers and other educators during the study and spring convening provides networking opportunities that can lead to future collaborations and professional growth.
• Strategic Planning and Policy Development: Districts will benefit from comprehensive data and analysis regarding effective teacher selection and evaluation strategies, which will aid in strategic planning and policy development.
• Scaling of Promising Practices: The study’s findings will help districts understand which aspects of the selection and evaluation process are more likely to be effective for achieving program outcomes, leading to systemic improvements in teacher quality and student performance.

There is minimal risk associated with participating in this study. While there is always a risk of a breach of data security, extensive measures will be taken to ensure participant anonymity and confidentiality. There is also minimal risk of psychological or emotional harm when participating in the interviews. While the questions and topics discussed during the interviews are not directly related to psychological or emotional concerns, individual emotional responses to being asked about the selection and evaluation of Advanced Teacher may vary.

The information that you share with us will be held in confidence to the fullest extent allowed by law. Protecting participant privacy as related to this research is of utmost importance to us. There are very rare circumstances related to confidentiality where we may have to share information about you. Your information collected in this research study could be reviewed by representatives of the University, research sponsors, or government agencies (for example, the FDA) for purposes such as quality control or safety. In other cases, we must report instances in which imminent harm could come to you or others.
How we manage, protect, and share your data are the principal ways that we protect your personal privacy. Data that will be shared with others about you will be de-identified. De-identified data is information that at one time can directly identify you, but that we will record and report this data in such a way that your identity will be separated from the data. When the research concludes, there will be no way your real identity will be linked to the data we publish.

Yes. To help maximize the benefits of your participation in this project by further contributing to science and our community, your de-identified information will be stored for future research and may be shared with other people without additional consent from you.

If you have questions at any time about the study itself or the procedures implemented in this study, you may contact the Principal Investigator, Dr. Shaun Kellogg at sbkellog@ncsu.edu or (919) 513-8563.