In 2010, Jefferson County (Jeffco) Public Schools in Colorado was awarded a five-year U.S. Department of Education Teacher Incentive Fund Grant to implement the Jeffco Strategic Compensation (StratComp) Initiative. StratComp, which provides professional supports to all educators, is a new teacher evaluation observation system in which administrators and peers conduct observations, job-embedded professional development is aligned to observation ratings and delivered by master teachers and instructional coaches, and groups of teachers work together in teacher-teaming activities to improve instructional and student achievement goals. These supports were implemented in 20 of the highest poverty schools in the district, 10 of which were randomly assigned to have the additional opportunity to receive performance pay if teacher performance outcomes were met.

Project Summary

AIR is in the final year of a five-year mixed-methods evaluation of the Jefferson County Strategic Compensation (StratComp) initiative in Colorado. This initiative provides professional supports to teachers in 20 of the highest poverty schools in the district, 10 of which were randomly assigned to have the additional opportunity to receive performance pay if teacher performance outcomes were met. AIR is conducting a formative evaluation to study StratComp implementation and a summative evaluation to estimate the impact of the StratComp program on teacher performance and student achievement.

Project Goals

• Measure the level, quality, and relevance of implementation. 
• Examine whether the initiative is associated with improvements in key intermediary outcomes (e.g., teacher performance).
• Assess the program’s effectiveness in improving student outcomes. 
• Identify the mechanisms and dosage associated with the largest gains in student achievement.

Methods

To collect information on the extent to which the Jeffco StratComp Initiative has been implemented with fidelity and with high quality, the evaluation team relied on a variety of data sources, including site visits, teacher and school administrator interviews, staff focus groups, administrator and teacher surveys, and extant data on program implementation. Data from these sources provide a rich and triangulated picture of program implementation. The data inform program improvement efforts and districts’ ability to understand the mechanisms and dosage associated with greater student improvement.

To examine the effect of the program on student achievement, the evaluation team has used a randomized experiment and a comparative interrupted time series design, one of the most rigorous quasi-experimental designs. This evaluation has compared the effects of the program among three groups of schools: (1) schools implementing “business-as-usual” policies, (2) schools implementing new StratComp supports, and (3) schools implementing StratComp supports in addition to a pay-for-performance incentive system that was randomly assigned to half of the StratComp schools. By using all three groups of schools, the evaluation team is able to measure the overall impact of StratComp as well as the marginal effect of the compensation reforms that were being implemented in half of the StratComp schools.

To inform future district decisions, district administrators want to know which program supports and implementation factors have been associated with the greatest growth in student achievement. The evaluation team has used data from the surveys in a statistical model to examine which factors are associated with greater gains in student achievement. The district also selected eight case-study schools (of varying implementation levels) where we collected qualitative data about which components supported and hindered implementation.