Evaluation Review, Ahead of Print.
BackgroundA recent stream of literature recognizes the impact of good/poor implementation on the effectiveness of programs. However, implementation is often disregarded in randomized controlled trials (RCTs) because they are run on a small scale. Replicated RCTs, although rare, provide a unique opportunity to study the relevance of implementation for program effectiveness.ObjectivesEvaluating the effectiveness of an at-scale professional development program for lower secondary school math teachers through two repeated RCTs.Research DesignThe program lasts a full school year and provides innovative methods for teaching math. The evaluation was conducted on two cohorts of teachers in the 2009/10 and 2010/11 school years. The program and RCTs were held at scale. Participating teachers and their classes were followed for 3 years. Impact is estimated by comparing the math scores of treatment and control students.SubjectsThe evaluation involved 195 teachers and their 3940 students (first cohort) and 146 teachers and their 2858 students (second cohort).MeasuresThe key outcome is students’ math achievement, measured through standardized assessment.ResultsIn the first wave, the program did not impact on students’ achievement, while in the second wave, a positive, persistent, and not insignificant effect was found. After excluding other sources of change, different findings across waves are interpreted in the light of improvements in the program implementation, such as enrollment procedure, teacher collaboration, and integration of innovation in daily teaching.ConclusionsRepeated assessment of interventions already at-scale provides the opportunity to better identify and correct sources of weak implementation, potentially improving effectiveness.