Evidence Based Research Brochure

Achievement of STARS Participants versus Non-Participants The most critical part of the impact evaluation involved the development of models to account for growth in academic achievement by students enrolled in STARS versus those who were not enrolled in the program. This entailed a mixed-model analysis of covariance that included a within-subjects factor in which the students were assessed twice on the SAT- (fall and spring). The between-subjects factor of the mixed model accounted for differences according to whether students participated in the STARS program. The analysis of the SAT-9 score gains indicated that Saturday STARS program participants experienced small but statistically significant gains over those experienced by students who were not in the program. Executive Summary - Impact & Implementation Impact Evaluation - The purpose of the impact evaluation component was to estimate the extent to which students benefited academically from participation in the STARS program. Changes in Participants Math Skills Paired t-test analyses indicated that STARS participants’ performance on the mathematics post-tests was statistically significantly higher than their performance on the pretests in each of the target grades. T-test analyses also determined that STARS participants’ scale scores on the spring SAT-9 were statistically significantly higher than their corresponding scores from the previous fall. Relation of STARS Curricula Mastery to SAT-9 Scores Hierarchical multiple regression analyses revealed that STARS participants’ performance on Moving with Math Post-Test scores were statistically significantly related to scaled scores on the Total Math section of the spring SAT-9 for students in grades 3 and 8. Table 5-15. SAT-9 Math—Overall Model: Tests of Within-Subjects Contrasts Type III Eta Source Sum of Squares df Mean Square F p Squared MATH 1636205.854 1 1636205.854 5062.587 .000 .316 MATH*AGETESTS 515873.977 1 515873.977 1596.166 .000 .127 MATH*STARS 2635.364 1 2635.364 8.154 .004 .001 Error (MATH) 3547395.082 10976 323.196 Table 5-16. SAT-9 Math—Overall Model: Tests of Between-Subjects Effects Type III Eta Source Sum of Squares df Mean Square F p Squared AGETESTS 17929961.537 1 17929961.537 6047.584 .000 .355 STARS 1588376.852 1 1588376.852 535.743 .000 .047 Error 32541796.340 10976 2964.814 Table 5-3. Pre- and Post-Test Scores on STARS Math Curricula Math Curricula (Grade) n Pre-Test Mean Post-Test Mean t Moving with Math (Grade 3) 357 15.62 22.26 20.31*** Moving with Math (Grade 5) 365 21.43 30.47 19.86*** Moving with Math (Grade 8) 110 24.67 31.05 6.57*** ***p < .000 Table 5-4. Total Math Scale Scores on the SAT-9 Grade n Pre-Test Mean Post-Test Mean t Grade 3 1436 550.25 593.03 42.64*** Grade 5 1203 606.87 635.30 45.52*** Grade 8 474 654.29 667.81 16.72*** Grade 11 136 675.93 689.25 6.42*** ***p < .001 Tables 5-8, 5-9, 5-10. Relation of Mastery of Math Curriculum to SAT-9 Performance Unstandardized Standard Standardized Variables b Error Beta Step 1: Age -.383 .200 -.087 Gender -.768 2.874 -.012 Step 2: Fall Total Math Score .635 .060 .502*** Step 3: Moving with Math Post-Test Score .641 .191 .159*** Step 1: Age -.323 .148 -.100 Gender -.418 2.473 -.008 Step 2: Fall Total Math Score .587 .053 .500*** Step 3: Moving with Math Post-Test Score .127 .098 .059 Step 1: Age .128 .190 .041 Gender 1.878 3.197 .036 Step 2: Fall Total Math Score .749 .078 .611*** Step 3: Moving with Math Post-Test Score .742 .132 .359*** ***p < .001 GRADE 3 GRADE 5 GRADE 8