A mixed-method multi-level randomized evaluation of the implementation and impact of an audio -assisted reading program for struggling readers

Joy K Lesnick, University of Pennsylvania


This dissertation describes the methods and findings of a mixed-methods multi-level randomized evaluation study of the impact of the New Heights Reading Program on the fluency and comprehension of struggling readers. The impact analysis is a randomized complete block experiment (Meter, Kutner, Nachtsheim, & Wasserman, 1996) that uses a combination of individual and group randomization (Boruch, May, Lavenberg, et al., 2004; Raudenbush, 1997). A complementary process evaluation provides context for the quantitative outcomes, helping to explain the varying degrees of implementation of the program and the corresponding outcomes for participating students (Scheirer, 1994). Originally developed in New Zealand under the name Rainbow Reading, this is the first evaluation of the American version of the curriculum. The study sample consists of 233 students in 30 third grade classrooms and 29 fifth grade classrooms in nine schools across two school districts. All students reading nine or more moths below grade level in the participating classrooms were randomly assigned to the treatment or control groups with a maximum of six students assigned to the treatment group per class. Students completed three pre- and post-test standardized measures of fluency and comprehension. Program providers participated in pre- and post-intervention teacher interviews, completed implementation surveys, and allowed the researcher to observe an average of eight sessions each during the eighteen weeks of intervention. Data from the process evaluation revealed that although the majority of teachers implemented the program with high quality, the quantity with which they used the program was quite low, as was their overall fidelity of implementation to the intended design. A rubric was developed to categorize the implementation practices of teachers across five dimensions: quality of practice, frequency of practice, quality of teacher conferences, frequency of teacher conferences, and goodness of fit. The impact evaluation found no significant impacts of the program on student fluency, comprehension, or attitudes as measured by the adjusted post-test differences between treatment and control groups. Furthermore, there were no detectable differences in impact of the New Heights Reading Program across the implementation levels of providers. The paper concludes with implications for policy and future research.

Subject Area

Literacy|Reading instruction|Elementary education

Recommended Citation

Lesnick, Joy K, "A mixed-method multi-level randomized evaluation of the implementation and impact of an audio -assisted reading program for struggling readers" (2006). Dissertations available from ProQuest. AAI3211103.