This report was originally published 14 December 2015.

Summary

Background

The New South Wales (NSW) Government has a strong focus on improving students' literacy and numeracy skills in the early years of schooling. In 2011 the NSW Government committed $261 million under the Literacy and Numeracy Action Plan to improve literacy and numeracy for students in the early years (Kindergarten to Year 2). This focus on early literacy is important as students who are not reading well by the time they reach Year 3 face significant challenges for the remainder of their schooling (Willms 2003). Early identification of students who are having difficulty reading and the introduction of effective remediation strategies are both critical elements of a school’s role in developing the reading capabilities of their students.

Reading Recovery Overview

There are many different interventions used in NSW primary schools to assist young students improve their literacy outcomes. One of the most widely used interventions is Reading Recovery (RR), which has been at the forefront of the remediation effort in the NSW Department of Education for over two decades. In 2012, RR was offered in over half (approximately 60%) of NSW government primary schools, with approximately 14% of all Year 1 students participating in the intervention.

RR was developed in New Zealand in the 1970s by Dame Marie Clay as an intensive individualised literacy intervention that aims to accelerate literacy learning for students performing in the bottom 20 per cent of Year 1 (Department of Education and Communities 2015; What Works Clearinghouse 2008). RR tuition is provided on a one-to-one basis over 12-20 weeks with the intention of raising students’ performance to the average level of their Year 1 peers, thereby enabling them to benefit from classroom instruction and sustain achievement throughout the early years of school (Department of Education and Communities 2015; May et al. 2013; 2015).

While the intention of RR is to be responsive to each individual student’s needs, the intervention typically addresses several aspects of reading and writing processes that support the comprehension of texts. These include: vocabulary, fluency, comprehension, writing, phonemic awareness, phonics, motivation and oral language (What Works Clearinghouse 2013). RR does not claim to align itself explicitly with a particular classroom-based approach and is substantively based on the notion that students draw on multiple sources of information (e.g., visual, linguistic, text-based) when learning to read (Chapman & Tunmer 2011; Reading Recovery Council of North America 2015, see http://readingrecovery.org/).

Eligibility for RR is assessed using the Observation Survey of Early Literacy Achievement, a standardised assessment based on teacher observations of student performance on six tasks related to early literacy skills (Clay 2002; Reading Recovery Council of North America 2015, see http://readingrecovery.org/). These include: Letter Identification, Word Test, Concepts About Print, Writing Vocabulary, Hearing and Recording Sounds in Words, and Text Reading (see http://readingrecovery.org/reading-recovery/teaching-children/observation-survey, 2015). The aim of this teacher-administered assessment is to identify the lowest 20 per cent of text readers in Year 1.

Following the administration of the Observation Survey, final selection of participating students is made in consultation with the school. Students begin participating in RR continuously across a school year when a place becomes available with a trained RR teacher. They ‘successfully discontinue’ RR when they have achieved the average reading level for their grade, typically a score of 16 or above on a re-test of the Observation Survey, as administered by an independent assessor. Students who do not reach this level after 12-20 weeks are referred for further specialist support or for long-term literacy support. Students who do not complete their series of lessons within a calendar year may have their lessons carried over to the next year. Students may also stop participating in RR if they transfer schools and are not able to continue with RR lessons at their new school.

Objectives of the current study

While the balance of the evidence suggests that RR is an effective intervention for raising student literacy levels, most evaluations of RR have been conducted outside Australia. This raises the question of whether RR is equally effective in NSW. Programs can be implemented in different ways across jurisdictions, which can lead to variability in the outcomes achieved. Moreover, the capabilities of teachers and the other strategies employed to remediate low levels of literacy may differ greatly across countries. Second, the extant evidence has not resolved the issues raised by critics regarding the effectiveness of RR for low performing students. Further research that carefully accounts for student baseline achievement is needed to assess whether RR is differentially effective for students at low versus high starting points. Finally, the long-term sustainability of the results achieved by RR have not been well considered using rigorous methodologies in the existing literature.

In light of these concerns, a rigorous and up-to-date sector-level evaluation of RR is critically important. This is particularly important considering the current educational policy environment in NSW. Under Local Schools, Local Decisions, school leaders have much greater authority to make local decisions about the programs that best suit the needs of their schools. While the evidence suggests that RR is an effective intervention in some contexts (e.g. compared to no supplemental intervention), understanding more about how RR works, for whom and under what circumstances will provide principals with the information they need to make informed decisions at the local level.

The primary objectives of the current evaluation were to determine the impact of RR on students’ literacy outcomes and whether any benefits associated with participating in RR are sustained over the longer term. This analysis was conducted at the sector-level (across NSW government schools) and focussed on identifying the impact of RR compared to students who had similar characteristics but who attended a school that did not offer RR. An important aim of the current study was to determine whether there was any interaction between baseline achievement levels and the effectiveness of RR. The key research questions addressed in this evaluation were:

1. What proportion of students participating in RR reach the minimum reading levels expected of Year 1 students, and achieve literacy outcomes equal to or greater than those of their peers?
2. In the short-term, are literacy outcomes for students who participate in RR greater than those for comparable students who do not participate in RR?
3. Are any benefits of RR sustained over longer periods of time (i.e. to Year 3)?

Summary of results

The results from the current analysis provide some evidence that RR is an effective short-term intervention for remediating reading text skills among the lowest performing students. However, RR does not appear to be an effective intervention for students who begin Year 1 with more proficient literacy skills, at least compared to other interventions or initiatives that are available in non-RR schools. These findings do not necessarily call into question the validity of RR, as the intervention is designed specifically to remediate reading skills among the poorest performing students. However, the implication of these findings is that currently, the most cost-effective method of implementing RR in NSW may be to target only the students performing at the lowest levels at the end of Kindergarten (at a sector- not a school-level) or to restrict RR to schools that are identified as having a proportionately higher number of students who are not meeting performance benchmarks in Kindergarten or early Year 1.

The limitations of this analysis also highlight the strong need to collect better information on the teaching practices and interventions being offered in non-RR schools and to develop valid and reliable measures of students’ literacy progress throughout the early years of school. Perhaps more importantly, these findings suggest that the true effect of the relative impact of RR can only be assessed through a rigorous prospective trial where RR is compared to other interventions, using random assignment to treatment condition and valid and reliable outcome measures. Indeed, only a rigorous prospective trial can effectively control for other potentially confounding factors (e.g. teacher quality, classroom instruction) that may influence student outcomes and can provide policy makers and school leaders with the critically needed information on which intervention is the most effective and cost-effective approach to literacy remediation.