This report was originally published 03 November 2015.

Summary

In 2008, the NSW Government entered into three major Smarter Schools National Partnership (SSNP) agreements with the Australian Government. The SSNPs came into effect in 2009 and included funding for government and non-government schools.

The Partnerships included:

• National Partnership on Literacy and Numeracy (NPLN)
• Low Socio-economic Status (SES) School Communities National Partnership (LSESNP) and
• Improving Teacher Quality National Partnership (ITQNP).

A brief description of the three SSNPs and the focus of the evaluations can be found at the end of this Learning Curve.

The timing of the SSNP agreements coincided with a new era of transparency and accountability in the NSW education system. Major independent evaluations of each of the SSNP agreements were commissioned to identify whether they had been implemented as intended and whether they had achieved their objectives. This Learning Curve shares some of the lessons learned along the way.

Lesson 1. Increased funding for low SES schools can make a difference to student outcomes

The end goal in all of the SSNP agreements was to lift student achievement. Not all of the evaluations were able to provide a robust answer to the question of whether the increased funding resulted in improved student outcomes. However, the LSESNP evaluation did provide a very robust analysis of the impact of the LSESNP on student outcomes. The results were modest but positive:

• Students in LSESNP schools had NAPLAN scores that were significantly higher than students in non-NP schools, ranging from 1.79 points for Spelling to 3.64 points for Grammar and Punctuation
• The gap in NAPLAN Reading growth between LSESNP primary schools and the state average significantly reduced over time
• Students in LSESNP schools were more likely to remain at or above benchmark NAPLAN standards from Year 3 to Year 5 and from Year 7 to Year 9 compared to students in non-NP schools
• Students in LSESNP schools had greater improvements in HSC attainment and apparent retention (both Year 7 to 12 and Year 10 to 12) than similar students in non-NP schools.

Qualitative survey findings were also positive with teachers and principals reporting school improvements in several areas. For example:

• 97 per cent of principals reported that LSESNP initiatives had led to more innovative and tailored learning initiatives for students
• Two-thirds of principals strongly agreed that Partnership reforms had assisted in the development of their staff
• Teachers were positive about changes to teaching with 91 per cent reporting use of more effective strategies to improve
• student learning and 90 per cent reporting better use of data to inform practice since Partnership involvement
• 92 per cent of teachers agreed that access to professional development had increased as a result of the LSESNP

The LSESNP evaluation is ongoing and the researchers are currently conducting research to identify the key drivers of this improvement in performance.

Lesson 2. The longer the reforms are sustained, the better the outcome

The SSNP evaluations provide direct evidence that sustained investment leads to better outcomes. Some schools entered into the LSESNP partnership earlier than others and were therefore funded for a longer period of time. The LSESNP evaluation found that greater gains in NAPLAN achievement were achieved by schools that entered the partnership earlier. NAPLAN Reading, Spelling, Grammar and Numeracy scores increased by between 0.4 and 1.4 points per additional year of participation in the program (see table 2). In addition, there was a positive correlation between length of time in the partnership and how positively the reforms were viewed by teachers.

Responses from the LSES NP teacher survey (2013) were more positive overall from schools that had been involved in the Partnership for longer. This was statistically significant in the areas of classroom management and parent and community relations. The findings suggest that it takes time to change school culture and embed new reforms across the entire school. In open ended responses to the 2014 principal survey, many respondents referred to cultural change resulting in positive outcomes for students. For example, one principal noted that there was a “cultural shift in behaviour to [the] point that learning became more important than behaviour management. In 2013 we finally saw the positive growth in NAPLAN reading that we had been working towards.” (Victoria University, 2015, p.38)

Similarly, longitudinal analysis conducted as part of the Cross- Sectoral Impact Survey (CSIS) indicated that responses were more positive the longer a school had been involved in a Partnership. The CSIS reported that making a four-year investment improved the chances of embedding practices into schools and that withdrawal of funding had a particular impact on the ability of schools to sustain professional learning.

In contrast, there was little evidence to suggest that the NPLN led to significant improvements in students’ literacy and numeracy over the course of the National Partnership (NP). This could come about for a number of reasons. The evaluators used aggregated measures of literacy and numeracy from the NAPLAN assessment to measure change over time, which may have lacked the sensitivity to detect subtle changes in achievement. However, it is also important to note that the funding (and evaluation) period for this NP was significantly shorter than the LSESNP. The evaluators concluded that lasting and widespread impact on student learning outcomes can only be achieved if reform initiatives and associated funding are sustained over a considerable length of time.¹

Lesson 3. Instructional leadership and professional development are seen to be highly effective

Many of the initiatives implemented under all three SSNPs involved training, mentoring, support and professional development for teaching staff. One of the key methods used in the ITQNP and, to a more limited extent, the LSESNP schools was the appointment of Highly Accomplished Teachers (HATs) and their non-government equivalents.² In ITQNP schools, HATs operated out of a ‘hub’ school that was designated as a Centre for Excellence to work with allocated ‘spoke’ schools. In LSESNP schools, the HATs worked with teachers in their own school.

The HAT (or equivalent) performed numerous functions that were often context-specific. However, their central roles were to model best practice and mentor other teachers. The HAT was almost universally seen by their colleagues, members of the school executive and school principals to be a positive initiative. Survey data from the ITQNP evaluation suggested that the HAT increased the quality of teaching and the level of support for teachers, strengthened the teaching and learning culture in the school, improved teacher capacity and was pivotal to building positive relationships within and between school networks.

The evaluators were unable to objectively link the HAT or its non-government school equivalent to student outcomes but the triangulation of evidence from school leaders and from teacher colleagues gives some confidence that the HAT role had a positive impact. One feature of the HAT or equivalent role that was singled out for particular attention was their capacity to mentor colleagues in the use of performance data to inform their teaching practice.

The NPLN also included instructional leadership via literacy and numeracy co-ordinators, who helped guide and mentor teachers in literacy and numeracy teaching practices. Inspirational instructional leadership was considered to be very successful and was one of the nine key drivers of improved literacy and numeracy identified through the NPLN evaluation.

Professional development that was not directly associated with the HAT or other paraprofessional roles was also seen to be highly effective by principals and teachers across the SSNPs. For example, teachers and leaders in ITQNP schools reported that a strategic, longitudinal approach to professional learning was important and effective. They also reported enhanced job satisfaction as a result of the support provided by HATs and paraprofessional positions. The critical role that effective professional development plays in improving teacher quality is reinforced by evidence reviews and other primary research carried out by CESE.³

Teachers, particularly at NPLN schools, were also trained to have a greater understanding of analysis tools and techniques, as well as greater appreciation of the role of data in the measurement of change (for example in the use of pre and post-testing). Some teachers were trained in the use of explicit teaching practices, which requires clearly identified outcomes and targeted strategies to produce results.

Lesson 4. Schools can’t do it alone – partnerships are critical

Evidence strongly suggests that partnerships between schools and parents, the community and other education providers are critical in supporting improved student learning outcomes (NSW Department of Education and Communities 2011). One LSESNP evaluation showed that partnerships that routinely exist between schools and the community or other education providers are extensive and diverse. They serve a range of purposes but usually aim to support student learning and professional learning among teaching staff.

There is some evidence to suggest that schools were more likely to engage in partnerships as a result of the LSESNP activities. Seventy-nine per cent of surveyed principals reported that their school was more frequently providing additional programs and services to support students in their learning. This included the employment of home-school liaison officers and the establishment of homework centres to build a relationship between the school and community. The impact of this activity is challenging to measure but the reported increases in engagement are a positive sign that real change has been achieved.

The ITQNP was slightly different to the other SSNPs insofar as it focussed on improving the quality of the teacher, not just the quality of teaching. All hub schools were partnered with a university and the evaluation highlighted the critical role that university Initial Teacher Education institutions play in lifting teacher quality.

Improving teacher quality is an issue of ongoing importance for the NSW and other Australian education systems. Evidence shows that having a quality teacher in front of the classroom explains 30 per cent of the variation in student achievement (Hattie, 2003). The NSW Government has signalled its ongoing commitment to improve teaching quality through its Great Teaching, Inspired Learning reforms. A crucial part of these reforms is to improve initial teacher education programs.

One of the most startling findings from the ITQNP evaluation was the variation between practicum supervisors in the extent to which they appropriately referenced the Australian Professional Standards for Teachers when assessing student teachers. A content analysis of comments in preservice practicum reports revealed that a majority of reports discussed practices that were irrelevant to the element being assessed.

Where the Australian Standards were appropriately referenced, it is clear that some Standards are referenced more than others. As Figure 1 shows, supervising teachers showed a greater propensity to make commentary in relation to students’ professional practice, with a relative lack of emphasis on professional commitment. Interestingly, practices that research shows to be highly effective (see CESE 2014 publication)⁴ were some of the lesser referenced standards by supervisors, including: communicating clear learning goals (4.1); providing oral and written feedback to students (3.7); and, use of student achievement data to evaluate learning (3.10). Communicating with parents and carers, and using educational research were also some of the least referenced professional standards.

The ITQNP also included a survey of 800 teaching students or recent graduates. Respondents were asked the extent to which their university preparation and professional experience had contributed to attainment of the Professional Teaching Standards. The respondents’ ratings of the extent to which they felt prepared to meet the standards is shown in Figure 2. Like the practicum reports, students felt most prepared to manage the classroom. Of concern are the low levels of preparedness to teach in schools with high Aboriginal enrolments, and rural and remote settings.

These data suggest a clear gap in teacher preparedness in some areas that will require ongoing commitment from both school systems and universities to address.

Lesson 5. To improve evaluation, education systems need better outcome measures

The final lesson was not a finding of any of the NP evaluations, but rather a lesson learnt indirectly: rigorous outcome evaluations require suitable and reliable outcome measures. Outcome measures also need to be recorded and collected systematically.

The SSNP evaluations relied heavily on survey data to determine the effectiveness of the Partnership activities. This was necessary but unfortunate from a measurement perspective. Survey data usually afford a low level of certainty in the causal attributions that can be made between a set of activities and intended outcomes. Unless they are very carefully constructed, survey data are often susceptible to response biases. For example, in the Cross-Sectoral Impact Survey and other surveys conducted under the SSNP evaluations, principals consistently reported the greatest extent of change, followed by school leadership members, with teachers reporting the smallest gains.

This could be because principals have a more holistic view of the change process. However, it is more likely that they have an incentive to respond favourably by virtue of their accountability to implement the activities they’re being asked to comment upon.

NSW and Australia are in the fortunate position of having regular measures of student literacy and numeracy progression. Students are assessed on NAPLAN at four times, in Years 3, 5, 7 and 9. Monitoring growth from one time point to the next can provide a very powerful test of whether there are any changes in student achievement on these measures as a result of a given set of reforms. The LSESNP evaluators were able to rigorously analyse student-level data to detect relatively subtle changes in literacy and numeracy growth over the course of the evaluation.

While NAPLAN is an important asset, it is often criticised by educators on the basis that it only captures a limited range of the capabilities children learn at school. This is a legitimate criticism but it will only be resolved if educators are willing to assess a wider range of capabilities in a standardised way. There is a live debate underway nationally as to whether schools should be assessing some of the other general capabilities outlined in the Australian Curriculum, such as critical and creative thinking, personal and social capacity, and ethical understanding (e.g. Fraillon & Mendelovits, 2015). These issues should continue to be debated in terms of their implications for outcome measurement.

Other outcomes are not measured systematically at all. In light of the continued focus on teacher quality at the State and Commonwealth level, the absence of any systematic measurement of teacher quality is an obvious gap. While NSW and other jurisdictions are introducing standards-based teacher classifications, these are insufficient to measure teacher quality. Teachers are also able to self-assess against the Australian Professional Standards for Teachers using an online tool developed by the Australian Institute for Teaching Standards and Leadership (AITSL). However, these self-assessments are voluntary, of unknown psychometric reliability and unavailable to school systems.

NSW has rolled out a teacher self-assessment survey to more than 1,000 schools in 2015. The Focus on Learning teacher survey, which is a commercial product developed and administered by The Learning Bar, will be a useful tool for principals to understand their teachers’ strengths and weaknesses. However, self-assessments are relatively limited in what they can say about actual quality of teaching practice. More rigorous ways of measuring teaching quality may be required, such as lesson observations, linking teachers to student outcomes measures.