This report was originally published 01 March 2013.

Summary

Aims and research questions

In 2012, in NSW government schools, approximately 230,000 students were from language backgrounds other than English (LBOTE). These students made up around 30% of total enrolments. Over 136,000 students (18%) were learning English as an additional language or dialect (EAL/D).

The relationship between language background and educational disadvantage has been analysed for more than 30 years. Given limited resources, policy makers seek to target resources effectively to overcome specific aspects of disadvantage, such as language proficiency, to improve both the quality and equality of education outcomes.

Since 30% of students in NSW government schools are LBOTE but only 18% require additional English language support, it is evident that LBOTE does not by itself indicate educational disadvantage or support needs. Various measures have been developed over the last 10 years for diagnostic purposes to identify suitable students for specialist programs. Other measures focus on resource allocation and contribute to school funding formulas that take account of the varying needs of schools.

In 2011 ACARA developed the English as an Additional Language or Dialect (EAL/D) Learning Progression to support the implementation of the Australian Curriculum. NSW Department of Education and Communities (DEC) carried out a trial of the EAL/D Learning Progression in government schools in May and June 2012. The aim was to investigate whether the EAL/D Learning Progression (the instrument) was sufficiently valid and reliable for teachers to use to assess English language proficiency of EAL/D students, primarily as a broad resource allocation mechanism.

The trial considered three research questions:

1. Can teachers with a diversity of experiences and expertise in ESL education, assess each of the four language modes consistently using the EAL/D Learning Progression, across a broad range of EAL/D students?
2. Is there sufficient evidence to support the intended interpretations and uses of teachers’ EAL/D Learning Progression phase assessments?
3. What are the successful elements and useful resources identified by teachers from the trial process?

Methodology

The trial included 97 teachers, both specialist ESL teachers and classroom teachers with diverse ESL (English as a Second Language) teaching and assessment backgrounds. The teachers came from 56 schools, including primary, secondary, and central schools as well as

Intensive English centres. Most of these schools were from metropolitan regions, with a few from provincial areas.

A stratified sample of 944 students, across key target grades (Kindergarten, Years 3, 5, 7 and 9), gender groups, sub-demographic groups (i.e., Aboriginal, international student, refugee), and representing the range of English proficiency levels based on the current NSW ESL phase assessment tool, was selected by participating teachers for inclusion in the trial.

A detailed program of professional learning prepared teachers to take part in the trial. Workshops and supporting documentation showed teachers how to identify suitable students, and collect work samples, observations and assessments for the trial.

Each student was assessed on four language modes – listening, speaking, reading and writing. Each language mode had four phases of proficiency – beginning, emerging, developing or consolidating. Teachers submitted assessment results of the four language modes for every student to a purpose-built website.

The trial used a double-marking process to investigate the reliability of assessments using the EAL/D Learning Progression: 639 of the sample of 944 students were assessed by two teachers on every language mode (listening, speaking, reading and writing) using the EAL/D instrument. Teachers’ assessments for each student were then compared. Various types of inter-rater statistics, such as exact and adjacent agreement rates, correlations, Kappa rates and Dependability Index (score reliability coefficient) were examined.

The validity of the EAL/D Learning Progression for the purpose of determining resource allocation in NSW government schools was analysed in relation to four aspects of score validity – concurrent, discriminant, structural and measurement.

Following the trial, teachers were surveyed to assess the usefulness of the instrument and the quality of support provided throughout the trial.

Results

Results are reported against each of the three research questions.

1. Can teachers assess each of the four language modes consistently using the EAL/D Learning Progression?

The trial results showed that, on the whole, the EAL/D Learning Progression enabled teachers to make consistent judgements of English language proficiency across the four language modes (listening, speaking, reading and writing). There was also strong evidence that teachers were able to use the EAL/D Learning Progression to consistently discriminate between the four phases within each mode and between each of the modes.

However, some variations in the consistency of teacher judgements were also observed. For example, teacher judgements were less consistent with some student cohorts (e.g., boys and students of relatively higher English language proficiency) and in speaking and listening modes. Teacher judgements were more consistent for other student groups, such as those at Intensive English centres. Reasons for variations could be attributable to factors including but not limited to: differences in teachers’ prior experience in similar assessment tasks, in particular in assessing informal student interactions which dominate the speaking and listening mode indicators of the EAL/D instrument; teacher knowledge of students; professional learning and training received; and time available to collect and assess work samples.

2. Is there sufficient evidence to support the intended interpretations and uses of teachers’ EAL/D Learning Progression phase assessments?

The trial has collected sufficient evidence to support the claim that the EAL/D Learning Progression provides a balanced and accurate reflection of English language development. The instrument can be used to derive a single measure of English language proficiency for each EAL/D student, for the purpose of allocating ESL funding to schools.

The trial found that there was a reasonable level of congruence between teachers’ assessments using the EAL/D instrument and those based on a similar construct – the current NSW ESL Phase tool. The EAL/D instrument was identified as being a more refined and discriminating tool than the current ESL Phase tool and allowed better discrimination between students at the low end of the English proficiency continuum.

The trial analysed the relationship between EAL/D assessments and NAPLAN results for each matched student. As expected, higher NAPLAN results on reading or writing were generally associated with higher EAL/D reading or writing phases, for the same students assessed.

The clear evidence of the concordance between the EAL/D assessments and those from other similar or related constructs strengthens the argument that the EAL/D instrument is functioning as intended.

There was also strong measurement evidence from the trial that the four modes were measuring a single underlying proficiency and that the four EAL/D phases (beginning, emerging, developing and consolidating) were being used meaningfully and consistently by the teachers. This means that EAL/D assessments over all four modes can be summarised to derive a single measure of English language proficiency for each student.

The analysis of the structural patterns of teacher judgements showed that some EAL/D students were at different levels for academic aspects of language (reading and writing) than conversational aspects of the language (listening and speaking). These findings corroborate earlier studies on the differential pattern of development across different modes for some EAL/D students. This further supports the validity of the assessments using the EAL/D instrument.

3. What are the successful elements and useful resources identified by teachers from the trial process?

Teachers generally supported the use of the EAL/D Learning Progression instrument for resource allocation in place of the current ESL Phase tool. Teachers also indicated considerable interest in using the instrument for informing mainstream program planning.

Survey results indicated that the professional learning provided was adequate to support confident and consistent judgements using the EAL/D Learning Progression. However, teachers required substantially more time than expected to plan and complete student assessments to make informed judgements on the appropriate phase. The teachers also indicated that they would like more sample assessment tasks and annotated work samples to support effective EAL/D Learning Progression implementation.

Teachers provided detailed feedback regarding the layout and wording used in the EAL/D Learning Progression. This feedback may be useful for ACARA and the ACARA English Language Proficiency Working Group for future enhancements to the EAL/D Learning Progression and its supporting packages.

Conclusions and recommendations

In conclusion, the NSW trial of the EAL/D Learning Progression has provided sufficient reliability and validity evidence for the instrument to be used in NSW government schools as a broad measure of English language proficiency for resource allocation. In addition, the NSW trial has national implications in terms of informing the design and the associated cost- benefit analysis of a prospective national trial. Such a national trial would provide recommendations about the potential use of the progression to report English language proficiency across jurisdictions.

The following recommendations were made based on evidence, feedback and insight from the NSW trial.

Recommendations for NSW government schools

1. Implementation for resource allocation

1.1 Implementation of the EAL/D Learning Progression is recommended as a replacement for the ESL Phase tool currently used in NSW government schools. If adopted, the EAL/D Learning Progression will become the broad measure of English language proficiency used for EAL/D students, and will become the basis of the allocation formula used for the ESL funding component of the new Resource Allocation Model in NSW government schools.

1.2 Full implementation of the EAL/D Learning Progression is recommended for 2014 so that data can be collected and used in the resource allocation process for 2015.

1.3 An implementation plan is recommended that builds on the learning from the trial, and includes:

a) developing and conducting a program of professional learning in government schools during 2013 to prepare teachers to use the EAL/D Learning Progression phase assessments for all EAL/D students from the beginning of 2014

b) ensuring professional learning programs emphasise the importance of judging students’ phases based on ongoing student assessment

c) developing resources to support teacher assessment using the EAL/D Learning Progression instrument, including samples of assessment tasks and annotated work samples demonstrating the evidence required to judge students against the EAL/D Learning Progression indicators

d) providing participants in professional learning programs with clear guidelines as to the purpose and potential uses of the EAL/D Learning Progression

e) providing participants in professional learning programs with clear guidelines as to the process of deriving an overall phase judgement from the four language mode phase judgements for each EAL/D student

f) building capacity in the new student administration and learning management system (SALM) to allow teachers to enter EAL/D mode and phase data. Teachers should be able to update at any time the EAL/D phase judgements for individual students in SALM by 2014

g) revising school data collection tools to collect EAL/D Learning Progression phases in each mode for every EAL/D student. This would replace the current ESL phase data collection tool.

2. Classroom support

The EAL/D Learning Progression should be mapped against the new NSW syllabuses and the literacy continuum. On the basis of this mapping, teaching resources should be developed where appropriate to support classroom teachers to program and plan for EAL/D learners.

National recommendations

3. National trial of the EAL/D Learning Progression

A national trial is recommended to test the generalisability of the evidence from the NSW trial. This national trial should include:

• a broader range of student and teacher demographic groups, including larger cohorts of specific student groups, in particular Aboriginal students
• examination of differential reliability in teachers’ judgements across different types of schools, students and language modes to inform future programs of teacher training and professional development
• recommendations about the potential use of the EAL/D Learning Progression to report English language proficiency across jurisdictions.

4. Detailed information to ACARA

It is recommended that ACARA is provided with this report of the NSW trial and with access to the detailed feedback provided by teachers related to the instrument itself. This may be useful to ACARA as authors of current and possible future versions of the EAL/D Learning Progression.