This report was originally published 16 September 2021.

Summary

The move to online testing for NAPLAN in 2019 brought many benefits to teachers, schools and education systems. However, prior to implementation, concerns were raised amongst stakeholders relating to the validity, comparability, equity and fairness of online testing. One key issue was the online assessment of writing, for Year 3 students in particular. The main concern was whether Year 3 students would have sufficient typing skills to produce online texts in timed conditions that were a valid reflection of their underlying writing proficiency. This report on the research conducted in 2016 was used to inform decision making concerning the move of NAPLAN writing tests from paper to online tests.

Informed by a literature review of research into online assessment of writing, this study used a mixed methods approach to investigate whether primary students in NSW schools perform differently according to the mode of writing test (computer‑based versus pen and paper based), and if there is a difference, whether it is uniform across different groups of students. In addition, the study examined the extent to which typing proficiency accounts for any differences observed in students’ performance in a computer-based writing test versus in a pen and paper test.

Whilst some students performed better and some performed worse on the computer-based test than the paper-based test, statistical analysis indicated that:

• after holding constant the effects of various task, student and school level factors on students’ writing performance, on average, students scored lower on computer-based writing tests than they did on paper-based tests, across all years examined (Years 2-5), and for both narrative and persuasive writing genres.
• for an average student, the estimated gap between paper-based and computer-based test results varied from 15 to 20 NAPLAN scaled points, depending on the student’s scholastic year. This represents roughly 0.2 to 0.3 of one standard deviation in writing results.
• over above other factors, typing proficiency was significantly associated with the mode effect, such that the faster students could type, the smaller the difference between their computer-based and paper-based results. For an increase of 5 words per minute in typing speed, the gap is reduced by approximately 7.5 scaled points.
• after taking into account all other factors including typing, the mode effect was found to be slightly worse for students with higher literacy ability than those with lower ability. The mode effect was approximately 5 scaled points larger for students whose literacy level was one standard deviation above the average for their year.
• the mode effect also appeared to be smaller (in size) for boys than for girls, and for Aboriginal than for non-Aboriginal students, although the differences did not reach statistical significance. Larger studies are needed to confirm any differential mode effect for these demographic groups.

In addition, qualitative analyses of researcher observations and teacher and student interview responses indicate that:

• students responded positively to using computers but there was a disconnect between the mode students said they preferred and the mode that best supported students’ performance in writing tests.
• students appeared to undertake less planning for the computer-based writing test compared to the paper-based test.
• there was considerable variability in technical readiness from school to school. A number of schools, particularly some in low SES areas, had an insufficient number of working computers, limited technical support available, and students who were not as familiar with using computers and accessing the internet. These issues are currently being addressed through the NSW NAPLAN online transition program, and are therefore expected to diminish over time.
• most trial schools do not explicitly teach keyboarding skills.
  

For Year 3 students, the study found that the median typing speed was 9 words per minute. The literature suggests this is lower than the handwriting speed for this age group, hence it is likely that many Year 3 students would struggle to produce online texts comparable to handwritten texts in a timed condition. Most research recommends that typing instruction is best commenced in the upper primary years, and there is evidence of cognitive and educational benefits of teaching handwriting to students in early learning years. For these reasons it is recommended Year 3 students continue to participate in NAPLAN writing tests on pen-and-paper.

Given the importance of typing proficiency for computer-based writing assessments, schools should consider their local contexts and identify an effective method for developing students’ typing fluency and to monitor the development of their typing proficiency over time, for students beyond Year 3.

Finally, further research is planned in NSW schools to investigate how the teaching of writing, and the writing process itself, can be enriched using new technologies to further develop students’ writing skills.