How well are we learning from NAPLAN?

Some Australian schools and school systems have seen greater improvements in NAPLAN results than others. How well do we understand where improvements are occurring and why?

There has been no higher priority in Australian school education over the past two decades than the improvement of students' literacy and numeracy levels. This has been the goal of a range of state, territory and Australian government initiatives and programs, including national partnership agreements, the National Assessment Program-Literacy and Numeracy (NAPLAN) and the My School website. Other initiatives to close gaps for Indigenous and disadvantaged students and to improve outcomes in the early years of school have shared the objective of increasing students' literacy and numeracy levels.

Despite the high priority given to improving literacy and numeracy skills, evidence from the OECD's Programme for International Student Assessment (PISA) shows a steady decline in the average reading and mathematical literacy levels of Australian 15-year-olds since 2000. These declines occurred in an absolute sense, not simply in relation to performances in other countries. In mathematical literacy the decline was dramatic; while Australia was one of a handful of high performing countries in 2000, by 2015 mathematical literacy levels had declined to about the OECD average.

Interestingly, these declines are not reflected in Year 9 NAPLAN performances, which were unchanged between 2008 and 2016, raising a question about why these two assessment programs reach different conclusions.

One difference between PISA and NAPLAN is that, while PISA assesses a national sample of 15-year-olds, NAPLAN assesses all students in Years 3, 5, 7 and 9. The most likely explanation for the different findings of these two programs, however, is that they assess different kinds of skills. The focus in PISA is not on basic reading and numeracy skills, but on students' abilities to apply higher-level skills in reading and mathematics to the solution of real-world problems.

NAPLAN makes it possible to explore the reading and numeracy skills of Australian students in some detail. Because NAPLAN assesses all students rather than samples of students, it is possible to study changes in performance at the level of individual schools as well as school systems – including schools and systems in which results have improved since 2008. This, in turn, introduces the possibility of identifying policies and practices that may have led to these improvements.

But how well are we capitalising on this opportunity? Do we know where improvements have occurred and do we understand why?

A starting point is to consider overall national trends in NAPLAN since 2008. These are shown for reading and numeracy in Figure 1. It can be seen from these graphs that, in Years 7 and 9, there was no significant change in mean reading or numeracy levels between 2008 and 2016. The lines for these year levels are essentially flat.

This is not the case at Years 3 and 5. Between 2008 and 2016, the national means in Year 5 reading and numeracy increased by 17 points on the NAPLAN scale. An even greater improvement (25 points) occurred in Year 3 reading, however there was no improvement in numeracy over this period.

Figure 1. National reading and numeracy means (NAPLAN)

In the context of Australia's PISA results for 15-year-olds, these improvements in basic literacy and numeracy skills in primary schools are encouraging. In Year 3, in particular, there have been strong improvements in average reading levels.

But how well do we understand the reasons for these improvements? Do we believe that they are sustainable into the future? Are they the result of changed educational policies since 2008? Are they the result of more effective teaching practices and, if so, do we understand the changes that were made and the factors that influenced those changes? Are improvements due to changes that occurred in schools or can they be traced to the years before school?

Another interesting question is whether these gains were uniform across the nation. Did some states or territories experience greater improvements than others?

The answer to this question is that one state, Queensland, experienced significantly greater improvements in Year 3 and Year 5 reading and numeracy levels than other jurisdictions. Figure 2 shows the relative performances of Queensland students – that is, how far the Queensland mean was below the national mean – in each year between 2008 and 2016.

Figure 2. Queensland reading and numeracy means relative to national mean (NAPLAN)

It can be seen from Figure 2 that there were very significant improvements in the reading and numeracy levels of Queensland Year 3 students between 2008 and 2016. It seems likely that the introduction of a Preparatory year in Queensland from 2008 provided part of the reason for this improvement, however much of the observed improvement occurred prior to this first cohort entering Year 3 in 2011 and Year 5 in 2013. It is also clear that the strong gains in Queensland primary schools were not reflected in the performances of secondary school students.

How well do we understand why improvements in Queensland primary schools were so much greater than in the rest of the country? Were there particular policies and/or practices that produced these improvements? Are there lessons for other jurisdictions?

The other state in which there were unusually large improvements in NAPLAN results between 2008 and 2016 was Western Australia. However, the gains in that state were largely in secondary schools and, in particular, in Year 9 (see Figure 3).

Figure 3. Western Australian reading and numeracy means relative to national mean (NAPLAN)

In 2008, the mean Year 9 performance in Western Australia was significantly below the national mean in both reading and numeracy. By 2016, performance was significantly above the national mean. (Recall from Figure 1 that, for the country as a whole, there was very little change in Year 9 performances over this period.)

Once again, there is a question about the reasons for this improvement. During this period, Western Australia introduced a requirement that students demonstrate a minimum standard of literacy and numeracy for the award of the Western Australian Certificate of Education. Students can meet this requirement through their performance on Year 9 NAPLAN. However, this requirement is unlikely to explain fully the steady improvements in Year 9 performances in Western Australia dating back as far as 2008.

Learning as a profession

Results from NAPLAN and from international studies such as PISA and the IEA's Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) provide a variety of national learning opportunities.

What can we learn as a profession from national improvements in NAPLAN in Year 3 reading; exceptional improvements in Queensland primary schools since 2008; and exceptional improvements in Year 9 in Western Australia?

What can we learn from the fact that Australian secondary students, while maintaining their average performances in NAPLAN, have become steadily less able to apply higher-level reading and mathematics skills to solve real-world problems? Do we know why this is happening? Does it require a policy response?

What can we learn from the fact that, while levels of reading and mathematical literacy as measured by PISA have declined nationally, a much smaller decline has occurred in Victoria (and in the case of scientific literacy, there has been no decline at all)? Is there an explanation? Are there lessons that could be learnt from Victoria?

Are we learning from individual schools' improvements in NAPLAN? The My School website was introduced in part as an opportunity to compare the performances of schools serving students from similar socioeconomic backgrounds. What national lessons are we learning from such comparisons? While individual schools and school systems may have views about why improvements have occurred, how reliable are these?

With evidence that reading and numeracy levels have stagnated or declined in many Australian jurisdictions, perhaps it is time for a more systematic effort to identify and understand where gains are being made in our schools.