Computer Adaptive Testing – challenging traditional thinking

Achievement tests traditionally have been designed to assess all students’ performances on the same curriculum content, using the same test or examination, at the same time. The thinking behind this has been that, if every student attempts the same test questions under identical (‘standardised’) test conditions, then all students will be treated equally and can be compared fairly.

Tests of this kind usually are conducted at specific times during the schooling process – for example, at the end of a topic, school term, school year or stage of school. Year 12 examinations are an example. The purpose is to establish how much of a body of taught content students are able to demonstrate, which is then converted to a percentage or other numerical mark or letter grade.

In addition to grading and certifying students’ performances, tests of this kind sometimes determine entry into the next stage of education.

In his 2019 article, The future of learning and the future of assessment, Harvard’s Richard Elmore describes traditional assessments as delivering ‘verdicts’ on students’ performances for the purposes of rationing and allocating scarce benefits. He argues that such assessments serve institutional interests rather than the interests of learning:

The role of assessment in the existing institutional structure of education is to deliver a summary verdict on how competent students are, a verdict they will carry through their subsequent development as learners. This verdict is reinforced and legitimated by a merit-based attainment structure designed to ration and allocate social status, rather than to develop competency in learning.

In the future, assessments will increasingly be formed around the radical idea that inquiry, measurement, and analysis should serve the interests of the individual learner, rather than the institutions that claim authority over the learner. (Elmore, 2019)

Adaptive assessments have a very different starting point from traditional tests and examinations. They do not set out to assess and grade every student’s performance on the same body of taught content. Instead, they are designed to establish the points individuals have reached in their long-term progress in an area of learning. This usually means establishing what they know, understand and can do at the time of assessment – information that can then be used to identify next steps in teaching and learning and to monitor individual progress over time.

Adaptive assessments are a response to the observation that students in the same year of school are at very different points in their learning. At any given time, some students are much more advanced than others. Adaptive assessments ‘adapt’ to these differences in students’ levels of attainment. Students who are less advanced, and so experience lower rates of success during an assessment, are automatically administered less difficult tasks. Students who are more advanced, and so experience higher rates of success, are administered more difficult tasks. In this way, each student is automatically administered a test tailored to where they are in their learning.

Traditional achievement tests report results against a common body of taught content. Performance expectations or ‘standards’ may be set in relation to this content and every student’s performance reported against these standards – a practice sometimes referred to as ‘standard-based’ assessment.

Adaptive assessments do not report students’ results against a common body of taught content, but against a described and illustrated continuum or progression of learning. The primary outcome of assessment is not a grade, but an estimate of the point an individual has reached in their learning, regardless of their age or year level. A well-constructed continuum of this kind is a requirement for all computer adaptive assessment and also enables the monitoring of individual progress over time –something that is not possible with assessments focused only on grading performances on defined bodies of taught content.

As the OECD has noted, this fundamentally different approach to assessment is a central element of the new Welsh curriculum:

As part of the new curriculum in Wales, student assessment has been associated with a new approach to learning progression itself. Wales describes progression as a continuum from ages 3 to 16 rather than a succession of ‘Key Stages’ (Wales’ groupings of Years of primary and secondary education). As a result, the system aims to move away from the standard-based assessment approach that measured how well each student fitted the attainment goals for each Year and Key Stage…
An important step forward in this regard is the ongoing development of a system of adaptive online personalised assessments… Some OECD countries, including Denmark and the Netherlands, are using such computer-based adaptive technology, which presents students with test items sequentially according to their performance on previous test items. (OECD, 2020)

Computer adaptive tests in mathematics and reading are now available to schools in the form of ACER’s Progressive Achievement (PAT Adaptive) assessments. These online adaptive assessments are alternatives to existing PAT Mathematics and PAT Reading test forms, which also continue to be available to schools. Each PAT Adaptive assessment analyses how each student performs during testing and automatically administers easier or harder sets of questions (‘testlets’) based on that student’s performances on prior questions.

The result is an estimate of the point each student has reached on a progression of learning in mathematics or reading. These mathematics and reading progressions describe and illustrate increasing proficiency, enabling teachers to interpret each student’s test result in terms of the knowledge, skills and understandings typical of students at that level. Students are located on the progressions by their PAT scores, which have been adjusted for the difficulties of the questions they were administered. In this way, each student’s PAT score can be interpreted as a level of proficiency on a learning progression and also compared with the locations of other students on the same progression.

Having established the points individuals have reached on learning progressions in mathematics and reading, online PAT Teaching Resources provide guidance on teaching strategies likely to be appropriate to individuals’ current levels of attainment. Teachers can then administer PAT Adaptive assessments later in the school year to measure students’ progress.

Computer adaptive tests offer a glimpse into the future of learning, the curriculum and assessment. This future will be based on identifying and responding to individual progress and learning needs, rather than on traditional group-based solutions.

References

Elmore, R. F. (2019). The future of learning and the future of assessment. ECNU Review of Education, 2(3), 328-341.

OECD. (2020). Achieving the New Curriculum for Wales, Implementing Education Policies. OECD Publishing. https://doi.org/10.1787/4b483953-en