The Research Files Episode 55: Assessing the General Capabilities

Hello and thanks for downloading this podcast from Teacher magazine. I'm Rebecca Vukovic.

How can we teach and assess general capabilities including critical thinking, creative thinking, collaboration, and problem solving skills? This is a challenge that teachers are faced with in 21st Century classrooms. A project, funded by the Centre for Assessment Reform and Innovation (CARI) at the Australian Council for Educational Research (ACER), is working to develop an assessment framework for measuring and monitoring these skills in the classroom. Through a combination of curriculum-oriented assessment tools, learning progressions, and professional development, the project aims to equip teachers with the skills to integrate teaching and assessing of general capabilities into their classrooms.

Here to discuss this today is Dr Claire Scoular, a Research Fellow at ACER and the person charged with leading the project. In today's episode, she shares more about the assessment tool that's been developed, and we discuss some practical ways teachers can measure these skills, as well as some the challenges involved in doing so. Claire also shares some of her early analyses of data to come from the trials completed in schools across Australia. Let's dive in.

Rebecca Vukovic: Dr Claire Scoular, thanks for joining Teacher magazine.

Claire Scoular: Thanks for having me.

RV: So in education, we hear the term ‘21st Century Skills' used quite a lot and it can mean a number of things. So for the context of this interview, when you say ‘21st Century skills', what do you mean exactly?

CS: That's been one of the major hurdles I think in this field of research is a lot of emphasis is put on what do we call them and not enough emphasis on what are they and what do they look like in the classroom? So here at ACER we adopting the term General Capabilities because that's the Australian term for them, but that can be synonymous with ‘21st Century skills' or ‘transversal competences' as they're called in the wider Asia Pacific. What we mean by that is a set of tools that we can give students to assist with their learning, support their learning. And those things look like collaboration, critical thinking, creative thinking, problem solving skills, self-regulation skills. So, we're really providing a set of tools that students can harness to improve, and teachers can provide learning to students with these tools to assist with their development.

RV: And then why is it important that we have some practical solutions for measuring these skills and that it's an integrated approach?

CS: At this point there aren't enough resources for teachers in schools to do that intervention with students. I think we're well beyond now the realisation that these skills are important. We know that the workforce is demanding students coming out of schools and universities with these skills, so we need to really merge that thinking with a practical solution for integration. The integration though has been a slow-burner. Some education systems have started by integrating it in assessment, others in the curriculum reform, but very few have looked at the teaching aspect or development of resources aspect. But really, what we're going to need is a harmonisation of all of those three things – the assessment, the curriculum and the pedagogy – so that we've got a harmonious approach to integration.

RV: In your research, you've found that teachers recognise the value of teaching 21st Century skills and are open and enthusiastic, but aren't necessarily prepared to teach these skills. Could you tell me more about this?

CS: Yes, you're right. Teachers are enthusiastic to get into this space and many, many schools are already finding their own ways of doing that. Teachers are doing this and they want to do it, the problem is the limited resources or more specifically, the lack of evidence for the resources out there at this point. And our approach at ACER has been to provide evidence base for the resources that we're putting out there – so, learning progressions, and the rubrics, and the skill definitions. We want to ensure that those have a really substantial evidence base, so that teachers can feel confident that they have something they can use and that will work.

RV: And I want to ask you a little bit more about that. Because you've been working on a project funded by the Centre for Assessment Reform and Innovation here at the Australian Council for Educational Research. What are the aims of this project?

CS: The centre itself has three aims and one of those is new metrics. So, we're thinking about measuring different things than your standard knowledge domains or subject areas. And therefore, that led us down the path of the General Capabilities. We've started by identifying three skills, because you have to start somewhere. We selected these three skills on the basis of requests we were getting from teachers, conversations we were having with teachers about what they thought they were already doing in the classroom but needed a little bit more explicit awareness about. And those skills were the critical and the creative thinking and the collaboration.

RV: And so you've developed an assessment framework for measuring and monitoring those three things – collaboration, critical thinking and creative thinking – in the classroom. Just to be really clear for those listening, when you refer to collaboration, what is it you mean exactly?

CS: The biggest distinction with collaboration from group work is this idea of the necessity of students to work together. So we are thinking about the real world, in the sense of if you work for other people you do it for a purpose, as opposed to working individually. But you would work with other people because they have a different skillset, they bring something to the table, maybe they have different information or resources to you. So, in the classroom, we would fabricate that scenario by giving different students different pieces of the puzzle, so to speak, so that it's necessary for every, single student in the group to contribute. There's a shared goal as well. So it gives the students motivation to work together. And so therefore it gives a purpose to working in a group, rather than just putting students in a group for the sake of it. And by this sense, it really harnesses students' ability to collaborate and produces those social skills that we want to see emerge.

RV: And in the context of this research, what is critical thinking then?

CS: Critical thinking is the ability to really analyse and synthesise information, but also identify the gaps that you have in your information or your knowledge. And to be able to go away and seek out information that would then plug those gaps. But going beyond that, it's the ability to analyse that information in a way that allows students to identify bias or discriminate different types of information. And I think in this world that's a very important thing that teachers want to get behind, because we have the internet, of course, and students have access to that readily available now. And so how can we teach students information discrimination? How can we get them to look at sources of information and work out if there's a tendency for bias in that? Or look at the news and work out what the perspective is of the presenter or the news writer. I think that's been a big element or focus of teachers in how they get behind that and get students up front and understanding those concepts.

RV: And of course this is different to creative thinking. So what's your definition of creative thinking?

CS: We took the stance early on that creative thinking is a bit distinct from creativity. And we did that deliberately because, in the Arts and Humanities there's this idea of creativity as potentially an innate ability, or something that produces something that can be measured in very subjective terms – it's very aesthetically dependent. And so we wanted to make a distinction there between that and something that was teachable and something that was objectively assessable. And so therefore we're looking from the perspective of a creative thinker.

And so in that sense we're looking at the ability to think of novel ideas, but novel in the sense of, what's novel for that student? … it doesn't need to be solving a big world problem, or inventing a new product, it could just be something that's novel to a Grade 2 or a Grade 5 student – something that they've never heard of before.

We're also thinking about the quality of ideas and the quantity of ideas as well. Looking at whether students can solve problems, for example, with a range of solutions and come up with different types of ideas. Also getting behind different perspectives. Can students pull a problem apart and then look at it from different angles and then create a new way of solving that problem, that maybe wasn't apparent to begin with?

RV: Really interesting. And these skills, they can be quite complex and they're made up of lots of small chunks and there's also a fair bit of overlap. So, how do you deal with that overlap? And that's also overlap between teachers, lessons and subject areas.

CS: I think what it comes down to first and foremost is defining these skills in a really clear, detailed way. Our initial thinking around this was that teachers want the simplified version of what these skills are – a couple of paragraphs would do. In fact, the feedback that we got from teachers is that they want as much detail as possible and that led us down the route of our detailed skill definitions.

We divide each of the skills up into strands, and then each strand has various aspects that contribute to that. So, for example, an aspect of collaboration might be negotiation. What we found in both teaching and assessment development is that teachers have been targeting the aspects, rather than the bigger skill itself. And, of course, that's been making it easier to teach and assess because they've got a narrower focus for which to integrate. They'll integrate negotiation skills into their lesson rather than trying to integrate collaboration into their lesson.

Yes, there's overlap between a lot of the skills. Our upcoming psychometric analysis is hopefully going to allow us to delve into potentially identifying what some of those formal overlaps are. I think what we're finding in the classroom is that those overlaps aren't so important if you are taking the approach that we've taken, in the sense of being explicit about needing the skills, working with students to identify what those skills look like. And any type of awareness, in that sense, is allowing for some impact to be made in the classroom.

I think going back to this idea of overlap over grades, teachers and within schools, we really need that common language and that consistency in terminology to allow us to be able to talk about these skills in an explicit, deeper way, rather than just that surface level understanding. And I think these detailed skill descriptions really [provide us] with that common language that teachers can use to discuss the skills – much in the same way that we've been discussing literacy and numeracy for centuries.

RV: And it's also important to note that you can't just decide to ‘do creativity' in a lesson, isn't?

CS: Yes and I think this is the reservation, understandably, for teachers is that they don't want to invest their really limited and valuable time in an approach that they aren't sure is going to have any impact. And so we are finding that the upfront requirement of the integration within lesson plans does take a bit of practice and does take some time to begin with. However, the feedback from teachers is that, as they get to understand these skills from skill descriptions, it does become more of a second nature to integrate the skills into the lesson plans. And, in fact, what we're finding is teachers are saying, particularly with these three skills, ‘we're already doing that, we just didn't identify those components of the lesson as being those skills'. For example, many teachers use probing questioning to further interrogate students and service responses, and we see that very much under the umbrella of critical thinking. And so just making that more explicit in the lesson has really helped those teachers and students to get to grips with what critical thinking is.

RV: And you've been working in quite a number of schools. Could you share any practical examples of how educators are doing this well?

CS: For schools that are looking at that harmonisation of alignment, they've been particularly successful at integration. So not just looking at integrating into the lesson plans, but looking at assessment and then trying to wrap their head around this idea of identifying resources that would help. Going back again to that idea of the definitions, focusing on particular aspects and breaking down the constructs, rather than trying to tackle the whole skill in one 50 minute lesson.

Other schools have also taken a problem-based learning approach – which is the approach taken in our assessment tasks – where you centre the lesson, quite often it's an interdisciplinary lesson, around this idea of having a problem scenario to solve, and then they integrate one or two of the skills into that lesson. So students might have to think critically about this problem and collaborate with others. This type of interdisciplinary problem-based lesson format might run, you know, for several weeks. What this allows teachers to do is to still have subject-orientated lessons, still targeting the curriculum topics and the learning outcomes, but to integrate those skills into that in quite a streamlined and authentic way.

RV: What role does technology play in the assessment of these skills?

CS: Technology can be beneficial, but only if it's used in a particular way. Our assessments are online and our hope is that we can move to a more sophisticated, custom-built platform in the near future. The reason for doing that would be to harness the benefits of processed data – being able to capture student behaviours in real time. That would lead us to be able to automatically score and generate reports really efficiently. Now, we would hope that the benefits of that process data in providing us information about the student behaviours would really allow us to understand better the skills and how those skills are playing out in a processed way.

The automation, of course, of scoring some of these assessments takes some of that burden off of teachers for scoring it themselves. So, really, from a teachers' perspective, the technology would provide ready access, being online, something they don't have to print out or download, they would just put students onto the computers. And something that would generate that in an efficient and quick way would generate the outcomes. So from, potentially from an efficiency perspective for teachers, and for us as researchers, an information gathering endeavour to get more information about those skills.

But I don't think technology is essential to seeing these skills played out, and it doesn't provide the whole picture either, particularly for something like collaboration. We can attempt to measure students' collaboration online and of course we can have certain measures to do that. But we still need to be filling gaps to create a bigger picture with having offline, classroom-based, collaborative tasks as well, so that we can observe students' body language and tone of language and suchlike.

RV: And Claire, you've touched on this already but is there anything else you could tell me about assessment tool and how it works?

CS: Our assessment tool is centred around this problem-based learning idea. So we have a problem scenario presented to students. We want to keep that authentic, something that is relevant to students' learning and their own lives. That ensures students stay engaged and make it purposeful for them.

So we have one assessment that has 10 tasks. Each of these tasks focuses on measuring one of those three skills. And students would work either individually or in groups of three to complete each of these tasks. Within each task, we have rubrics that measure the skill in that task. So we're looking at the students' output, the students working together to measure those skills being applied in each particular assessment. And at the end of that assessment we can look at how a student has performed over each of those 10 tasks and provide some estimate of ability for where students are at with those three skills. And we can look at those in regards to the levels of progression and try to place students on that progression to give them an idea.

Where we want to go with this assessment then is to be able to generate reports that provide teachers with intervention ideas or tasks ideas that they can use in the classroom to target students at particular levels of progression and increase their ability, so they move up to the next level of progression.

RV: And the assessment tools are being trialled across Australia. Could you share any of your early analyses of the trial data?

CS: It's good to remember that only maybe eight years ago we weren't even certain that we should, or could, measure these skills and certainly we had a camp of doubters completely. So, it's really nice for us to be able to say that we have developed a tool and we are gathering data that is providing information about these skills and we are seeing these skills playing out, being demonstrated by students in a way that we had initially conceived.

What's extra beneficial, and what we hoped but weren't certain of to being with, was not just being able to see these skills, but seeing them being played out at various levels of proficiency. So we can actually see what a ‘low' to a ‘mid' to a ‘high' critical thinker looks like, based upon the student responses that we're getting. What we're hoping over the next few months with our psychometric analysis, is to really further validate those progressions and those skill definitions in this iterative way. So every piece of data that we get is informing more and more our skill definitions, our learning progressions and our rubrics. So that, as I say, we can really be building on that evidence base for our resources, and ensuring that everything we have in writing is based upon what we've been seeing in these assessments but other assessments that we're able to tap into as well.

RV: And so, for teachers or school leaders listening to this podcast, is there any way they could find out some more information about the project or where they go to learn more about what you're doing?

CS: That's a really great question because we've just formed a research consortium for General Capabilities here at ACER. What that looks like is going to be initially a quarterly newsletter that we'll send out to people who sign up. What we'll be able to share is updates on our research here on the General Capabilities, as well updates on our resources being released and where people can access those. We had great success with the masterclass being run at the ACER Research Conference, we would love to run more of those masterclasses. So we're going to be running at least one in each state over the next six months – and we will be able to provide that information about those as the newsletter comes out as well.

RV: Fantastic. Well we'll be following the project with interest. Dr Claire Scoular, thanks for sharing your work and your insights with Teacher magazine.

CS: Thanks again.

That's all for this episode. If you'd like to listen to more from Teacher, you'll find us by searching ‘Teacher ACER' wherever you get your podcasts – on Apple Podcasts, SoundCloud or Spotify. And while you're there, make sure you subscribe to the channel by clicking the purple ‘Subscribe' button on Apple, the green ‘Follow' button on Spotify or the orange ‘Follow' button on SoundCloud. By subscribing, you ensure that new podcasts land in your feed as soon as they're available. You'll also get a notification straight to your device, which is really handy for making sure you never miss an episode.