The case for organic assessment: 5 reasons to not outsource marking to AI

They screamed with terror as the dinosaurs savaged their expedition.

What had we done?

The deafening of the Tyrannosaurus froze the same thought in the minds of its prey:

We’ve made a huge mistake.

Jurassic Park is perhaps one of the greatest parables for the modern age. It speaks to the deadly cocktail of innovation without wisdom, and is home to one of my favourite lines in cinema:

Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.

As we approach AI and assessment, I want to argue we are in the horns of the Jurassic Dilemma.

The Jurassic Dilemma is the difficult delineation between could and should. With AI moving at the speed of a bullet out of a gun, we could do all sorts of things. We could automate, outsource, and delegate large chunks of our life and work.

But could is not should. Capability is not obligation.

And I want to give you five reasons we should not outsource assessment to Artificial Intelligence.

Let’s get some terminology right at the outset. By assessment I mean the high-value (GET BETTER DEFINITION), summative assessment that represents the high-water mark of a student’s achievement. I’m not arguing that we should all mark our weekly retrieval practice quizzes by hand. And by outsourcing I mean allowing the AI to make the critical and decisive evaluation of the work sample.

So here are five reasons we should not outsource assessment to AI.

The AI and assessment conversation is raging, and I can’t help but notice an obvious blunder made by many in the outsourcing camp. It’s debating 101:

Define your terms.

Our word ‘assessment’ comes from the Latin word ‘assidere’ - ad (“beside”) and sedere (“to sit”).

Assessment means to sit beside.

It’s a physical word. A tangible word. A word that speaks to connection, relationship, and knowledge of the other.

When we outsource our assessment to AI, we are leaving the metaphorical table and politely ushering a robot into our seat. We vacate the seat beside our learner and, in doing so, contort the very thing we’re doing.

As an educator, I’ve noticed that the more I bring the ‘sitting beside’ image into practice, the richer assessment has become. A written comment beats a stand-alone letter grade. A 1-1 meeting about the work with feedback beats them both. The more human, the more authentic the assessment.

It’s 5:06 am as I write this paragraph.

I take pride in my writing - and I’m grateful that you lend me your attention for a few minutes each week. But what if no one was going to read what I wrote? What if I werejust getting assigned a percentage grade by an algorithm. Even a very good algorithm?

Well, I wouldn’t be up at 5:06 am writing this paragraph.

Audience matters. How we do our work is shaped by who we are doing our work for.

A few weeks ago in “5 theses against AI humanoid teachers” I discussed a study by Cohen and Riel. They found that students who were writing for their peers wrote better essays than when writing for teachers for a semester grade.

‘Who’ shapes ‘how’.

And if the who is an AI algorithm, the how will quickly deteriorate.

Let me give you a hypothetical scenario.

Imagine if AI could mark student work accurately (and that’s a big ‘if’).

Imagine that it gave good feedback at the right level and this feedback was actually understood by the learner and not rejected as an insurmountable wall of text.

Even in that scenario, I would still argue outsourcing assessment to AI is wrong.

It’s automating the wrong thing.

There are many things which can (and should) be outsourced to AI:

• Creating simpler versions of texts for low-literacy readers

• Finding a variety of sources for students to examine

• Crafting sentence-stems to help struggling writers

• Creating checklists for assessment completion

• Specifying definitions for key vocabulary

• Drafting exemplars and non-exemplars

• Creating unit readers

All these tasks can be given to AI and then used once a human has done a low-altitude flyover to ensure accuracy and suitability.

AI can help us speed up the busywork so we can spend more time on the things that shouldn’t be rushed.

And assessment is one of the things that shouldn’t be rushed.

There’s an old saying in education:

It’s the student’s fault if they fail a test. But it’s the teacher’s fault if everyone fails the test.

And the simple truth is that assessment doesn’t just give us feedback on student learning. It gives us feedback on our teaching.

Marking by hand (or by brain) allows me to see what stuck. Do you know what I mean when I say ‘stuck?’ There are some ideas that are just sticky. Some things you labour over time after time, yet you see one throwaway line present in 20 student essays.

(I had this teaching WWII, some students forgot the oft-repeated basics but remembered the one story I told about Operation Konserve.)

As I read multiple assessments I get a feel for what got absorbed and what didn’t - what needs more oxygen and what can afford to have less air-time.

In short, not outsourcing assessment to AI gives me an opportunity to learn about my own teaching, and everyone benefits when teachers improve their practice.

Two days ago I was walking around the classroom as students were working.

I like to take my pen and make little annotations on students’ work or give them a tick etc. It’s good to show I’m engaged and I always see diligence and quality improve when I do it.

I call it the cruise and peruse.

(apparently someone else out there calls it the tour to be sure - can you let me know who that is if you know?)

But on my cruise and peruse I remembered there was a student whose essay I’d just marked who really struggled to integrate evidence in his argument. He had good evidence and a good argument, he just couldn’t get them to work together well.

Now this was not the sort of thing I would have noticed with a quick peruse.

But because I already had the data points in mind, I was able to look for it, spot it, and give some quick, targeted advice.

In short, time spent marking gave me valuable knowledge of my learners that then informed my quality differentiated teaching practice.

We are in the horns of the Jurassic dilemma.

When what we could do is changing so quickly we must think very carefully about what we should do.

The good news is that if we get this wrong we aren’t going to release a new species of flesh eating dinosaurs. But we will slowly erode the humanity of education. And that’s why I think for the major tasks and summative assessments, we should assess organically.

The five reasons I’ve given you are as follows:

• Assessment, by definition, should be ‘unoutsourceable’.

• Outsourcing assessment reduces the quality of the work being assessed.

• Outsourcing assessment seeks efficiency in the wrong area.

• Organic marking is fantastic professional development.

• Organic marking informs in-the-moment high-quality teaching.

And if you’re finding that this is a live discussion in your school, I’ve got something that I think you’ll find really useful.

Just click on the poster below to find out more - I hope to see you there!