For the reasonable price of just $5 AUD per month or $50 per year, you can be a full subscriber to Filling the Pail with complete access to my extensive archive on a range of topics, as well as access to the unabridged version of my weekly Curios posts.
It may surprise you that I am usually a live-and-let-live kind of guy. People make mistakes. If those mistakes are unimportant, then who cares? These days, when I quote from blogs or articles that I then go on to criticise, I often quietly correct typos rather than add a “[sic]”. The latter can feel like a distraction from the more important question: is the claim right or wrong?
Nevertheless, I am certainly capable of pedantry, especially in response to the pedantry of others.
The Australian Education Research Organisation (AERO) has many critics in the faculties and bureaucracies of Australian education. I suspect these are the people who prefer problems to solutions and see AERO, with its clear, practical guidance, as a direct threat. In a world where it’s all too hard, AERO proves you can just do things. How is a savvy administrator expected to hold on to that lucrative sinecure if problems are simply solved?
The latest combatant to enter the fray is, as far as I know, anonymous, and so I have no idea if the above description applies to him, her, or them. On Friday, I was made aware by one of my numerous sources of a website calling itself, with all due pomposity and self-regard, the Australian Centre for the Public Understanding of Education (ACPUE—an unfortunate acronym for a website with academic pretensions). That very evening, Adam Voigt, CEO of Real Schools and sometime columnist for The Age was posting a screenshot from this website in my timeline. How interesting. He must have his sources, too.
The thing is, as of the time of writing, the ‘About’ page on the ACPUE website lists no CEO, team members or board. It could be one dude prompting Claude or it could be a team of disgruntled and marginalised education lecturers. We don’t know. Voigt did not know, either, and could not vouch for their data.
ACPUE lists numerous forthcoming projects, but the one completed so far is an attempted hatchet-job on a 2023 publication by AERO, something they present as ‘Evaluating AERO’s evidence claims’ as if it is an analysis of AERO’s entire output. This analysis largely consists of a table of claims made in the AERO report alongside ACPUE’s attempts to check the citations.
This table is unwieldy and the ‘expand all’ and ‘collapse all’ buttons do not work for me, so I used AI to export it to Excel.
In the table, the reference for each claim is analysed and a judgement is made by ACPUE as to whether the reference ‘matches’, is ‘plausible’, or ‘does not match’. This analysis reveals some genuine, if minor, issues. For example, there is one missing citation for early childhood and the development of executive functions—AERO 2023c is not listed in the references despite being cited in the body of the text. Yet the claim being made is hardly controversial.
Many claims have multiple references and each individual reference is analysed independently. This leads to an odd phenomenon. If AERO writes, say, “(Smith 2012, Jones 2015, Brown 2021)” at the end of a passage of text, and if ACPUE judges that each of Smith and Brown ‘matches’ the claim, it will still list Jones as ‘does not match’ unless it encompasses all parts of the claim. This is uncharitable because the AERO authors have supported the claim with the other two papers and Jones may only be relevant to a part of it.
For example, the AERO paper claims that:
“This approach to building and sustaining high expectations fosters belonging, positive relationships and effective teaching and learning (AERO 2023a; Cobb & Krownapple, 2019; Healey & Stroman, 2021; Miller & Steele, 2021).”
ACPUE judge that AERO 2023a and Healey & Stroman match the claim. So, that should probably be the end of the matter. But ACPUE cannot find the Cobb & Krownapple paper and so they list the result for this citation check only as ‘plausible’ and they decide the Miller & Steel reference ‘does not match’ because:
“They do not say that high expectations foster positive relationships but treat positive relationships as equally foundational. However they do say that high expectations are vital to improving academic achievement.”
Seriously? And what, exactly, are we arguing about here? Is there a counterclaim that building and sustaining high expectations does not foster positive relationships that needs to be addressed, despite the other supporting references?
Uncharitable judgements are present throughout the analysis. ACPUE does not think that a paper on Load Reduction Instruction, which suggests ‘guided independence’ as the fifth stage, supports claims about the value of guided, structured inquiry once students have gained sufficient expertise. This is partly due to the nature of the paper but also because ACPUE do not seem to view ‘guided independence’ and ‘guided, structured inquiry’ as the same thing. It is reasonable to have different views on such definitions, but this is a difference of opinion and not an example of a citation error.
Similarly, ACPUE do not view a Brown et al. 1989 paper, that mentions cognitive apprenticeship and students completing an ‘authentic domain activity’ as supporting a claim about inquiry learning because the authors ‘do not mention inquiry’. Again, this is a difference of opinion rather than a citation error.
Now, consider the following AERO claim:
“Education should not restrict First Nations students to having to choose between either pursuing academic success or sustaining their identity and cultural knowledge base (Price, 2019; Vass et al., 2019)”
ACPUE finds that the Price reference ‘matches’ and so, again, we should probably stop here. But it claims the Vass et al. reference does not match because “AERO’s claim does not match Vass et al., (2019) say.” I don’t know what this means, it is probably a typo or mistake, but I am mentioning it because if ACPUE are going to engage in an exercise in pedantry then it is fair game. And zooming out, what are we arguing about, here? The term ‘should not’ denotes an opinion and is anyone really of the opinion that First Nations students should have to choose between pursuing academic success and sustaining their identity? I’d love to see the references for that.
At the top of this report, ACPUE cite some statistics—the ones Voigt cited to me. They claim 47% of claims do not match, 42% of citation checks do not match and 67% of references are misused at least once. I cannot reverse-engineer these figures. The table lists all the citation checks and does not give results by claim. The citation checks show that 40.7% supposedly ‘do not match’ rather than 42%. And like the claims, there is no data on references being ‘misused’.
There is an irony in this exercise coming to light in the wake of Cambridge University—and its education faculty—being consumed by an alleged plagiarism scandal, with attempts by some in academia to wave away genuine, large-scale citation irregularities. I wonder what the author(s) of the ACPUE analysis made of that? I am dying to know who they are because they have now given me licence to subject their own work to a similar, if more robust, citation analysis, something I would relish but would otherwise avoid out of politeness.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.