3 Things That Affect AI-Powered Assessment in 2026

AI-powered assessment has potential to provide students more practice, free up faculty time, and give more consistent and thorough feedback. But it presents many challenges for faculty and technologists. I find—understandably—that faculty have a difficult time explaining their process for giving feedback in a way that and AI can repeat. Technologists and faculty alike are likely to under-evaluate their assessments, leading to LGTM “Looks Good to Me” analysis that frustrates students.

And so it is always a good exercise to have a handle on what is actually important for the most accurate AI assessment, and what doesn’t matter. When we have limited energy and time, at least we can focus that energy in the most impactful places. I recently did a dive on this and was surprised to learn that some of my previously held habits are less relevant today.

Here is the TL;DR: To build a reliable AI assessment pipeline, you must over-invest in rubrics, human-graded examples, and model size. You can de-prioritize persona prompting, temperature settings, and the perceived “premium” of commercial models.

My dive involved two recent studies examining LLM scoring in both university bioinformatics courses (Baylor College of Medicine) and medical clinical reasoning simulations (Harvard Graduate School of Education) give us an empirical look at the architecture of automated assessment.

Automated assignment grading with large language models: insights from a bioinformatics course - This study evaluated the effectiveness of automated LLM graders in a university bioinformatics course and demonstrated that clear rubrics and human-graded examples together lead to better accuracy, even using open-source models.

How Model Size, Temperature, and Prompt Style Affect LLM-Human Assessment Score Alignment - This research examined how different LLM configurations evaluate medical students’ clinical reasoning skills, finding that larger model sizes are crucial for aligning with human expert scores, while practices like “expert persona” prompting actually inflate grades and reduce reliability.

Based on my experience and supported by the experiments from the articles, here are three things that matter when it comes to increasing the accuracy of your AI-powered assessment.

1. Rubrics: You cannot simply feed an assignment into a model and ask for a grade; structured rubrics are required to build proper scaffolding. However, in a 2025 study on bioinformatics grading, researchers found that feeding an LLM a rubric alone resulted in overly strict grading.

2. Examples: Providing manual, human-graded examples is the other half of the framework, but using them on their own made the model far too lenient. When examples and rubrics are deployed together, they properly anchor the model, achieving accuracy comparable to human teaching assistants.

3. Model Size: In an evaluation of clinical reasoning assessments, researchers compared a frontier model (GPT-4o) against a smaller model (GPT-4o-mini). The larger model demonstrated consistently better alignment with human expert scores, while the smaller model showed poor alignment, despite agreeing internally with its own repeated ratings.

Much of the advice circulating about prompt engineering is anecdotal. The data shows that several common practices actually degrade the reliability of your pipeline.

1. Persona Prompting: Persona prompting (i.e. “Act as ____”) is a strong habit and one that has been backed up by evidence in the 2022-2023 era of LLMs. However, in the recent Harvard study this style of prompting actually backfired. Testing showed that persona prompting significantly inflates average scores and degrades alignment with actual human expert ratings. Skip the roleplay and give the model direct, structural criteria.

2. Temperature Fussing: Temperature, or the “creativity measure” of an LLM, was something I fretted over in 2023-2024. Since then, it seemed like temperature has become less important and the Harvard study seems to support that. In that study, temperature has a relatively minor effect on scoring alignment especially with larger models.

3. The Commercial Premium: Smart models do matter, but it doesn’t matter where you get that model from. Open-source models perform just as well as commercial alternatives in both grading accuracy and feedback satisfaction. Furthermore, researchers found that quantization—compressing a model so it uses less computational memory—has a negligible effect on an LLM’s grading capabilities.

To pivot from theory to practice, every educational technologist should adopt this mental model when designing an AI assessment pipeline:

• Always include rubrics and examples in your AI-Powered Assessment

• Drop elaborate persona prompting and temperature tweaking.

• Consider open source models as a way to get higher accuracy for less cost.