Kappa scores considered harmful

Measures of how well people agree about data they've labelled are often used to estimate data quality. And that’s OK.

Let’s step back a bit. Machine learning models are trained using labelled data. This is called supervised learning. The supervision is both an item, i.e. a single instance from the data, and a label, together. The goal for the system is to be able to accurately predict the label of instances.

Where do the labels come from? Often (especially in NLP), humans manually add labels. This is a major source of knowledge for the model: any labels that could be automatically added, could be labelled automatically at run time, so there’s no point making data points for a machine learning algorithm to generalise over; that’s less efficient.

Humans don’t always agree about labels, though. Sometimes, instances might just be ambiguous. Other times, the labelling scheme might be ambiguous. Ambiguities are often addressed by developing annotation guidelines and, hopefully, iterating over them a few times using input from annotators to develop a consistent, clear definition of the labels. This definition also serves as a description of what the labels that the model eventually predicts, are intended to be.

It’s hard to know how well all these layers are actually working, and often, edge cases require a decent amount of domain- and problem-specific knowledge before they can even be described. This is hard to compare and hard to discuss, so instead the whole thing is, sometimes, reduced to a single figure: inter-annotator agreement.

Higher agreement is interpreted as better quality data. This is a bit of a fallacy; it could just mean that the particular data making up the dataset was particularly clear and simple to label, or that the group of annotators used were quite homogeneous and all had the same interpretation of the phenomenon at hand. We saw in one paper relating annotation of named entities in English social media that expert annotators would agree well (>80% of the time), but actually only recognise 30% of entities; in effect, the experts all made the same mistakes and shared the same blindspots. In contrast, crowd workers over the same data picked up over 80% of the entities, but rarely agreed with each other.

Agreement also gets a bit tricky to calculate once there are more than two annotators: pairwise agreement can produce unintuitive results (for example, if two of three annotators agree on a binary labelling task, e.g. labelling A-A-B, then two thirds of the pairwise agreements are disagree, A/A-A/B-A/B, relating to one third agreement). There are plenty of metrics in this area, and the kappa family of measures tend to get good recommendations: Cohen’s kappa for when there are two raters, Fleiss’ kappa for when there is any number of raters. Kappa measures correct for chance agreement, are flexible and well-defined for varying numbers of categories, and can tolerate various numbers of raters.

It’s just that the scores they give are wildly problematic.

So back in 2020 we got a complaint about one of our papers on automatic hate speech detection. That goes with the territory - nothing new to see there. It was explicitly designed to burn researcher time. The core of the complaint was that the we stated we reached “good agreement”, but the kappa scores reported corresponded to “moderate agreement” according to a table on Wikipedia (one rank better, in that table, than “fair agreement”). The Wikipedia screenshot included a disclaimer that the table shouldn’t be applied anywhere beyond an example in a 1977 paper, and so was easy to see off: we weren’t working in that two-annotator two-class known-class-balance situation, and the authors of that paper (Landis & Koch) explicitly state that they pulled their evaluations of arbitrary ranges of kappa values out of thin air:

In order to maintain consistent nomenclature when describing the relative strength of agreement associated with kappa statistics, the following labels will be assigned to the corresponding ranges of kappa:

• Kappa statistic - Strength of Agreement

• Kappa < 0.00 - Poor

• Kappa 0.00-0.20 - Slight

• Kappa 0.21-0.40 - Fair

• Kappa 0.41-0.60 - Moderate

• Kappa 0.61-0.80 - Substantial

• Kappa 0.81-1.00 - Almost Perfect

and the kicker:

Although these divisions are clearly arbitrary, they do provide useful "benchmarks" for the discussion of the specific example in Table 1.

However, despite the authors’ best efforts to state what was going on, their evaluations have been popular. Like, really popular. A brief look through the 75,000+ papers citing of Landis & Koch 1977 shows how this work is used, and it’s almost always to connect an evaluatory word from Table 1 to the kappa score someone has for their experiments.

“results were: 0.609 and 0.821 for positive and negative aspects respectively, suggesting substantial to outstanding inter-coder agreement”

κ = 0.629 (p < 0.001), suggesting “substantial agreement” (Landis and Koch, 1977)

All models showed substantial agreement (0.61 ≤ k ≤ 0.80; Landis and Koch, 1977) among their annotations

NOS-E scores (ICC ¼ 0.675, CI 0.49–0.79) were shown to be more reliable with a fair to good agreement (Landis & Koch, 1977)

interrater reliability for the independent full-text screening of two authors was strong (Kappa = 0.85, p < .001; Landis and Koch 1977).

With a kappa-statistic of 0.81 and above in three of the four cases, we achieved near perfect agreement between the coders (Landis & Koch, 1977).

(kappa = 0.92) presenting a high level of agreement between the reviewers under the Kappa criteria

(source: the top 20 most-cited papers citing Landis & Koch, published since 2019)

Too bad, Richard and Gary, we can see you tried, and I’m sorry this happened!

But why is this problematic?

Well, kappa scores aren’t directly connected to annotator accuracy. A number of confounding factors, which frequently vary between annotation situations, shift and distort how kappa scores will look for a given accuracy. And changes in of those factors render kappa scores incomparable.

Take for example chance correction. One great thing that kappa scores (and many other good agreement metrics) do is to correct for chance annotation - that is, the chance that two annotators will just randomly agree regardless of the data that’s being processed. Annotations from people giving random responses have some probability of agreeing with each other - if for example there are two classes and two annotators assigning them with equal probability, there’s a 50% chance they will agree in any case (out of the four possible responses A/A, A/B, B/A, and B/B, two responses indicate annotator agreement). Often one is only interested in above-random agreement; with kappa scores, the chance agreement gets a score of zero. A side effect of this is that the fewer annotators there are, and the fewer classes there are, the higher the chance of random agreement.

Similarly, the more annotators there are, the higher the chance of any disagreement at all - the same goes for the number of classes. Just as it’s harder to get high accuracy scores with multi-class classification vs. binary classification, so it is harder to get precisely the right category when the number of categories is larger. To pick a gambling analogy, you get lower odds at Roulette when you pick numbers instead of colours. This means that the underlying annotation task becomes harder as the number of classes increases, and so achieving a kappa of 0.44 in binary annotation is not as impressive as achieving kappa of 0.44 with seven or eight classes. But this isn’t reflected in the Landis and Koch table. Because that’s explicitly just for two classes.

Beyond class count and annotator count, one particularly tricky thing to correct for is the latent skew of classes in the data. One can readily count the number of annotators working on an instance, and the number of classes is known a priori for discriminative classification annotation (though defining those classes can be a difficult, qualitative, iterative exercise, especially with language data). What often isn’t known, is the relative frequency of the classes in the data (the class skew). This is often discovered through annotation! While the calculation of kappa scores corrects for class skew, the skew isn’t visible until some data is annotated. And that skew can change across data source, even if the guidelines and class definitions are kept static, so class skew priors from one annotation exercise aren’t guaranteed to apply to other annotation exercises.

Class skew affects the chance correction: the assumption made is that it’s more likely for errors to land on majority classes than rare classes, and so having skewed classes tends to affect the chance agreement, thus altering the kappa observed even when true annotator accuracy is kept static. Class skew is also not really known until the exercise is complete, making it hard to interpret kappa.

So when can kappa scores be compared and interepreted in absolute ranges across studies? Not often. Kappas are comparable only iff

• the number of annotators is the same; and

• the number of classes is the same; and

• the data is the same (fixing class skew).

When all three conditions are met, kappa scores become comparable across scenarios. When any of these changes, kappa scores become incomparable.

And Landis & Koch (1977)’s arbitrary interpretation is only ever applicable to the synthetic data in Landis & Koch (1977), as the illustrative subjective guideline they intended it to be.