Comparing Labeling Behaviour of Annotator Groups

Learning objectives

By the end of this tutorial, you will be able to:

• Understand the basics of inter-annotator agreement measures.
• Tell the differences of prescriptive and descriptive annotation data as well as aggregated and non-aggregated annotation.
• Calculate inter-annotator agreement for both data annotation paradigms.
• Interpret the outcomes in context of the different annotation paradigms.

Target audience

This tutorial is targeted at:

• Researchers in the social sciences interested in inter-annotator agreement measures (also known as inter-rater reliability) and their application to cases involving coexisting perspectives on social constructs.
• NLP practitioners that are interested in capturing human label variation.
• Students or professionals beginning their journey in NLP and text analysis.
• Anyone interested in subjective annotation tasks.

Duration

60-120 mins

Use cases

• Assessing the reliability of annotations within homogeneous groups in data annotation of social constructs, such as hate speech, polarization, or misinformation.
• Measuring and comparing label variation for different sets of annotators (e.g., different demographic groups) in data annotation of social constructs, such as hate speech, polarization, or misinformation.
• Supporting hypotheses regarding how different groups perceive social constructs.
• Demonstrating the importance of a well-considered selection of annotator for training useful machine learning models (see Section 5.2 in Romberg et al., 2025).

Setting up the computational environment

To run all the code in this tutorial, you need to install several packages:

#!pip install -r requirements.txt

We import each of these packages as follows:

import pandas as pd
import numpy as np
import krippendorff
from scipy.stats import wasserstein_distance
import matplotlib.pyplot as plt

pd.set_option('display.max_colwidth', None)  # show full cell content

Basic concepts you should know for this tutorial

• What is data annotation? Enriching raw datasets with additional information tags is a common method of data analysis and builds the foundation for training and testing many machine learning algorithms. For example, to study hate speech in social media, posts (=raw data) are manually reviewed for whether they contain hate speech (=additional information tags). These tags are called annotations, labels, or codes (interchangeably).

• Two paradigms of data annotation (“objective” vs. “subjective”) Dataset annotation can be conducted with two different paradigms in mind (Röttger et al., 2022):

  • Prescriptive annotation is the longstanding and more common paradigm of data annotation. The goal is to achieve a consistent annotation across the dataset, e.g., through meticulous decision rules predefined in codebooks. The general credo is that there is always exactly one correct assessment; for example, in hate speech, each statement can be clearly categorized as such or not containing hate speech. The phenomenon is therefore assumed to be capturable in an objective manner.
  • Descriptive annotation has gained attention more recently, as it has become more and more obvious that subjective phenomena may more accurately be reflected by capturing a multitude of annotators’ perspectives. The reason behind is that certain properties of an individual can shape the perception of a phenomenon. For instance, consider a white man and a black woman—it seems reasonable to expect that there may be cases of hate speech that these individuals assess differently. According to the paradigm, there can be multiple valid assessments, which are kept as additional source of information for real-world modeling.

  With our goal of comparing group-specific annotation behaviour, we follow the descriptive annotation paradigm in this tutorial.

• Ground truth formats Related to the two paradigms of annotation, the “end product” of a data annotation pipeline looks different.

  • Aggregated ground truth: In prescriptive annotation, the annotations from multiple annotators are consolidated to create a single ground truth—typically by using methods such as majority voting or averaging the assigned scores. We refer to this as the aggregated ground truth.
  • Non-Aggregated ground truth: In descriptive annotation, the labels provided by multiple annotators may all be considered valid and are therefore not merged. Instead, each data point is associated with multiple labels. We refer to this as the non-aggregated ground truth.

• What this tutorial means by annotator groups A set of individuals who share similar traits (such as demographics, experiences, values, or beliefs) and are thus expected to annotate more similarly to one another than to those with different traits, under the descriptive annotation paradigm.

• Inter-annotator agreement measures To assess how uniformly or diversely annotators work, inter-annotator agreement measures are typically used. These measures provide a score indicating how likely it is that multiple annotators make the same annotation decision on the same data items.

  Note: While inter-annotator agreement measures inform us about the degree of agreement between annotators, they are commonly used as a proxy for reliability in prescriptive annotations: the more the annotators agree, the more consistent the annotation.

  For a general overview of inter-annotator agreement measures and mathematical details, see: Artstein & Poesio (2008). We will use two selected measures in this tutorial (in other use cases than provided in this tutorial, a different choice of measure might be more suitable):

  • Krippendorff’s Alpha is one of the most well-known inter-annotator agreement measure. It can handle sparse annotations and varying amounts of annotators per item (i.e., cases in which not all annotators worked on every data item; as the tutorial progresses and we look at datasets, the reason for this requirement will become clear).

    Conceptually, Krippendorff’s Alpha integrates the observed disagreement between annotators with the amount of disagreement that is expected to happen by chance, thus providing a chance-corrected agreement coefficient in the range from -1 to 1. A value of 1 indicates perfect agreement between annotators, -1 systematic disagreement, and 0 random agreement.

  • Wasserstein distance, also known as Earth Mover’s distance, is a distance measure for comparing probability distributions. Conceptually, it quantifies the minimum amount of “work” needed to transform one distribution into another, where “work” is defined as the amount of mass transported multiplied by the distance it is moved. The higher the distance value, the more dissimal the distributions are; 0 indicates identical distributions. Unlike Krippendorff’s Alpha, the Wasserstein distance does not have a fixed scale for possible interpretation, as its values are data-dependent. For instance, if annotations are on a 1–5 scale, a Wasserstein distance of 1.5 indicates a significant difference between annotators. But on a broader scale, like 0–100, that same value would reflect only a small disagreement.

    Note: The Wasserstein distance is not a classical inter-annotator agreement measure. Unlike Krippendorff’s Alpha, it does not include chance correction, and may therefore overestimate the similarity between two distributions. The development of distribution-based inter-annotator agreement measures is an active area of research.

Which example tasks and datasets are we using?

We exemplify how to measure in-group and cross-group annotation behaviour by means of two different annotation tasks: hate speech detection and argument quality assessment.

The datasets are pre-cleaned and available in the datasets folder of the repository.

# Import the MHS dataset annotations by the female annotators
df_hatespeech_female = pd.read_csv('datasets/MHS_female_prep.csv')
df_hatespeech_female.iloc[20:23,:].dropna(how='all', axis=1)  # iloc is used to select rows and columns by their index numbers; only show columns that contain an entry for the 3 selected rows (to see a bit more than NaN values)

# Import the MHS dataset annotations by the female annotators
df_hatespeech_male = pd.read_csv('datasets/MHS_male_prep.csv')
df_hatespeech_male.iloc[20:23,:].dropna(how='all', axis=1)  # iloc is used to select rows and columns by their index numbers; only show columns that contain an entry for the 3 selected rows (to see a bit more than NaN values)

# Import the Dagstuhl dataset annotated by the expert group
df_argumentquality_experts = pd.read_csv('datasets/dagstuhl_expert_prep.csv')
df_argumentquality_experts.head()

# Import the Dagstuhl dataset annotated by the novice group
df_argumentquality_novices = pd.read_csv('datasets/dagstuhl_novice_prep.csv')
df_argumentquality_novices.head().dropna(how='all', axis=1)  # only show columns that contain an entry for the 5 head rows (to see a bit more than NaN values)

How uniform do the groups of interest label?

def k_alpha_per_dimension(df, dimension):
    """
    The function selects those columns from the dataframe that contain annotations for the 
    dimension of interest. It then computes the agreement between annotators according to 
    Krippendorff's Alpha formula, yielding a single alpha score per dimension.
    
    Args:
        df (pd.dataframe): Dataframe containing rows of data points with columns of annotations
        dimension (str): Name of the dimension as contained in df
    
    Returns:
        float: The computed Krippendorff's Alpha for the specified dimension.
    """
    # Get the columns that correspond to the dimension (e.g., columns starting with "clarity")
    cols = [col for col in df.columns if col.startswith(dimension)]
    # Calculate alpha for the specific dimension using all correspondent columns
    alpha = krippendorff.alpha(df[cols].values.T, level_of_measurement="ordinal")  # transpose for function format
    return alpha

hatespeech_dimensions = ["respect", "insult", "humiliate", "violence"]
argumentquality_dimensions = ["clarity", "credibility", "effectiveness", "emotional_appeal"]

def all_k_alpha(df, dimensions, group):
    alphas = [k_alpha_per_dimension(df, d) for d in dimensions]
    return pd.DataFrame(alphas, index=dimensions, columns=[group])

# Compute Krippendorff's Alpha for each group
female = all_k_alpha(df_hatespeech_female, hatespeech_dimensions, "female")
male = all_k_alpha(df_hatespeech_male, hatespeech_dimensions, "male")

# Compute Krippendorff's Alpha for the full sample of annotators (female + male) as a reference value
all = all_k_alpha(pd.merge(df_hatespeech_female, df_hatespeech_male, on=["comment_id", "text"]), hatespeech_dimensions, "all")

# Join output for easier overview
pd.concat([female, male, all], axis=1)

# Compute Krippendorff's Alpha for each group
experts = all_k_alpha(df_argumentquality_experts, argumentquality_dimensions, "expert")
novices = all_k_alpha(df_argumentquality_novices, argumentquality_dimensions, "novice")

# Compute Krippendorff's Alpha for the full sample of annotators (female + male) as a reference value
all = all_k_alpha(pd.merge(df_argumentquality_experts, df_argumentquality_novices, on=["#id", "argument"]), argumentquality_dimensions, "all")

# Join output for easier overview
pd.concat([experts, novices, all], axis=1)

def mean_scores(df, dimension):
    """
    The function selects those columns from the dataframe that contain annotations for the 
    dimension of interest. It then computes the mean score per data point.
    
    Args:
        df (pd.dataframe): Dataframe containing rows of data points with columns of annotations
        dimension (str): Name of the dimension as contained in df
    
    Returns:
        pd.Series: The mean scores for all data points.
    """
    cols = [col for col in df.columns if col.startswith(dimension)]
    return df[cols].mean(axis=1)

data1 = mean_scores(df_argumentquality_experts, "effectiveness")
data2 = mean_scores(df_argumentquality_novices, "effectiveness")

plt.boxplot([data1, data2], tick_labels=["Experts", "Novices"])
plt.title('Differences in labeling decisions (mean score per item)')
plt.ylabel('Values')
plt.show()

How do label distributions differ between groups?

So far, we have investigated the inter-annotator agreement scores for different reference populations: for all annotators, regardless of demographics, as well as separately for female and male annotators. We have also compared these scores across the different groups. To sum up: we have learnt how we can analyze and interpret agreement within and across groups.

We now go one step further and ask: can we directly compare how similar or dissimilar annotation decisions in two distinct groups are? The goal is to take into account the full range of non-aggregated label decisions, assuming that different groups have unique distributions across labels. Fully following the descriptive approach of data annotation, we therefore directly compare the label distributions between the two annotator groups.

Note: For the following analysis, it is assumed that all label decisions are valid assessments.

def get_annotations(df, dimension, row):
    """
    Extract all label decisions from a row (invidiual data point).

    Args:
        df (pd.Dataframe): Dataframe containing rows of data points with columns of annotations
        dimension (str): Name of the dimensions as contained in the dataframe
        row (int): Row id of the data point

    Returns:
        list: Labels for the data point
    """
    # Get the columns that correspond to the dimension (e.g., columns starting with "clarity")
    cols = [col for col in df.columns if col.startswith(dimension)]
    # Filter those columns that contain annotations (in case of sparse annotation)
    annotations = [df.iloc[row][col] for col in cols if not np.isnan(df.iloc[row][col])]
    return annotations

def calc_Wasserstein_dist(df1, df2, dimensions, id_col, text_col):
    """
    Compute Wasserstein distances between two groups for each data point.
    
    Args:
        df1 (pd.Dataframe): Dataframe with annotations of group 1
        df2 (pd.Dataframe): Dataframe with annotations of group 2
        dimensions (list): Dimensions for which the Wasserstein distances should be computed
        id_col (str): Name of the id column in the passed dataframes
        text_col (str): Name of the text column in the passed dataframes

    Returns:
        pd.Dataframe: Per data point Wasserstein distance scores
    """
    # Init the outcome dataframe with id and text columns
    wasserstein_df = pd.DataFrame({
        "id": df1[id_col], 
        "text": df1[text_col]
        })

    # Iterate over all dimensions to be accounted for
    for dimension in dimensions:
        dimension_results = []

        # Iterate over all data points
        for row in range(len(df1)):
            # Extract the annotations of both groups for the data point
            annotations1 = get_annotations(df1, dimension, row)
            annotations2 = get_annotations(df2, dimension, df2.index[df2[id_col] == df1[id_col][row]][0])

            # Compute Wasserstein distance if both groups annotated
            if not (len(annotations1) == 0 or len(annotations2) == 0):
                dimension_results.append(wasserstein_distance(annotations1, annotations2))
        
        wasserstein_df["Wasserstein distance for "+dimension] = dimension_results
    
    return wasserstein_df

# 1. Extract the annotations from both groups.
ann_female = get_annotations(df_hatespeech_female, "insult", 0)
ann_male = [float(v) for v in get_annotations(df_hatespeech_male, "insult", 0)]

# Note: the list comprehension is just for more accessible formatting but not needed in general
print("female group", [float(v) for v in ann_female])
print("male_group", [float(v) for v in ann_male])

female group [3.0, 3.0, 1.0]
male_group [2.0, 2.0]

float(wasserstein_distance(ann_female, ann_male))  # float() just for formatting, not needed in general

1.0

# Running this script takes some time (~15 min)
result_hatespeech = calc_Wasserstein_dist(
    df_hatespeech_female,
    df_hatespeech_male,
    hatespeech_dimensions,
    "comment_id",
    "text"
)

result_hatespeech.iloc[20:23,:]  # iloc is used to select rows and columns by their index numbers

result_hatespeech.iloc[:, 2:].describe()

result_argumentquality = calc_Wasserstein_dist(
    df_argumentquality_experts,
    df_argumentquality_novices,
    argumentquality_dimensions,
    "#id",
    "argument"
)

result_argumentquality

result_argumentquality.iloc[:, 2:].describe()

Discussion and Takeaways:

In this tutorial, we explored two methods for comparing groups of annotators based on their labeling behavior. We began with the established Krippendorff’s Alpha to measure agreement within selected annotator groups and in the overall population of annotators, analyzing whether differences exist between groups. We then applied the Wasserstein distance as a metric to directly measure the distance between group-specific label distributions, allowing us to compare differences in labeling behavior at a non-aggregated level per annotation unit.

References

Artstein, R, & Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 2008; 34 (4): 555-596. https://doi.org/10.1162/coli.07-034-R2

Romberg, J., Maurer, M., Wachsmuth, H., & Lapesa G. (2025). Towards a perspectivist turn in argument quality assessment. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). https://arxiv.org/abs/2502.14501

Röttger, P., Vidgen, B., Hovy, D., & Pierrehumbert, J. (2022). Two contrasting data annotation paradigms for subjective NLP tasks. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. https://doi.org/10.18653/v1/2022.naacl-main.13

Wachsmuth, H., Naderi, N., Hou, Y., Bilu, Y., Prabhakaran, V., Thijm, T. A., Hirst, G., & Stein, B. (2017a). Computational argumentation quality assessment in natural language. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. https://aclanthology.org/E17-1017/

Datasets

Kennedy, C. J., Bacon, G., Sahn, A., & von Vacano, C. (2020). Measuring Hate Speech Dataset. Hugging Face. https://huggingface.co/datasets/ucberkeley-dlab/measuring-hate-speech

Mirzakhmedova, N., Goshen, Chang, C. H., & Stein, B (2024). Are Large Language Models Reliable Argument Quality Annotators? Dataset. Zenodo. https://zenodo.org/records/13692561

Wachsmuth, H., Naderi, N., Hou, Y., Bilu, Y., Prabhakaran, V., Thijm, T. A., Hirst, G., & Stein, B. (2017b). Dagstuhl-15512-ArgQuality Dataset. Zenodo. https://doi.org/10.5281/zenodo.3973285