Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability

This is the official repo of our CVPR 2026 paper Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability. Check out our 5-minute blog post on the project website.

Introduction

In this work, we conduct a large-scale crowd-sourced evaluation of automated interpretability methods for describing neurons in vision models, and develop methods to improve evaluation efficiency by 40×.

• While previous work has focused on only measuring whether the explanation matches the highest neuron activations, we instead measure the correlation between neuron activations and the explanation concept, giving us a more complete picture.
• To make our evaluation economically feasible, we introduce Model-Guided Importance Sampling (MG-IS) to select most important inputs to show raters, leading to ∼15× reduction in labeling cost over uniform sampling.
• We develop a Bayes Rater Aggregation (BRAgg) to aggregate predictions of different raters to deal with noisy labels, further reducing the number of ratings required to reach a certain accuracy by ∼3×.

Results

Overall we find that Linear Explanations produces the most accurate neuron descriptions for models/layers we studied.

Setup (uv)

pip install uv
uv run jupyter lab        # or: uv run python your_script.py

On some platforms (especially Windows) PyTorch needs a platform/CUDA-specific index; it is already set up for Windows under [tool.uv.sources] in pyproject.toml — adjust the URL there for your CUDA version if needed.

Data

To run most notebooks you will need the ImageNet validation dataset, which we do not share here. To run, you need to specify the path to your local copy in the DATASET_ROOTS dictionary in data_utils.py.

For the DinoV3 experiments in Appendix Fig12 and 13 you need to additionally download the dino model and zero-shot fine-tuned head following the instructions from https://github.com/facebookresearch/dinov3 and add DINO_PATH in data_utils.py.

Core modules

Reproducing the paper

Main paper figures 2-5 can be reproduced by notebooks fig2_*-fig5_*, and Tables 1 and 2 are reproduced in table_1_2_automated_eval.ipynb. Appendix results are reproducible with files in the appendix/ directory.

Some notebooks run from the CSVs shipped in this repo with no model, dataset or GPU, but the most recompute activations and need:

• Datasets / model weights — ImageNet-val. Point to them with the DATA_ROOT env var (default data/) or edit DATASET_ROOTS in data_utils.py. Optionally add Dino weights (not needed for most results).
• Cached activations — written to saved_activations (the activations_dir argument) on first run.

Running your own study

The repo runs a crowdsourced evaluation end to end. To evaluate your own model and explanations, the workflow is built around mturk/generate_input_csv_large_scale.ipynb (build the study) and analyze_mturk.py (score the results):

1. Prepare the explanations to evaluate. Put them in data/explanations/<method>/<model_layer>.csv, one row per neuron with at least unit and description columns (see the shipped files for the format).

2. Generate the per-task input CSV. In mturk/generate_input_csv_large_scale.ipynb, set your target_name / target_layer and the methods you want to compare, and run it. It computes the target-model activations and the SigLIP similarity of every input to each explanation, draws the most informative inputs per neuron with model-guided importance sampling, and writes a study CSV (15 inputs per task, carrying input_id and the importance weights needed for scoring). Use generate_input_csv_test.ipynb for a small pilot.

3. Host the images. Upload your probing images where raters' browsers can reach them (e.g. an S3 bucket) and set IMAGE_BASE_URL in the notebook to that root, so each image_url{j} resolves to IMAGE_BASE_URL/<synset>/<file>.

4. Launch the study. On your crowdsourcing platform (we used Amazon MTurk), create a task from mturk/interface.html and upload the generated input CSV as the task data. We collect 3 independent ratings per input.

5. Score the results. Download the platform's results CSV (one row per completed task, with the Answer.image{j}.on selections) into mturk/large_scale/outputs/, then aggregate and score it:

   import analyze_mturk
   correlations, descriptions = analyze_mturk.analyze_large_scale(
       "mturk/large_scale/outputs/your_results.csv", aggregation="bragg_siglip")
   # correlations[method][neuron] -> correlation score for that explanation

   aggregation chooses how noisy ratings are combined: "majority" / "avg" baselines, or "bragg_uniform" / "bragg_siglip" for Bayesian rating aggregation. fig5_large_scale_results.ipynb shows the full analysis and plotting, and fig4b_mturk_test_analysis.ipynb does the same for a pilot study

Crowdsourced study data

mturk/ holds the two studies as parallel subtrees plus the generation notebooks and the rater-facing template:

mturk/
  test/{inputs,outputs,results_analyzed,processed}   small validation study (Fig. 4b)
  large_scale/{inputs,outputs,processed}             main study (Fig. 5, tables)
  generate_input_csv_test.ipynb                      build study inputs via MG-IS
  generate_input_csv_large_scale.ipynb
  interface.html                                     rater-facing study template

Cite this work

T. Oikarinen, G. Yan, A. Kulkarni and T.-W. Weng, Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability, CVPR 2026.

@inproceedings{oikarinen2026beyond,
    title={Beyond Top Activations: Efficient and Reliable Crowdsourced
           Evaluation of Automated Interpretability},
    author={Oikarinen, Tuomas and Yan, Ge and Kulkarni, Akshay and Weng, Tsui-Wei},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer
               Vision and Pattern Recognition (CVPR)},
    year={2026}
    }