This repository implements an evaluation script for measuring the mean BISON accuracy on the val2014 split of the COCO-caption dataset. If you use BISON to analyze your image-captioning system, please cite:

Hexiang Hu, Ishan Misra, and Laurens van der Maaten. Binary Image Selection (BISON): Interpretable Evaluation of Visual Grounding. _arXiv:1901.06595, 2019.

Requirements

Python 3+
Numpy 1.10.0+

Usage

Please put your prediction file (format as shown in later section) in the folder predictions/, and specify the current annotation filepath, as well as the prediction filepath. The usage is listed as following:

python bison_eval.py [-h] [--anno_path ANNO_PATH] [--pred_path PRED_PATH]
optional arguments:
  -h, --help            show this help message and exit
  --anno_path ANNO_PATH
                        Path to the annotation file
  --pred_path PRED_PATH
                        Path to the prediction file

File format for predictions

The model predictions used as input into the BISON evaluation script should be in the following file format:

[
	{
		"bison_id": 0,
		"predicted_image_id": 50965,
	},
	...
]

References

License

BISON is CC-BY-NC 4.0 licensed, as found in the LICENSE file.