Paper || Demo || Artifacts

News

• September 2024: added a bunch of fine-tuning datasets. Full list here
• August 2024: added the Dolma v1.7 dataset to our index
• August 2024: added the Tulu2 dataset to our index
• August 2024: added functionality to the es search that allows flexible queries in between text terms
• August 2024: pypi package release of the elasticsearch wrapper. Install by running pip install wimbd
• April 2024: WIMBD was accepted to ICLR!
• October 2023: first release

This repository contains the code for running What's In My Big Data (WIMBD), which accompanies our recent paper (with the same name).

What is WIMBD?

WIMBD is composed of two components

1. A set of tools for analyzing and revealing the content of large-scale datasets
2. A set of analyses we apply to those datasets, using the aforementioned tools

WIMBD tools consist of two parts:

1. Count
2. Search

The count follows a map-reduce functionality, which divides the task into smaller chunks, applies the operation (e.g., extract the domain from a URL) and then aggregates the counts. We have two implementations for this. One through python functions (e.g., for domain counts) which is easily extendable and scalable, and one through a Rust CLI for faster processing. The Rust implementation covers the summary statistics (presented in Table 2 in the paper) such as the corpus size, number of tokens, etc. In addition, it computes the most & least common