Everyone Knows It Scores Half. Nobody Checks Which Half.

The public leaderboard scores half the test set and says so on the page. I measured which half. It was not a random sample, it was the first 1,974,995 rows in time order, and that one fact invalidated six weeks of tuning.