The macro-F1 went up. Did the model get better?

We taught our parser the hardest street cases it gets wrong, the aggregate score climbed, and we almost shipped it. Then a held-out probe caught it forgetting what a town is.