Why PostgreSQL major version upgrades are hard

Upgrades between PostgreSQL major versions are famously annoying. You can’t just install the server binaries and restart, because the format of the data directory is incompatible. Why is that? Why can’t we just keep the data format compatible?