Mind ordering cycles in systemd: how systemd breaks them can brick server startup

I've been building a service for a month, and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigorous" manual tests so I launched 100 EC2 instances. Surprise! Around 28 instances failed to launch. What's going on? All failed instances were stuck in the "initializing" state, and the only way to connect to them was through EC2 Serial Console. There, I noticed…