Apply sysctl before systemd: a lesson from reserving ephemeral ports too late

We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. Sometimes it self-healed, sometimes it kept restarting and we lost logs. This post…