Avoid panic on expected errors: lessons from operating journald-to-cwl

We've been using journald-to-cwl to ship journal logs from EC2 instances to CloudWatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panicking on expected errors in Go is generally a bad idea. Where Do False Positive Alarms Come From? We run many Go programs…