Telegram Alerts for Training

I often use Weights & Biases for training. Others might use something like Tensorboard or mlflow . While these are great for tracking experiments and sweeps or monitoring logs, sometimes you want a simple alert system that'll wake you up when a run starts or fails (but with a little more control so you can quickly get a view on experiment status). While SLURM allows you to setup email alerts,…