Training Small Digit Adder Transformer on my Laptop

I explore how very small ML models might in some cases be better trained on CPU than GPU. I use the recent AdderBoard as a source for models and inspiration. In one of these I was able to achieve an approximately 8-9x per-iteration speedup on my laptop (CPU, custom C++) compared with the PyTorch A100 runs below: 17.9/2.2 for the compiled CUDA-Graphs run and 20.1/2.2 for the baseline. These are…