Superintelligence is imminent and model welfare matters more than you think

I started this essay to investigate where reasoning came from, and I ended it caring about model welfare. Let me take you through this ride. Quick run through the training pipeline: We start with pretraining on large amounts of data, which anneals towards the end with higher quality data (sometimes called mid-training), as the later stages have a more direct effect on the final weights. Then we…