Nothing to be said
title
152334H Personal Blog
title
These are quick self-notes I wrote this week, while scrolling various AI Alignment related posts online. There are no insights here, just mundane reading & paraphrasing to ensure I ‘get it’.
Post Orgasmic Illness Syndrome ( POIS ) is, allegedly, a Rare Sexual Disorder whose sufferers experience some combination of flu/fever/fatigue/irritability/aphasia/… rapidly after any orgasm.
Instead of a garbage monthly post, let’s look at how different models respond to my garbage.
Most of my life has been dedicated towards intellectual excellence, but continuing to do so would be a mistake.
For various reasons, I had to develop a kernel n-day exploit last week.
It has been, overall, a pretty shit year.
I don’t remember what it’s like to think with more than 60 seconds of context.
Recently, GDM released a great paper titled, Scaling Exponents Across Parameterizations and Optimizers , in which they conduct over 10,000 LLM training runs to obtain optimal hyperparameters under different regimes. After reading it (it was great), I wanted to test my understanding of the paper by tallying up all experiments conducted within, calculating the total compute cost it would take to…
Paper summary: 1.3B to 33B LLMs on 1/2T code tokens (87 langs) w/ FiM and 16K seqlen. Strong effort in constructing pretraining data from Github from scratch, with repository-level samples. Evals beat OSS code models solidly + GPT-3.5 a bit; Coder-7B > CodeLlama-33B often. They don’t spend much effort on Instruction tuning. They commit a continued pretrain of DeepSeek LLM -> Coder: I believe…