The next generation of AI systems for reasoning will likely rely on a powerful base language model at its core, with a number of inference time techniques that make it more useful. Next token prediction is awesome I’m in the next token-prediction is awesome camp. I think despite whatever the naysayers say, the ability to coherently complete a series of tokens grounded in some context is highly…
Or How to Think About Prompting with an Imprecise but Hopefully Helpful Analogy Originally presented as a tutorial at the CMU-LTI Seminar TLDR: Large Language models can be thought of as compilers, with prompts being programs written in a high-level language. Like programs, prompts can be written in different styles, such as by specifying instructions or a few examples or using alternative formats…
Tl; dr Outline Motivation Investigating the Sequence of Multiplications A first intuitive solution Xavier Initialization Why \(\sqrt(512)\)? | Intuition Why \(\sqrt(512)\)? | Proofs 1. Proof that \(Y \sim \mathcal{N}(0, 512)\) 1.1 Expectation (Mean) of Y 1.2 Variance of Y 2. Proof that Y is \(\sim \mathcal{N}(0, 1)\) when A \(\sim \mathcal{N}(0, 1 / 512)\) 2.1 Expectation (Mean) of Y 2.2 Variance…
TL; DR Exits on the US 101 are not numbered sequentially, and are based on distance. Two subsequent exits that are 5 miles apart will have exit numbers that differ by 5. Various visualizations show that the exits are densely packed in the cities and become increasingly sparse as we move north. Contents TL; DR Background Collecting Exit Numbers Visualizing the Gaps Transformations 1. Present/Absent…
Contents Introduction Dataset RNNs Cross Seeding Looking at the Data Top 5 Most Popular Names Name Length Distributions Input Representation a) Encoding b) Standardization c) Embeddings Model Transferring Name Styles Results Introduction Dataset A few days ago I was enticed into thinking if there is an app for generating names ( no, not a startup idea ). You know, something along the lines of…
It’s Simpler than You Think Writing bad code is simpler than you think. Yada-yadas will make you believe that it takes special skills to craft junk code, that only a “chosen few” among us are capable of doing it. They may trick you into thinking that it takes a lot of apathy, the ability to hand wave, the special gut feeling to make bad assumptions, and a keen, sharp focus on the short term (or…