Recent content in Home on Aelias
ArXiV and excellent video explainer from Jia-Bin Huang. Summary: This paper extends on the [[Hyper-Connections|Hyper Connections]] paper. Hyper connections were introduced as a way to essentially apply the idea of a mixture of experts (MoE) to residual connections. Instead of having a single residual connection, let’s have multiple different residual connections each with a linear mapping to…
Original Paper Summary: This paper views attention as: a particular instance of tensor lifting : a hidden vector is mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. As an alternative, they propose “an attention-free sequence model built around Grassmann flows .” Instead of lifting from…
Howdy 👋🤠 I’m Elias, I’m a software engineer currently living in SF. I spend a lot of my time thinking about machine learning and programming languages, keeping up with current state of the art research and techniques. I am also currently open to work, so if you think I might be a good fit for your team please reach out 1 . Most recently I was a Founding Engineer at Nectar Climate . I…
Introduction I recently read this 1 tweet, which referenced the LLaDa paper ( Large Language Diffusion Models ). And with Inception’s recent announcement of their Mercury models , I’ve been thinking a lot about Diffusion architectures for large language models, and I’m really excited about them. I harbor very similar sentiments that diffusion feels far more analogous to how…
One big problem is holding back 3D Printed Construction: Materials
Government provides small insurance policies constantly. Once such example is walk signs. The key however is that you don’t pay with money, you pay with time ! Every time you want to cross an intersection and you don’t see any cars coming, but the walk sign still has that orange hand telling you to stop, you have two options: Ignore the sign and cross anyways Wait until the sign says you can cross…
This is a multi part post about the programming language I’ve been working on in my free time called Errand (lang). Part 1 - Why another language? : Why I thought there was a need for a new language and what it aims to do Part 2: Design: How this language will try to solve these problems Part 3: Progress & Roadmap: What’s been written so far, and what’s next
Intro Whenever I see new programming languages, the #1 question I see people ask, and one I ask myself, is why? What problems does this language solve that aren’t solved by an existing language? Why is this language the solution to those problems? Why can’t this be added to an existing language or as a library? The most common answer to this is simply that they wanted to learn about what it takes…
Posts to write: Large / multi-part post on Errand lang. What the goals are, why those goals (problems w other languages or possible improvements) How I’m going about trying to reach those goals Current progress & roadmap Large post on Multivectors Why they’re actually extremely simple How most of math and science makes many concepts way more complicated than they ought to be Why we…
ArXiV Summary Amos Tversky’s seminal paper Features of Similarity , introduced a similarity metric for how humans perceive similarity. Tversky’s similarity metric is non-symetric as human preferences empirically are. By using this non-symmetric metric in neural networks, its able to more-easily learn non-linear functions such as XOR . The primary similarity function they introduce is:…