Emulating ALiBi with RoPE
LLMs usually use some sort of positional encoding for helping the model understand where different tokens — or more precisely, KV cache entries — are. Two of the main strategies for doing this are RoP...
Alex's Blog
Thoughts and writings from Alex Litzenberger
Latest posts
Unconditional Love
Unconditional love is a very talked-about but much misunderstood concept. It sticks in people's minds as something highly desirable and ideal but limits and refutations abound. Would you still love me...
Building a Minimal Transformer for 10-digit Addition
I saw [a post](https://x.com/DimitrisPapail/status/2024555561199480918?s=20) on Twitter/X about training a model with fewer than 1,000 parameters to do 10-digit addition, and how Claude and Codex were...
Audiobooks are Just as Good
Audiobooks vs. physical books, many go as far as to say vs. real reading, I do not, I'll try to explain why.
The Morality of Being Stubborn
Being stubborn and how stubborn to be is an extremely important ethical question. It relates to being able to credibly pre-commit to following certain policies and plays a part in determining the beha...
Deontology as Pareto Optimization
Morality is divided broadly between deontologists and utilitarians. Some characterization of the two and their differences are along the following lines. Deontology broadly considers some things just ...
Why the Book Determined Totally Misses the Point on Free Will
I read the book Determined by Robert Sapolsky and disagreed with the core thesis on numerous accounts. I think it misses the point of what is meaningful about free will and the only things it can sens...
Autobiography the Autobiography
Since I was a child I have had a great fear perhaps my greatest fear of living a tale that has already been told.
Contrarianism as a Proxy for Agency
Peter Thiel praises contrarianism why? There may be plenty of reasons to bet on contrarians besides, e.g. unpopular hypotheses may be under priced in the market, or under-explored relative to their in...
Chip and Bowl Curvature in Guac Scrapping
We are all faced with a problem, we order some chips and guac or some chips and salsa and once we get towards the end of the dip we must resort to a sort of scrapping to most efficiently get at what r...
Adaptive Speed Audio
I like to listen to audiobooks and podcasts at several times speed. I find I am able to absorb the information well at a faster speed and I am able to enjoy more audiobooks and podcasts than I would o...
Privacy at the Poker Table and in the Bathroom Stall -- Caring About Privacy When You Have Nothing to Hide
People talk often about privacy. They value it, often care deeply about it but may not have an explicit grasp of what it actually is and of why it is important. Importantly, privacy describes a state ...
Thoughts on Lava Lamp Shapes
So you want to make a lava lamp what shape?
Feeling Anti-Rundown -- If you really hate running you should probably run
If you really hate running it is probably because your cardio is bad or because you are going way to hard or both
Indoor Humidity
Why is indoor humidity so low in the winter?
The Problem with Counterfactuals is they can be Counterfactual
The problem with counterfactuals is they can be counterfactual
Making a Steel Story Bookcase
I made a bookcase.
Agency Diversification and Concentration
You want to diversify to maximize your expected return.
Logits as a Measure of Effort
I was struck reading some materials on cognitive biases [Allias paradox](https://www.lesswrong.com/posts/zJZvoiwydJ5zvzTHK/the-allais-paradox) specifically not by the existence of the bias but by the ...
Subtleties of Edit Distance
Normally ASR is measure by edit distance measured in words or charecters (also refered to as levenstien distance). It measures how many insertions, deletions, substitutions.
A Chess Engine Deserves to be Punished for a Bad Move
It is bad for agents to act against their values. And hell of a lot more follows from just that than is commonly acknowledged.
Retrocausality and Competence
Causality runs forward in time right? Not when you are talking about some one or something competent.