Hi, my name is Kosti (you can also find me at kgourgou.me or Twitter). This blog consists of my personal notes & research. My interests currently are in: ...
https://arxiv.org/html/2507.16003v1 I liked this paper, I think it gives a nice way to think about how self-attention (and equivalent constructions) operate. Unless otherwise stated, all ideas below are from the authors and I'm just simplifying a bit. The authors come up with an abstraction called a "contextual layer", i.e., a function A that can map either a vector x to another vector or a…
In this post, we will first summarize what we discussed in part 1 and part 2 , then add a few extra thoughts. Summary of the two posts Part 1: we picked a function f : ℝ → ℝ and built a simple vector field such that for any trajectory ( x , y ) : ℝ > 0 → ℝ 2 , the trajectory is attracted to the graph of f , in the…
In the previous post we talked about creating attracting manifolds and how to get the corresponding vector field given a manifold. In this one, we talk about optimization. Suppose we have g : ℝ → ℝ for which we want to find a minimum and g is sufficiently smooth. For example, we can consider def g(x): return 0.25 * (x * x - 1.0) ** 2 + 0.18 * x That's a simple function with…
TL;DR : This is a math post (and part 1 of a theme, here's part 2 ). I was wondering how easy it is to build a dynamical system with a user-defined attracting curve, e.g., a sine wave. You pick a point and if it starts on the curve, then it remains there, otherwise it smoothly gets attracted to the curve as t increases. I don't recall where I first saw this, but I think it's a standard…
I discovered this nice set of notes by Paige Bright, Alan Edelman, and Steven G. Johnson (from an MIT course). Great if you have forgotten what differential calculus looks like.
I was listening to StarTalk and Neil deGrasse Tyson told the story of the discovery of Neptune, which first happened mathematically and then observationally! Essentially, the effect of Neptune on the trajectories of other planets caused discrepancies that shouldn't have been there. This clued-in two mathematicians of the 19th century, John C. Adams and Urbain Le Verrier , who looked into what…
TL;DR: My notes on the “Greedy Coordinate Gradient”, first read from Zou, et al, 2023 1 , but slightly abstracted. Suppose we have the set Σ = { 1 , … , N } , a function from sequences of n elements to a vector space: f : Σ n → ℝ n × d , and another one g : ℝ n × d → ℝ ≥…
TL;DR This paper studies a subset of possible transformer models called “compact transformers” (CT). Those are transformers that have compact input embeddings and compact positional encodings in every layer. The authors prove that such transformers cannot learn to predict every possible sequence with confidence – instead the sequence space is separated into equivalence classes and a CT can only…
First of all, happy new year to all! I read this short paper by Daskalakis, et al., 1 on translating stories to game-theory representations, e.g., game trees, payoff functions, etc., and then using game theory solvers to learn more about the strategic interactions of the story, e.g., by finding equilibria. The paper applies this to Romeo & Juliet as an example, generating different game paths…
TL;DR: This post collects useful facts about homogeneous functions—functions satisfying f ( a x ) = a r f ( x ) . Key insight: they decompose into spherical behavior plus radial scaling. I also connect to two references in ML. A function f is (positively) r-homogeneous if f ( a x ) = a r f ( x ) for all a ≥ 0…