How Modern LLMs Rebuilt Attention
GQA, MLA, SWA, MoE, Gated DeltaNet, DSA, KV sharing — and which model actually ships each one.
pile of thoughts
Huijo Kim's personal site — machine learning, computer vision, MLOps, and reflections.
Latest posts
AI is everywhere, even in interview
It is expected to use AI in the technical interview, but it is even used when not allowed.
Attention has four matrices, not three
Q, K, V, and the output projection the name leaves out. What each weight does to the same input, and why the differences between them decide what an inference server has to cache.
What’s inside that token? JWT explained with a boarding pass
I built an OAuth 2.0 server with Keycloak for our MCP setup. Here is what a JWT actually carries — issuer, audience, roles — and how we pack our own facts into it: environment, customer, and fine-grained permissions.
My tips for improving an agent-driven work environment
Three limits decide how much you get out of coding agents: how many diffs you can verify, how much context you feed them, and how much attention each loop costs you.
Designing a RAG retrieval system: from ingestion to grounded generation
How to trace evidence through dense search, BM25, fusion, filtering, reranking, context construction, and evaluation.
What Claude Code taught me about compaction
Claude Code clears old tool output, summarizes the decisions that remain, and reloads working state. That suggests a practical compact strategy for long coding sessions.
Chunking in RAG: why 512 tokens is not a strategy
Chunking involves four separate decisions: which units exist, how search represents them, what context the LLM receives, and how the whole policy is tested.
KL Divergence, Practically — What I Got Wrong
Why treating KL as 'just cross-entropy' breaks down inside variational inference, and what its asymmetry actually does to optimization (mode-covering vs mode-seeking).
VAE, Practically — What I Got Wrong
A corrected mental model for VAEs, written as 'mis-belief → correction' notes after re-reading Rezende & Mohamed (2015) and reframing the encoder/decoder as variational inference rather than autoencoding.