Essays, books, wiki on technologies, career, markets, and more.
I made HuggingFace tokenizers 3.3x faster by parallelizing single-input tokenization with overlapping chunks, zero-copy offset operations, SIMD-accelerated boundary detection, and cache-hierarchy-aware chunking. The result is bit-identical to serial encoding. Fast tokenization is critical to achieve low TTFT given long context. The Long Context Revolution Has a Bottleneck LLM models now routinely…
Many inference speedup techniques mirror the classic systems regime—such as caching, paging, tiling, pipelining, and speculative execution (e.g. branch prediction and cache prefetch). Speculative decoding, generalizing speculative execution to stochastic settings, produces several tokens in each forward pass, without changing the output distribution (model quality) or model parameters. This post…
In the multi-head attention mechanism, why after reshaping the projection matrices for Q/K/V from 3 dimensions to 4, we need to transpose the tokens dimension with the heads dimension? Using the canonical example code for Attention Heads below as an example, why do we need Q = self.W_q(x).reshape(B, T, self.num_heads, self.head_dim).transpose(1, 2) ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19…
Careful planning and execution work for general management but not for startups. Perfect execution is futile if you end up building something nobody wants (waste). The real progress for startups is not how many JIRA tickets we closed but how fast we gain validated learnings—what creates value for customers and their willingness to pay—while minimizing waste. Systematically break down a business…
Before Fundraise : Allow minimum three to six months to raise money. Have a clean cut from last job to avoid IP disputes. Prepare data site (Certificate of Incorporation, Bylaws, board minutes, cap table, customer list, product roadmap, org chart, employment agreements, budgets, financial statements). Some VC deals fail to close because of one missing IP assignment agreement. If you want money,…
In a large, busy cluster, how do you scale Istio to address Istio-proxy Container being OOM-Killed and Istiod crashes if too many connected istio-proxies? Istio-proxy Container OOM-Killed Problem If istio-proxy dies, Pod disconnects from the world, because istio routes the Pod’s ingress and egress through the istio-proxy container. Thus, the main application container cannot communicate with other…
AWS EKS on VPC networks need AWS Security Group Rules (SG) to receipt ingress traffic. But what if you reach the max rules count in your SG? Background LoadBalancer-type Service and Security Group Rules Kubernetes users can expose a Service in two ways: Register with the Istio ingress gateways—the golden path for most tenants Create a dedicated LoadBalancer-type Service object, which tells the…
Your Kubernetes cluster probably has a shared ingress for north-south traffic, coming from a cloud load balancer and lands on your favorite proxies like Envoy, or Istio gateways, or Nginx. If you use a LoadBalancer-type Service to create a Layer-4 Load Balancer fronting your Kubernetes ingress retain source IP address by setting externalTrafficPolicy: Local Then horizontal autoscaling (scale-in)…
How to do product-led growth and hands-on outbound sales at the same time? Every PLG company eventually has to embrace enterprise. – Annie Pearl, Chef Product Officer at Calendly The upper limit of PLG seems to be $100M to $200M ARR (e.g. DataDog around IPO). Beyond triple-digit million ARR, you quickly saturate the market of users who buy things by pulling out their credit cards. The growth of…
Make sure to check out the previous post: Advices I wish I got at the start of my career . Ask for help Son, your ego is writing checks your body can’t cash. – Captain Tom “Stinger” Jordan, Top Gun The number one reason why senior people fail is that they do not ask for help. We are all shareholders. You do whatever you can to unblock yourself. It is about time to market and showing results.…