Life, programming, and other such frivolity
Modeling Voice Pipelines as Scalable Agents: A Kubernetes Approach In the world of Voice AI agents, one of the key challenges is creating agent architectures such that it can scale to support concurrent calls while maintaining low latency. Voice interactions need very low latencies because it becomes a bad experience for the users if there are lags. In this post, I will share how I approached ...
Balancing Act: Using Structured Output with Streaming in LLMs When building applications with LLMs, one of a very important trade off comes is - how to use structured responses while using streaming responses. This choice significantly impacts user experience, system architecture, and overall application performance. The Core Challenge Streaming and structured outputs represent two powerful ...
Large Language Models (LLMs) have become essential tools for various data processing tasks, including data labeling, classification, and enrichment. However, when working with substantial datasets, context windows size starts getting in the way/ Also, as the dataset itself grows, the hallucinations increase too. Effective way I have found to mitigate this problem is using batched queries with L...
Practical insights from integrating Large Language Models into production systems, including handling non-JSON outputs, evaluation loops, model switching, and more.
Taming Latency: Strategies to Handle Latencies in Scalable Systems The most difficult bugs I have solved are always about latencies. In production systems, based on different factors like traffic patterns, cache invalidations, resource allocations can cause unpredictable latency distributions. With strict SLA agreements around latencies for critical systems, it becomes very important to be a...