More than 70% of organizations are now using three or more AI models in production. And people uses OpenAI takes up 63% of the market. But recently people are adopting Anthropic and Gemini.
I myself shifted from using cursor to Claude Code. But in production OpenAI and Gemini dominates.
Organizations are building a model portfolio — selecting different models based on workload, cost, operational risk, latency, and task-specific requirements.
Model Portfolio is one of the most important things for any agentic workflow. You should be aware of what models are capable of.
We already know one thing: you don’t need Opus 4.8 for everything.
As the AI market matures and subsidies from major providers eventually decline, token costs are likely to increase. This makes it increasingly important to understand cost optimization techniques, token management, and intelligent model selection.
One of the simplest ways to reduce costs is switching between models based on the task. A premium reasoning model may be ideal for complex analysis, while a cheaper and faster model can handle summarization or classification workloads just as effectively.
A multi-model architecture introduces significant engineering overhead.
You now need to manage:
Multiple LLM integrations
Different SDKs and APIs
Retry logic and provider-specific edge cases
Fallback mechanisms when a model or provider fails
All of this slows down the speed of iteration.
As teams move toward a multi-model strategy, they also introduce more moving parts into the system. Over time, this increases operational complexity and creates technical debt that becomes harder to maintain.
According to Datadog’s observations, many organizations are also slow to retire older models, further increasing system complexity and maintenance costs.
LLM Gateways!
Instead of integrating multiple providers directly into your application — and manually handling switching, retries, and fallbacks — teams are increasingly adopting LLM gateways as an abstraction layer.
An LLM gateway typically provides:
Unified API: A single interface to interact with 100+ LLM providers, eliminating the need to write separate integration logic for each model.
Automatic Fallbacks: If a primary provider fails or experiences downtime, requests can automatically be routed to a backup model.
Smart Routing: Requests can be intelligently directed to different models depending on the task — coding, summarization, reasoning, or general Q&A.
Caching: Repeated prompts can be identified and cached, reducing latency and potentially cutting inference costs by 40–60%.
Some strong options in this space include LiteLLM, OpenRouter, and Portkey.
With multiple models you need to implement multiple LLM handlers, retry logic and fallbacks. This slows the speed of iteration.
As teams are moving to multi-modal approach, we are adding multiple moving components to the system. This increases the complexity of maintaining them.
Datadog has seen them organization are slow in retiring old models.
LLM Gateways! Instead of integrating multiple LLM providers in your app and handling switching, fallback in your app, it is great idea to use LLM Gateways. LLM Gateways provides you:
Unified API: single function call to interact with over 100 different LLM providers, eliminating the need to write separate integration code for each.
Automatic Fallbacks: If a primary API provider fails or goes down, the gateway can automatically route requests to a secondary or backup model
Smart Routing: Gateways can intelligently direct incoming requests to specific models based on the nature of the task (e.g., coding, summarization, or general inquiries)
Caching: It identifies and stores the responses to repeated queries, which can significantly reduce latency and decrease costs by 40–60%.
LiteLLM, OpenRouter, Portkey are some of the good options for LLM Gateways.
Datadog has seen adoption of agentic frameworks rise from 8% to 18%, with tools like LangChain, Pydantic AI, LangGraph, and the Vercel AI SDK gaining strong momentum. These frameworks accelerate development by providing boilerplate implementations for common agentic patterns, allowing teams to build AI applications significantly faster.
This speed comes with trade-offs.
While these frameworks reduce development effort, they also introduce operational complexity. Under the hood, many of them add multiple layers of abstraction and execution steps that are often difficult to track. As systems scale, this can lead to agent performance degradation, increased latency, and rising inference costs.
As adoption grows, teams are increasingly investing in agent telemetry and observability to better understand how LLM-powered systems behave in production. They want visibility into how agents make decisions, where failures occur, why unexpected behaviors emerge, and what is driving cost spikes.
By investing in LLM observability and agent telemetry.
Modern AI systems are no longer simple API calls — they involve prompts, tool calls, retrieval pipelines, memory, multi-step reasoning, and fallbacks. Traditional application monitoring tools are often insufficient to debug these workflows.
Teams are now tracking signals such as:
Token usage and cost per request
Latency across agent steps
Prompt and response traces
Tool call success/failure rates
Hallucination or quality degradation patterns
Model routing and fallback behavior
User feedback and agent success metrics
Several tools are emerging to solve this problem:
LangSmith — Deep tracing and debugging for LangChain/LangGraph workflows. Useful for visualizing execution paths and diagnosing failures.
Langfuse — Open-source LLM observability platform for tracing, prompt management, evaluations, and cost tracking.
Helicone — Lightweight observability layer for monitoring request latency, costs, caching, and model performance.
Datadog LLM Observability — Useful for teams already using Datadog, enabling production monitoring for LLM applications alongside existing infrastructure telemetry.
OpenTelemetry (OTEL) — Increasingly becoming the standard for instrumenting AI applications, allowing teams to trace requests across services, agents, and model calls.
In short, as agentic systems become more powerful, observability is becoming just as important as orchestration. Teams that invest early in telemetry will debug faster, control costs better, and iterate more confidently in production.
In this section, I want to highlight two interesting signals from how organizations are building AI systems today:
69% of input tokens are consumed by system prompts
59% of agentic applications still make a single service call (monolithic prompt architecture)
One surprising insight is that most input tokens are not user input — they are system instructions.
Nearly 69% of input tokens are spent on system prompts, meaning organizations are repeatedly sending the same instructions, context, and formatting rules with every request.
This introduces a major optimization opportunity.
If your system instructions remain mostly unchanged across requests, prompt caching can significantly improve both speed and cost efficiency. Instead of repeatedly processing the same input, supported models can reuse previously computed prompt states, reducing latency and token costs.
However, despite the benefits, adoption remains surprisingly low.
Even among models that support prompt caching, only 28% of LLM call spans show any cached-read input tokens. This suggests that many teams are either unaware of the feature, have not implemented it effectively, or are missing opportunities to optimize production workloads.
As inference costs continue to matter, prompt engineering is no longer just about quality — it is also about efficiency.
We have already learned from traditional software engineering that monoliths do not scale well. Yet, in the AI world, most production systems are still built around monolithic prompts.
In fact, 59% of agentic applications still operate through a single service call, where one large prompt attempts to handle planning, reasoning, tool use, memory, and response generation all at once.
While this approach helps teams ship quickly, it often becomes difficult to scale, debug, and optimize as complexity grows.
As organizations mature, many are moving toward dedicated agent services and multi-agent architectures, where responsibilities are split across specialized agents or workflows. For example, one agent may focus on retrieval, another on reasoning, and another on execution or validation.
This shift introduces new platform requirements:
Better orchestration between agents
Observability across workflows
Shared memory and context management
Smarter routing and fallback mechanisms
Stronger cost and latency controls
The transition away from monolithic prompts may take time, but the direction is becoming increasingly clear: AI systems are slowly moving toward modular architectures — much like traditional software did years ago.
One of the biggest shifts in modern LLMs is the massive increase in context window size.
Just a year ago, context limits were a major bottleneck. Teams had to aggressively summarize conversations, chunk documents, and carefully optimize every token sent to the model.
That constraint is rapidly disappearing.
Modern models can now handle massive context windows, allowing us to include conversation history, documents, retrieved knowledge, memory, guardrails, and detailed instructions in a single request.
And organizations are taking full advantage of it.
According to Datadog’s findings, the average number of tokens used in customer requests more than doubled for median customers and quadrupled for 90th-percentile power users year over year.
This tells us something important:
Context window size is no longer the bottleneck. Context quality is.
The challenge is no longer “How do I fit everything into the context window?”
The challenge is now “What is the minimum high-quality context the model actually needs to succeed?”
The best AI systems are increasingly focused on:
Selective retrieval instead of dumping everything
Dynamic context injection based on user intent
Smarter memory management
Prompt compression and summarization
Removing noisy or redundant context
In short, context quality — not context volume — is becoming the new limiting factor for LLM agents.
If you want to go deeper into practical techniques, trade-offs, and implementation patterns, read my article on Context Engineering.
This was one of the most interesting insights from the report.
In February, 5% of all LLM call spans reported an error, and surprisingly, 60% of those failures were caused by exceeded rate limits.
This highlights an uncomfortable reality:
The reliability of your AI application is increasingly tied to the capacity limits of model providers.
As demand grows, provider-side bottlenecks are starting to impact production reliability. When rate limits are hit, agents can fail mid-execution, workflows can stall, and downstream systems can become unstable.
This means building AI systems is no longer just about prompt engineering — it is increasingly becoming a distributed systems problem.
1. Use LLM Gateways
LLM gateways help reduce provider dependency by enabling:
Automatic failover to backup providers or models
Load balancing across providers
Retry handling and intelligent routing
Rate limit management at the platform layer
Instead of tightly coupling your application to one provider, gateways create a reliability layer between your app and model vendors.
2. Avoid spikes in loop length and tool fanout
Many reliability issues come from uncontrolled agent behavior.
For example:
Long agent loops can unexpectedly multiply API calls
Aggressive tool fanout can trigger sudden traffic spikes
Recursive planning patterns may unintentionally create runaway execution paths
As a result, teams are increasingly designing agents with bounded execution patterns to prevent unpredictable scaling.
3. Avoid reliability failures by bringing back traditional engineering patterns
Ironically, many of the solutions look very familiar.
To make AI systems production-ready, teams are reintroducing battle-tested distributed systems patterns:
Queues to smooth traffic spikes
Exponential backoff and retries for transient failures
Fallback models/providers when capacity is constrained
Circuit breakers to prevent cascading failures
Rate limiting and concurrency controls
4. Avoid runaway loops with agent budgets
By implementing budgets that force agent loops to terminate after a maximum number of calls or token usage, teams can prevent runaway execution from exhausting provider capacity and affecting downstream services.
Common limits include:
Maximum number of model calls
Token budgets
Execution timeouts
Tool invocation caps
The takeaway is clear: as AI systems become more autonomous, reliability engineering is becoming just as important as model intelligence.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.