One Client, Every LLM: Provider Fallback Without Five SDKs

If your product does anything real with an LLM — drafting replies, scoring leads, summarizing calls, extracting JSON — then routing every one of those calls through a single provider is a quiet bet you didn’t mean to make. You’re betting that one company’s uptime, rate limits, and pricing will all stay convenient forever. They won’t: providers throw 503s, you hit a rate limit at the worst moment, and costs balloon because a trivial parsing task is somehow running on the flagship model.

The fix used to mean integrating several vendor SDKs and abstracting over their differences. It doesn’t anymore, because of one quietly enormous development: almost every LLM provider now speaks OpenAI’s dialect. Their chat endpoints accept the same request shape and return the same response shape as OpenAI’s. That means you can talk to all of them through one client, switch providers by changing a URL, and fail over between them in a plain loop. Here’s how that pattern works — and, just as important, where “compatible” quietly stops meaning “identical.”

One client, many base URLs

The OpenAI Chat Completions API has become the de-facto standard surface for talking to a language model. Groq, Mistral, DeepSeek, xAI, Together, Fireworks, and aggregators like OpenRouter all expose an OpenAI-compatible endpoint — and so do local runtimes like Ollama, vLLM, and llama.cpp. The practical upshot: you don’t need five SDKs. Take the official openai client and point it at a different baseURL:

const groq     = new OpenAI({ apiKey: GROQ_KEY,     baseURL: "https://api.groq.com/openai/v1" });
const mistral  = new OpenAI({ apiKey: MISTRAL_KEY,  baseURL: "https://api.mistral.ai/v1" });
const deepseek = new OpenAI({ apiKey: DEEPSEEK_KEY, baseURL: "https://api.deepseek.com" });
const local    = new OpenAI({ apiKey: "ollama",     baseURL: "http://localhost:11434/v1" });
const openai   = new OpenAI({ apiKey: OPENAI_KEY,   baseURL: "https://api.openai.com/v1" });

After that, the exact same client.chat.completions.create(...) call works against every one of them. (This isn’t a TypeScript trick — the Python and Go OpenAI SDKs take the same base_url/BaseURL option, so the pattern ports verbatim.) Create a client only when its key is present, otherwise leave it null; the bonus is that an unconfigured provider simply drops out instead of crashing anything, so the identical code runs whether you’ve wired up three providers or seven.

Route by task, not by model

The calling code shouldn’t know or care which model answers. What it knows is the kind of job: write something high-quality, parse something cheaply, return strict JSON, transcribe audio. So put a layer in between that maps each task class to an ordered chain of provider-plus-model options:

const chains = {
  QUALITY:    [groqReasoning, mistralMedium, openai, deepseek], // best prose / reasoning first
  FAST:       [groqSmall, mistralSmall, openai],                // cheap intent parsing, JSON extraction
  STRICT_JSON:[groqStructured, openai],                         // models good at structured output
  TRANSCRIBE: [groqWhisper, openaiWhisper],                     // audio → text
};
// drop any (provider+model) whose client is null because its key is missing

Notice that one provider can sit behind several entries, because a single provider often hosts several models with different strengths — a strong reasoning model, a long-context multilingual one, one tuned for JSON, and a tiny fast-and-cheap one:

The win is that callers ask for QUALITY or FAST; you change which model serves that class in one place, and you can slot a new provider into a chain without touching a single feature.

Fallback is just a loop

With chains in place, resilience is unglamorous — which is exactly what you want. Walk the chain: try a provider, and on a transient failure give it one retry; on success, log the cost and return; otherwise move to the next. If the whole chain is exhausted, throw the last error up.

for (let i = 0; i < chain.length; i++) {
  try {
    const res = await callChat(chain[i], opts);
    if (i > 0) logger.warn(`AI fallback → ${chain[i].name}`);
    await logCost(chain[i], res);
    return res;
  } catch (err) {
    lastError = err;
    if (i === 0) { await sleep(2000); try { return await callChat(chain[i], opts); } catch (e) { lastError = e; } }
  }
}
throw lastError;

Two refinements make this production-worthy. First, only retry the failures worth retrying. A 503, a 429 rate-limit, or a timeout is transient — retry, then fail over. A 400 (malformed request) or 401 (bad key) is not — retrying it against the next provider just burns time and money on a request that’s broken everywhere; surface it immediately. Second, treat a successful-looking empty answer as a failure. A model can return a finish_reason with empty content — formally a success, actually garbage. Hand that up as an empty string and your user gets a reply made of nothing. Count empty content as an error and fall through to the next provider.

Speed, streaming, and timeouts

Providers are not interchangeable on speed, and that should shape your chains. Some specialized inference providers serve hundreds of tokens per second while a flagship reasoning model can take many seconds to first token. So order chains with the use case in mind: put a fast model first for anything interactive, and reserve the slow-but-strong model for work that can wait. For user-facing replies, stream the response (stream: true) so the user sees tokens as they arrive instead of staring at a spinner.

Two operational guards belong here. Set a per-call timeout so one hung provider can’t stall the whole chain — the SDK takes a timeout option, and a request that blows past it should be treated as a transient failure and fall through:

const res = await client.chat.completions.create(params, { timeout: 30_000 });

And cache what you can. A lot of LLM traffic is repetitive — the same classification prompt over similar inputs — and an exact-match response cache in front of the cheap classes cuts both cost and latency to zero on a hit. Several providers also offer server-side prompt caching that discounts repeated prompt prefixes; it’s worth turning on for long, stable system prompts.

“OpenAI-compatible” is not “OpenAI-identical”

This is the part that eats afternoons, and it’s the real reason a thin layer of your own often beats a one-line SDK swap. Providers implement most of the OpenAI surface, not all of it, and the gaps are exactly where things silently misbehave.

Structured output and tool calling vary. JSON mode, strict schema enforcement, and function/tool calling are supported unevenly — a model that advertises JSON output may still wrap it in prose, and tool-calling formats differ. Don’t assume a feature works on a new provider until you’ve tested it there.

Reasoning models don’t take the usual parameters. Newer reasoning models often reject the familiar max_tokens and temperature and instead want max_completion_tokens and a reasoning_effort setting — and the reasoning tokens are billed from the same completion budget. Set the budget too tight and the model “thinks” until it hits the cap and never reaches an answer. So branch on it and leave headroom:

if (isReasoningModel(model)) {
  params.max_completion_tokens = (opts.maxTokens ?? 2048) * 2; // room to think AND answer
  params.reasoning_effort = "low";                              // cheap/fast for non-creative work
} else {
  params.temperature = opts.temperature ?? 0.5;
  params.max_tokens  = opts.maxTokens ?? 2048;
}

Models smuggle extra text into the output. Some reasoning models prepend their chain-of-thought wrapped in <think>…</think>; strip it before returning. And when you ask for JSON, a model may answer “Sure, here’s your JSON: { … }” — so run the response through a step that pulls the first valid object or array out of any chatty preamble rather than trusting the whole string to parse.

Usage fields and streaming differ too. Token-usage accounting and streaming chunk formats aren’t perfectly uniform, so normalize them at the edge instead of sprinkling provider-specific checks through your code. The pattern that keeps you sane: one adapter layer that swallows all these differences, so the rest of your app sees a single clean interface.

Pin your model versions. A bare model name is often an alias that the provider quietly re-points to a newer snapshot — great until a “minor” update shifts your outputs and a prompt that worked yesterday regresses today. Where a provider offers dated or versioned model IDs, pin them, and treat a model upgrade as a change you test, not one that happens to you.

And don’t assume your prompts are portable. A prompt carefully tuned for one model can underperform on another — different models want different phrasing, react differently to system vs user roles, and vary in how strictly they follow format instructions. The chain hides which model answers, which is exactly why you should test your prompts against every model in a chain, not just the one you developed against. Otherwise a fallback that “works” still quietly degrades quality the moment it kicks in.

Don’t trust “success” — validate the output

API success is not the same as a usable result, and the cleanest example is transcription. Speech-to-text models hallucinate on silence or noise, typically by repeating one phrase dozens of times. The call returns 200; the transcript is junk. So validate before you accept it — a length floor plus a unique-word ratio catches the looping case — and if it fails, fall through the chain like any other error:

function isTranscriptValid(text: string): boolean {
  if (text.length < 50) return false;
  return uniqueWordRatio(text) >= 0.15; // mostly-repeated text → reject
}

The general principle outlives the transcription example: decide what a valid answer looks like for each task, check it, and treat a structurally-wrong-but-200 response as a failure worth failing over. The model giving you an answer and the model giving you a useful answer are different events.

Count the money on every call

The moment you have several providers and models in play, “what does this feature cost?” and “which provider is actually carrying load?” become unanswerable unless you record them. So log every successful call: the operation, the chosen provider and model, input and output tokens, the computed cost, and the latency.

const costUsd = tokensIn / 1e6 * price.inputPerM + tokensOut / 1e6 * price.outputPerM;
await db.llmCallLog.create({ operation, provider, model, tokensIn, tokensOut, costUsd, ms });

Without this you’re flying blind: you can’t see that one feature quietly costs ten times another, and you can’t tell which providers earn their place in a chain versus which are just listed there for decoration. A line in your logs reading AI fallback → mistral at 3 a.m. is also how you learn a vendor fell over — instead of learning it from forty customer complaints in the morning.

Putting it together

Stacked up, the whole router is small — one function the rest of your app calls, with everything else hidden behind it:

async function chat(taskClass: TaskClass, opts: ChatOpts) {
  const chain = chains[taskClass].filter(Boolean);   // configured providers only
  let lastError: unknown;
  for (let i = 0; i < chain.length; i++) {
    const target = chain[i];
    try {
      const params = normalizeParams(target.model, opts);        // reasoning vs classic params
      const res = await callChat(target, params, { timeout: 30_000 });
      const text = cleanOutput(res);                             // strip <think>, unwrap JSON
      if (!isValid(taskClass, text)) throw new Error("invalid output");
      await logCost(target, res);                                // operation, model, tokens, cost, ms
      if (i > 0) logger.warn(`AI fallback → ${target.name}`);
      return text;
    } catch (err) {
      if (!isRetryable(err)) throw err;                          // 400/401 → stop, don't fail over
      lastError = err;
    }
  }
  throw lastError;
}

Everything the rest of the codebase sees is chat("QUALITY", { … }). Which providers exist, which model serves which class, how parameters are normalized, how output is cleaned and validated, what a call costs — all of it lives in this one layer. Adding a provider is a new OpenAI({ baseURL }) and a slot in a chain; the features that call chat() never change.

Build it, or buy it?

You don’t have to hand-roll any of this — and for many teams you shouldn’t. The honest build-vs-buy map:

• OpenRouter — hosted: one key, one OpenAI-compatible endpoint, and it routes across providers with fallback and pricing built in. Zero infrastructure; the fastest start.
• LiteLLM — the de-facto open-source standard: an SDK plus a proxy gateway covering 100+ providers with fallback, retries, budgets, caching, cost tracking, and logs. It does everything above and more.
• Portkey — a similar AI gateway (open-source and hosted) that adds guardrails and observability.
• Vercel AI SDK — if you’re on TypeScript, it gives unified providers and fallback out of the box.
• Semantic routers (Not Diamond, Martian, Unify) — these go a step further and pick the model per request rather than via a static chain.

The rule of thumb: if you need budgets, caching, dashboards, and a hundred providers out of the box, reach for LiteLLM or OpenRouter — rolling your own won’t pay off. The case for a thin homegrown layer is narrow but real: you have only three to five providers, you want the cost log sitting in your own database next to your domain data, and you need custom edge logic (the reasoning-model handling, the transcript validation) without an extra proxy hop in the critical path. Under the hood you’re still leaning on the official SDK’s low-level retries and HTTP — you’re just adding a thin routing brain on top.

The takeaway

The thing worth internalizing is that the OpenAI-compatible API turned provider independence from a migration project into a configuration detail. One client, a swapped base URL, and a fallback loop is genuinely most of the way to a system that shrugs off a vendor outage and routes cheap work to cheap models. The difficulty was never the client; it’s the edges — the parameters that differ, the outputs that aren’t quite JSON, the 200s that aren’t really answers, and the bill you can’t see unless you log it. Treat your providers as what they now are: interchangeable, fallible suppliers behind one interface — wire them so any one of them can disappear and your product keeps answering.