Streaming from OpenRouter in production: what actually works

OpenRouter is a language-model gateway: one API key, dozens of providers behind it. For a product that supports multiple models this is beautiful. In production it has two gotchas that cost us a day.

Here they are, so they cost you an hour instead.

What OpenRouter is

OpenRouter proxies requests to the underlying providers (OpenAI, Anthropic, Google, and many others). The API surface is OpenAI-compatible chat/completions. You pass a model like openai/gpt-5-mini, anthropic/claude-sonnet-5 or minimax/minimax-m3:free, and OpenRouter routes appropriately.

Pricing is per token, marked up over the underlying providers by a small margin. In exchange you get: one bill, key rotation in one place, automatic failover across providers, and — this is the real win — access to models before their direct SDK is public.

Wiring it up with the AI SDK

The Vercel AI SDK has excellent OpenAI support and no OpenRouter provider. But OpenRouter speaks the OpenAI wire format, so you point the OpenAI provider at their URL:

import { createOpenAI } from "@ai-sdk/openai";

const gateway = createOpenAI({
  apiKey: process.env.OPENROUTER_API_KEY,
  baseURL: "https://openrouter.ai/api/v1",
  headers: {
    "HTTP-Referer": "https://your-app.example",
    "X-Title": "Your App",
  },
});

The HTTP-Referer and X-Title headers aren’t credentials. OpenRouter uses them to attribute traffic in dashboards.

Gotcha 1: /responses vs /chat/completions

The AI SDK’s OpenAI provider now defaults to OpenAI’s /responses API, not the older /chat/completions one. OpenRouter does not implement /responses. If you use gateway(modelId), the AI SDK sends the request to /responses and you get:

APICallError [AI_APICallError]: User not found.
  url: 'https://openrouter.ai/api/v1/responses'

That error message is misleading — the user is fine. The endpoint just doesn’t exist on OpenRouter.

The fix: pin the chat-completions endpoint by using .chat():

return gateway.chat(modelId);

The SDK’s docs mention this in passing but the error message doesn’t point at it, so it burns time.

Gotcha 2: streamText fails through onError, not throws

Here’s the subtle one. The AI SDK’s streamText doesn’t throw when the provider fails. It reports failures through an onError callback and returns an empty stream that ends normally.

This means the naïve implementation:

const result = streamText({ model, ... });
try {
  for await (const delta of result.textStream) {
    yield delta;
  }
} catch (err) {
  // fall back to mock provider
}

looks like it works. In tests it does. In production, an invalid OpenRouter key produces an empty stream, no exception, and the visitor sees a chatbot that returns nothing.

You have to explicitly capture errors:

let providerError: string | undefined;
const result = streamText({
  model,
  ...,
  onError: ({ error }) => {
    providerError = error instanceof Error ? error.message : String(error);
  },
});
for await (const delta of result.textStream) yield delta;
if (providerError) {
  // NOW you can degrade to a fallback
}

Without this, silent failures are the default.

Graceful degradation

We use OpenRouter as a universal gateway and keep direct provider keys where available. Priority order:

1. Direct key for the requested provider (cheaper, lower latency).
2. OpenRouter (universal fallback).
3. Deterministic mock provider that answers from retrieved context (last resort).

The mock provider is what makes the free plan honest — it produces grounded-looking replies from the actual chunks the retriever returned, so the UX doesn’t collapse when there are no keys. It also runs in tests without hitting any provider.

When a real provider fails mid-request, we degrade to the mock and set degradedReason on the result so the operator can see what happened without the visitor seeing a broken widget.

Free-tier reality

OpenRouter’s free-tier models exist (models with the :free suffix), but they’re rate-limited per day per key. Fine for evaluation, terrible for production traffic. If you’re serving real users, budget for real credits.

The free tier is exactly what you want during development: real completions, no bill, honest behaviour.

The short version

• Pin .chat() on the OpenRouter-backed client.
• Capture errors via onError — streamText will not throw.
• Have a deterministic mock so a bad key doesn’t collapse the UX.
• Free tier for evals; paid tier for production.

If you want to see the graceful-degradation pattern implemented, Anserra’s providers.ts does exactly this.