AI Gateway

finding 1·before: no gateway

The third API key is the one that hurts

Our first AI feature shipped with a provider key in a secret, and that was fine. By the third I could see where it was heading: keys scattering across repos, and no way to say what any of it cost.

None of that is an AI problem. It's the same problem as unmanaged database credentials, and the fix is the same one: put the credential somewhere central and hand out scoped access instead.

LiteLLM is an open-source LLM proxy. I didn't write a proxy, I ran that one on Kubernetes through GitOps like everything else we deploy. The work worth describing is the key and allowlist model, the attribution scheme, and the runbook.

finding 2·request · auth + allowlist

Virtual keys, with explicit model allowlists

Services don't hold provider credentials any more. They hold a virtual key with an explicit list of models it may call. Ask for anything else and you get a 401.

A gateway that helpfully substitutes another model gives you something that works and behaves differently from the thing you tested. I'd rather it broke.

# 1. the gateway has to know the model exists
#    (model_list entry in the gateway config, shipped via GitOps)

# 2. every key that should reach it has to be told, separately
curl -X POST "$GATEWAY/key/update" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"key": "sk-…", "models": ["gemini-2.5-flash", "gemini-3.6-flash"]}'

# skip step 2 and consumers get 401 for a model that
# demonstrably exists. what can this key actually reach?
curl -H "Authorization: Bearer $VIRTUAL_KEY" "$GATEWAY/v1/models"

Those two halves being separate is the sharp edge, and it has caught me more than once: a model can be fully deployed and still 401 for everybody, because being deployed and being permitted are separate facts. It's in the runbook now.

finding 3·request · spend tags

Whose spend is it?

Every call carries its tenant, plus tags for environment and feature. Chat, scheduled estate summaries and the nightly schema compile all bill to one service but answer completely different questions about cost.

So "what is the AI costing us" became answerable per tenant and per feature, which matters the first time somebody asks about cost to serve.

finding 4·postmortem · attached

The 4x pricing bug

Attribution being right is not the same as the number being right. This one is the reason the last action item is still open.

finding 5·after: what got cheap

What it made cheap

There's an automated pull request review agent running on PRs across eight production services now, and building it needed no provider credentials, no billing conversation, no new secret.

Model upgrades became config too. Moving a consumer between versions is an allowlist entry and an env var, which is how every consumer followed three generations of the same model family without anyone rewriting one.

What the review tracked, and the one item still open:

resolution·200 OK · monitoring

Where it stands

It's dull infrastructure now, which is what I wanted. I built it while there were two consumers rather than twelve, so adding an AI feature is a config change. Doing it later would have meant unpicking a dozen scattered provider keys first.