AI Gateway

The third API key is the one that hurts

Our first AI feature shipped with a provider key in a secret and nobody minded. By the third I could see where it was heading: keys scattering across repos, and nobody able to say what any of it cost.

None of that is an AI problem. It's unmanaged database credentials wearing a hat, and it has the same fix. I built this while we had two consumers, which is the only reason it was a small job.

Virtual keys, and a model list you can't talk your way past

Services don't hold provider credentials any more. They hold a virtual key with an explicit list of models it may call. Ask for anything else and you get a 401.

A gateway that helpfully substitutes another model gives you something that works and behaves differently from the thing you tested. I'd rather it broke.

# 1. the gateway has to know the model exists
#    (model_list entry in the gateway config, shipped via GitOps)

# 2. every key that should reach it has to be told, separately
curl -X POST "$GATEWAY/key/update" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"key": "sk-…", "models": ["gemini-2.5-flash", "gemini-3.6-flash"]}'

# skip step 2 and consumers get 401 for a model that
# demonstrably exists. what can this key actually reach?
curl -H "Authorization: Bearer $VIRTUAL_KEY" "$GATEWAY/v1/models"

Those two halves being separate is the sharp edge, and it's caught me more than once. A model can be fully deployed and still 401 for everybody, because existing and being permitted are different facts. It's written down now.

Whose spend is it?

Every call carries its tenant, plus tags for environment and feature. Chat, scheduled estate summaries and the nightly schema compile all bill to one service but answer completely different questions about cost.

So "what is the AI costing us" became answerable per tenant and per feature, which matters the first time somebody asks about cost to serve.

What it made cheap

The point of a seam is what gets easy afterwards. There's an automated pull request review agent running on PRs across eight production services now, and building it needed no provider credentials, no billing conversation, no new secret.

Model upgrades became config too. Moving a consumer between versions is an allowlist entry and an env var, which is how the estate followed three generations of the same model family without anyone rewriting a consumer.

Why it's shaped like this

Where it stands

It's an unglamorous piece of infrastructure and that's roughly the recommendation. Built before the sprawl rather than after it, which is the only reason adding an AI feature here is now a config change instead of a procurement conversation.