Cut the AI bill, then prove nothing broke.
You already know roughly where the money goes. Nothing changes, because whoever moves the traffic owns the risk if quality drops.
One thread. Invoice to proof.
Scroll. The product on the right is the same replay, moving one step at a time — the way a change actually gets made.
You have a bill. You don’t have an owner.
OpenAI, Anthropic, Bedrock, Azure, Vertex. Five PDFs. None of them know Support from Checkout. A provider invoice has no idea your run existed.
Nothing moves, because quality is someone’s neck.
Cheaper models exist. Nobody routes to them. Whoever swaps the traffic owns the regression. So the expensive path stays the default.
Read-only. First snapshot in 48 hours.
Billing access and an IAM role. No production access, no code, nothing in the request path. We replay the 30 days that already happened.
Stamp the call. Then the bill can be read.
Team, feature, customer at the call site — explicit tags, not inferred. You see what stays un-attributed instead of a total that pretends otherwise.
A budget that can refuse.
Per-run caps checked at the proxy before the provider is called. Runaway loops get caught at five times their own baseline. Shadow first. Not armed until you say so.
Every swap, twice, on your traffic.
Answers swapped so position bias cancels. A safety regression in either pass fails it outright. A sequential test decides when the evidence is enough — and reverts if quality slips.
Measured against a control group of your own spend.
Thirty days either side. Comparable workloads that did not change. It isn’t “spend fell while we happened to be installed.” The ledger can read negative.
A GPU-hour and a token, same day, same team.
Most teams live in the LLM lane. That is a complete product. Owned hardware is an escalation: utilization, real $/GPU-hour, build against buy — joined to the same model.
Then it runs like a conversation.
Not a dashboard dump. The replay shows up, a policy stays in shadow, you arm it, the ledger reports back. Same thread.
Replay of the last 30 days is in. $184,200 across five providers. 91% attributed. Could-move band $11.4k–$18.2k — mostly Support tickets still on gpt-4o.
could_move Support/tickets $6k–$9k hold Checkout/retry quality unlabeled 9%
Arm checkout-router? Swap opus → haiku on retries.
No. Shadow only. Two judge passes on your real traffic first. Pass A hold, pass B hold, safety clear. A policy cannot arm until that exists.
policy checkout-router mode shadow armed false
Arm it.
Armed. Thirty days in: changed cohort −12.4%, control −1.1%. The difference is the swap, not the season. Ledger is positive. If it flips, the policy reverts itself.
First snapshot in 48 hours.
Not minutes. Read-only billing. Written replay inside two weeks.
Connect
Billing APIs and an IAM role. No production access, no code changes, nothing in the request path.
Name them
Production, staging, the names you already use. Dev eval spend stops quietly inflating prod.
The replay
What could have moved, the band it would have saved, and the honest reasons the rest couldn’t. If it doesn’t clear the fee, the report says so.
The day-to-day, once you’re in.
The story above is the argument. This is the rest of the surface.
Chargeback finance will send
Month-end PDF and CSV per team, customer, or feature. Engineering owns its number. Finance stops chasing people.
Budgets and burn
Warn at 75, 90, 100. The cap that actually refuses a call sits in the gate above — this is the early-warning half.
Tied out daily
What we measured, against each provider’s own reported usage. Drift outside tolerance flags. Adding a provider doesn’t add a blind spot.