The comparison everyone makes
Search for "self-host LLM vs API cost" and you'll find the same spreadsheet everywhere: GPU-hour price divided by tokens-per-hour throughput, versus a hosted API's price per million tokens. Above some volume threshold, self-hosting wins. Below it, the API wins. Simple.
It's also missing the column that actually decides the answer for most teams: what does it cost you in time when the infrastructure misbehaves, and how often does that happen.
The column nobody puts in the spreadsheet
We ran the self-hosting side of this comparison for real, on a rented GPU running an open-weight model behind an OpenAI-compatible endpoint. The GPU-hour math looked exactly like the tutorials promised. The actual week looked different:
- An instance that wouldn't restart because the host ran out of free GPU capacity — a support-ticket-shaped problem, not a code problem.
- A driver mismatch on one host that crashed the server at boot, with no way to specify the driver version up front — discovered only after deploying, not before.
- A five-minute cold start on every fresh deployment, which needed to be designed around rather than ignored.
None of these show up in a GPU-hour calculation. All of them are hours of someone's time, and none of them happen when you're paying a hosted API to make the problem disappear.
How to actually price it
The honest way to compare isn't "GPU-hour cost" vs "token cost." It's:
GPU-hour cost + (expected incidents per month × time-to-resolve) × your hourly cost, compared against the hosted API's price, plus whatever premium the API charges for not having to think about any of the above.
For workloads at real scale, that math still tips toward self-hosting — the incident rate drops as you get better at the infrastructure, and the per-token savings compound. For a one-off project or a small team without spare ops capacity, the "premium" a hosted API charges is often just paying someone else to have already solved the problems above. That's a legitimate thing to pay for.
The actual decision rule we use
Self-host when the volume is high enough that the token savings dwarf a realistic number of bad weeks, and when someone on the team is willing to own GPU infrastructure as an ongoing responsibility, not a one-time setup task. Otherwise, the hosted API's premium is buying exactly the thing it looks like it's buying: somebody else's on-call rotation.
