Why bother self-hosting at all
Calling a hosted LLM API is one HTTP request and a credit card. Renting a GPU and running the model yourself is neither of those things — so why do it?
Control, mostly. No per-token bill to a third party, no rate limits set by someone else's business model, full say over context window size and which weights you're actually running. For workloads where you want that control, self-hosting on a rented GPU (rather than a "local" laptop) is a real middle ground: you get root access to the box, but you're not buying hardware.
We spent a few days doing exactly this — deploying an open-weight model behind an OpenAI-compatible API on a rented GPU instance — and it taught us more about the operational reality of "just run your own model" than any amount of reading.
The gotchas nobody puts in the tutorial
1. Ephemeral infrastructure will silently strand your app. The endpoint URL for a rented GPU instance often bakes in a unique instance ID. Every time you redeploy, that ID — and therefore the URL — changes. If nothing re-points your client at the new address, you don't get an error. You get an app that quietly talks to a dead host until someone notices nothing is responding.
2. "Restart" is not guaranteed to mean restart. A stopped GPU instance can fail to restart simply because the host machine ran out of free GPU capacity in the meantime. If your setup has no persistent volume, fighting a restart wastes more time than just recreating the instance from a template.
3. The underlying host's software stack is not something you control. On community-tier cloud GPU providers, we hit hosts running a CUDA driver version that flat-out crashed the inference server at startup. There was no way to filter for the right driver version at instance-creation time — the only fix was to create the instance, inspect what we got, and redraw if it was wrong. Paying a premium for a "reliable" tier of the same provider consistently avoided the problem, which tells you the premium is buying you determinism, not just speed.
4. Cold starts are minutes, not milliseconds. Downloading model weights, compiling the inference graph, and initializing the serving engine took roughly five minutes end to end on our setup. A hosted API never makes you think about this. Self-hosting means it becomes part of every deploy, and every workflow depending on the model needs to tolerate it.
5. Context window and concurrency are a tradeoff you now own. Long context windows eat GPU memory that would otherwise go to serving multiple requests at once. With a hosted API, someone else absorbs that tradeoff invisibly. Running it yourself means you're staring directly at the tradeoff and have to make the call.
The takeaway
None of this is a reason to avoid self-hosting — it's a reason to go in with eyes open. The dollar cost of a rented GPU is genuinely lower than API token costs at any real volume. The time cost is in babysitting infrastructure that a hosted provider normally hides from you entirely. Whether that trade is worth it depends entirely on how much you value control versus how much you value not thinking about any of this.
