Cold starts are the quiet tax on every serverless inference platform. When a request lands on a model that isn’t already resident on a GPU, the user waits while weights are pulled, the runtime spins up, and the first token finally appears. For most workloads the median looks fine — it’s the tail that hurts, and the tail is what customers remember.
We set ourselves a hard target: keep p99 cold-start latency under 100ms across every EU region, without paying for idle capacity we don’t need. Getting there meant rethinking two things we had treated as separate problems — where a model runs, and when we evict it.
Placement, reconsidered
Our original scheduler optimised for packing density: fit as many models as possible onto each node. That was great for utilisation and terrible for tail-latency, because popular models kept landing on whichever node had a spare slot, far from the data they needed.
The new placement layer is locality-aware. It keeps a rolling estimate of each model’s request rate per region and biases replicas toward the regions actually driving traffic. Weights are staged on local NVMe before the first request arrives, so a “cold” start is really just a runtime spin-up, not a network fetch.
Warm pools that evict intelligently
The second half of the win came from how we manage warm pools. A naive least-recently-used policy evicts exactly the models that are about to be needed again — anything with bursty, periodic traffic gets punished. We replaced it with a cost-aware policy that weighs reload cost against recency and observed periodicity.
The result: p99 cold starts dropped from roughly 640ms to 94ms, and warm-pool memory pressure fell at the same time because we stopped hoarding models nobody was calling. The tail is no longer the thing customers remember.