Sai Ram

Where 7m 40s of vLLM startup went

In my August autoscaling run, the HPA that KEDA manages asked for a second vLLM replica about 10 seconds after the load started. The new pod was Ready 7 minutes 40 seconds later, on a GPU node that was already up. The timeline below comes from the run's Kubernetes events and the pod's recorded Ready time.

The second replica, minute by minute

Source: the published run. Pull and start times are from operator/events-final.json, Ready from operator/pod-ready-time.txt, and the last request from the run's README. vLLM 0.27.1 serving Qwen2.5-7B-Instruct-AWQ, one NVIDIA L4 per replica on g6.xlarge nodes, EKS 1.34, us-east-1. Image and model sizes checked against Docker Hub and Hugging Face on 11 October 2026.

What the timestamps show

The pod was created and scheduled in the same second as the HPA's request. Its node was already up, so none of this was waiting for a machine.

The kubelet reported 4m 38s to pull vllm/vllm-openai from Docker Hub, a figure that covers downloading and unpacking. The pinned amd64 image is 9.1 GB of compressed layers, and one layer alone is 4.6 GB. The first replica, on the other node, reported 4m 40s for the same pull, so the two took about the same time.

The container started 5 seconds after the pull. From there to Ready took 2m 56s. In this lab the Hugging Face cache is an emptyDir, so each new pod starts without the 5.6 GB of weights and has to fetch them. That window also covers loading onto the GPU, vLLM's warmup, and the health checks, which run every 10 seconds. The run didn't keep vLLM's logs, so I can't split the 2m 56s further.

Why it matters

Until the new pod is Ready, the existing replica carries all the traffic. In this run it coped: requests queued, and p95 end-to-end latency was 832 ms against a 6 second target. The second replica arrived as the test was ending, about 38 seconds before the last request finished. Under heavier traffic, the service could have run short of capacity during that wait. This was sustained load; I didn't test bursts.

Both GPU nodes stayed up the whole time, so adding a pod didn't change the bill. Faster startup saves money only when it means paying for fewer GPU node-hours, for example by scaling nodes down sooner or keeping fewer spare replicas warm.

What I'm testing next

One change at a time against this baseline, starting with the biggest piece:

• The same pinned image already on the node before the pod needs it, compared with pulling it from scratch.
• Keeping the model weights on the node or a shared volume instead of downloading them on every start.
• Keeping timestamped vLLM logs to narrow down the 2m 56s, and timing the weight download on its own.

Results will go into the lab repo next to this run.

Check your own cluster

Run these against a running pod that started recently, replacing POD, NAMESPACE and CONTAINER. Kubernetes keeps events for an hour by default, so look soon after a scale-up. Pod conditions and the start time describe the latest transition, so a restarted container shows its last start.

1. How long the image pull took. The kubelet reports it in the event message.

   kubectl get events -n NAMESPACE \
     --field-selector involvedObject.name=POD,reason=Pulled

2. When the pod was scheduled, started and became Ready.

   kubectl get pod POD -n NAMESPACE -o jsonpath=\
   '{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.lastTransitionTime}{"\n"}{end}'
   kubectl get pod POD -n NAMESPACE -o jsonpath=\
   '{.status.containerStatuses[?(@.name=="CONTAINER")].state.running.startedAt}{"\n"}'

3. What vLLM spent its time on. vLLM 0.27.1 logs model loading (including the weight download, when there is one), CUDA graph capture and engine startup. The timers overlap, so don't add them up.

   kubectl logs POD -n NAMESPACE -c CONTAINER --timestamps | grep -E \
     "Model loading took|Graph capturing finished|init engine"

4. Whether each new pod downloads the model again. If the Hugging Face cache is an emptyDir or isn't mounted, a new pod starts without the weights. An emptyDir does survive container restarts inside the same pod, and some setups bake the weights into the image. To be sure, look inside a freshly started pod: if the cache directory starts empty and fills up during startup, it's downloading.
5. Where the image comes from and how big it is. A multi-gigabyte image pulled from a public registry onto a fresh node is worth timing on its own.