Inference lab on EKS, 23 August 2026
The slowest pod took 7m 40s from creation to its latest Ready transition, 4m 38s of it pulling the image.
Startup, pod by pod
model-serving/vllm
Deployment · vLLM · 1 to 2 replicas (KEDA ScaledObject vllm) · template image vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967
Times as a table
| Pod | Image pull | Start to latest Ready | Created to latest Ready | State |
|---|---|---|---|---|
vllm-5f577ccf85-llq95 | 4m 40s | unknown | unknown | Running |
vllm-5f577ccf85-j27ct | 4m 38s | 2m 56s | 7m 40s | Ready |
Findings
Startup
Image pull took 4m 40s
The kubelet reported pulling vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967 (9.1 GB) in 4m 40s for pod vllm-5f577ccf85-llq95 on node ip-10-42-10-131.ec2.internal.
Why it matters. A new replica on a node without this image waits for the pull before the server can start. How long depends on the registry, the network and what the node already has cached.
Next. Time the same image from a registry in the cluster's region, and with the image already on the node, then compare.
Container start to latest Ready took 2m 56s
Pod vllm-5f577ccf85-j27ct started its vllm container at 14:18:35 UTC, and its Ready condition last turned true at 14:21:31 UTC.
Why it matters. Model download, loading and warmup usually happen in this window. Pod status keeps only the latest Ready transition, so if readiness dropped and came back since, the startup itself was shorter.
Next. Read the engine's startup log lines to see which part dominates. For vLLM, look for "Model loading took" and "init engine".
The model cache isn't kept between pods
vLLM serves Qwen/Qwen2.5-7B-Instruct-AWQ, and the Hugging Face cache at /root/.cache/huggingface/hub is on an emptyDir.
Why it matters. If the weights come from the Hub, every new replica downloads them again before it can load them, which adds to scale-up time.
Next. Confirm where the weights come from (the image, an init container or the Hub), then compare startup with a cache that outlives the pod.
Reliability
Can drop to a single replica
KEDA ScaledObject vllm is set to 1 to 2 replicas.
Why it matters. Losing the only serving replica interrupts requests until a replacement is Ready, which can take as long as the startups above.
Next. If this model needs to stay up, keep a second replica or a fallback route.
GPU nodes
| Node | Instance type | GPUs | Requested here | Requested share |
|---|---|---|---|---|
ip-10-42-10-131.ec2.internal | g6.xlarge | 1 | 1 | |
ip-10-42-11-99.ec2.internal | g6.xlarge | 1 | 1 |
What this report can't see
- GPU utilisation. Requests show what pods reserved, not how busy the GPUs are. Only nvidia.com/gpu and amd.com/gpu are counted. With one namespace read, requests from other namespaces aren't counted, so spare GPUs aren't reported.
- First starts. Pod status keeps only the latest Ready transition, so a pod whose readiness dropped and came back looks slower than it started, and a restarted pod's first start is gone.
- Settings outside the pod spec. vLLM flags set inside a shell script or a config file aren't parsed, and only one serving container per pod is checked.
- Network reachability. Exposure findings come from Service settings; nothing was probed.
- Older starts. Kubernetes keeps events for about an hour by default, so image pull times for older pods are usually gone.