Infrastructure notes

Production stories,
written in the calm after.

I do platform work at a startup. This is where I write down the things I had to figure out the hard way, so future-me at 3am doesn't have to.

Read the latest →Browse TILs (11)

Currentlyantenna building

The workbench: SDR, Flipper Zero, Pwnagotchi, a Pi Zero, and an hourglass.

The workbench, in detail.

SDR, a Flipper, a Pwnagotchi, a few antennas I tuned in bad lighting, and an hourglass for measuring how long a deploy actually takes.

See the lair →

Latest

Browse all →

A dark terminal screenshot. First command: kubectl get pods -l app=vllm-llama3-70b shows two pods with STATUS CrashLoopBackOff and RESTARTS 4. Second command: kubectl describe pod shows Last State Terminated, Reason Error, Exit Code 137, and an Events section with a Warning 'Liveness probe failed: Get http://10.0.3.14:8000/health: connect: connection refused' followed by 'Container vllm failed liveness probe, will be restarted'.

jul 23, 2026

Your model isn't crashing, your probe is

A model server that takes five minutes to load and a liveness probe that gives it ten seconds is a crash loop waiting to happen. Probes, drains, and safe…

A two-panel diagram contrasting request routing. Left panel titled 'round-robin': four vLLM replicas, incoming requests that share a common prefix drawn as colored tokens, arrows scattering them evenly across all four replicas; each replica shows a mostly-full KV cache and a low prefix-cache-hit badge (~11%), with a note 'every replica re-runs prefill for prefixes its neighbors already cached'. Right panel titled 'prefix-cache-aware': the same requests and four replicas, but arrows route each request to the replica that already holds its prefix; caches show headroom and a high hit badge (~93%), with a note 'a cache hit skips prefill entirely'.

jul 18, 2026

The cheapest speedup is your load balancer

Same GPUs, same model, same replica count. Swap round-robin for prefix-cache-aware routing and the fleet gets 2.3x faster. The router was throwing the…

A horizontal stacked-bar diagram titled 'the cold-start tax: what the first user waits for'. A single bar is divided into four segments sized by time: node provisioning (1–5 min), image pull (2–4 min for a 5–15GB image), model weight load (several minutes for ~140GB), and warmup (tens of seconds). The total is labeled 'naive cold start: 5+ minutes'. Below, a second much shorter bar labeled 'after the fixes in this post' shows the same stages compressed, annotated with 'lazy image pull + model streaming + warm nodes → tens of seconds'.

jul 16, 2026

Scaling GPU inference to zero and back

Idle GPUs at six dollars an hour are a bonfire. Scaling to zero saves the money, but the first user back waits minutes unless you kill the cold start.

A timeline diagram of a single inference request showing where each latency metric is measured. A horizontal track runs left to right: the request arrives and sits in a queue (labeled 'queue time'), then prefill runs (labeled 'prefill'), then the first token emerges (a marker labeled 'TTFT ends here: queue + prefill'), then a series of output tokens stream out with the gaps between them labeled 'ITL / TPOT', and the far right marks 'end-to-end latency'. Below, a note reads 'TTFT is what the user waits for; ITL is how smooth the stream feels'.

jul 11, 2026

What a green GPU dashboard hides

The GPU dashboard says 92% busy and users are waiting eight seconds for the first token. Monitoring an LLM server means watching the queue, not the…

The next post, in your inbox.

Long-form infra writing, every few weeks. No tracking pixels, no marketing sequences, no LinkedIn-isms.

Harshit Luthra. Senior SRE, infrequent essayist, occasional source of production incidents. More about me →

4M+

Daily active users

1M/min

Peak throughput

99.99%

Uptime on 95% spot

$200K

Cloud spend trimmed

What I work in

kubernetes · terraform · aws · next.js · typescript · postgres

28 posts · 11 TILs · writing here since 2018

Production stories,written in the calm after.