Infrastructure notes

Production stories,
written in the calm after.

I do platform work at a startup. This is where I write down the things I had to figure out the hard way, so future-me at 3am doesn't have to.

Read the latest →Browse TILs (11)

Currentlyantenna building

The workbench: SDR, Flipper Zero, Pwnagotchi, a Pi Zero, and an hourglass.

The workbench, in detail.

SDR, a Flipper, a Pwnagotchi, a few antennas I tuned in bad lighting, and an hourglass for measuring how long a deploy actually takes.

See the lair →

Latest

Browse all →

A timeline diagram of a single inference request showing where each latency metric is measured. A horizontal track runs left to right: the request arrives and sits in a queue (labeled 'queue time'), then prefill runs (labeled 'prefill'), then the first token emerges (a marker labeled 'TTFT ends here: queue + prefill'), then a series of output tokens stream out with the gaps between them labeled 'ITL / TPOT', and the far right marks 'end-to-end latency'. Below, a note reads 'TTFT is what the user waits for; ITL is how smooth the stream feels'.

jul 11, 2026

What a green GPU dashboard hides

The GPU dashboard says 92% busy and users are waiting eight seconds for the first token. Monitoring an LLM server means watching the queue, not the…

A network topology diagram titled 'rail-optimized fabric'. It shows four GPU nodes at the bottom, each with eight GPUs and eight NICs. The NICs are grouped into rails: NIC-0 of every node connects up to the same leaf switch, NIC-1 of every node to the next leaf switch, and so on across eight leaf switches. The leaf switches connect up to a row of spine switches forming a non-blocking fat-tree. An annotation notes 'GPU_i on any node reaches GPU_i on any other node in one switch hop'.

jul 9, 2026

Scaling GPUs past one box

Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.

$A diagram contrasting two eight-GPU server layouts. Top, labeled 'HGX / DGX baseboard': eight GPUs each connected to a bank of four NVSwitches so every GPU reaches every other at full 900 GB per second, drawn as a dense fully-connected mesh. Bottom, labeled 'PCIe box': eight GPUs hanging off PCIe switches and the CPU root complexes, GPU-to-GPU traffic forced through PCIe at a fraction of the bandwidth, drawn as a sparse tree routed through the CPUs.$

jul 4, 2026

One box, eight GPUs, and the wires between them

Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.

A vertical layer-stack diagram titled 'what a GPU pod actually sits on', showing twelve stacked layers from bottom to top: GPU silicon and NVSwitch, firmware (VBIOS and GSP), kernel driver nvidia.ko, userspace driver libcuda, CUDA runtime and cuDNN and NCCL, NVIDIA Container Toolkit, container runtime containerd, device plugin, node feature discovery, GPU Operator, scheduler, and the workload pod on top. Each layer has a short label describing its one job and a red note describing its characteristic failure.

jul 2, 2026

The dozen layers under a GPU pod

A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…

The next post, in your inbox.

Long-form infra writing, every few weeks. No tracking pixels, no marketing sequences, no LinkedIn-isms.

Harshit Luthra. Senior SRE, infrequent essayist, occasional source of production incidents. More about me →

4M+

Daily active users

1M/min

Peak throughput

99.99%

Uptime on 95% spot

$200K

Cloud spend trimmed

What I work in

kubernetes · terraform · aws · next.js · typescript · postgres

25 posts · 11 TILs · writing here since 2018

Production stories,written in the calm after.