Infrastructure notes

Production stories,
written in the calm after.

I do platform work at a startup. This is where I write down the things I had to figure out the hard way, so future-me at 3am doesn't have to.

Read the latest →Browse TILs (11)

Currentlyantenna building

The workbench: SDR, Flipper Zero, Pwnagotchi, a Pi Zero, and an hourglass.

The workbench, in detail.

SDR, a Flipper, a Pwnagotchi, a few antennas I tuned in bad lighting, and an hourglass for measuring how long a deploy actually takes.

See the lair →

Latest

Browse all →

A network topology diagram titled 'rail-optimized fabric'. It shows four GPU nodes at the bottom, each with eight GPUs and eight NICs. The NICs are grouped into rails: NIC-0 of every node connects up to the same leaf switch, NIC-1 of every node to the next leaf switch, and so on across eight leaf switches. The leaf switches connect up to a row of spine switches forming a non-blocking fat-tree. An annotation notes 'GPU_i on any node reaches GPU_i on any other node in one switch hop'.

jul 9, 2026

Scaling GPUs past one box

Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.

$A diagram contrasting two eight-GPU server layouts. Top, labeled 'HGX / DGX baseboard': eight GPUs each connected to a bank of four NVSwitches so every GPU reaches every other at full 900 GB per second, drawn as a dense fully-connected mesh. Bottom, labeled 'PCIe box': eight GPUs hanging off PCIe switches and the CPU root complexes, GPU-to-GPU traffic forced through PCIe at a fraction of the bandwidth, drawn as a sparse tree routed through the CPUs.$

jul 4, 2026

One box, eight GPUs, and the wires between them

Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.

A vertical layer-stack diagram titled 'what a GPU pod actually sits on', showing twelve stacked layers from bottom to top: GPU silicon and NVSwitch, firmware (VBIOS and GSP), kernel driver nvidia.ko, userspace driver libcuda, CUDA runtime and cuDNN and NCCL, NVIDIA Container Toolkit, container runtime containerd, device plugin, node feature discovery, GPU Operator, scheduler, and the workload pod on top. Each layer has a short label describing its one job and a red note describing its characteristic failure.

jul 2, 2026

The dozen layers under a GPU pod

A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…

A newspaper-style poster titled The Daily Eight, listing the eight git aliases I run most: gst, glola, gd/gds, gcam, gpsup, gco/gcb, gpf, gfa, each paired with its expansion in mono caps.

may 20, 2026

The git commands I actually run every day

Ten years of git, distilled to the daily eight, an fzf branch picker, and the weekly pruning ritual.

The next post, in your inbox.

Long-form infra writing, every few weeks. No tracking pixels, no marketing sequences, no LinkedIn-isms.

Harshit Luthra. Senior SRE, infrequent essayist, occasional source of production incidents. More about me →

4M+

Daily active users

1M/min

Peak throughput

99.99%

Uptime on 95% spot

$200K

Cloud spend trimmed

What I work in

kubernetes · terraform · aws · next.js · typescript · postgres

24 posts · 11 TILs · writing here since 2018

Production stories,written in the calm after.