A working miniature of a bare-metal GPU cloud: four pods of 8-GPU servers, 1,024 GPUs in all. Racked hardware becomes customer capacity through a real provisioning pipeline: discovery over the BMC, firmware, network install and burn-in. Outside air follows real Lidköping weather and energy follows the real SE3 day-ahead price. Place customer orders, roll firmware behind a canary, then run the drills: trip the cooling, spike the price, storm the fleet with faults. Every button is an API call logged with its HTTP status, and an advisor explains what to do next.
Rows face each other across cold aisles and exhaust into hot aisles, where the haze follows each row's cooling load. Click a node to inspect it, or switch to 2D for the full rack grid and bulk provisioning.
Collecting samples. Let the clock run.
| racked | 32 | |
|---|---|---|
| discovering | 0 | |
| firmware | 0 | |
| imaging | 0 | |
| burnin | 0 | |
| available | 40 | |
| allocated | 56 | |
| draining | 0 | |
| maintenance | 0 | |
| rma | 0 |
Running a customer workload.
HereA typed state machine with timed steps that can fail into maintenance.
In productionDiscovery and power control over Redfish on each BMC, network boot (PXE/iPXE) into an image service, driven by a bare-metal manager such as MAAS, Ironic or Tinkerbell. Each step persisted and retryable in a workflow engine, so a crashed controller resumes instead of orphaning a node.
HereA four-hour stress step with a fixed failure rate.
In productionGPU diagnostics (DCGM level 3), NCCL all-reduce across the pod to prove the fabric, memory and InfiniBand link checks, with results stored per node so repeat offenders go to RMA.
HerePoisson faults scaled by GPU temperature, auto-drain and backfill.
In productionDCGM exporter and node exporter into Prometheus, XID and ECC events from kernel logs, and alert rules that call the same drain endpoint the operator button does.
HerePod-contiguous best fit with cooling and power admission.
In productionTopology read from the fabric manager, so jobs land on one non-blocking leaf group, with the same admission checks fed by live facility data instead of a model.
HereA pure reducer returning REST-shaped responses and audit events.
In productionThe same contract served by a Go or TypeScript service with an OpenAPI spec, idempotency keys on every mutation, per-tenant auth and an append-only audit table.
HereCanary, waves, and an automatic halt on burn-in failures.
In productionUpdates through Redfish UpdateService, gated by the same burn-in signal, with the halt wired to paging so a bad build stops at the canary without anyone watching.
HerePUE from outside air, lumped row cooling, a site power cap.
In productionLive data from the building and DCIM systems (cooling units and PDUs over Modbus or BACnet), and power capping enforced per node through the BMC or GPU power limits.
HereRule-based, same interface as the other control rooms.
In productionThe rules stay as guardrails, with an open-weights model served on the hall’s own GPUs drafting explanations and runbook steps from the same structured state.