gpu-fleet
gpu-fleet is the software that turns racks of GPU servers into something tenants can reserve. It provisions nodes over Redfish, places reservations by fabric topology, proves to each tenant that the previous one left nothing behind, and pulls failing GPUs out of service on its own. Simulators stand in for the hardware, so the whole system runs on a laptop with one Docker Compose command.
The control plane is written in Go: a pure lifecycle state machine, lease-based provisioner workers that claim nodes with FOR UPDATE SKIP LOCKED, a leader-elected reservation reconciler and a REST API. Each node runs a small Rust agent that exposes dcgm-exporter compatible GPU metrics. Prometheus scrapes the agents, Alertmanager turns a fatal GPU XID into a webhook, and the control plane quarantines the node and swaps in a replacement for the tenant.
Tenant isolation is checked, not assumed. After every wipe the node is re-inspected: a nonce-bound, TPM-style quote and per-GPU reports are verified against reference values, and the signed result can be checked offline against a published Ed25519 key. A kernel implant left by a previous tenant survives a disk wipe, but not attestation, so the node is quarantined before anyone else can rent it.
The hosted demo on Railway is kept busy by a chaos service that books reservations, plants implants and injects GPU faults. In one recorded incident a GPU fell off the bus under a running tenant. The platform went from fault to quarantine in 88 seconds and had a replacement in the same pod 2 seconds later, with no operator involved. The public API is read-only by construction, so anyone can poke it with curl.
- Go for the control plane
- The control plane is mostly network I/O and concurrency: fanning out to hundreds of BMCs, talking to Postgres and serving an API. Goroutines fit that shape, and the infrastructure ecosystem (Kubernetes client-go, controller-runtime, Terraform SDK, Redfish libraries) is Go first.
- Rust for the node agent
- The agent runs on every GPU host next to latency-sensitive training jobs, so it must be small, predictable and hard to crash. Rust gives a single static binary with no GC pauses, and unsafe code is confined to the NVML FFI boundary and forbidden elsewhere by crate lints.
- Postgres as the concurrency engine
- Instead of a separate lock service, correctness lives in the database: row locks around every lifecycle event, SKIP LOCKED leases for workers, and an exclusion constraint that rejects overlapping reservations. Losing a placement race becomes a retry, not a bug.
- Prometheus and Alertmanager for remediation
- The agent speaks the dcgm-exporter metric format, so the same dashboards and alerts would work against real NVIDIA hardware. Alertmanager already handles grouping and retries, so remediation is just an authenticated webhook into the control plane.
- Simulators instead of hardware
- A Redfish BMC simulator, a fake GPU backend and a chaos service let the whole system run and be tested without GPUs. Each has a written contract, so swapping in real hardware means implementing an interface, not rewriting the platform.
- One repository, two languages
- Both sides change together while the API between them is still moving, and one CI pipeline keeps them consistent. The cost is two toolchains and an explicit contract, since types cannot be shared across Go and Rust.