Bare-metal GPU cloud control plane · Go + Rust

gpu-fleet

Software that turns racks of GPU servers into something tenants can reserve: it provisions nodes over Redfish, places reservations by fabric topology, proves to each tenant that the previous one left nothing behind, and pulls failing GPUs out of service on its own. A simulated fleet makes all of it run on a laptop or on the live demo below.

Live dashboard Live API Source on GitHub
Fleet floor · 50 nodes · 2 pods · 7 racks · illustration
available tenant workload inspecting / provisioning wiping quarantined
steady stateTenant acme holds 12 nodes in pod p2, placed best-fit so whole racks in p1 stay free for rack-scale requests. Press replay to watch the two incidents below play out.
Recorded from live runs

Two incidents, no operator

A GPU fails under a running tenant

  1. 12:29:08The node's agent reports XID 79 on GPU 3: the GPU fell off the bus.
  2. 12:29:21Prometheus sees gpu_agent_gpu_healthy == 0; GPUFatalXid goes pending.
  3. 12:30:36Alertmanager calls the authenticated webhook; the node gets hardware_fault and is quarantined.
  4. 12:30:38The reservation reconciler marks the reservation degraded and provisions a replacement in the same pod.

88 s from fault to quarantine · replacement 2 s later

A tenant leaves an implant behind

  1. activeWhile renting the node, the tenant modifies its kernel image. Disk wipes cannot remove this.
  2. releaseThe reservation ends; the node is wiped: wipe_succeeded.
  3. inspectRe-inspection boots the node and requests a nonce-bound TPM-style quote and GPU reports.
  4. verifyThe event log replays to the quoted PCRs, but the kernel digest is not in the reference values: attestation_failed.

quarantined before any other tenant can get it · signed result verifiable offline

Design

Decisions worth asking about

Pure lifecycle core, database as backstop

Node states are a pure state machine. Postgres CHECK constraints repeat its invariants, so a bug elsewhere cannot leave an active node without a tenant.

Leases with fencing

Workers claim nodes with FOR UPDATE SKIP LOCKED. A result is applied only while the worker still holds the lease, so a stalled worker cannot overwrite its successor.

No double booking, by constraint

An exclusion constraint on (node_id, period) rejects overlapping reservations in the database itself; losing a placement race becomes a retry, not a bug.

Attestation that fails closed

The verifier checks every source of evidence and reports every failure. A policy file that omits the confidential-computing requirement is rejected, not read as "off".

Remediation with a blast radius

Automatic quarantines are capped at 5 per 10 minutes, counted in the audit table under an advisory lock, so the cap holds across replicas and one noisy rule cannot empty the fleet.

Read-only public demo, by construction

The control plane serves the full API on a private listener and a GET-only copy on the public one. The public side has no route to a write path at all.

Architecture

How the pieces connect

flowchart TB
  agent["gpu-agent (Rust)
dcgm-exporter metrics"] -->|scrape| prom[Prometheus]
  prom -->|GPUFatalXid| am[Alertmanager]
  am -->|webhook| cp["control plane (Go)"]
  cp <--> pg[(Postgres)]
  cp -->|Redfish + in-band agent| bmc["bmcsim
50 simulated nodes"]
  cp -.->|reconciler| cp
  chaos[chaos service] -->|reservations, implants, XIDs| cp
  prom --> graf[Grafana]
Try it

Poke the live system

The public API is read-only. Every attestation result is signed, so you can check it against the published key without trusting the API.

Free capacity per pod
curl -s https://controlplane-production-6f4e.up.railway.app/v1/capacity
Automatic quarantines
curl -s https://controlplane-production-6f4e.up.railway.app/v1/remediations
A node's signed attestation result
curl -s https://controlplane-production-6f4e.up.railway.app/v1/nodes/node-001/attestation
Run everything locally
git clone https://github.com/Isidorsson/gpu-fleet
cd gpu-fleet
docker compose -f deploy/compose.yaml --profile demo up -d --build
Go 1.27Rust · axum · tokioPostgres · pgx · goose Prometheus · Alertmanager · GrafanaRedfishEd25519 · SHA-256 Docker · Kubernetes · RailwayGitHub Actions