The RelOps Fleet Dashboard — one place to see everything, fix anything, and lose no more sleep over mystery workers.

Hangar pulls data from Taskcluster, SimpleMDM, Puppet, and Google Sheets and stitches it into a single live view of your entire test infrastructure. Pool health, hardware generations, task failures, quarantined machines, missing workers — all in one dark-themed dashboard. No more tab soup.

Currently tracking Mozilla's CI fleet. Built to grow with the rest of the infrastructure.

🛠️ Stack

Layer	Technology
Frontend	React 19 + TypeScript, Vite, Tailwind CSS, Recharts, TanStack Table
Backend	FastAPI (Python 3.11), SQLAlchemy 2, APScheduler
Database	PostgreSQL 16
Infrastructure	GCP (Cloud Run, Cloud SQL, Cloud Load Balancing, IAP, Cloud Build)

🌐 Production

Resource	Value
URL	https://hangar.relops.mozilla.com
GCP Project	`relops-dashboard`
Cloud Run	`hangar` (us-central1)
Auth	Google IAP — `@mozilla.com` accounts only, no GCP access required
CI/CD	Cloud Build triggers on push to `main`

🚀 Quick Start (local dev)

# 1. Configure environment
cp .env.example .env
# edit .env — see Environment Variables below
# 2. Start the database
docker compose up -d db
# 3. Build frontend
cd frontend && npm install && npm run build && cd ..
# 4. Start backend (serves built frontend at /)
docker compose up backend

Open http://localhost:8000.

Frontend dev (hot reload)

cd frontend
npm install
npm run dev        # http://localhost:5173 → proxies API to :8000

Backend dev (no Docker)

cd backend
pip install -r requirements.txt
export DATABASE_URL=postgresql://relops:relops@localhost:5432/relops
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
# requires: docker compose up -d db

🔑 Environment Variables

All variables are optional unless marked required.

Variable	Default	Description
`DATABASE_URL`	`postgresql://relops:relops@localhost:5432/relops`	Required. PostgreSQL DSN
`SIMPLEMDM_API_KEY`	—	SimpleMDM REST API key
`TC_ROOT_URL`	`https://firefox-ci-tc.services.mozilla.com`	Taskcluster root URL
`TC_CLIENT_ID`	—	TC client ID (public pools work unauthenticated)
`TC_ACCESS_TOKEN`	—	TC access token
`GOOGLE_SHEETS_ID`	—	Master inventory spreadsheet ID
`GOOGLE_EXPORT_SHEET_ID`	—	Export/dashboard spreadsheet ID
`GOOGLE_CREDENTIALS_JSON`	—	Path to service account JSON
`PUPPET_REPO_URL`	`https://github.com/mozilla-platform-ops/ronin_puppet`	Puppet repo
`PUPPET_REPO_PATH`	`/tmp/ronin_puppet`	Local clone location
`SYNC_INTERVAL_TC`	`300`	Taskcluster sync interval (seconds)
`SYNC_INTERVAL_SIMPLEMDM`	`900`	SimpleMDM sync interval
`SYNC_INTERVAL_SHEETS`	`1800`	Google Sheets sync interval
`SYNC_INTERVAL_PUPPET`	`3600`	Puppet sync interval
`TC_MISSING_THRESHOLD_HOURS`	`24`	Hours before raising a `missing_from_tc` alert

🏗️ Architecture

┌─────────────────────────────────────────────────────┐
│                   React SPA (Vite)                  │
│  Overview · Workers · Alerts · Pools · Tart VMs      │
└──────────────────────┬──────────────────────────────┘
                       │ REST
┌──────────────────────▼──────────────────────────────┐
│                  FastAPI backend                     │
│  /api/workers /api/fleet /api/alerts /api/tart-health│
│                                                     │
│  APScheduler ─── sync/taskcluster.py   (5 min)      │
│               ├── sync/simplemdm.py    (15 min)      │
│               ├── sync/google_sheets.py(30 min)      │
│               └── sync/puppet.py       (60 min)      │
└──────────────────────┬──────────────────────────────┘
                       │ SQLAlchemy
┌──────────────────────▼──────────────────────────────┐
│              PostgreSQL 16                          │
│  workers · alerts · sync_logs · failure_events      │
│  tart_slot_health (pushed by the on-network agent)  │
└─────────────────────────────────────────────────────┘

GCP architecture

User (mozilla.com Google account)
  │
  ▼
Global Load Balancer (34.54.129.77)
  │  Cloud Armor (OWASP rules + rate limiting)
  │  Identity-Aware Proxy (IAP) — google auth gate
  ▼
Cloud Run — hangar (us-central1, min 1 instance)
  │  INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER (direct URL blocked)
  ▼
Cloud SQL — Postgres 16 (private IP via VPC connector)

Data sources

Source	What it provides	Sync
Taskcluster	Worker state, quarantine status, last active, latest task	GraphQL + REST
SimpleMDM	MDM enrollment, OS version, serial number, custom attributes	REST (paginated)
Puppet	Worker role, pool assignment	Git clone of `ronin_puppet`
Google Sheets	Canonical state (production / loaner / defective / spare / staging), notes	Sheets API v4

📺 Pages

🗺️ Overview

Your morning briefing. Fleet-wide summary stats, sync health, top-10 failing machines and tests over the last 7 days, generation breakdown chart, workers-by-pool bar chart.

🖥️ Workers

The full roster. Filterable and sortable across generation, state, pool, and MDM/TC status. Search by hostname or serial number. Up to 2,000 rows.

🔍 Worker Detail

Everything Hangar knows about a single worker — Puppet role, sheet state, MDM enrollment, TC history. Edit notes and jump out to Taskcluster.

🚨 Alerts

The stuff that needs your attention. Types: missing_from_tc, quarantined, mdm_unenrolled, pool_mismatch. Add notes, acknowledge, resolve.

🏊 Pool Health

Per-pool health scores, staleness breakdowns (active <24 h / 1–7 d / 7–30 d / 30 d+ / never seen), and job source distribution. Branch-override visibility (counts and which pools/branches are pinned).

📦 Consolidation

Side-by-side hardware generation comparison — state breakdowns, inactive machines, and retirement candidates.

🖲️ Tart VMs

Per-slot health for the tart VM hosts behind gecko-t-osx-1500-m-vms (macOS → Tart VMs in the sidebar).

The unit is the slot, not the host: each host runs two VMs that fail independently, and Taskcluster's view — which is what Pool Health shows — can't see inside one. In July 2026 five of 26 slots were out of production for weeks because three guests were crash-rebooting every ~84 s while tart run on the host stayed up 11+ days, so every host-level signal read green.

Columns cover the worker id, TC registration, tart run uptime, whether credentials are host-injected or image-baked, and guest disk headroom; rows expand to the problem list plus VM state, clock skew, configured-vs-actual identity, puppet SHA and cert expiry.

Two behaviours worth knowing:

Stale reads as unknown, never ok. Rows older than 30 minutes are reported stale — a wedged collector must not leave the fleet looking green.
Cert expiry is judged against the cert's own lifetime, not a fixed number of days. These hosts hold 168 h certs, so "expires within 7 days" is the steady state; a warning means the renew daemon has actually missed its window.

Data arrives from an on-network agent, not from Hangar — see Tart VM slot health below.

🔁 Reprovision — one-click EACS → prod

The headline capability: wipe, re-enroll, and fully re-provision a Mac worker end-to-end from a button in the dashboard — no SSH, no console, no pasted secrets.

On a worker's detail page, authorized operators (IAP allowlist) get a Reprovision cockpit: live readiness (quarantined? idle? M4/EACS-supported?), an Execute button behind a confirm, and a live Apple-rainbow terminal timeline that streams every phase of the run.

How it reaches the datacenter safely. Hangar runs on Cloud Run and can't SSH into MDC1 — so it never does:

Execute enqueues a ReprovisionJob (one open job per host).
An on-network runner — Puppet-managed on an MDC1 mini — polls Hangar over mTLS (outbound only, no inbound to the datacenter), claims the job, and runs the reprovision CLI.
Each phase streams back as a job event and renders live: quarantine → drain → EACS wipe → DEP re-enroll → SecureToken mint → Bootstrap Token → self-provision (mTLS vault · puppet · Taskcluster).

The runner holds all SSH/admin creds; Hangar holds none. A server-side reaper closes any job whose runner vanishes, so a lost run never wedges a host.

Proven in prod: a Hangar-enqueued job reprovisioned macmini-m4-80 end-to-end, driven by the Puppet-managed runner on macmini-m4-81, streaming the whole flow into the cockpit.

🖲️ Tart VM slot health

Same architectural constraint as Reprovision: Cloud Run can't reach MDC1, so Hangar never SSHes a tart host. An on-network agent does the collecting and pushes results in.

macmini-m4-81 (Puppet-managed)                      Hangar
  com.mozilla.hangar-tart-health-agent
    every 600s:
      ssh admin@<13 tart hosts>  ──collect──▶
      POST /api/tart-health/agent/push   ─────▶  require_runner (mTLS client cert)
                                                   ↓
                                                 evaluate() → ok/warn/crit/unknown
                                                   ↓
  browser ──▶ GET /api/tart-health  ─────────▶  require_access (IAP)

The agent is hangar-tart-health-agent in relops-bootstrap, the third daemon on the reprovision runner host, reusing its venv, env file and mTLS cert.

Severity is derived server-side, in evaluate(), not by the agent. The agent is a dumb collector, so thresholds can be retuned by deploying Hangar alone — nothing on-network changes — and the API and UI can't disagree about what crit means.

Routing is load-bearing

Only /api/tart-health/agent/* is routed to the non-IAP mTLS backend (terraform/lb_runner.tf). GET /api/tart-health deliberately stays on the IAP frontend.

require_runner does no per-path scoping, so that URL map is the only thing keeping the read rollup behind human auth. Routing a broad /api/tart-health/* to the runner backend as a convenience would expose fleet data to anything inside runner_source_cidrs holding a valid cert, with no Google identity attached.

Diagnostic, if a push ever stops landing: on the runner hostname a path that reaches the right backend returns 403 (Cloud Armor source deny from anywhere but MDC1). A 301 to the IAP domain means the path isn't in the allowlist at all.

Known gap

tart_health_guests is currently false, so guest-level checks — disk headroom, clock skew, worker identity — are not collected and those columns are blank. Those are the checks that catch a crash-looping guest inside a healthy host, i.e. the July 2026 failure mode. Until it's enabled, treat a green Tart VMs page as "the hosts look fine", not "the fleet is fine".

🗄️ Database Schema

workers          — one row per hostname, columns from all four sources
alerts           — active/resolved per-worker alerts
sync_logs        — audit trail for each sync run (source, duration, records updated, errors)
failure_events   — TC task failures indexed by hostname and task name
tart_slot_health — latest health row per tart VM slot (PK hostname+slot), pushed by the agent

Worker state precedence: sheet_state (if set) → inferred from TC/Puppet membership → unknown

Health score: fraction of production workers that are MDM-enrolled, not quarantined, and active within 24 hours.

📡 API Reference

GET    /api/workers                      list + filter + search + sort
GET    /api/workers/{hostname}           full worker record
PATCH  /api/workers/{hostname}/notes     update dashboard notes
GET    /api/fleet/summary               dashboard stats
GET    /api/fleet/pools                 per-pool health
GET    /api/fleet/pending-counts        TC pending tasks per pool
GET    /api/fleet/pool-sources          running task project breakdown
GET    /api/fleet/failures?days=7       top failing machines + tests
GET    /api/fleet/consolidation         hardware generation analysis
GET    /api/alerts                      list (filter: type, active_only)
PATCH  /api/alerts/{id}/acknowledge
PATCH  /api/alerts/{id}/resolve
GET    /api/prs/ronin                   ronin_puppet PR queue + voting
POST   /api/prs/ronin/{n}/upvote
POST   /api/prs/ronin/{n}/downvote
GET    /api/tart-health                 tart VM slot rollup, worst-first (IAP)
GET    /api/tart-health/{hostname}      one host's slots (IAP)
POST   /api/tart-health/agent/push      on-network agent push (mTLS runner cert)
POST   /api/sync/run                    trigger manual sync
GET    /api/health                      liveness check

📁 Project Structure

hangar/
├── backend/
│   ├── app/
│   │   ├── main.py              FastAPI app, lifespan, route registration
│   │   ├── config.py            Pydantic Settings — all env vars
│   │   ├── database.py          SQLAlchemy engine + table init
│   │   ├── models.py            ORM models
│   │   ├── api/
│   │   │   ├── workers.py
│   │   │   ├── fleet.py
│   │   │   ├── alerts.py
│   │   │   ├── tart_health.py   tart slot rollup + agent push, severity logic
│   │   │   └── prs.py           ronin_puppet PR voting
│   │   └── sync/
│   │       ├── scheduler.py     APScheduler job registration
│   │       ├── taskcluster.py   GraphQL + REST sync, alert generation
│   │       ├── simplemdm.py     Paginated REST sync + custom attributes
│   │       ├── puppet.py        Git clone + inventory.d parse
│   │       └── google_sheets.py Sheets API v4 read
│   ├── Dockerfile
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.tsx              React Router setup
│   │   ├── api.ts               Typed API client + all TS interfaces
│   │   ├── components/
│   │   │   ├── Layout.tsx       App shell — sidebar nav (collapses to hamburger drawer on mobile)
│   │   │   ├── CommandPalette.tsx  ⌘K quick search
│   │   │   └── KeyboardShortcuts.tsx
│   │   └── pages/
│   │       ├── Overview.tsx
│   │       ├── Workers.tsx
│   │       ├── WorkerDetail.tsx
│   │       ├── Alerts.tsx
│   │       ├── Pools.tsx
│   │       ├── TartVMs.tsx      per-slot tart VM health
│   │       └── Consolidation.tsx
│   ├── vite.config.ts           Dev proxy → :8000
│   └── package.json
├── terraform/                   All GCP infrastructure
│   ├── main.tf                  Provider, APIs, VPC, VPC connector
│   ├── run.tf                   Cloud Run service
│   ├── lb.tf                    Load balancer, IAP, Cloud Armor, SSL cert
│   ├── lb_runner.tf             mTLS frontend — runner/agent path allowlist
│   ├── iam.tf                   Service accounts, IAM bindings, IAP access
│   ├── sql.tf                   Cloud SQL Postgres 16
│   ├── secrets.tf               Secret Manager secrets
│   ├── variables.tf             All configurable vars
│   └── terraform.tfvars         Local var overrides (gitignored)
├── cloudbuild.yaml              CI/CD — triggers on push to main
├── docker-compose.yml           Local dev (postgres + backend)
└── .env.example                 Template for local dev

⌨️ Keyboard Shortcuts

Shortcut	Action
`⌘ K`	Open command palette (jump to any worker)
`⌘ /`	Open keyboard shortcuts help
`Esc`	Close any modal