Logs answer "what happened to this one request". Metrics answer "how is the system doing in aggregate and how is that changing over time".
Two methods
- RED (Rate, Errors, Duration), for services that handle requests
- USE (Utilization, Saturation, Errors), for finite resources (CPU, memory, disk, file descriptors, goroutines, DB pool)
If you only have time for one in a service, it's RED. If you only have time for one on a host, it's USE.
Business metrics (signups, churn, share-clicks) are a third category. They live in your warehouse, not Prometheus. Different consumers, freshness, stakes.
The must-have
The single most useful counter for any web service: a count of HTTP requests broken down by method, path, and status. Everything else (error rates, latency percentiles, top endpoints) is derived from this.
In linko: middleware.go:22-28 declares it as a CounterVec, and middleware.go:72-85 increments it via a statusRecorder that wraps the response writer.
Prometheus is pull-based
Prometheus is a database for metrics, not a push endpoint. Your app exposes /metrics, Prometheus scrapes it on a schedule, stores the data, you query with PromQL. The app does not "send to" Prometheus.
To get data in you need two things:
- A scrape target, a
/metricsendpoint, or a host with Node Exporter running, or a Pushgateway for batch jobs - A scrape config, the
prometheus.ymlthat tells Prometheus where to scrape and how often
In linko: server.go:47 exposes /metrics via promhttp.Handler(), and prometheus.yml tells the Prometheus container where to find it. Node Exporter is in the same compose file for host metrics, see the Node Exporter bliki for the full setup. See the Prometheus configuration reference for the full scrape config options.
Build info as a label
Tie every metric to the build that produced it. git_sha as a label means a "regression in error rate" query can immediately show which build introduced it. In linko this lives next to the logger's With(...) (see Foundations), same GitSHA and BuildTime variables, two different consumers. See the Prometheus data model docs for the full label-naming rules and gotchas.
Visualizing
Grafana on top of Prometheus for the dashboards. Two patterns:
- Line chart for things-over-time (error rate, request rate, latency p95)
- Bar chart for things-across-categories (HTTP status code distribution)
The "Status Code Bars" lesson makes this concrete: a bar chart grouped by status code is a faster read of "is this healthy" than a line chart of the same data.