Prometheus with Thanos ruler

TL;DR As a Kubernetes cluster grows, a single Prometheus cannot handle all scrape targets, so Prometheus is sharded into multiple instances. Once Prometheus is sharded, each shard only has a partial view of the metrics, so recording/alerting rules that need a global view no longer evaluate correctly on individual shards. Thanos Ruler evaluates PrometheusRules against Thanos Query, which provides a…