Observe · Metrics & alerting
Prometheus
Prometheus scrapes metrics endpoints, stores time series locally and evaluates alerting rules sent to Alertmanager. It is the de facto metrics standard for Kubernetes via the kube-prometheus-stack.
On this page
What it does
Collects metrics by scraping /metrics endpoints, supports service discovery for Kubernetes, and drives alerts through Alertmanager.
Where it fits
Observe stage — the source for SLO burn-rate alerts, capacity signals and Grafana dashboards.
Quick start
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090
# PromQL: error ratio over 5 minutes
# sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))Common pitfalls
- High-cardinality labels (user IDs, request IDs) exhaust memory; review label sets before shipping.
- Alerts without runbook links get ignored during incidents.
- Long-term retention needs Thanos/Mimir or remote write; local storage is not a backup.
On this site
- Kubernetes Observability with Prometheus, Grafana and Loki: kube-prometheus-stack on kind, the PromQL for restarts, OOM kills and memory-versus-limit, and a
PrometheusRuleloaded through the operator.
Articles about Prometheus
Kubernetes Observability with Prometheus, Grafana and Loki
kube-prometheus-stack, Loki in single-binary mode and Alloy on a kind cluster; the PromQL and LogQL that answer operational questions; low-cardinality log labels; a real OOM restart loop diagnosed end to end.
Logging, Audit and Security Visibility: Who Did What, From Where
Application, infrastructure and audit logs and what each answers; Kubernetes API audit logging enabled on a real cluster, tuned from lease noise to the events that matter, and read for denied and privileged requests.