ALERT_NAME_HERE¶

Template

Copy this file to mk_runbook_<alert-kebab-name>.md (see naming below), fill in the tables, then add runbook_url to the PrometheusRule. Delete this note block when done.

Use this runbook when the alert fires. Start with Triage (confirm it is real), then Diagnose, then Resolve. Document anything unique to your environment in the app-specific mk_runbook.md next to the HelmRelease if needed.

Field	Value
Alert	`ALERT_NAME_HERE`
Severity	`warning` / `critical`
Team	`platform`
PrometheusRule	`homelab-*.yaml` (file name)

What this means¶

One or two sentences: what condition PromQL detects and what breaks for users.

Triage¶

Triage (first 5 minutes)¶

Acknowledge the alert (note time, alertname, namespace/release from ntfy).
Check if something changed recently (Git push, chart bump, node drain, storage outage).
Confirm the alert is still firing in Prometheus / Grafana (Alerting → Alert rules).
Decide: transient (wait one reconcile interval) vs sustained (continue below).

# Recent events for the namespace (replace NAMESPACE)
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' | tail -20

Diagnose¶

Diagnose (Flux HelmRelease)¶

# Replace NAME and NAMESPACE from the alert
export NAME=helmrelease-name
export NAMESPACE=target-namespace

flux get helmrelease "$NAME" -n "$NAMESPACE"
kubectl describe helmrelease "$NAME" -n "$NAMESPACE"
kubectl get helmrelease "$NAME" -n "$NAMESPACE" -o yaml | less

# Helm install job / chart fetch errors
kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$NAME"
kubectl logs -n flux-system deploy/helm-controller --tail=80

Common causes

Symptom	Likely cause
Chart not found	Wrong chart name/version or HelmRepository not ready
Install timeout	PVC pending, image pull, or resource limits
Upgrade failed	Values breaking upgrade; check `helm release` history
Stuck progressing	Stuck Helm hook or pre-upgrade job

# App-specific commands

Resolve¶

Resolve (GitOps)¶

Homelab changes must go through Git — do not kubectl apply or patch live resources.

Identify the manifest under clusters/main/kubernetes/.
Fix chart version, values, dependencies, or suspend/resume as appropriate.
Commit, push, and watch Flux:

flux get helmrelease -A | rg -i 'false|unknown'
flux logs -n flux-system --tail=30

Wait for Ready=True and confirm the alert clears (allow for: duration + scrape interval).

Escalation¶

Escalation / close-out¶

Alert resolved in Alertmanager (or silenced with a documented reason and expiry).
Root cause noted (link PR/commit if GitOps change).
Update this runbook or the app mk_runbook.md if you learned something new.

If the issue is upstream (TrueCharts chart bug, Flux bug), capture logs and open an issue; avoid permanent silences without a ticket.

Applies to¶

Document which services this runbook is for (must match front matter above). With matching releases:, the runbook appears as a sub-page under that chart in the sidebar (below the auto-generated HelmRelease page). Platform-wide alerts use scope: all-helmreleases and stay at the area level.

Binding	Example
`releases`	`downloaders/nzbget` → only that release’s doc
`areas`	`downloaders` → every release under `my-apps/downloaders/`
`charts`	`nzbget` → any release installing chart `nzbget`
`scope: all-helmreleases`	Listed under “Platform alert runbooks” on every chart page

Per-service on-call steps: add docs/homelab/kubernetes/my-apps/<workload>/app/mk_runbook.md (shows under “Service documentation” on that chart’s page).

ntfy buttons (alertmanager-ntfy)¶

Homelab ntfy notifications expose three view actions when configured: Runbook, Alert (Grafana), Dashboard (only if dashboard_url is set on the alert). Tapping the notification opens the homelab-alerts topic in the ntfy app.

Naming and ntfy link¶

Item	Convention
Filename	`mk_runbook_<slug>.md` — run the helper first; use the path it prints
Annotation	`runbook_url: https://nerd.dad/latest/homelab/kubernetes/my-apps/observability/runbooks/mk_runbook_.../`
URL helper	Resolves `alertname` / `alertnames` from front matter — not only kebab-guessing
Published	Last updated (Git) on the page; Site build info for the nginx image date

annotations:
  runbook_url: https://nerd.dad/latest/homelab/kubernetes/my-apps/observability/runbooks/mk_runbook_ALERT-KEBAB-HERE/