HomelabFluxHelmReleaseNotReady¶
Use this runbook when the alert fires. Start with Triage (confirm it is real), then Diagnose, then Resolve. Document anything unique to your environment in the app-specific mk_runbook.md next to the HelmRelease if needed.
| Field | Value |
|---|---|
| Alert | HomelabFluxHelmReleaseNotReady |
| Severity | critical |
| Condition | gotk_resource_info{kind=HelmRelease, ready="False"} for 10m |
| Typical impact | App not installed/upgraded; workload may be missing or on old revision |
What this means¶
A Flux HelmRelease has been Not Ready for at least ten minutes. The chart install or upgrade failed, a dependency (HelmChart/HelmRepository) is broken, or values are invalid.
Triage¶
Triage (first 5 minutes)¶
- Acknowledge the alert (note time,
alertname, namespace/release from ntfy). - Check if something changed recently (Git push, chart bump, node drain, storage outage).
- Confirm the alert is still firing in Prometheus / Grafana (Alerting → Alert rules).
- Decide: transient (wait one reconcile interval) vs sustained (continue below).
# Recent events for the namespace (replace NAMESPACE)
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' | tail -20
Use labels from the alert: exported_namespace, name, chart, chart_version, chart_source.
Diagnose¶
Diagnose (Flux HelmRelease)¶
# Replace NAME and NAMESPACE from the alert
export NAME=helmrelease-name
export NAMESPACE=target-namespace
flux get helmrelease "$NAME" -n "$NAMESPACE"
kubectl describe helmrelease "$NAME" -n "$NAMESPACE"
kubectl get helmrelease "$NAME" -n "$NAMESPACE" -o yaml | less
# Helm install job / chart fetch errors
kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$NAME"
kubectl logs -n flux-system deploy/helm-controller --tail=80
Common causes
| Symptom | Likely cause |
|---|---|
| Chart not found | Wrong chart name/version or HelmRepository not ready |
| Install timeout | PVC pending, image pull, or resource limits |
| Upgrade failed | Values breaking upgrade; check helm release history |
| Stuck progressing | Stuck Helm hook or pre-upgrade job |
# Example from alert labels
flux get helmchart -A | rg -i false
flux get helmrepository -A | rg -i false
Resolve¶
Resolve (GitOps)¶
Homelab changes must go through Git — do not kubectl apply or patch live resources.
- Identify the manifest under
clusters/main/kubernetes/. - Fix chart version, values, dependencies, or suspend/resume as appropriate.
- Commit, push, and watch Flux:
- Wait for
Ready=Trueand confirm the alert clears (allowfor:duration + scrape interval).
Silence only if you intentionally suspended the release (spec.suspend: true) and expect Not Ready.
Escalation¶
Escalation / close-out¶
- Alert resolved in Alertmanager (or silenced with a documented reason and expiry).
- Root cause noted (link PR/commit if GitOps change).
- Update this runbook or the app
mk_runbook.mdif you learned something new.
If the issue is upstream (TrueCharts chart bug, Flux bug), capture logs and open an issue; avoid permanent silences without a ticket.