Skip to content

HomelabVolsyncMetricsDown

Use this runbook when the alert fires. Start with Triage (confirm it is real), then Diagnose, then Resolve. Document anything unique to your environment in the app-specific mk_runbook.md next to the HelmRelease if needed.

Field Value
Alert HomelabVolsyncMetricsDown
Severity warning
Condition up{namespace="volsync"} == 0 or absent for 20m
Impact volsync_* backup alerts may not fire; operator may still run

What this means

Prometheus is not scraping the VolSync operator metrics Service (HTTPS on port 8443). Backup health rules in homelab-storage.yaml depend on metrics like volsync_missed_intervals_total and volsync_volume_out_of_sync. kube-state alerts (volsync-src jobs/pods) may still work.

Triage

Triage (first 5 minutes)

  • Acknowledge the alert (note time, alertname, namespace/release from ntfy).
  • Check if something changed recently (Git push, chart bump, node drain, storage outage).
  • Confirm the alert is still firing in Prometheus / Grafana (Alerting → Alert rules).
  • Decide: transient (wait one reconcile interval) vs sustained (continue below).
# Recent events for the namespace (replace NAMESPACE)
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' | tail -20

kubectl -n volsync get pods,svc,servicemonitor
kubectl -n observability get prometheusrules homelab-storage
# Prometheus UI → Status → Targets → filter volsync

Diagnose

# Operator pod healthy?
kubectl -n volsync get pods -l app.kubernetes.io/name=volsync
kubectl -n volsync logs deploy/volsync --tail=50

# ServiceMonitor (chart should enable metrics)
kubectl -n volsync get servicemonitor volsync -o yaml

# In-cluster scrape test
kubectl run curl-vs --rm -i --restart=Never --image=curlimages/curl:latest -- \
  curl -sk https://volsync-metrics.volsync.svc:8443/metrics | head

Common causes

  • VolSync pod crash loop or not ready after upgrade (system/volsync/app/helm-release.yaml).
  • ServiceMonitor label mismatch with Prometheus serviceMonitorSelector.
  • TLS / network policy blocking scrape from Prometheus namespace.
  • Prometheus or operator restarted; wait 20m before assuming sustained outage.

Resolve

  1. Fix the VolSync HelmRelease via Git (clusters/main/kubernetes/system/volsync/app/helm-release.yaml); let Flux reconcile.
  2. Confirm ServiceMonitor exists and target shows UP in Prometheus.
  3. If only metrics are broken but backups run, prioritize restoring scrape before relying on missed-interval alerts.

Resolve (GitOps)

Homelab changes must go through Git — do not kubectl apply or patch live resources.

  1. Identify the manifest under clusters/main/kubernetes/.
  2. Fix chart version, values, dependencies, or suspend/resume as appropriate.
  3. Commit, push, and watch Flux:
flux get helmrelease -A | rg -i 'false|unknown'
flux logs -n flux-system --tail=30
  1. Wait for Ready=True and confirm the alert clears (allow for: duration + scrape interval).

Escalation

Escalation / close-out

  • Alert resolved in Alertmanager (or silenced with a documented reason and expiry).
  • Root cause noted (link PR/commit if GitOps change).
  • Update this runbook or the app mk_runbook.md if you learned something new.

If the issue is upstream (TrueCharts chart bug, Flux bug), capture logs and open an issue; avoid permanent silences without a ticket.

While metrics are down, use kubectl get jobs,pods -A | rg volsync-src and Longhorn disk alerts as backup signals.