Skip to content

HomelabImmichframeAssetApiDown

Use this runbook when the alert fires. Start with Triage (confirm it is real), then Diagnose, then Resolve. Document anything unique to your environment in the app-specific mk_runbook.md next to the HelmRelease if needed.

Field Value
Alert HomelabImmichframeAssetApiDown
Severity warning
Team media
PrometheusRule homelab-immichframe.yaml

What this means

Blackbox cannot get HTTP 200 from ImmichFrame’s /api/Asset/RandomImageAndInfo for 5+ minutes. The frame’s / probe can still be healthy while album/image loading is broken (common after Immich major API changes).

Triage

Triage (first 5 minutes)

  • Acknowledge the alert (note time, alertname, namespace/release from ntfy).
  • Check if something changed recently (Git push, chart bump, node drain, storage outage).
  • Confirm the alert is still firing in Prometheus / Grafana (Alerting → Alert rules).
  • Decide: transient (wait one reconcile interval) vs sustained (continue below).
# Recent events for the namespace (replace NAMESPACE)
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' | tail -20
  1. Confirm versions:
kubectl get deploy -n media immichframe-app-template -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl get deploy -n media immich -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
flux get helmrelease immichframe -n media
  1. Check ImmichFrame logs for Immich API / deserialize errors (e.g. AlbumUserRole, owner):
kubectl logs -n media deploy/immichframe-app-template --tail=80
  1. Confirm the blackbox probe:
probe_success{target="homelab-immichframe-asset"}
probe_http_status_code{target="homelab-immichframe-asset"}

Diagnose

Diagnose (Flux HelmRelease)

# Replace NAME and NAMESPACE from the alert
export NAME=helmrelease-name
export NAMESPACE=target-namespace

flux get helmrelease "$NAME" -n "$NAMESPACE"
kubectl describe helmrelease "$NAME" -n "$NAMESPACE"
kubectl get helmrelease "$NAME" -n "$NAMESPACE" -o yaml | less

# Helm install job / chart fetch errors
kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/instance="$NAME"
kubectl logs -n flux-system deploy/helm-controller --tail=80

Common causes

Symptom Likely cause
Chart not found Wrong chart name/version or HelmRepository not ready
Install timeout PVC pending, image pull, or resource limits
Upgrade failed Values breaking upgrade; check helm release history
Stuck progressing Stuck Helm hook or pre-upgrade job

Typical causes:

  • ImmichFrame still on an Immich 2.x-era image (v1.0.33 and older) while Immich is v3+
  • Flux HelmRelease Stalled / rolled back after a failed upgrade (timeout during node upgrade)
  • Wrong ApiKey / Albums config, or Immich unreachable from the ImmichFrame pod
  • AuthenticationSecret mismatch (probe and clients get 401)

Resolve

Resolve (GitOps)

Homelab changes must go through Git — do not kubectl apply or patch live resources.

  1. Identify the manifest under clusters/main/kubernetes/.
  2. Fix chart version, values, dependencies, or suspend/resume as appropriate.
  3. Commit, push, and watch Flux:
flux get helmrelease -A | rg -i 'false|unknown'
flux logs -n flux-system --tail=30

  1. Wait for Ready=True and confirm the alert clears (allow for: duration + scrape interval).

  2. Keep ImmichFrame on v1.0.34+ when Immich is v3 (see ImmichFrame releases).

  3. If the HelmRelease is Stalled / RetriesExceeded, fix the image tag in Git, push, then:
flux reconcile helmrelease immichframe -n media --with-source

Escalation

Escalation / close-out

  • Alert resolved in Alertmanager (or silenced with a documented reason and expiry).
  • Root cause noted (link PR/commit if GitOps change).
  • Update this runbook or the app mk_runbook.md if you learned something new.

If the issue is upstream (TrueCharts chart bug, Flux bug), capture logs and open an issue; avoid permanent silences without a ticket.

Applies to

  • HelmRelease media/immichframe
  • Probe target homelab-immichframe-asset (blackbox-exporter)