db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion

- Migrations 010 (content_hash) + 011 (search tsvector column) - new: internal/knowledge/seed.go — knowledge seed ingest engine - new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge - wire knowledge ingest into oikos seed pipeline - convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml - archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/ - delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py - remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00
parent b2bfa26f64
commit 6b75f7302d
125 changed files with 7557 additions and 1938 deletions
--- a/archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md
+++ b/archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md
@@ -0,0 +1,89 @@
+# 2026-04-21 — Hubris crash loop (thermal + USB drive)
+
+## Summary
+
+[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
+
+## Timeline
+
+### 2026-04-19 — drive attached
+External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
+
+### 2026-04-19 → 2026-04-21 — first crashes
+Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
+
+### 2026-04-21 — crash cadence escalates
+23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`.
+
+### 2026-04-21 — first set of mitigations applied
+
+- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
+- **Crash capture**: `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`).
+- **`rasdaemon`** installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query `ras-mc-ctl --summary`.
+- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge.
+- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
+
+### 2026-04-22 — recurrence after 30 h 37 m
+Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
+
+### 2026-04-22 — `cpu-epp.service` design bug fixed
+Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
+
+### 2026-04-22 — drive removed (A/B test)
+User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
+
+### 2026-04-23 — SSD cooling + thermal pads installed
+Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
+
+#### 2026-04-23 thermal-pad verdict
+Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold.
+
+### 2026-04-28 — A/B test passing so far
+3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.
+
+### 2026-04-29 — physical relocation
+User moved the host to a spot with better airflow. Post-move idle baseline (45 min uptime, light load): Tctl **47.2 °C**, GPU 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C. Notably nvme0 sensor1 is **27 °C cooler** than the 2026-04-23 thermal-pad steady-state of 60–61 °C. Watch lifetime warning-time counters (`nvme smart-log` → "Warning Comp. Temperature Time") over the next week — flat trend confirms the move helped. Doesn't change the drive-removal A/B test, but reduces residual thermal headroom risk independently.
+
+## Root cause
+
+Two-contributor:
+1. **Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them.
+2. **Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).
+
+Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.
+
+## Diagnostic notes
+
+- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
+- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`.
+- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`.
+- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
+- `mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement.
+
+## BIOS update path
+
+Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.
+
+## Mitigations applied
+
+| Mitigation                                | Status   | Notes                                                |
+| ----------------------------------------- | -------- | ---------------------------------------------------- |
+| `cpu-epp.service` (powersave + balance_power) | Active   | Unit ordering fixed 2026-04-22 to run before guests |
+| Crash capture (sysctl + softdog + watchdog) | Active   | Catches everything but pure silicon hangs           |
+| `rasdaemon`                               | Active   | Persistent telemetry for MCE / memory / PCIe AER    |
+| UAS blacklist for `090c:2320`             | Active (drive currently absent) | `/etc/modprobe.d/usb-storage-quirks.conf` |
+| Mount-on-demand (`backup-usb.sh`)         | Active (drive currently absent) | Backup unit `ExecStartPre`/`ExecStopPost` hooks |
+| Drive physically removed                  | Active   | A/B test in progress                                 |
+| Thermal pads on NVMes                     | Active 2026-04-23 | Steady-state composite 47 °C / sensor1 60 °C |
+| `pcie_aspm=off pci=nomsi`                 | NOT applied | Reserved for if crashes recur without the drive    |
+
+## Affected nodes
+- [Hubris host](../../../wiki/hosts/hubris.md)
+- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
+- [Monitoring](../../../wiki/infrastructure/monitoring.md)
+
+## Open questions
+- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
+- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
+- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
+- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.
--- a/archive/knowledge/investigations/2026-05-31-authentik-vps-migration.md
+++ b/archive/knowledge/investigations/2026-05-31-authentik-vps-migration.md
@@ -0,0 +1,109 @@
+# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
+
+## Summary
+
+The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
+
+Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
+
+The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
+
+## Timeline
+
+### Trigger
+`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
+```
+Error: failed reading provided config file: /etc/netbird/management.json:
+OIDC configuration request returned status 504 with response: Gateway Timeout
+```
+The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
+
+### Fix — Phase A (restore the mesh)
+1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
+2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
+3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
+4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
+
+### Fix — Phase B (full data migration)
+Once the mesh was back, the homelab DB was reachable:
+1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
+2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
+3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
+4. Restarted Authentik. All users / apps / passwords / groups present.
+
+Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
+
+### Fix — DNS cutover (the long tail)
+The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
+
+- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
+- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
+
+## Issues hit during the migration (and fixes)
+
+1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
+2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
+3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
+4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
+5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
+
+## Architecture delta
+
+| | Before | After |
+|---|---|---|
+| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
+| Version | `2026.2.2` | `2026.5.2` |
+| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
+| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
+| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
+| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
+| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
+
+> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
+
+## Lessons learned
+
+1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
+2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
+3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
+4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
+5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
+6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
+7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
+8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
+
+## Forward-auth outpost — RESOLVED 2026-06-01
+
+Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
+
+Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
+
+### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
+
+The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
+
+**Fix:** Deleted from Postgres:
+```sql
+DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
+DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
+DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';
+```
+The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
+
+**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
+
+## Outstanding
+
+- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
+- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
+- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
+- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
+- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
+- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
+- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
+
+## Related
+- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
+- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
+- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
+- [Mesh migration](../../../wiki/infrastructure/mesh.md)
--- a/archive/knowledge/investigations/2026-06-01-mac-mini-onboarding.md
+++ b/archive/knowledge/investigations/2026-06-01-mac-mini-onboarding.md
@@ -0,0 +1,191 @@
+# mac-mini onboarding — post-mortem & lessons learned
+
+Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
+homelab context system with the `--with-hermes` profile. What follows is a
+chronological recap of every hitch, the fix, and the systemic improvements
+needed so the next workstation takes 5 min instead of an hour.
+
+## Session log
+
+### Step 1 — clone + symlink
+- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
+- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink.
+- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
+  already cloned by hand. The bootstrap's `clone exists; pulling` path handled
+  it gracefully.
+
+### Step 2 — hostname mismatch
+- `scutil --get LocalHostName` → `Davids-Mac-mini`
+- `hostname -s` → `Mac`
+- Inventory file: `hosts/mac-mini.yaml`
+- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
+- **Lesson:** The bootstrap and `homelab whoami` use different hostname
+  resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
+  but the `homelab` CLI binary uses `hostname -s`. Both need to match the
+  inventory key. On a fresh macOS machine, neither does.
+
+### Step 3 — bootstrap dependencies
+- pyyaml was missing → `pip install pyyaml`
+- age and sops were missing → `brew install age sops`
+- Netbird was already installed and connected ✓
+- **Lesson:** The bootstrap preflight handles these, but only if running
+  `bootstrap.sh` from the start. Since we ran it after manual setup, some
+  steps (netbird install) were correctly skipped as already-present.
+
+### Step 4 — full bootstrap with `--with-mcp --with-hermes`
+- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
+- Age key issued ✓
+- Launchd sync timer installed ✓
+- Goose binary installed ✓
+- Hermes CLI linked ✓
+- MCP config merged ✓
+- `refresh-creds` skipped (not yet a recipient) ⚠️
+- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
+  group-naming quirk)
+
+### Step 5 — finalize from hubris
+- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
+  hubris
+- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
+  stale, bootstrap had already pushed from mac-mini
+- **Fix:** `git pull --rebase && git push` on hubris
+- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
+  creating a race. The `homelab client add --finalize-pubkey` command should
+  pull before committing/pushing.
+
+### Step 6 — sops couldn't find the age key
+- `homelab secret hello` failed because sops looks in
+  `/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
+- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
+  but this requires passwordless sudo and the correct env var passthrough
+- **Fix:**
+  1. Added NOPASSWD sudo rules
+  2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
+- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
+  agent-enrollment troubleshooting. New clients can't assume `homelab secret`
+  works out of the gate — the sudo re-exec chain has permission pitfall.
+
+### Step 7 — OpenRouter key was a placeholder
+- `secrets/openrouter-api-key.yaml` contained
+  `api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
+- User ran `sops` on hubris, but got the same error (age key not found on
+  hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
+- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
+  Later: the user pasted the real key, but the sops file showed
+  `sk-or-...5c55` — the literal content was truncated with ellipsis.
+
+### Step 8 — editor loaded the wrong data
+- Neovim on the system is configured with `clipboard+=unnamedplus`, which
+  points `*` and `+` registers to the macOS clipboard manager rather than
+  X11. When editing SOPS files, this caused the **system clipboard** to be
+  pasted instead of the actual ciphertext.
+- This wasn't diagnosed during the session — the sops file would load empty
+  or show the wrong content because the editor's idea of "paste" was
+  disconnected from what sops expected.
+- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
+  hijack OS clipboards:
+  ```bash
+  EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
+  ```
+- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
+  macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
+  because the paste register reads from the system clipboard instead of the
+  sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
+  interactively. Alternatively, override the clipboard option with
+  `EDITOR='nvim -c "set clipboard="'`.
+- Also useful for the troubleshooting table in `agent-enrollment.md` under a
+  new row: "sops file loads empty / wrong content on macOS"
+
+### Step 9 — model doesn't support tool use
+- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
+- Error: `No endpoints found that support tool use`
+- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
+  `~/.config/goose/config.yaml`
+- Also updated `operations/hermes-agent.md` with the correct model
+- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
+  never validated on OpenRouter for tool-use capability. Need to either:
+  (a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
+  (b) Or have the bootstrap probe OpenRouter at install time
+
+### Step 10 — MCP homelab extension returns 404
+- Goose's homelab extension configured as `streamable_http` at
+  `https://mcp.hubris.network/mcp` returns HTTP 404
+- The actual MCP server runs on SSE (`/sse`), not streamable_http
+- **Known follow-up** from `hermes-agent.md` — server needs migration
+- **Workaround:** The agent works fine via the built-in `developer` extension
+  (shell + file) and the `homelab` CLI
+
+### Step 11 — git credential dance
+Several layers of issues with git authentication:
+
+1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
+   couldn't access the credential file, and git's `store` helper couldn't
+   create its `.lock` file.
+   - **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
+   - **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`
+
+2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
+   as a default credential helper. Even after configuring the `store` helper,
+   git calls osxkeychain after every successful `get`, which prompts for
+   macOS keychain access.
+   - `credential.helper = ""` does NOT disable the compiled-in default
+   - The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
+     silently, causing fallback behaviour
+   - **Fix:** Embed the credential in the remote URL directly:
+     `https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
+   - This bypasses all credential helpers entirely
+
+3. **refresh-creds:** The credential is now managed by the remote URL.
+   If the PAT is rotated, `homelab refresh-creds` won't update the URL.
+   - **Workaround:** Run `git remote set-url origin` after `refresh-creds`
+
+### Step 12 — `homelab` CLI hostname detection
+- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
+  `hostname -s` (returning `Mac`) while the bootstrap uses
+  `scutil --get LocalHostName` (which we fixed to `mac-mini`)
+- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
+- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
+  LocalHostName` like the bootstrap does, or at least try both and pick the
+  one that matches an inventory key. This is a CLI bug.
+
+## Improvement backlog
+
+### High priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
+| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
+| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
+| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
+| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
+
+### Medium priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
+| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
+| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
+| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
+
+### Low priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
+| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
+| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
+
+## Changelog
+
+### 2026-06-01 — initial post-mortem
+Captured the full onboarding of mac-mini into the hubris homelab context
+system, including the hostname mismatch, sudo/sops credential chain, model
+selection, git credential gymnastics, and editor clipboard interference.
+
+### 2026-06-02 — MCP server migrated to streamable-http
+Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
+not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
+and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
+re-deployed on apps LXC (105).
--- a/archive/knowledge/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md
+++ b/archive/knowledge/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md
@@ -0,0 +1,84 @@
+# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
+
+## Summary
+
+[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
+
+## Timeline
+
+### Pre-existing state
+- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
+- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
+- Both on the same subnet via a consumer router at 192.168.178.1
+- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
+- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
+
+### 2026-06-03 — investigation
+**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
+- TCP throughput: 49–86 Mbps, average ~65 Mbps — highly variable
+- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
+- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
+- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
+
+**Additional findings:**
+- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
+- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
+- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
+- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
+
+## Root cause
+
+mac-mini is on WiFi. WiFi introduces:
+1. **Throughput variance** — 49–86 Mbps TCP fluctuation
+2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
+3. **Jitter** — 28 ms stddev with spikes to 138 ms
+
+Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
+
+## Mitigations applied
+
+### mac-mini — Moonlight (via `defaults write`)
+
+| Setting | Before | After | Rationale |
+||---------|--------|-------|-----------|
+|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
+|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
+|| fps | 30 | 60 | Restored to game-appropriate value |
+
+### mac-mini — Moonlight (current, wired — 2026-06-04)
+
+| Setting | WiFi value | Wired value | Rationale |
+||---------|-----------|-------------|-----------|
+|| bitrate | 60 Mbps | 80 Mbps | Wired 1 GbE can sustain comfortably |
+|| framepacing | on (1) | off (0) | Wired latency is consistent, no smoothing needed |
+|| fps | 60 | 60 | Unchanged |
+
+### ludo-mini — Sunshine (via SSH)
+
+| Setting | WiFi value | Wired value | Rationale |
+||---------|-----------|-------------|-----------|
+|| bitrate | 40 Mbps | 80 Mbps | Wired can handle full bandwidth |
+|| fec_percentage | 2 | 5 | Restored to default — wired has no dropout concern |
+|| packetsize | 1024 | 1316 | Restored to default (Ethernet MTU) |
+
+Sunshine restarted after changes; confirmed active.
+
+## Resolution
+
+### 2026-06-04 — mac-mini wired to LAN
+
+mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
+
+## Open questions
+
+~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~ **RESOLVED** — mac-mini now wired via en0.
+2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
+3. **Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
+
+## Changelog
+
+### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
+- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
+- Moonlight: bitrate 60→80 Mbps, framepacing disabled
+- Sunshine: bitrate 40→80 Mbps, fec_percentage 2→5, packetsize restored to default (1316)
+- First root cause in investigation (Open questions #1) resolved
--- a/archive/knowledge/investigations/2026-06-06-authentik-session-lifetime.md
+++ b/archive/knowledge/investigations/2026-06-06-authentik-session-lifetime.md
@@ -0,0 +1,101 @@
+# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
+
+## Summary
+
+User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
+
+## Timeline
+
+- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
+- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
+- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
+- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
+- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
+- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
+- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
+- **~23:45** — Verified both changes took effect
+
+## Root cause
+
+Two independent but compounding issues:
+
+1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
+
+2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
+
+3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
+
+### Why forward-auth worked but OAuth2 didn't
+
+- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
+- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
+
+## Changes applied
+
+### 1. Database — `authentik_stages_user_login_userloginstage`
+
+```sql
+UPDATE authentik_stages_user_login_userloginstage
+SET session_duration = 'days=30'
+WHERE stage_ptr_id = (
+    SELECT stage_uuid FROM authentik_flows_stage 
+    WHERE name = 'default-authentication-login'
+);
+```
+
+This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
+- Sets `_session_expire_at_browser_close=False` for that session
+- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
+- Sets the server-side session expiry to 30 days
+
+### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
+
+Added to `/opt/authentik.env`:
+
+```
+AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
+```
+
+This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
+
+```python
+SESSION_COOKIE_AGE = timedelta_from_string(
+    CONFIG.get("sessions.unauthenticated_age", "days=1")
+).total_seconds()
+```
+
+Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
+
+## Verification
+
+```python
+# Inside authentik-server container:
+import os
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
+import django
+django.setup()
+from django.conf import settings
+print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)  # → 2592000.0
+print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)  # → True
+
+from authentik.stages.user_login.models import UserLoginStage
+stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
+print("session_duration:", stage.session_duration)  # → "days=30"
+```
+
+## Open questions
+
+- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
+- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
+
+## Related
+
+- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
+- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
+- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
+- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
+
+## Changelog
+
+### 2026-06-06 — created
+Documented the session lifetime investigation, root cause, and applied fixes.
--- a/archive/knowledge/investigations/2026-06-06-caddyfile-truncation.md
+++ b/archive/knowledge/investigations/2026-06-06-caddyfile-truncation.md
@@ -0,0 +1,61 @@
+# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
+
+**Date:** 2026-06-06
+**Status:** resolved
+**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
+
+## Symptom
+
+All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
+
+## Root cause
+
+The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
+
+### Timeline
+
+| Time (UTC+2) | Event |
+|---|---|
+| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
+| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
+| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
+| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
+| Jun 06 22:34 | Investigation began |
+| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
+
+### Evidence
+
+- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
+- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
+- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
+- `git stash list` shows one auto-stash entry
+- `origin/master` at `1b977aa`: 260 lines, all site blocks present
+
+### Secondary root cause found during investigation
+
+**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
+
+This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
+
+## Fix applied
+
+1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
+2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
+
+## Permanent safeguards (all deployed)
+
+| Safeguard | Location | What it does |
+|---|---|---|
+| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
+| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
+| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
+| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
+| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
+
+## Related
+
+- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
+- [Caddy (121)](../../wiki/containers/121-caddy.md)
+- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
+- [dns-sync script](../../../scripts/dns-sync.py)
+- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)
--- a/archive/knowledge/investigations/index.md
+++ b/archive/knowledge/investigations/index.md
@@ -0,0 +1,29 @@
+# Investigations
+
+Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
+
+## Active / recent
+
+| Date         | Title                                                              | Status        |
+| ------------ | ------------------------------------------------------------------ | ------------- |
+| 2026-06-06   | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
+| 2026-06-06   | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
+| 2026-06-03   | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
+| 2026-06-01   | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
+
+## Resolved (archived)
+
+See [`archive/`](archive/):
+
+| Date         | Title                                                              |
+| ------------ | ------------------------------------------------------------------ |
+| 2026-04-21   | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
+| 2026-05-31   | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
+
+## Conventions
+
+- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
+- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
+- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
+- Link back from every node's changelog that's affected.
+- Move to `archive/` when the incident is fully resolved and no longer actively referenced.