docs: Authentik session lifetime investigation, fix docs, changelog

.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md

@@ -0,0 +1,130 @@
+# Plan: Fix Frequent Authentik Login Prompts
+
+## Goal
+
+Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 7–30 days, or until explicit logout).
+
+## Current Context
+
+Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
+
+```
+Browser → Caddy (LXC 121) → VPS Traefik → Authentik
+```
+
+Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
+
+## Root Cause Found
+
+### Primary: `SESSION_EXPIRE_AT_BROWSER_CLOSE = True`
+
+The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
+
+### Secondary: `SESSION_COOKIE_AGE = 86400` (24 hours)
+
+Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
+
+### Session configuration (from Docker Python environment):
+
+| Setting | Current Value | Default in Django |
+|---------|---------------|-------------------|
+| `SESSION_EXPIRE_AT_BROWSER_CLOSE` | `True` | `False` |
+| `SESSION_COOKIE_AGE` | `86400` (24h) | `1209600` (14d) |
+| `SESSION_SAVE_EVERY_REQUEST` | `False` | `False` |
+| `SESSION_COOKIE_SAMESITE` | `Lax` | `Lax` |
+
+### What ISN'T the problem:
+
+- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
+- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
+- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
+- **Caddy configuration** — The forward-auth chain is correctly set up.
+- **Outpost health** — All containers healthy, up for 6 days.
+
+## Proposed Approach
+
+Change two Django session settings via Authentik environment variables:
+
+1. **`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
+2. **`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
+
+This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
+
+## Step-by-step Plan
+
+### Step 1: Add environment variables to Docker compose
+
+Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
+
+```yaml
+authentik-server:
+  environment:
+    # ... existing vars ...
+    AUTHENTIK_SESSION_COOKIE_AGE: "604800"      # 7 days (was 86400 / 24h)
+    AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE: "false"  # was true
+```
+
+Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
+
+### Step 2: Verify variable naming
+
+Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
+
+**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
+
+### Step 3: Restart Authentik server
+
+```bash
+ssh root@82.165.190.79
+docker compose -f /opt/docker-compose.yml restart authentik-server
+```
+
+### Step 4: Verify the fix
+
+```bash
+# Check session settings took effect
+ssh root@82.165.190.79 'docker exec -i authentik-server python3 << "PYEOF"
+import os
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
+import django
+django.setup()
+from django.conf import settings
+print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)
+print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)
+PYEOF'
+```
+
+### Step 5: Functional test
+
+1. Login to Authentik at `auth.hubris.network`
+2. Close the browser completely
+3. Re-open browser, navigate to a forward-auth-gated service (e.g., paperless.hubris.network)
+4. Verify you're NOT redirected to login
+5. Verify OAuth2 services (Gitea) also maintain the session
+
+## Files Likely to Change
+
+| File | Change |
+|------|--------|
+| `/opt/docker-compose.yml` | Add `AUTHENTIK_SESSION_COOKIE_AGE` and `AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` env vars |
+
+## Tests / Validation
+
+1. **Config verification** — Run Python snippet inside container to confirm Django settings changed
+2. **Browser test** — Close/reopen browser, verify session persists (Step 5 above)
+3. **24-hour test** — Check session is still alive after 24h of normal use
+
+## Risks, Tradeoffs, and Open Questions
+
+| Risk | Mitigation |
+|------|------------|
+| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
+| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
+| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
+| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
+
+## Open Questions
+
+1. **What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
+2. **Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
+3. **Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.

.hermes/plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md

@@ -0,0 +1,149 @@
+# Plan: Fix Caddyfile truncation + prevent recurring outages
+
+**Date:** 2026-06-06
+**Slug:** caddyfile-truncation-permanent-fix
+
+---
+
+## Goal
+
+Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
+
+## Root cause
+
+The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
+
+**Evidence:**
+- `git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
+- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
+- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
+- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
+- `origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
+
+**Why "third time this week":**
+| Incident | Date | Cause |
+|---|---|---|
+| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
+| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
+| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
+
+The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
+
+## Immediate fix
+
+### Step 1: Restore Caddyfile from origin/master and reload
+
+On LXC 121:
+
+```bash
+cd /etc/caddy
+# Stash any local changes
+git stash
+# Reset to origin/master
+git checkout --force origin/master -- Caddyfile
+# Caddyfile now has all 30+ sites
+caddy validate --config /etc/caddy/Caddyfile
+systemctl reload caddy
+```
+
+This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
+
+### Step 2: Add `photos2.hubris.network` via git (if still needed)
+
+The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
+
+### Step 3: Verify
+
+- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
+- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
+- Flush mac-mini DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
+
+## Permanent safeguards
+
+### Layer 1: Caddyfile integrity check (deploy hook)
+
+Add a site-count validation to the deploy script (`/etc/caddy/scripts/deploy.sh`):
+
+```bash
+# Count site blocks (lines matching *.hubris.network {)
+SITE_COUNT=$(grep -c '^[a-z].*hubris.network {' Caddyfile)
+if [ "$SITE_COUNT" -lt 20 ]; then
+  echo "[deploy] ERROR: Only $SITE_COUNT sites found (expected 20+). Refusing to reload."
+  exit 1
+fi
+```
+
+This catches any future truncation before `caddy reload` runs.
+
+### Layer 2: Caddyfile backup on deploy
+
+Add to deploy script before git pull:
+
+```bash
+cp Caddyfile "Caddyfile.bak.$(date +%s)"
+```
+
+Keep last 3 backups, auto-rotate.
+
+### Layer 3: Dirty-tree handling in deploy webhook
+
+The deploy webhook currently hard-fails when the working tree is dirty. Change the receiver script to handle this gracefully:
+
+```bash
+cd /etc/caddy
+# If dirty, stash local changes
+if ! git diff --quiet; then
+  echo "[deploy] Working tree dirty — stashing"
+  git stash push -m "auto-stash by deploy webhook $(date)"
+fi
+git pull --ff-only
+```
+
+This prevents the webhook from blocking on future local edits.
+
+### Layer 4: Scheduled Caddyfile health check
+
+Add a homelab cron job that runs `check-caddy-backends.sh` every 10 minutes and notifies if any Caddy backend is unreachable.
+
+```yaml
+# In homelab context: cronjob
+schedule: "*/10 * * * *"
+script: /opt/homelab-context/scripts/check-caddy-backends.sh
+```
+
+### Layer 5: DNS sync cron (fix already-deployed sync)
+
+The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
+
+```bash
+echo "*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1" > /etc/cron.d/dns-sync
+```
+
+## Files likely to change
+
+| File | Change |
+|------|--------|
+| `/etc/caddy/Caddyfile` on LXC 121 | Restore from origin/master |
+| `/etc/caddy/scripts/deploy.sh` on LXC 121 | Add site-count validation + backup + dirty-tree handling |
+| `caddy-conf` git repo | PR with deploy.sh improvements + photos2 (if wanted) |
+| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
+| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
+
+## Verification
+
+1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
+2. `check-caddy-backends.sh` exits 0 on hubris
+3. `systemctl status caddy` shows active on LXC 121
+4. `dns-sync` runs and writes to `/var/log/dns-sync.log`
+
+## Risks / Tradeoffs
+
+- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
+- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
+- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
+
+## Open questions
+
+1. Keep `photos2.hubris.network`? If yes, add via proper git push.
+2. `prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
+3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?

containers/106-auth-outpost.md

@@ -49,5 +49,8 @@ Fix: the LAN outpost gets its **own** domain.
 
 ## Changelog
 
+### 2026-06-06 — Authentik session lifetime extended to 30 days
+VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
+
 ### 2026-06-01 — created; forward-auth cut over from LXC 124
 New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).

containers/107-dns.md

@@ -48,6 +48,9 @@ Replaces the DHCP that was previously served by the Slate AX router. Static-IP L
 
 ## Changelog
 
+### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
+Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
+
 ### 2026-06-03 — DHCP pool narrowed to `.241–.254`
 Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
 

containers/118-elementsynapse.md

@@ -43,6 +43,19 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
 
 ## Changelog
 
+### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
+**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
+
+**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
+
+**Fix:**
+- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
+- `ifdown eth0 && ifup eth0` applied the static IP
+- Killed lingering dhclient process
+- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
+
+**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
+
 ### 2026-05-15 — phantom-notification cleanup for `@admin`
 After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
 1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.

containers/121-caddy.md

@@ -65,6 +65,22 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
 
 ## Changelog
 
+### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
+**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
+
+**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
+
+**Fix:**
+- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
+- `systemctl reload caddy`
+
+**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
+- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
+- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
+- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
+
+Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
+
 ### 2026-06-02 — caddy.service unit missing; recreated
 After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
 

hosts/elementsynapse.yaml

@@ -7,7 +7,7 @@ os: linux
 role: matrix-server
 host: hubris
 pve_id: 118
-lan_ip: 192.168.8.239
+lan_ip: 192.168.8.239 # static; was DHCP drifting to .244 — fixed by setting iface eth0 inet static
 mesh:
   tailscale:
     fqdn: elementsynapse

infrastructure/dns.md

@@ -93,6 +93,13 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was
 
 > Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
 
+### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
+The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
+
+**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
+
+Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
+
 ### 2026-06-02 — 8 LXCs moved from DHCP to static IP
 All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
 

investigations/2026-06-06-authentik-session-lifetime.md

@@ -0,0 +1,101 @@
+# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
+
+## Summary
+
+User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
+
+## Timeline
+
+- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
+- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
+- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
+- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
+- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
+- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
+- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
+- **~23:45** — Verified both changes took effect
+
+## Root cause
+
+Two independent but compounding issues:
+
+1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
+
+2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
+
+3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
+
+### Why forward-auth worked but OAuth2 didn't
+
+- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
+- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
+
+## Changes applied
+
+### 1. Database — `authentik_stages_user_login_userloginstage`
+
+```sql
+UPDATE authentik_stages_user_login_userloginstage
+SET session_duration = 'days=30'
+WHERE stage_ptr_id = (
+    SELECT stage_uuid FROM authentik_flows_stage 
+    WHERE name = 'default-authentication-login'
+);
+```
+
+This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
+- Sets `_session_expire_at_browser_close=False` for that session
+- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
+- Sets the server-side session expiry to 30 days
+
+### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
+
+Added to `/opt/authentik.env`:
+
+```
+AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
+```
+
+This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
+
+```python
+SESSION_COOKIE_AGE = timedelta_from_string(
+    CONFIG.get("sessions.unauthenticated_age", "days=1")
+).total_seconds()
+```
+
+Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
+
+## Verification
+
+```python
+# Inside authentik-server container:
+import os
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
+import django
+django.setup()
+from django.conf import settings
+print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)  # → 2592000.0
+print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)  # → True
+
+from authentik.stages.user_login.models import UserLoginStage
+stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
+print("session_duration:", stage.session_duration)  # → "days=30"
+```
+
+## Open questions
+
+- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
+- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
+
+## Related
+
+- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
+- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
+- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
+- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
+
+## Changelog
+
+### 2026-06-06 — created
+Documented the session lifetime investigation, root cause, and applied fixes.

investigations/2026-06-06-caddyfile-truncation.md

@@ -0,0 +1,61 @@
+# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
+
+**Date:** 2026-06-06
+**Status:** resolved
+**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
+
+## Symptom
+
+All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
+
+## Root cause
+
+The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
+
+### Timeline
+
+| Time (UTC+2) | Event |
+|---|---|
+| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
+| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
+| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
+| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
+| Jun 06 22:34 | Investigation began |
+| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
+
+### Evidence
+
+- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
+- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
+- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
+- `git stash list` shows one auto-stash entry
+- `origin/master` at `1b977aa`: 260 lines, all site blocks present
+
+### Secondary root cause found during investigation
+
+**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
+
+This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
+
+## Fix applied
+
+1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
+2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
+
+## Permanent safeguards (all deployed)
+
+| Safeguard | Location | What it does |
+|---|---|---|
+| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
+| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
+| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
+| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
+| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
+
+## Related
+
+- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
+- [Caddy (121)](../containers/121-caddy.md)
+- [elementsynapse (118)](../containers/118-elementsynapse.md)
+- [dns-sync script](../scripts/dns-sync.py)
+- [check-caddy-backends script](../scripts/check-caddy-backends.sh)

investigations/index.md

@@ -6,7 +6,8 @@ Time-stamped incident reports and experiments. One entry per incident; the entry
 
 | Date         | Title                                                              | Status        |
 | ------------ | ------------------------------------------------------------------ | ------------- |
-| 2026-05-31   | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
+|| 2026-06-06   | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
+|| 2026-05-31   | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
 | 2026-06-03   | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
 | 2026-04-21   | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |