docs: Authentik session lifetime investigation, fix docs, changelog
This commit is contained in:
130
.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md
Normal file
130
.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# Plan: Fix Frequent Authentik Login Prompts
|
||||
|
||||
## Goal
|
||||
|
||||
Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 7–30 days, or until explicit logout).
|
||||
|
||||
## Current Context
|
||||
|
||||
Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
|
||||
|
||||
```
|
||||
Browser → Caddy (LXC 121) → VPS Traefik → Authentik
|
||||
```
|
||||
|
||||
Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
|
||||
|
||||
## Root Cause Found
|
||||
|
||||
### Primary: `SESSION_EXPIRE_AT_BROWSER_CLOSE = True`
|
||||
|
||||
The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
|
||||
|
||||
### Secondary: `SESSION_COOKIE_AGE = 86400` (24 hours)
|
||||
|
||||
Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
|
||||
|
||||
### Session configuration (from Docker Python environment):
|
||||
|
||||
| Setting | Current Value | Default in Django |
|
||||
|---------|---------------|-------------------|
|
||||
| `SESSION_EXPIRE_AT_BROWSER_CLOSE` | `True` | `False` |
|
||||
| `SESSION_COOKIE_AGE` | `86400` (24h) | `1209600` (14d) |
|
||||
| `SESSION_SAVE_EVERY_REQUEST` | `False` | `False` |
|
||||
| `SESSION_COOKIE_SAMESITE` | `Lax` | `Lax` |
|
||||
|
||||
### What ISN'T the problem:
|
||||
|
||||
- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
|
||||
- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
|
||||
- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
|
||||
- **Caddy configuration** — The forward-auth chain is correctly set up.
|
||||
- **Outpost health** — All containers healthy, up for 6 days.
|
||||
|
||||
## Proposed Approach
|
||||
|
||||
Change two Django session settings via Authentik environment variables:
|
||||
|
||||
1. **`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
|
||||
2. **`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
|
||||
|
||||
This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
|
||||
|
||||
## Step-by-step Plan
|
||||
|
||||
### Step 1: Add environment variables to Docker compose
|
||||
|
||||
Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
|
||||
|
||||
```yaml
|
||||
authentik-server:
|
||||
environment:
|
||||
# ... existing vars ...
|
||||
AUTHENTIK_SESSION_COOKIE_AGE: "604800" # 7 days (was 86400 / 24h)
|
||||
AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE: "false" # was true
|
||||
```
|
||||
|
||||
Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
|
||||
|
||||
### Step 2: Verify variable naming
|
||||
|
||||
Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
|
||||
|
||||
**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
|
||||
|
||||
### Step 3: Restart Authentik server
|
||||
|
||||
```bash
|
||||
ssh root@82.165.190.79
|
||||
docker compose -f /opt/docker-compose.yml restart authentik-server
|
||||
```
|
||||
|
||||
### Step 4: Verify the fix
|
||||
|
||||
```bash
|
||||
# Check session settings took effect
|
||||
ssh root@82.165.190.79 'docker exec -i authentik-server python3 << "PYEOF"
|
||||
import os
|
||||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
|
||||
import django
|
||||
django.setup()
|
||||
from django.conf import settings
|
||||
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)
|
||||
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)
|
||||
PYEOF'
|
||||
```
|
||||
|
||||
### Step 5: Functional test
|
||||
|
||||
1. Login to Authentik at `auth.hubris.network`
|
||||
2. Close the browser completely
|
||||
3. Re-open browser, navigate to a forward-auth-gated service (e.g., paperless.hubris.network)
|
||||
4. Verify you're NOT redirected to login
|
||||
5. Verify OAuth2 services (Gitea) also maintain the session
|
||||
|
||||
## Files Likely to Change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/opt/docker-compose.yml` | Add `AUTHENTIK_SESSION_COOKIE_AGE` and `AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` env vars |
|
||||
|
||||
## Tests / Validation
|
||||
|
||||
1. **Config verification** — Run Python snippet inside container to confirm Django settings changed
|
||||
2. **Browser test** — Close/reopen browser, verify session persists (Step 5 above)
|
||||
3. **24-hour test** — Check session is still alive after 24h of normal use
|
||||
|
||||
## Risks, Tradeoffs, and Open Questions
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
|
||||
| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
|
||||
| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
|
||||
| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
|
||||
2. **Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
|
||||
3. **Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Plan: Fix Caddyfile truncation + prevent recurring outages
|
||||
|
||||
**Date:** 2026-06-06
|
||||
**Slug:** caddyfile-truncation-permanent-fix
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
|
||||
|
||||
## Root cause
|
||||
|
||||
The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
|
||||
|
||||
**Evidence:**
|
||||
- `git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
|
||||
- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
|
||||
- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
|
||||
- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
|
||||
- `origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
|
||||
|
||||
**Why "third time this week":**
|
||||
| Incident | Date | Cause |
|
||||
|---|---|---|
|
||||
| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
|
||||
| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
|
||||
| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
|
||||
|
||||
The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
|
||||
|
||||
## Immediate fix
|
||||
|
||||
### Step 1: Restore Caddyfile from origin/master and reload
|
||||
|
||||
On LXC 121:
|
||||
|
||||
```bash
|
||||
cd /etc/caddy
|
||||
# Stash any local changes
|
||||
git stash
|
||||
# Reset to origin/master
|
||||
git checkout --force origin/master -- Caddyfile
|
||||
# Caddyfile now has all 30+ sites
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
systemctl reload caddy
|
||||
```
|
||||
|
||||
This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
|
||||
|
||||
### Step 2: Add `photos2.hubris.network` via git (if still needed)
|
||||
|
||||
The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
|
||||
|
||||
### Step 3: Verify
|
||||
|
||||
- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
|
||||
- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
|
||||
- Flush mac-mini DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
|
||||
|
||||
## Permanent safeguards
|
||||
|
||||
### Layer 1: Caddyfile integrity check (deploy hook)
|
||||
|
||||
Add a site-count validation to the deploy script (`/etc/caddy/scripts/deploy.sh`):
|
||||
|
||||
```bash
|
||||
# Count site blocks (lines matching *.hubris.network {)
|
||||
SITE_COUNT=$(grep -c '^[a-z].*hubris.network {' Caddyfile)
|
||||
if [ "$SITE_COUNT" -lt 20 ]; then
|
||||
echo "[deploy] ERROR: Only $SITE_COUNT sites found (expected 20+). Refusing to reload."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
This catches any future truncation before `caddy reload` runs.
|
||||
|
||||
### Layer 2: Caddyfile backup on deploy
|
||||
|
||||
Add to deploy script before git pull:
|
||||
|
||||
```bash
|
||||
cp Caddyfile "Caddyfile.bak.$(date +%s)"
|
||||
```
|
||||
|
||||
Keep last 3 backups, auto-rotate.
|
||||
|
||||
### Layer 3: Dirty-tree handling in deploy webhook
|
||||
|
||||
The deploy webhook currently hard-fails when the working tree is dirty. Change the receiver script to handle this gracefully:
|
||||
|
||||
```bash
|
||||
cd /etc/caddy
|
||||
# If dirty, stash local changes
|
||||
if ! git diff --quiet; then
|
||||
echo "[deploy] Working tree dirty — stashing"
|
||||
git stash push -m "auto-stash by deploy webhook $(date)"
|
||||
fi
|
||||
git pull --ff-only
|
||||
```
|
||||
|
||||
This prevents the webhook from blocking on future local edits.
|
||||
|
||||
### Layer 4: Scheduled Caddyfile health check
|
||||
|
||||
Add a homelab cron job that runs `check-caddy-backends.sh` every 10 minutes and notifies if any Caddy backend is unreachable.
|
||||
|
||||
```yaml
|
||||
# In homelab context: cronjob
|
||||
schedule: "*/10 * * * *"
|
||||
script: /opt/homelab-context/scripts/check-caddy-backends.sh
|
||||
```
|
||||
|
||||
### Layer 5: DNS sync cron (fix already-deployed sync)
|
||||
|
||||
The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
|
||||
|
||||
```bash
|
||||
echo "*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1" > /etc/cron.d/dns-sync
|
||||
```
|
||||
|
||||
## Files likely to change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/etc/caddy/Caddyfile` on LXC 121 | Restore from origin/master |
|
||||
| `/etc/caddy/scripts/deploy.sh` on LXC 121 | Add site-count validation + backup + dirty-tree handling |
|
||||
| `caddy-conf` git repo | PR with deploy.sh improvements + photos2 (if wanted) |
|
||||
| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
|
||||
| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
|
||||
|
||||
## Verification
|
||||
|
||||
1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
|
||||
2. `check-caddy-backends.sh` exits 0 on hubris
|
||||
3. `systemctl status caddy` shows active on LXC 121
|
||||
4. `dns-sync` runs and writes to `/var/log/dns-sync.log`
|
||||
|
||||
## Risks / Tradeoffs
|
||||
|
||||
- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
|
||||
- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
|
||||
- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Keep `photos2.hubris.network`? If yes, add via proper git push.
|
||||
2. `prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
|
||||
3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?
|
||||
Reference in New Issue
Block a user