docs: streamline & consolidate the tree (phase 6)
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -7,3 +7,4 @@ each page's `## Changelog` and the Oikos change ledger, not here.
|
||||
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
|
||||
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
|
||||
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
|
||||
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Sources
|
||||
|
||||
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
|
||||
incident evidence lives in [`investigations/`](../../investigations/index.md) (written once at
|
||||
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
|
||||
incident time, then linked from the changelogs of the nodes they implicate).
|
||||
|
||||
| Slug | Reference | Summary |
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
# mac-mini onboarding — post-mortem & lessons learned
|
||||
|
||||
Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
|
||||
homelab context system with the `--with-hermes` profile. What follows is a
|
||||
chronological recap of every hitch, the fix, and the systemic improvements
|
||||
needed so the next workstation takes 5 min instead of an hour.
|
||||
|
||||
## Session log
|
||||
|
||||
### Step 1 — clone + symlink
|
||||
- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
|
||||
- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink.
|
||||
- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
|
||||
already cloned by hand. The bootstrap's `clone exists; pulling` path handled
|
||||
it gracefully.
|
||||
|
||||
### Step 2 — hostname mismatch
|
||||
- `scutil --get LocalHostName` → `Davids-Mac-mini`
|
||||
- `hostname -s` → `Mac`
|
||||
- Inventory file: `hosts/mac-mini.yaml`
|
||||
- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
|
||||
- **Lesson:** The bootstrap and `homelab whoami` use different hostname
|
||||
resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
|
||||
but the `homelab` CLI binary uses `hostname -s`. Both need to match the
|
||||
inventory key. On a fresh macOS machine, neither does.
|
||||
|
||||
### Step 3 — bootstrap dependencies
|
||||
- pyyaml was missing → `pip install pyyaml`
|
||||
- age and sops were missing → `brew install age sops`
|
||||
- Netbird was already installed and connected ✓
|
||||
- **Lesson:** The bootstrap preflight handles these, but only if running
|
||||
`bootstrap.sh` from the start. Since we ran it after manual setup, some
|
||||
steps (netbird install) were correctly skipped as already-present.
|
||||
|
||||
### Step 4 — full bootstrap with `--with-mcp --with-hermes`
|
||||
- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
|
||||
- Age key issued ✓
|
||||
- Launchd sync timer installed ✓
|
||||
- Goose binary installed ✓
|
||||
- Hermes CLI linked ✓
|
||||
- MCP config merged ✓
|
||||
- `refresh-creds` skipped (not yet a recipient) ⚠️
|
||||
- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
|
||||
group-naming quirk)
|
||||
|
||||
### Step 5 — finalize from hubris
|
||||
- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
|
||||
hubris
|
||||
- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
|
||||
stale, bootstrap had already pushed from mac-mini
|
||||
- **Fix:** `git pull --rebase && git push` on hubris
|
||||
- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
|
||||
creating a race. The `homelab client add --finalize-pubkey` command should
|
||||
pull before committing/pushing.
|
||||
|
||||
### Step 6 — sops couldn't find the age key
|
||||
- `homelab secret hello` failed because sops looks in
|
||||
`/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
|
||||
- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
|
||||
but this requires passwordless sudo and the correct env var passthrough
|
||||
- **Fix:**
|
||||
1. Added NOPASSWD sudo rules
|
||||
2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
|
||||
- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
|
||||
agent-enrollment troubleshooting. New clients can't assume `homelab secret`
|
||||
works out of the gate — the sudo re-exec chain has permission pitfall.
|
||||
|
||||
### Step 7 — OpenRouter key was a placeholder
|
||||
- `secrets/openrouter-api-key.yaml` contained
|
||||
`api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
|
||||
- User ran `sops` on hubris, but got the same error (age key not found on
|
||||
hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
|
||||
- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
|
||||
Later: the user pasted the real key, but the sops file showed
|
||||
`sk-or-...5c55` — the literal content was truncated with ellipsis.
|
||||
|
||||
### Step 8 — editor loaded the wrong data
|
||||
- Neovim on the system is configured with `clipboard+=unnamedplus`, which
|
||||
points `*` and `+` registers to the macOS clipboard manager rather than
|
||||
X11. When editing SOPS files, this caused the **system clipboard** to be
|
||||
pasted instead of the actual ciphertext.
|
||||
- This wasn't diagnosed during the session — the sops file would load empty
|
||||
or show the wrong content because the editor's idea of "paste" was
|
||||
disconnected from what sops expected.
|
||||
- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
|
||||
hijack OS clipboards:
|
||||
```bash
|
||||
EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
|
||||
```
|
||||
- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
|
||||
macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
|
||||
because the paste register reads from the system clipboard instead of the
|
||||
sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
|
||||
interactively. Alternatively, override the clipboard option with
|
||||
`EDITOR='nvim -c "set clipboard="'`.
|
||||
- Also useful for the troubleshooting table in `agent-enrollment.md` under a
|
||||
new row: "sops file loads empty / wrong content on macOS"
|
||||
|
||||
### Step 9 — model doesn't support tool use
|
||||
- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
|
||||
- Error: `No endpoints found that support tool use`
|
||||
- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
|
||||
`~/.config/goose/config.yaml`
|
||||
- Also updated `operations/hermes-agent.md` with the correct model
|
||||
- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
|
||||
never validated on OpenRouter for tool-use capability. Need to either:
|
||||
(a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
|
||||
(b) Or have the bootstrap probe OpenRouter at install time
|
||||
|
||||
### Step 10 — MCP homelab extension returns 404
|
||||
- Goose's homelab extension configured as `streamable_http` at
|
||||
`https://mcp.hubris.network/mcp` returns HTTP 404
|
||||
- The actual MCP server runs on SSE (`/sse`), not streamable_http
|
||||
- **Known follow-up** from `hermes-agent.md` — server needs migration
|
||||
- **Workaround:** The agent works fine via the built-in `developer` extension
|
||||
(shell + file) and the `homelab` CLI
|
||||
|
||||
### Step 11 — git credential dance
|
||||
Several layers of issues with git authentication:
|
||||
|
||||
1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
|
||||
couldn't access the credential file, and git's `store` helper couldn't
|
||||
create its `.lock` file.
|
||||
- **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
|
||||
- **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`
|
||||
|
||||
2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
|
||||
as a default credential helper. Even after configuring the `store` helper,
|
||||
git calls osxkeychain after every successful `get`, which prompts for
|
||||
macOS keychain access.
|
||||
- `credential.helper = ""` does NOT disable the compiled-in default
|
||||
- The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
|
||||
silently, causing fallback behaviour
|
||||
- **Fix:** Embed the credential in the remote URL directly:
|
||||
`https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
|
||||
- This bypasses all credential helpers entirely
|
||||
|
||||
3. **refresh-creds:** The credential is now managed by the remote URL.
|
||||
If the PAT is rotated, `homelab refresh-creds` won't update the URL.
|
||||
- **Workaround:** Run `git remote set-url origin` after `refresh-creds`
|
||||
|
||||
### Step 12 — `homelab` CLI hostname detection
|
||||
- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
|
||||
`hostname -s` (returning `Mac`) while the bootstrap uses
|
||||
`scutil --get LocalHostName` (which we fixed to `mac-mini`)
|
||||
- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
|
||||
- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
|
||||
LocalHostName` like the bootstrap does, or at least try both and pick the
|
||||
one that matches an inventory key. This is a CLI bug.
|
||||
|
||||
## Improvement backlog
|
||||
|
||||
### High priority
|
||||
|
||||
| # | Issue | Fix |
|
||||
|---|-------|-----|
|
||||
| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
|
||||
| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
|
||||
| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
|
||||
| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
|
||||
| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
|
||||
|
||||
### Medium priority
|
||||
|
||||
| # | Issue | Fix |
|
||||
|---|-------|-----|
|
||||
| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
|
||||
| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
|
||||
| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
|
||||
| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
|
||||
|
||||
### Low priority
|
||||
|
||||
| # | Issue | Fix |
|
||||
|---|-------|-----|
|
||||
| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
|
||||
| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
|
||||
| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-01 — initial post-mortem
|
||||
Captured the full onboarding of mac-mini into the hubris homelab context
|
||||
system, including the hostname mismatch, sudo/sops credential chain, model
|
||||
selection, git credential gymnastics, and editor clipboard interference.
|
||||
|
||||
### 2026-06-02 — MCP server migrated to streamable-http
|
||||
Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
|
||||
not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
|
||||
and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
|
||||
re-deployed on apps LXC (105).
|
||||
@@ -0,0 +1,84 @@
|
||||
# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
|
||||
|
||||
## Summary
|
||||
|
||||
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
|
||||
|
||||
## Timeline
|
||||
|
||||
### Pre-existing state
|
||||
- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
|
||||
- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
|
||||
- Both on the same subnet via a consumer router at 192.168.178.1
|
||||
- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
|
||||
- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
|
||||
|
||||
### 2026-06-03 — investigation
|
||||
**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
|
||||
- TCP throughput: 49–86 Mbps, average ~65 Mbps — highly variable
|
||||
- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
|
||||
- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
|
||||
- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
|
||||
|
||||
**Additional findings:**
|
||||
- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
|
||||
- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
|
||||
- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
|
||||
- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
|
||||
|
||||
## Root cause
|
||||
|
||||
mac-mini is on WiFi. WiFi introduces:
|
||||
1. **Throughput variance** — 49–86 Mbps TCP fluctuation
|
||||
2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
|
||||
3. **Jitter** — 28 ms stddev with spikes to 138 ms
|
||||
|
||||
Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
|
||||
|
||||
## Mitigations applied
|
||||
|
||||
### mac-mini — Moonlight (via `defaults write`)
|
||||
|
||||
| Setting | Before | After | Rationale |
|
||||
||---------|--------|-------|-----------|
|
||||
|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
|
||||
|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
|
||||
|| fps | 30 | 60 | Restored to game-appropriate value |
|
||||
|
||||
### mac-mini — Moonlight (current, wired — 2026-06-04)
|
||||
|
||||
| Setting | WiFi value | Wired value | Rationale |
|
||||
||---------|-----------|-------------|-----------|
|
||||
|| bitrate | 60 Mbps | 80 Mbps | Wired 1 GbE can sustain comfortably |
|
||||
|| framepacing | on (1) | off (0) | Wired latency is consistent, no smoothing needed |
|
||||
|| fps | 60 | 60 | Unchanged |
|
||||
|
||||
### ludo-mini — Sunshine (via SSH)
|
||||
|
||||
| Setting | WiFi value | Wired value | Rationale |
|
||||
||---------|-----------|-------------|-----------|
|
||||
|| bitrate | 40 Mbps | 80 Mbps | Wired can handle full bandwidth |
|
||||
|| fec_percentage | 2 | 5 | Restored to default — wired has no dropout concern |
|
||||
|| packetsize | 1024 | 1316 | Restored to default (Ethernet MTU) |
|
||||
|
||||
Sunshine restarted after changes; confirmed active.
|
||||
|
||||
## Resolution
|
||||
|
||||
### 2026-06-04 — mac-mini wired to LAN
|
||||
|
||||
mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
|
||||
|
||||
## Open questions
|
||||
|
||||
~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~ **RESOLVED** — mac-mini now wired via en0.
|
||||
2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
|
||||
3. **Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
|
||||
- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
|
||||
- Moonlight: bitrate 60→80 Mbps, framepacing disabled
|
||||
- Sunshine: bitrate 40→80 Mbps, fec_percentage 2→5, packetsize restored to default (1316)
|
||||
- First root cause in investigation (Open questions #1) resolved
|
||||
@@ -0,0 +1,101 @@
|
||||
# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
|
||||
|
||||
## Summary
|
||||
|
||||
User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
|
||||
|
||||
## Timeline
|
||||
|
||||
- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
|
||||
- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
|
||||
- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
|
||||
- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
|
||||
- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
|
||||
- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
|
||||
- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
|
||||
- **~23:45** — Verified both changes took effect
|
||||
|
||||
## Root cause
|
||||
|
||||
Two independent but compounding issues:
|
||||
|
||||
1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
|
||||
|
||||
2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
|
||||
|
||||
3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
|
||||
|
||||
### Why forward-auth worked but OAuth2 didn't
|
||||
|
||||
- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
|
||||
- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
|
||||
|
||||
## Changes applied
|
||||
|
||||
### 1. Database — `authentik_stages_user_login_userloginstage`
|
||||
|
||||
```sql
|
||||
UPDATE authentik_stages_user_login_userloginstage
|
||||
SET session_duration = 'days=30'
|
||||
WHERE stage_ptr_id = (
|
||||
SELECT stage_uuid FROM authentik_flows_stage
|
||||
WHERE name = 'default-authentication-login'
|
||||
);
|
||||
```
|
||||
|
||||
This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
|
||||
- Sets `_session_expire_at_browser_close=False` for that session
|
||||
- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
|
||||
- Sets the server-side session expiry to 30 days
|
||||
|
||||
### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
|
||||
|
||||
Added to `/opt/authentik.env`:
|
||||
|
||||
```
|
||||
AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
|
||||
```
|
||||
|
||||
This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
|
||||
|
||||
```python
|
||||
SESSION_COOKIE_AGE = timedelta_from_string(
|
||||
CONFIG.get("sessions.unauthenticated_age", "days=1")
|
||||
).total_seconds()
|
||||
```
|
||||
|
||||
Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
|
||||
|
||||
## Verification
|
||||
|
||||
```python
|
||||
# Inside authentik-server container:
|
||||
import os
|
||||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
|
||||
import django
|
||||
django.setup()
|
||||
from django.conf import settings
|
||||
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE) # → 2592000.0
|
||||
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE) # → True
|
||||
|
||||
from authentik.stages.user_login.models import UserLoginStage
|
||||
stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
|
||||
print("session_duration:", stage.session_duration) # → "days=30"
|
||||
```
|
||||
|
||||
## Open questions
|
||||
|
||||
- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
|
||||
- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
|
||||
|
||||
## Related
|
||||
|
||||
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
|
||||
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
|
||||
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
|
||||
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-06 — created
|
||||
Documented the session lifetime investigation, root cause, and applied fixes.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
|
||||
|
||||
**Date:** 2026-06-06
|
||||
**Status:** resolved
|
||||
**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
|
||||
|
||||
## Symptom
|
||||
|
||||
All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
|
||||
|
||||
## Root cause
|
||||
|
||||
The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
|
||||
|
||||
### Timeline
|
||||
|
||||
| Time (UTC+2) | Event |
|
||||
|---|---|
|
||||
| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
|
||||
| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
|
||||
| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
|
||||
| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
|
||||
| Jun 06 22:34 | Investigation began |
|
||||
| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
|
||||
|
||||
### Evidence
|
||||
|
||||
- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
|
||||
- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
|
||||
- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
|
||||
- `git stash list` shows one auto-stash entry
|
||||
- `origin/master` at `1b977aa`: 260 lines, all site blocks present
|
||||
|
||||
### Secondary root cause found during investigation
|
||||
|
||||
**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
|
||||
|
||||
This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
|
||||
|
||||
## Fix applied
|
||||
|
||||
1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
|
||||
2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
|
||||
|
||||
## Permanent safeguards (all deployed)
|
||||
|
||||
| Safeguard | Location | What it does |
|
||||
|---|---|---|
|
||||
| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
|
||||
| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
|
||||
| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
|
||||
| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
|
||||
| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
|
||||
|
||||
## Related
|
||||
|
||||
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
|
||||
- [Caddy (121)](../../wiki/containers/121-caddy.md)
|
||||
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
|
||||
- [dns-sync script](../../../scripts/dns-sync.py)
|
||||
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)
|
||||
@@ -0,0 +1,89 @@
|
||||
# 2026-04-21 — Hubris crash loop (thermal + USB drive)
|
||||
|
||||
## Summary
|
||||
|
||||
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
|
||||
|
||||
## Timeline
|
||||
|
||||
### 2026-04-19 — drive attached
|
||||
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
|
||||
|
||||
### 2026-04-19 → 2026-04-21 — first crashes
|
||||
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
|
||||
|
||||
### 2026-04-21 — crash cadence escalates
|
||||
23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`.
|
||||
|
||||
### 2026-04-21 — first set of mitigations applied
|
||||
|
||||
- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
|
||||
- **Crash capture**: `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`).
|
||||
- **`rasdaemon`** installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query `ras-mc-ctl --summary`.
|
||||
- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge.
|
||||
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
|
||||
|
||||
### 2026-04-22 — recurrence after 30 h 37 m
|
||||
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
|
||||
|
||||
### 2026-04-22 — `cpu-epp.service` design bug fixed
|
||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||
|
||||
### 2026-04-22 — drive removed (A/B test)
|
||||
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
|
||||
|
||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
|
||||
|
||||
#### 2026-04-23 thermal-pad verdict
|
||||
Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold.
|
||||
|
||||
### 2026-04-28 — A/B test passing so far
|
||||
3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.
|
||||
|
||||
### 2026-04-29 — physical relocation
|
||||
User moved the host to a spot with better airflow. Post-move idle baseline (45 min uptime, light load): Tctl **47.2 °C**, GPU 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C. Notably nvme0 sensor1 is **27 °C cooler** than the 2026-04-23 thermal-pad steady-state of 60–61 °C. Watch lifetime warning-time counters (`nvme smart-log` → "Warning Comp. Temperature Time") over the next week — flat trend confirms the move helped. Doesn't change the drive-removal A/B test, but reduces residual thermal headroom risk independently.
|
||||
|
||||
## Root cause
|
||||
|
||||
Two-contributor:
|
||||
1. **Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them.
|
||||
2. **Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).
|
||||
|
||||
Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.
|
||||
|
||||
## Diagnostic notes
|
||||
|
||||
- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
|
||||
- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`.
|
||||
- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`.
|
||||
- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
|
||||
- `mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement.
|
||||
|
||||
## BIOS update path
|
||||
|
||||
Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.
|
||||
|
||||
## Mitigations applied
|
||||
|
||||
| Mitigation | Status | Notes |
|
||||
| ----------------------------------------- | -------- | ---------------------------------------------------- |
|
||||
| `cpu-epp.service` (powersave + balance_power) | Active | Unit ordering fixed 2026-04-22 to run before guests |
|
||||
| Crash capture (sysctl + softdog + watchdog) | Active | Catches everything but pure silicon hangs |
|
||||
| `rasdaemon` | Active | Persistent telemetry for MCE / memory / PCIe AER |
|
||||
| UAS blacklist for `090c:2320` | Active (drive currently absent) | `/etc/modprobe.d/usb-storage-quirks.conf` |
|
||||
| Mount-on-demand (`backup-usb.sh`) | Active (drive currently absent) | Backup unit `ExecStartPre`/`ExecStopPost` hooks |
|
||||
| Drive physically removed | Active | A/B test in progress |
|
||||
| Thermal pads on NVMes | Active 2026-04-23 | Steady-state composite 47 °C / sensor1 60 °C |
|
||||
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
|
||||
|
||||
## Affected nodes
|
||||
- [Hubris host](../../../wiki/hosts/hubris.md)
|
||||
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
|
||||
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
|
||||
|
||||
## Open questions
|
||||
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
|
||||
- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
|
||||
- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
|
||||
- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.
|
||||
@@ -0,0 +1,109 @@
|
||||
# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
|
||||
|
||||
## Summary
|
||||
|
||||
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
|
||||
|
||||
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
|
||||
|
||||
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
|
||||
|
||||
## Timeline
|
||||
|
||||
### Trigger
|
||||
`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
|
||||
```
|
||||
Error: failed reading provided config file: /etc/netbird/management.json:
|
||||
OIDC configuration request returned status 504 with response: Gateway Timeout
|
||||
```
|
||||
The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
|
||||
|
||||
### Fix — Phase A (restore the mesh)
|
||||
1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
|
||||
2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
|
||||
3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
|
||||
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
|
||||
|
||||
### Fix — Phase B (full data migration)
|
||||
Once the mesh was back, the homelab DB was reachable:
|
||||
1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
|
||||
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
|
||||
3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
|
||||
4. Restarted Authentik. All users / apps / passwords / groups present.
|
||||
|
||||
Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
|
||||
|
||||
### Fix — DNS cutover (the long tail)
|
||||
The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
|
||||
|
||||
- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
|
||||
- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
|
||||
|
||||
## Issues hit during the migration (and fixes)
|
||||
|
||||
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
|
||||
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
|
||||
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
|
||||
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
|
||||
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
|
||||
|
||||
## Architecture delta
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
|
||||
| Version | `2026.2.2` | `2026.5.2` |
|
||||
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
|
||||
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
|
||||
| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
|
||||
| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
|
||||
| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
|
||||
|
||||
> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
|
||||
|
||||
## Lessons learned
|
||||
|
||||
1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
|
||||
2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
|
||||
3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
|
||||
4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
|
||||
5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
|
||||
6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
|
||||
7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
|
||||
8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
|
||||
|
||||
## Forward-auth outpost — RESOLVED 2026-06-01
|
||||
|
||||
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
|
||||
|
||||
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
|
||||
|
||||
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
|
||||
|
||||
The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
|
||||
|
||||
**Fix:** Deleted from Postgres:
|
||||
```sql
|
||||
DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
|
||||
DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
|
||||
DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';
|
||||
```
|
||||
The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
|
||||
|
||||
**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
|
||||
|
||||
## Outstanding
|
||||
|
||||
- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
|
||||
- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
|
||||
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
|
||||
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
|
||||
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
|
||||
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
|
||||
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
|
||||
|
||||
## Related
|
||||
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
|
||||
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
|
||||
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
|
||||
- [Mesh migration](../../../wiki/infrastructure/mesh.md)
|
||||
29
knowledge/sources/investigations/index.md
Normal file
29
knowledge/sources/investigations/index.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Investigations
|
||||
|
||||
Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
|
||||
|
||||
## Active / recent
|
||||
|
||||
| Date | Title | Status |
|
||||
| ------------ | ------------------------------------------------------------------ | ------------- |
|
||||
| 2026-06-06 | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
|
||||
| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
|
||||
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
|
||||
| 2026-06-01 | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
|
||||
|
||||
## Resolved (archived)
|
||||
|
||||
See [`archive/`](archive/):
|
||||
|
||||
| Date | Title |
|
||||
| ------------ | ------------------------------------------------------------------ |
|
||||
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
|
||||
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
|
||||
|
||||
## Conventions
|
||||
|
||||
- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
|
||||
- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
|
||||
- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
|
||||
- Link back from every node's changelog that's affected.
|
||||
- Move to `archive/` when the incident is fully resolved and no longer actively referenced.
|
||||
@@ -54,7 +54,7 @@ We considered three options before building this:
|
||||
|
||||
| Option | Outcome |
|
||||
|---|---|
|
||||
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../../investigations/index.md)). Rejected. |
|
||||
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
|
||||
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
|
||||
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 106 — `auth-outpost`
|
||||
|
||||
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
||||
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
||||
|
||||
## At a glance
|
||||
- **Hostname:** `auth-outpost`
|
||||
@@ -12,7 +12,7 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co
|
||||
|
||||
## Role
|
||||
|
||||
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||
|
||||
## Service / port map
|
||||
| Service | Listen | Notes |
|
||||
@@ -45,12 +45,12 @@ Fix: the LAN outpost gets its **own** domain.
|
||||
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
|
||||
- [Caddy (121)](121-caddy.md) — forward-auth consumer
|
||||
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
|
||||
- [Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)
|
||||
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-06 — Authentik session lifetime extended to 30 days
|
||||
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md).
|
||||
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
|
||||
|
||||
### 2026-06-01 — created; forward-auth cut over from LXC 124
|
||||
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
|
||||
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
|
||||
|
||||
@@ -7,7 +7,7 @@ audio content against a local server instead of the official cloud.
|
||||
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
|
||||
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
|
||||
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
|
||||
[OIKOS.md](../../../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
|
||||
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
|
||||
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
|
||||
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
|
||||
for it — but it never got its own inventory entry or doc page until now.
|
||||
@@ -50,7 +50,7 @@ for the first time.
|
||||
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
|
||||
taken by this container
|
||||
- [Containers index](index.md)
|
||||
- [OIKOS.md](../../../OIKOS.md) — drift detector that caught this
|
||||
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@ after this set.
|
||||
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
|
||||
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
|
||||
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
|
||||
[agent-enrollment.md troubleshooting](../../../operations/agent-enrollment.md#troubleshooting) if it recurs
|
||||
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
|
||||
on future LXC bootstraps.
|
||||
|
||||
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
|
||||
|
||||
@@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
|
||||
## At a glance
|
||||
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
||||
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
||||
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
||||
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
||||
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
||||
@@ -113,8 +113,8 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
|
||||
- [Media permissions](../infrastructure/media-permissions.md)
|
||||
- [Monitoring](../infrastructure/monitoring.md)
|
||||
- [Backups (disabled)](../infrastructure/backups.md)
|
||||
- [Operations cheatsheet](../../../operations/commands.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||
- [strong — Proxmox host](strong.md)
|
||||
|
||||
## Changelog
|
||||
@@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`.
|
||||
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
||||
|
||||
### 2026-04-29 — relocated to better-ventilated spot
|
||||
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
||||
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
||||
|
||||
### 2026-04-28 — Phase 1 WiFi failover
|
||||
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
||||
@@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov
|
||||
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
||||
|
||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
||||
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
||||
|
||||
### 2026-04-22 — drive removal A/B test
|
||||
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||
@@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `
|
||||
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
||||
|
||||
### 2026-04-21 — `cpu-epp.service` deployed
|
||||
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
@@ -50,13 +50,13 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
|
||||
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
|
||||
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
|
||||
5-min sync timer are live; `homelab whoami` resolves correctly. See
|
||||
[agent-enrollment.md](../../../operations/agent-enrollment.md).
|
||||
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
|
||||
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
|
||||
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
|
||||
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
|
||||
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
|
||||
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
|
||||
["Granting a secret to a new client"](../../../operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
|
||||
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
|
||||
|
||||
## Cluster membership
|
||||
|
||||
@@ -79,7 +79,7 @@ needed going forward.
|
||||
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
|
||||
- [Network](../infrastructure/network.md)
|
||||
- [SSH access](../infrastructure/ssh-access.md)
|
||||
- [Agent enrollment](../../../operations/agent-enrollment.md)
|
||||
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
|
||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Operations cheatsheet](../../../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
|
||||
## Changelog
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design.
|
||||
|
||||
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
||||
|
||||
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
## Status
|
||||
|
||||
@@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl
|
||||
|
||||
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
||||
|
||||
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
||||
|
||||
@@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud
|
||||
|
||||
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
||||
|
||||
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||
|
||||
## Thermal monitoring
|
||||
|
||||
@@ -122,7 +122,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
|
||||
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||
- [Monitoring](monitoring.md)
|
||||
- [Auto-deploy](auto-deploy.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
@@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon
|
||||
Initial documentation. Status remains DISABLED.
|
||||
|
||||
### 2026-04-22 — DISABLED
|
||||
Drive removed as the A/B test in the [crash investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
||||
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
||||
|
||||
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
||||
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
||||
|
||||
@@ -110,7 +110,7 @@ All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float o
|
||||
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||||
|
||||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||||
|
||||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||||
|
||||
@@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
||||
self-locating and able to read the same source of truth.
|
||||
|
||||
Operational walkthrough for enrolling a new client lives in
|
||||
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md); this
|
||||
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
|
||||
page is the architecture reference.
|
||||
|
||||
## What's where
|
||||
@@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone
|
||||
|
||||
## Related
|
||||
|
||||
- [Operations: agent enrollment](../../../operations/agent-enrollment.md) — the
|
||||
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
|
||||
step-by-step for adding a new client
|
||||
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
||||
pipelines (and the rest of the lab's webhook pipelines)
|
||||
|
||||
@@ -62,5 +62,5 @@ are documented in their own pages. Each system below links to its full doc.
|
||||
|
||||
- [README](../../../README.md) — entry point
|
||||
- [Containers index](../containers/index.md)
|
||||
- [Operations cheatsheet](../../../operations/commands.md)
|
||||
- [OIKOS operating model](../../../OIKOS.md)
|
||||
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||
- [OIKOS operating model](../../../.agents/OIKOS.md)
|
||||
@@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8
|
||||
|
||||
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
||||
|
||||
Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
||||
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
||||
|
||||
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
||||
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
||||
@@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `
|
||||
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
|
||||
|
||||
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
||||
Authentik migrated onto the VPS ([investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
||||
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
@@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
|
||||
## Changelog
|
||||
|
||||
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
||||
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||
|
||||
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
||||
|
||||
|
||||
@@ -178,7 +178,7 @@ done
|
||||
|
||||
- [Mesh migration](mesh.md)
|
||||
- [VPS hardening](vps-hardening.md)
|
||||
- [Agent enrollment](../../../operations/agent-enrollment.md)
|
||||
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||
- [Homelab CLI](../../../bin/homelab)
|
||||
|
||||
## Changelog
|
||||
|
||||
Reference in New Issue
Block a user