docs: streamline & consolidate the tree (phase 6)
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -44,7 +44,7 @@ one pass through **Observe → Orient → Decide → Act**:
|
|||||||
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
| Signal | something needing attention, with lifecycle | `signals/` ledger (Week 3) |
|
||||||
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
| Change | one mutation: who, what, risk, approval, verification | `ledger/` (Week 2) |
|
||||||
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
| Approval | short-TTL signed grant for a gated action | approval engine (Week 3) |
|
||||||
| Incident | investigation narrative | `investigations/` |
|
| Incident | investigation narrative | `knowledge/sources/investigations/` |
|
||||||
| Plan | design doc for non-trivial work | `plans/` |
|
| Plan | design doc for non-trivial work | `plans/` |
|
||||||
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||||||
|
|
||||||
@@ -207,7 +207,7 @@ guesswork. Roughly ordered by what unblocks the most:
|
|||||||
populate more as specific services turn out to need non-default risk
|
populate more as specific services turn out to need non-default risk
|
||||||
classes.
|
classes.
|
||||||
- **Incident timeline generator** — stitch ledger + signal history into
|
- **Incident timeline generator** — stitch ledger + signal history into
|
||||||
a single narrative for `investigations/` entries instead of writing
|
a single narrative for `knowledge/sources/investigations/` entries instead of writing
|
||||||
them by hand.
|
them by hand.
|
||||||
- **Secret access audit** — who-can-decrypt-what report from
|
- **Secret access audit** — who-can-decrypt-what report from
|
||||||
`.sops.yaml` + inventory `age_pubkey`s, extending what
|
`.sops.yaml` + inventory `age_pubkey`s, extending what
|
||||||
|
|||||||
@@ -20,7 +20,7 @@ fixed paths, so the wiki reorganization never moves it.
|
|||||||
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
|
| Substrate — generated host records | `hosts/*.yaml` (root) | `mcp/server.py` (`HOSTS_DIR`), `bin/homelab`; written by `mcp/build_host_files.py` |
|
||||||
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
||||||
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
| Narrative — synthesized wiki | `knowledge/wiki/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||||
| Evidence — immutable sources | `knowledge/sources/`, `investigations/` | synthesis into wiki pages |
|
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
|
||||||
|
|
||||||
## Wiki pages
|
## Wiki pages
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ procedures), investigations (incident evidence), and plans (design docs for non-
|
|||||||
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
|
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
|
||||||
exception.
|
exception.
|
||||||
|
|
||||||
|
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
|
||||||
|
reference (command cheatsheet, enrollment, Hermes agent) lives in
|
||||||
|
[`.agents/operations/`](../../operations/); investigations are sources under
|
||||||
|
`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below).
|
||||||
|
|
||||||
## Plans always live in `plans/`
|
## Plans always live in `plans/`
|
||||||
|
|
||||||
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
|
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
|
||||||
@@ -39,12 +44,12 @@ transition: "<from> -> <to>" # only for lifecycle runbooks
|
|||||||
|
|
||||||
## Investigations
|
## Investigations
|
||||||
|
|
||||||
Incident records live in `investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
|
Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
|
||||||
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
|
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
|
||||||
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
|
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
|
||||||
incidents move to `investigations/archive/`.
|
incidents move to `knowledge/sources/investigations/archive/`.
|
||||||
|
|
||||||
## The operations log
|
## The operations log
|
||||||
|
|
||||||
`plans/log.md` and `investigations/log.md` are append-only records of documentation operations on
|
`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on
|
||||||
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.
|
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
|
|||||||
|
|
||||||
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
||||||
|
|
||||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
|
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../knowledge/wiki/containers/106-auth-outpost.md). Only needed once per account.
|
||||||
|
|
||||||
**Path A — setup-key (headless/scripted onboarding):**
|
**Path A — setup-key (headless/scripted onboarding):**
|
||||||
|
|
||||||
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
|
|||||||
### DNS prerequisite
|
### DNS prerequisite
|
||||||
|
|
||||||
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
||||||
([dns.md](../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
|
([dns.md](../../knowledge/wiki/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||||
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
||||||
workstations don't by default. Options:
|
workstations don't by default. Options:
|
||||||
|
|
||||||
@@ -261,7 +261,7 @@ arguments.
|
|||||||
|
|
||||||
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
|
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
|
||||||
visible to the classifier (it's not actually durable in 0.71.2, but the
|
visible to the classifier (it's not actually durable in 0.71.2, but the
|
||||||
ControlMaster block is — see [runbook-dpkg-interrupted](../.agents/skills/runbook-dpkg-interrupted/SKILL.md)
|
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
|
||||||
for context), drop a free-text rule into `autoMode.allow` describing the
|
for context), drop a free-text rule into `autoMode.allow` describing the
|
||||||
authorization. Optional.
|
authorization. Optional.
|
||||||
|
|
||||||
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
|||||||
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
||||||
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
||||||
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
||||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../knowledge/wiki/containers/106-auth-outpost.md). |
|
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md). |
|
||||||
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
||||||
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
||||||
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# Operations cheatsheet
|
# Operations cheatsheet
|
||||||
|
|
||||||
Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
Run from the [hubris host](../../knowledge/wiki/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||||
|
|
||||||
## Proxmox CLI
|
## Proxmox CLI
|
||||||
|
|
||||||
@@ -8,13 +8,13 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work
|
|||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `pct list` / `qm list` | List LXC containers / VMs |
|
| `pct list` / `qm list` | List LXC containers / VMs |
|
||||||
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||||||
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../knowledge/wiki/infrastructure/media-permissions.md)) |
|
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md)) |
|
||||||
| `pct enter <id>` | Shell into a container |
|
| `pct enter <id>` | Shell into a container |
|
||||||
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||||||
| `pvesm status` | Storage pools status |
|
| `pvesm status` | Storage pools status |
|
||||||
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||||||
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||||||
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Hermes cron — see [monitoring](../../knowledge/wiki/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||||
| `pveversion` | PVE version |
|
| `pveversion` | PVE version |
|
||||||
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||||||
|
|
||||||
@@ -22,22 +22,22 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work
|
|||||||
|
|
||||||
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||||||
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||||||
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../knowledge/wiki/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||||
|
|
||||||
## Reverse proxy
|
## Reverse proxy
|
||||||
|
|
||||||
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../knowledge/wiki/containers/121-caddy.md).
|
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../knowledge/wiki/containers/121-caddy.md).
|
||||||
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||||
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||||||
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||||||
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||||||
|
|
||||||
## DNS
|
## DNS
|
||||||
|
|
||||||
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../knowledge/wiki/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||||
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||||||
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||||||
- See [DNS](../knowledge/wiki/infrastructure/dns.md).
|
- See [DNS](../../knowledge/wiki/infrastructure/dns.md).
|
||||||
|
|
||||||
## Web access
|
## Web access
|
||||||
|
|
||||||
@@ -49,7 +49,7 @@ Run from the [hubris host](../knowledge/wiki/hosts/hubris.md) as root. When work
|
|||||||
- `ras-mc-ctl --errors` — full event log
|
- `ras-mc-ctl --errors` — full event log
|
||||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||||||
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||||||
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/archive/2026-04-21-hubris-crash-loop.md))
|
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||||
|
|
||||||
## Fleet apt operations
|
## Fleet apt operations
|
||||||
|
|
||||||
@@ -80,12 +80,12 @@ See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
|||||||
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Hermes, or the Oikos Console's `/approvals` page) |
|
||||||
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||||||
|
|
||||||
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../oikos/console/deploy/README.md).
|
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../oikos/console/deploy/README.md).
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Hubris host](../knowledge/wiki/hosts/hubris.md)
|
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
|
||||||
- [Containers index](../knowledge/wiki/containers/index.md)
|
- [Containers index](../../knowledge/wiki/containers/index.md)
|
||||||
- [DNS](../knowledge/wiki/infrastructure/dns.md)
|
- [DNS](../../knowledge/wiki/infrastructure/dns.md)
|
||||||
- [Monitoring](../knowledge/wiki/infrastructure/monitoring.md)
|
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
|
||||||
- [Auto-deploy](../knowledge/wiki/infrastructure/auto-deploy.md)
|
- [Auto-deploy](../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||||
- [Runbook: dpkg-interrupted recovery](../.agents/skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
||||||
@@ -169,9 +169,9 @@ every tool call, use `approve`. See
|
|||||||
Hermes flow assumes is done.
|
Hermes flow assumes is done.
|
||||||
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
|
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
|
||||||
session start (via `~/.config/goose/.goosehints`).
|
session start (via `~/.config/goose/.goosehints`).
|
||||||
- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key
|
- [`bin/hermes`](../../bin/hermes) — the wrapper that decrypts the OpenRouter key
|
||||||
and execs `goose session`.
|
and execs `goose session`.
|
||||||
- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block.
|
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-hermes` flag's install block.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
|
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
||||||
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
|
`sources / wiki / index / log` model: a durable synthesized layer (`knowledge/wiki/`) built on top
|
||||||
of immutable evidence (`knowledge/sources/`, `investigations/`), with pure-listing indexes and an
|
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
|
||||||
append-only operations log.
|
append-only operations log.
|
||||||
|
|
||||||
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
||||||
@@ -11,7 +11,7 @@ moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for t
|
|||||||
|
|
||||||
## Layers
|
## Layers
|
||||||
|
|
||||||
- **Sources** are immutable raw material: incident records (`investigations/`), external reference
|
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
|
||||||
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
||||||
Read them; do not rewrite them into other sources.
|
Read them; do not rewrite them into other sources.
|
||||||
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
|
- **Wiki** (`knowledge/wiki/`) is the synthesized, authoritative current-state layer: one page per
|
||||||
|
|||||||
@@ -96,7 +96,7 @@ What it looks like after.
|
|||||||
Changelog entries to write, index status to update.
|
Changelog entries to write, index status to update.
|
||||||
```
|
```
|
||||||
|
|
||||||
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
|
### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# YYYY-MM-DD — <title>
|
# YYYY-MM-DD — <title>
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
|
|||||||
Goal: bring a new host (workstation, LXC, VM) into inventory and the
|
Goal: bring a new host (workstation, LXC, VM) into inventory and the
|
||||||
secrets model, with mesh membership only where it's actually needed.
|
secrets model, with mesh membership only where it's actually needed.
|
||||||
This wraps the existing `homelab client add` flow — see
|
This wraps the existing `homelab client add` flow — see
|
||||||
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md) for
|
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
|
||||||
the full walkthrough; this runbook is the risk/lifecycle framing.
|
the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||||
|
|
||||||
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ own repo) and get it live, safely.
|
|||||||
`oikos/policy.yaml`.
|
`oikos/policy.yaml`.
|
||||||
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||||
directly — tracked configs change by commit + push, per
|
directly — tracked configs change by commit + push, per
|
||||||
[OIKOS.md](../../../OIKOS.md) conventions).
|
[OIKOS.md](../../OIKOS.md) conventions).
|
||||||
3. Make the change, commit, push to `main`.
|
3. Make the change, commit, push to `main`.
|
||||||
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
||||||
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
|
[infrastructure/auto-deploy.md](../../../knowledge/wiki/infrastructure/auto-deploy.md) for
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ Goal: understand what broke and why, before touching anything.
|
|||||||
5. Check the change ledger for recent mutations to the affected entity
|
5. Check the change ledger for recent mutations to the affected entity
|
||||||
or anything upstream of it: `homelab service <name> history` (once
|
or anything upstream of it: `homelab service <name> history` (once
|
||||||
populated) or grep `ledger/*.jsonl`.
|
populated) or grep `ledger/*.jsonl`.
|
||||||
6. Write findings to a new `investigations/<date>-<slug>.md` — symptom,
|
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
|
||||||
timeline, root cause, fix applied, prevention. This is the durable
|
timeline, root cause, fix applied, prevention. This is the durable
|
||||||
record; don't rely on chat history.
|
record; don't rely on chat history.
|
||||||
|
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Operations cheatsheet](../../../operations/commands.md)
|
- [Operations cheatsheet](../../operations/commands.md)
|
||||||
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
|
- [Auto-deploy pipelines](../../../knowledge/wiki/infrastructure/auto-deploy.md)
|
||||||
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
|
- [Hubris host page](../../../knowledge/wiki/hosts/hubris.md)
|
||||||
|
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ the operator to run `homelab client add <hostname>` from an existing client.
|
|||||||
- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via
|
- `/opt/homelab-context/knowledge/wiki/infrastructure/dns.md` — split-horizon DNS via
|
||||||
Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network`
|
Technitium on [dns (107)](knowledge/wiki/containers/107-dns.md). `*.hubris.network`
|
||||||
resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN.
|
resolves to 192.168.x.x on the LAN and to mesh addresses off-LAN.
|
||||||
- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet
|
- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet
|
||||||
for pct, caddy, DNS, and the Oikos command surface. Use these verbs when
|
for pct, caddy, DNS, and the Oikos command surface. Use these verbs when
|
||||||
you take actions.
|
you take actions.
|
||||||
|
|
||||||
@@ -76,9 +76,9 @@ Grep is fine for browsing or when MCP is unreachable.
|
|||||||
## 4. Wiki conventions
|
## 4. Wiki conventions
|
||||||
|
|
||||||
- Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`;
|
- Narrative pages live under `knowledge/wiki/{containers,hosts,vms,infrastructure}/`;
|
||||||
runbook procedures under `.agents/skills/<name>/SKILL.md`; other procedural
|
incident records under `knowledge/sources/investigations/`; runbook procedures under
|
||||||
pages under `investigations/`, `operations/`, `plans/`.
|
`.agents/skills/<name>/SKILL.md`; operator reference under `.agents/operations/`;
|
||||||
Cross-link liberally; orphans are bugs.
|
design docs under `plans/`. Cross-link liberally; orphans are bugs.
|
||||||
- Every page ends with a `## Changelog` section, entries in reverse-chrono
|
- Every page ends with a `## Changelog` section, entries in reverse-chrono
|
||||||
order:
|
order:
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +0,0 @@
|
|||||||
# CAVEMAN.md — moved
|
|
||||||
|
|
||||||
Agent chat-mode rules now live at [`.agents/shared/caveman.md`](.agents/shared/caveman.md).
|
|
||||||
@@ -1,7 +0,0 @@
|
|||||||
# CONTRIBUTING.md — moved
|
|
||||||
|
|
||||||
Documentation conventions are now split across `.agents/shared/`:
|
|
||||||
|
|
||||||
- [`page-templates.md`](.agents/shared/page-templates.md) — per-page-type structural templates, linking discipline, changelog hygiene, same-session update rule.
|
|
||||||
- [`writing-style.md`](.agents/shared/writing-style.md) — prose voice, banned vocabulary, page shape.
|
|
||||||
- [`llm-wiki.md`](.agents/shared/llm-wiki.md) — the sources / wiki / index / log layer model.
|
|
||||||
3
OIKOS.md
3
OIKOS.md
@@ -1,3 +0,0 @@
|
|||||||
# OIKOS.md — moved
|
|
||||||
|
|
||||||
The Oikos operating model now lives at [`.agents/OIKOS.md`](.agents/OIKOS.md).
|
|
||||||
10
README.md
10
README.md
@@ -55,12 +55,12 @@ See the full table with IPs, hosts, mounts, and status in
|
|||||||
- [Homelab context distribution](knowledge/wiki/infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
|
- [Homelab context distribution](knowledge/wiki/infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
|
||||||
|
|
||||||
### Investigations
|
### Investigations
|
||||||
Time-stamped incident notes / experiments in [`investigations/index.md`](investigations/index.md).
|
Time-stamped incident notes / experiments in [`investigations/index.md`](knowledge/sources/investigations/index.md).
|
||||||
Resolved cases move to [`investigations/archive/`](investigations/archive/).
|
Resolved cases move to [`investigations/archive/`](knowledge/sources/investigations/archive/).
|
||||||
|
|
||||||
### Operations
|
### Operations
|
||||||
- [Command cheatsheet](operations/commands.md)
|
- [Command cheatsheet](.agents/operations/commands.md)
|
||||||
- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
|
- [Agent enrollment](.agents/operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@ When you change a node:
|
|||||||
1. Update the relevant page (config snapshot, ports, mounts).
|
1. Update the relevant page (config snapshot, ports, mounts).
|
||||||
2. Add a changelog entry at the bottom of that page.
|
2. Add a changelog entry at the bottom of that page.
|
||||||
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
|
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
|
||||||
4. If it's an incident, add an entry to [`investigations/`](investigations/index.md).
|
4. If it's an incident, add an entry to [`investigations/`](knowledge/sources/investigations/index.md).
|
||||||
|
|
||||||
## See also
|
## See also
|
||||||
|
|
||||||
|
|||||||
@@ -1929,7 +1929,7 @@ def main() -> int:
|
|||||||
csub_add.add_argument("--with-hermes", action="store_true",
|
csub_add.add_argument("--with-hermes", action="store_true",
|
||||||
help="also grant secrets/openrouter-api-key.yaml so this "
|
help="also grant secrets/openrouter-api-key.yaml so this "
|
||||||
"host can run the Hermes agent (see "
|
"host can run the Hermes agent (see "
|
||||||
"operations/hermes-agent.md). Combine with --finalize-pubkey.")
|
".agents/operations/hermes-agent.md). Combine with --finalize-pubkey.")
|
||||||
csub_add.set_defaults(func=cmd_client_add)
|
csub_add.set_defaults(func=cmd_client_add)
|
||||||
csub_rm = csub.add_parser("remove")
|
csub_rm = csub.add_parser("remove")
|
||||||
csub_rm.add_argument("name")
|
csub_rm.add_argument("name")
|
||||||
|
|||||||
@@ -507,7 +507,7 @@ if [ "$WITH_HERMES" -eq 1 ]; then
|
|||||||
Linux) HERMES_LINK=/root/HERMES.md ;;
|
Linux) HERMES_LINK=/root/HERMES.md ;;
|
||||||
Darwin) HERMES_LINK=/etc/HERMES.md ;;
|
Darwin) HERMES_LINK=/etc/HERMES.md ;;
|
||||||
esac
|
esac
|
||||||
run "ln -sfn '$CLONE_DIR/HERMES.md' '$HERMES_LINK'"
|
run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$HERMES_LINK'"
|
||||||
echo "[bootstrap] linked HERMES.md → $HERMES_LINK"
|
echo "[bootstrap] linked HERMES.md → $HERMES_LINK"
|
||||||
|
|
||||||
# 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the
|
# 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the
|
||||||
@@ -563,7 +563,7 @@ PYEOF
|
|||||||
|
|
||||||
# 5. Symlink HERMES.md as the global .goosehints — Goose injects it into
|
# 5. Symlink HERMES.md as the global .goosehints — Goose injects it into
|
||||||
# the system prompt on every session start.
|
# the system prompt on every session start.
|
||||||
run "ln -sfn '$CLONE_DIR/HERMES.md' '$GOOSEHINTS'"
|
run "ln -sfn '$CLONE_DIR/.agents/HERMES.md' '$GOOSEHINTS'"
|
||||||
if [ "$DRY_RUN" -eq 0 ]; then
|
if [ "$DRY_RUN" -eq 0 ]; then
|
||||||
chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true
|
chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ services_hosted:
|
|||||||
systemd_unit: secrets-issuance
|
systemd_unit: secrets-issuance
|
||||||
public_host: secrets.hubris.network
|
public_host: secrets.hubris.network
|
||||||
endpoint: https://secrets.hubris.network/issue
|
endpoint: https://secrets.hubris.network/issue
|
||||||
doc_page: operations/agent-enrollment.md
|
doc_page: .agents/operations/agent-enrollment.md
|
||||||
config_repo: dtoro/Homelab-Docs
|
config_repo: dtoro/Homelab-Docs
|
||||||
note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS).
|
note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS).
|
||||||
risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class"
|
risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class"
|
||||||
|
|||||||
@@ -150,7 +150,7 @@ services:
|
|||||||
systemd_unit: secrets-issuance
|
systemd_unit: secrets-issuance
|
||||||
public_host: secrets.hubris.network
|
public_host: secrets.hubris.network
|
||||||
endpoint: https://secrets.hubris.network/issue
|
endpoint: https://secrets.hubris.network/issue
|
||||||
doc_page: operations/agent-enrollment.md
|
doc_page: .agents/operations/agent-enrollment.md
|
||||||
config_repo: dtoro/Homelab-Docs
|
config_repo: dtoro/Homelab-Docs
|
||||||
note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN
|
note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN
|
||||||
subnets in MESH_SUBNETS).
|
subnets in MESH_SUBNETS).
|
||||||
|
|||||||
@@ -7,3 +7,4 @@ each page's `## Changelog` and the Oikos change ledger, not here.
|
|||||||
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
|
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
|
||||||
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
|
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
|
||||||
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
|
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
|
||||||
|
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# Sources
|
# Sources
|
||||||
|
|
||||||
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
|
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
|
||||||
incident evidence lives in [`investigations/`](../../investigations/index.md) (written once at
|
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
|
||||||
incident time, then linked from the changelogs of the nodes they implicate).
|
incident time, then linked from the changelogs of the nodes they implicate).
|
||||||
|
|
||||||
| Slug | Reference | Summary |
|
| Slug | Reference | Summary |
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
[`ludo-mini`](../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
|
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
|
||||||
|
|
||||||
## Timeline
|
## Timeline
|
||||||
|
|
||||||
@@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30"
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Container 106 — auth-outpost](../knowledge/wiki/containers/106-auth-outpost.md)
|
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
|
||||||
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
|
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
|
||||||
- [Ingress (VPS Traefik)](../knowledge/wiki/infrastructure/ingress.md)
|
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
|
||||||
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
|
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
@@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
|
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
|
||||||
- [Caddy (121)](../knowledge/wiki/containers/121-caddy.md)
|
- [Caddy (121)](../../wiki/containers/121-caddy.md)
|
||||||
- [elementsynapse (118)](../knowledge/wiki/containers/118-elementsynapse.md)
|
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
|
||||||
- [dns-sync script](../scripts/dns-sync.py)
|
- [dns-sync script](../../../scripts/dns-sync.py)
|
||||||
- [check-caddy-backends script](../scripts/check-caddy-backends.sh)
|
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)
|
||||||
@@ -2,12 +2,12 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
[`hubris`](../../knowledge/wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
|
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
|
||||||
|
|
||||||
## Timeline
|
## Timeline
|
||||||
|
|
||||||
### 2026-04-19 — drive attached
|
### 2026-04-19 — drive attached
|
||||||
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../knowledge/wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
|
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
|
||||||
|
|
||||||
### 2026-04-19 → 2026-04-21 — first crashes
|
### 2026-04-19 → 2026-04-21 — first crashes
|
||||||
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
|
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
|
||||||
@@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly
|
|||||||
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
|
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
|
||||||
|
|
||||||
### 2026-04-22 — recurrence after 30 h 37 m
|
### 2026-04-22 — recurrence after 30 h 37 m
|
||||||
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../knowledge/wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
|
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
|
||||||
|
|
||||||
### 2026-04-22 — `cpu-epp.service` design bug fixed
|
### 2026-04-22 — `cpu-epp.service` design bug fixed
|
||||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
|
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||||
|
|
||||||
### 2026-04-22 — drive removed (A/B test)
|
### 2026-04-22 — drive removed (A/B test)
|
||||||
User physically removed the external USB drive. [Backup timers disabled](../../knowledge/wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
|
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
|
||||||
|
|
||||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||||
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
|
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
|
||||||
@@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu
|
|||||||
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
|
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
|
||||||
|
|
||||||
## Affected nodes
|
## Affected nodes
|
||||||
- [Hubris host](../../knowledge/wiki/hosts/hubris.md)
|
- [Hubris host](../../../wiki/hosts/hubris.md)
|
||||||
- [Backups (disabled)](../../knowledge/wiki/infrastructure/backups.md)
|
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
|
||||||
- [Monitoring](../../knowledge/wiki/infrastructure/monitoring.md)
|
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
|
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
|
||||||
@@ -2,9 +2,9 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
The NetBird management server (on the [VPS](../../knowledge/wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
|
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
|
||||||
|
|
||||||
Resolved by moving Authentik off [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
|
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
|
||||||
|
|
||||||
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
|
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
|
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
|
||||||
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
|
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
|
||||||
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
|
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
|
||||||
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
|
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
|
||||||
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
|
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
|
||||||
|
|
||||||
@@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
| | Before | After |
|
| | Before | After |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Authentik host | [LXC 124](../../knowledge/wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
|
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
|
||||||
| Version | `2026.2.2` | `2026.5.2` |
|
| Version | `2026.2.2` | `2026.5.2` |
|
||||||
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
|
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
|
||||||
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
|
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
|
||||||
@@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
|
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
|
||||||
|
|
||||||
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../knowledge/wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
|
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
|
||||||
|
|
||||||
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
|
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
|
||||||
|
|
||||||
@@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R
|
|||||||
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
|
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
|
||||||
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
|
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
|
||||||
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
|
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
|
||||||
- **Reconcile [124-authentik](../../knowledge/wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
|
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
|
||||||
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
|
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [124 — authentik](../../knowledge/wiki/containers/106-auth-outpost.md)
|
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
|
||||||
- [DNS split-horizon](../../knowledge/wiki/infrastructure/dns.md)
|
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
|
||||||
- [Public ingress (VPS traefik)](../../knowledge/wiki/infrastructure/ingress.md)
|
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
|
||||||
- [Mesh migration](../../knowledge/wiki/infrastructure/mesh.md)
|
- [Mesh migration](../../../wiki/infrastructure/mesh.md)
|
||||||
@@ -54,7 +54,7 @@ We considered three options before building this:
|
|||||||
|
|
||||||
| Option | Outcome |
|
| Option | Outcome |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../../investigations/index.md)). Rejected. |
|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
|
||||||
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
|
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
|
||||||
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
|
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# 106 — `auth-outpost`
|
# 106 — `auth-outpost`
|
||||||
|
|
||||||
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `auth-outpost`
|
- **Hostname:** `auth-outpost`
|
||||||
@@ -12,7 +12,7 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co
|
|||||||
|
|
||||||
## Role
|
## Role
|
||||||
|
|
||||||
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md).
|
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||||
|
|
||||||
## Service / port map
|
## Service / port map
|
||||||
| Service | Listen | Notes |
|
| Service | Listen | Notes |
|
||||||
@@ -45,12 +45,12 @@ Fix: the LAN outpost gets its **own** domain.
|
|||||||
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
|
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
|
||||||
- [Caddy (121)](121-caddy.md) — forward-auth consumer
|
- [Caddy (121)](121-caddy.md) — forward-auth consumer
|
||||||
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
|
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
|
||||||
- [Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)
|
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-06-06 — Authentik session lifetime extended to 30 days
|
### 2026-06-06 — Authentik session lifetime extended to 30 days
|
||||||
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md).
|
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
|
||||||
|
|
||||||
### 2026-06-01 — created; forward-auth cut over from LXC 124
|
### 2026-06-01 — created; forward-auth cut over from LXC 124
|
||||||
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
|
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ audio content against a local server instead of the official cloud.
|
|||||||
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
|
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
|
||||||
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
|
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
|
||||||
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
|
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
|
||||||
[OIKOS.md](../../../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
|
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
|
||||||
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
|
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
|
||||||
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
|
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
|
||||||
for it — but it never got its own inventory entry or doc page until now.
|
for it — but it never got its own inventory entry or doc page until now.
|
||||||
@@ -50,7 +50,7 @@ for the first time.
|
|||||||
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
|
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
|
||||||
taken by this container
|
taken by this container
|
||||||
- [Containers index](index.md)
|
- [Containers index](index.md)
|
||||||
- [OIKOS.md](../../../OIKOS.md) — drift detector that caught this
|
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
|||||||
@@ -95,7 +95,7 @@ after this set.
|
|||||||
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
|
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
|
||||||
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
|
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
|
||||||
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
|
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
|
||||||
[agent-enrollment.md troubleshooting](../../../operations/agent-enrollment.md#troubleshooting) if it recurs
|
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
|
||||||
on future LXC bootstraps.
|
on future LXC bootstraps.
|
||||||
|
|
||||||
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
|
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
|
|||||||
## At a glance
|
## At a glance
|
||||||
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
||||||
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
||||||
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
||||||
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
||||||
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
||||||
@@ -113,8 +113,8 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
|
|||||||
- [Media permissions](../infrastructure/media-permissions.md)
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../infrastructure/monitoring.md)
|
||||||
- [Backups (disabled)](../infrastructure/backups.md)
|
- [Backups (disabled)](../infrastructure/backups.md)
|
||||||
- [Operations cheatsheet](../../../operations/commands.md)
|
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||||
- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md)
|
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||||
- [strong — Proxmox host](strong.md)
|
- [strong — Proxmox host](strong.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
@@ -138,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`.
|
|||||||
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
||||||
|
|
||||||
### 2026-04-29 — relocated to better-ventilated spot
|
### 2026-04-29 — relocated to better-ventilated spot
|
||||||
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
||||||
|
|
||||||
### 2026-04-28 — Phase 1 WiFi failover
|
### 2026-04-28 — Phase 1 WiFi failover
|
||||||
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
||||||
@@ -147,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov
|
|||||||
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
||||||
|
|
||||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||||
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
||||||
|
|
||||||
### 2026-04-22 — drive removal A/B test
|
### 2026-04-22 — drive removal A/B test
|
||||||
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
||||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||||
@@ -159,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `
|
|||||||
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
||||||
|
|
||||||
### 2026-04-21 — `cpu-epp.service` deployed
|
### 2026-04-21 — `cpu-epp.service` deployed
|
||||||
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|||||||
@@ -50,13 +50,13 @@ Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
|
|||||||
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
|
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
|
||||||
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
|
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
|
||||||
5-min sync timer are live; `homelab whoami` resolves correctly. See
|
5-min sync timer are live; `homelab whoami` resolves correctly. See
|
||||||
[agent-enrollment.md](../../../operations/agent-enrollment.md).
|
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
|
||||||
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
|
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
|
||||||
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
|
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
|
||||||
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
|
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
|
||||||
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
|
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
|
||||||
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
|
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
|
||||||
["Granting a secret to a new client"](../../../operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
|
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
|
||||||
|
|
||||||
## Cluster membership
|
## Cluster membership
|
||||||
|
|
||||||
@@ -79,7 +79,7 @@ needed going forward.
|
|||||||
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
|
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
|
||||||
- [Network](../infrastructure/network.md)
|
- [Network](../infrastructure/network.md)
|
||||||
- [SSH access](../infrastructure/ssh-access.md)
|
- [SSH access](../infrastructure/ssh-access.md)
|
||||||
- [Agent enrollment](../../../operations/agent-enrollment.md)
|
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
|
|||||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||||
- [Backups (disabled)](backups.md)
|
- [Backups (disabled)](backups.md)
|
||||||
- [Operations cheatsheet](../../../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ See [132-rclone](../containers/132-rclone.md) for the full design.
|
|||||||
|
|
||||||
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
||||||
|
|
||||||
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
@@ -36,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl
|
|||||||
|
|
||||||
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
||||||
|
|
||||||
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
||||||
|
|
||||||
@@ -111,7 +111,7 @@ Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud
|
|||||||
|
|
||||||
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
||||||
|
|
||||||
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md).
|
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
## Thermal monitoring
|
## Thermal monitoring
|
||||||
|
|
||||||
@@ -122,7 +122,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
|
|||||||
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||||
- [Monitoring](monitoring.md)
|
- [Monitoring](monitoring.md)
|
||||||
- [Auto-deploy](auto-deploy.md)
|
- [Auto-deploy](auto-deploy.md)
|
||||||
- [Investigation: 2026-04-21 crash loop](../../../investigations/archive/2026-04-21-hubris-crash-loop.md)
|
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
@@ -133,7 +133,7 @@ Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclon
|
|||||||
Initial documentation. Status remains DISABLED.
|
Initial documentation. Status remains DISABLED.
|
||||||
|
|
||||||
### 2026-04-22 — DISABLED
|
### 2026-04-22 — DISABLED
|
||||||
Drive removed as the A/B test in the [crash investigation](../../../investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
||||||
|
|
||||||
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
||||||
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
||||||
|
|||||||
@@ -110,7 +110,7 @@ All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float o
|
|||||||
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||||||
|
|
||||||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||||||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||||||
|
|
||||||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||||||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
|||||||
self-locating and able to read the same source of truth.
|
self-locating and able to read the same source of truth.
|
||||||
|
|
||||||
Operational walkthrough for enrolling a new client lives in
|
Operational walkthrough for enrolling a new client lives in
|
||||||
[operations/agent-enrollment.md](../../../operations/agent-enrollment.md); this
|
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
|
||||||
page is the architecture reference.
|
page is the architecture reference.
|
||||||
|
|
||||||
## What's where
|
## What's where
|
||||||
@@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Operations: agent enrollment](../../../operations/agent-enrollment.md) — the
|
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
|
||||||
step-by-step for adding a new client
|
step-by-step for adding a new client
|
||||||
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
||||||
pipelines (and the rest of the lab's webhook pipelines)
|
pipelines (and the rest of the lab's webhook pipelines)
|
||||||
|
|||||||
@@ -62,5 +62,5 @@ are documented in their own pages. Each system below links to its full doc.
|
|||||||
|
|
||||||
- [README](../../../README.md) — entry point
|
- [README](../../../README.md) — entry point
|
||||||
- [Containers index](../containers/index.md)
|
- [Containers index](../containers/index.md)
|
||||||
- [Operations cheatsheet](../../../operations/commands.md)
|
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||||
- [OIKOS operating model](../../../OIKOS.md)
|
- [OIKOS operating model](../../../.agents/OIKOS.md)
|
||||||
@@ -49,7 +49,7 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8
|
|||||||
|
|
||||||
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
||||||
|
|
||||||
Since 2026-05-31 [Authentik runs on the VPS itself](../../../investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
||||||
|
|
||||||
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
||||||
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
||||||
@@ -91,7 +91,7 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `
|
|||||||
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
|
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
|
||||||
|
|
||||||
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
||||||
Authentik migrated onto the VPS ([investigation](../../../investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
### 2026-04-28 — wiki entry created
|
||||||
Initial documentation.
|
Initial documentation.
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
||||||
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../../investigations/archive/2026-05-31-authentik-vps-migration.md).
|
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||||
|
|
||||||
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
||||||
|
|
||||||
|
|||||||
@@ -178,7 +178,7 @@ done
|
|||||||
|
|
||||||
- [Mesh migration](mesh.md)
|
- [Mesh migration](mesh.md)
|
||||||
- [VPS hardening](vps-hardening.md)
|
- [VPS hardening](vps-hardening.md)
|
||||||
- [Agent enrollment](../../../operations/agent-enrollment.md)
|
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||||
- [Homelab CLI](../../../bin/homelab)
|
- [Homelab CLI](../../../bin/homelab)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
- backend: host:apps
|
- backend: host:apps
|
||||||
- url: https://secrets.hubris.network/issue
|
- url: https://secrets.hubris.network/issue
|
||||||
- doc: operations/agent-enrollment.md
|
- doc: .agents/operations/agent-enrollment.md
|
||||||
- config repo: dtoro/Homelab-Docs
|
- config repo: dtoro/Homelab-Docs
|
||||||
- risk notes: identity issuance — any change is security-sensitive; key operations are destructive-class
|
- risk notes: identity issuance — any change is security-sensitive; key operations are destructive-class
|
||||||
|
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ at `pct create` time.
|
|||||||
|
|
||||||
## Why
|
## Why
|
||||||
|
|
||||||
Week-3 reliability layer (see [OIKOS.md](../OIKOS.md)) wants trend
|
Week-3 reliability layer (see [OIKOS.md](../.agents/OIKOS.md)) wants trend
|
||||||
signals — "disk full in ~9 days at current rate", temperature creep —
|
signals — "disk full in ~9 days at current rate", temperature creep —
|
||||||
which need a real time-series store. The scheduler
|
which need a real time-series store. The scheduler
|
||||||
([oikos/scheduler.py](../oikos/scheduler.py)) currently does point-in-time
|
([oikos/scheduler.py](../oikos/scheduler.py)) currently does point-in-time
|
||||||
|
|||||||
@@ -14,7 +14,7 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
CONTEXT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
CONTEXT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
HERMES_MD="$CONTEXT_DIR/HERMES.md"
|
HERMES_MD="$CONTEXT_DIR/.agents/HERMES.md"
|
||||||
SOUL_MD="${HOME}/.hermes/SOUL.md"
|
SOUL_MD="${HOME}/.hermes/SOUL.md"
|
||||||
|
|
||||||
# Colors for output (only when connected to a terminal)
|
# Colors for output (only when connected to a terminal)
|
||||||
|
|||||||
Reference in New Issue
Block a user