Domain-level forward-auth needs its own external_host domain when the IdP core
and outpost are on different hosts. sso.hubris.network -> Caddy -> LAN outpost.
Includes the redirect_uris-regeneration gotcha. Carry the DNS record into
Technitium in DNS Phase 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).
- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Management crash-loop root cause and fix documented.
Phase 6 (Authentik as Netbird IdP) marked complete and live.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The IONOS netbird VPS held two credentials in plaintext that were the last
holdouts from the homelab's sops+age secrets pattern:
- /root/turn-pass.txt (coturn long-term-credential password)
- PKCEAuthorizationFlow.ProviderConfig.ClientSecret inline in
/opt/management.json (Authentik OIDC client secret)
This commit moves both into sops-encrypted YAML in the repo and adds a render
command that recreates the VPS config files from templates + decrypted secrets:
* secrets/turn-shared-secret.yaml — encrypted `password: <coturn pwd>`
* secrets/netbird-authentik-oidc.yaml — encrypted `client_secret: <...>`
Both recipients = hubris + apps + republic-laptop (same 3 as hello.yaml).
* vps/turnserver.conf.tmpl + vps/management.json.tmpl — templates with
{{TURN_PASSWORD}} + {{AUTHENTIK_CLIENT_SECRET}} placeholders.
* bin/homelab new subcommand `render-vps-configs`:
- Decrypts both secrets locally (works on any recipient).
- Substitutes placeholders into templates.
- Diffs against current VPS state via ssh, prompts, applies atomically
(write `.new` then mv), restarts coturn + netbird-mgmt.
- --dry-run + -y flags. Hops through hubris when not running on hubris
itself, since VPS sshd is locked to hubris's pubkey.
* inventory.yaml adds the VPS as `kind: external` (new kind; reserved for
ssh-managed hosts that aren't homelab clients themselves — no age key,
no /opt/homelab-context). hosts/netbird-vps.yaml regenerated.
* SHARED_SECRETS list includes both new secrets so re-keys on enrollment
changes pick them up automatically.
After this lands + the 5-min sync propagates to hubris, run from hubris PVE
shell (or any client; hubris just skips the extra ssh hop):
homelab render-vps-configs --dry-run # see plan, no changes
homelab render-vps-configs -y # apply + restart services
Once verified working, the plaintext `/root/turn-pass.txt` should be deleted
on the VPS (the rendered /etc/turnserver.conf no longer needs it as a
reference).
Three coordinated changes addressing follow-ups from the 2026-05-21 netbird vanilla
migration, plus a related off-LAN ergonomics fix:
bin/homelab:
- New ssh_target(name, force_mesh=False) helper resolves (addr, port, user)
from inventory, honoring ssh.netbird_port (forces mesh path) and ssh.user
(default "root"). Falls back to the netbird FQDN when LAN IP fails a
cached 1.5s TCP probe — helps off-LAN operators on VPN/symmetric-NAT
paths where the netbird subnet route doesn't reach 192.168.8.0/24.
- New ssh_base() builds the full `ssh ... user@addr` invocation; hubris_ssh()
is now a back-compat shim. cmd_ssh, cmd_logs, cmd_restart, cmd_nuke
refactored to use it — no more hardcoded "root@" anywhere.
inventory.yaml:
- New ssh.user convention (root by default, explicit per workstation).
- hubris.ssh.user=root (explicit, documents convention).
- republic-laptop, mac-mini: ssh.user=dtoro. ludo-mini left default (TODO).
- Comment block in the header explains the field + why it exists (netbird-ssh
defaults to LOCAL username; "user not found" on LXCs is the gotcha).
- hosts/*.yaml regenerated from build_host_files.py.
infrastructure/mesh.md:
- Migration changelog entry updated: Device Code Stage is now configured
(was "NOT yet" — landed in d41d73f); --setup-key no longer required.
- New subsection documenting the post-migration JWT-issuer cache bug:
client/internal/engine_ssh.go's updateSSH() bails out when sshServer is
already running, so systemctl restart and netbird down/up don't refresh
the SSH JWT validator. Full daemon stop/start is the fix.
- Companion username gotcha (`netbird ssh` defaulting to local username).
operations/agent-enrollment.md:
- Three new troubleshooting rows: JWT-issuer cache, user-not-found, and
LAN-unreachable-from-mesh-peer (the new homelab CLI behavior).
Verification: ssh_target resolution against the live inventory yields
- hubris → ssh -p 22022 root@proxmox-server.netbird.selfhosted (mesh-forced)
- jellyfin/gitea → ssh root@192.168.8.x (LAN reachable, probe passed)
- republic-laptop/mac-mini → ssh dtoro@<fqdn> (per ssh.user)
- ludo-mini → ssh root@<fqdn> (default)
Updates to four pages reflecting the combined → vanilla mgmt+signal+relay+coturn
cutover and the IONOS-3478-firewall-exception discovery:
* infrastructure/mesh.md — rewrites the ICE/STUN section to cover the new
TURN endpoint, the IONOS upstream TCP-3478 filtering (load-bearing,
undocumented before today), and the verification probe. New changelog
entry covering the migration outcome + Device Code Stage gap.
* infrastructure/vps-hardening.md — "At a glance" lists the new 6-service
docker stack + host coturn. Firewall section notes the new
`iifname ens6 tcp dport 3478 accept` rule plus the IONOS upstream
exception. New changelog entry.
* containers/124-authentik.md — replaces the "Netbird IdP integration —
DEFERRED" section with the LANDED state: Provider details (Public
client type — Confidential breaks PKCE on the dashboard SPA), the
first-time owner-promotion sqlite recipe, the missing Device Code
Stage gap + workaround (setup-keys), and a note that the old 2026-04-22
pre-work Provider/App is now obsolete and safe to delete. Updated
changelog (Phase 6 landed).
* operations/agent-enrollment.md — new "Getting onto Netbird" subsection
explaining the setup-key path (currently the only working flow until
Device Code Stage lands) and why direct OIDC from the public internet
fails (auth.hubris.network is mesh-only-reachable). Prerequisites table
row updated to point at the new section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundles the documentation slice of the apt-sweep backlog:
* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
recovery) + Path B (PVE web Shell when the netbird mesh broke
alongside the dpkg state, as happened during Wave 6 on hubris).
Closes B2.
* operations/commands.md — new "Fleet apt operations" section
documenting `homelab apt-audit` and `homelab apt-upgrade`
(--status / --safe / --force). Adds the dpkg-interrupted runbook to
Related.
* operations/agent-enrollment.md —
- new "Claude Code permissions for fleet ops" section with the
`permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
`Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
- two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
(G2) and the cosmetic netbird DNS-probe warning.
* infrastructure/auto-deploy.md — new "Custom-built binaries that
overlap apt-managed paths" section describing the two acceptable
patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
apt-mark hold) and the discovery path via `homelab apt-audit`'s
NONAPT column. Closes D3.
Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds infrastructure/homelab-context.md as the architecture reference for
the cross-client context + MCP + secrets-issuance system. Updates:
- 105-apps.md: two new ## Stacks sections (homelab-mcp, secrets-issuance)
with their deploy pipelines + a row each in the public-hostname table;
changelog entry.
- auto-deploy.md: both new pipelines added to the table (one repo, two
webhooks, same push); per-pipeline notes covering the clone-per-service
pattern and the deploy.sh self-restart caveat; changelog entry.
- README.md: link to the new infrastructure page.
Operational walkthrough already lives at operations/agent-enrollment.md;
this commit is the architecture side of the same story.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container had been stopped since 2026-04-21 and was never re-enabled.
pct destroy 109 --purge cleaned up vm-109-disk-0 on local-lvm and the
config file. /mnt/library/syncthing subtree was already empty at the
time of destruction and is retained as an empty dir (no real data to
migrate or back up).
- README.md, containers/index.md: removed row, moved to "recently
destroyed" table
- hosts/hubris.md: dropped from /mnt/library subtree list, updated
containers/index summary line, added changelog entry
- infrastructure/media-permissions.md: dropped from membership table
and onboarding example, generalised pct-exec gotcha hostname,
added changelog
- vms/100-zimaos.md: dropped from "existing fleet" enumeration
- containers/102-nfs-export.md: dropped from bind-mount sibling list
(7 LXCs now, not 8)
- containers/109-syncthing.md: deleted
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After Files-UI evaluation passed (library renders as folder, thumbnails
work), flipped /etc/exports from ro to rw on LXC 102. Tested: write from
ZimaOS appears on /mnt/library as www-data:media, confirming the
all_squash,anonuid=33,anongid=10000 design works.
Documented two architectural findings discovered this session:
- ZimaOS Drives panel sources from GET /v2/local_storage/storages (read-only
API). Network shares cannot become Drives — Files-as-folder is supported.
- Mesh peers reach ZimaOS via hubris's existing 192.168.8.0/24 netbird subnet
advertisement; no new infra needed, just DNS (Management nameserver group
for hubris.network or per-device /etc/hosts override).
User destroyed the heaper LXC on 2026-05-14. Removed it from the
container index, README quicktable, hubris host doc, and
media-permissions membership table; moved to the "recently destroyed"
archaeology list. /mnt/library/heaper retained (224 MiB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LXC 125 stood up as a Seafile CE 13.0 docker-compose deployment, behind
files.hubris.network. Authentik OAuth wired up via ak shell. No data
migration — exploration alongside Nextcloud (114).
Initial documentation of the hubris Proxmox homelab as a cross-linked
markdown wiki. Per-node pages, cross-cutting infrastructure pages, an
investigation log, and an operations cheatsheet. Each node and topic
ends with a Changelog section so changes can be tracked in-place going
forward.
Refreshed against live state on 2026-04-28 — 14 active LXCs (109
syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test
state of the 2026-04-21 hubris crash-loop investigation.
