- New investigation doc: 2026-06-03-moonlight-sunshine-wifi-jitter.md
- Updated investigations/index.md with link and status
- Added .gitignore for .DS_Store
- Saved Hermes planning docs from recent sessions
- scripts/dns-sync.py: reconcile Technitium named A-records -> NetBird managed
zone via API (cron */10 on dns LXC 107). Single authoring source; kills the
manual drift behind the auth/sso/nfs-export saga.
- secrets/netbird-pat.yaml: sops-encrypted NetBird API PAT for the sync.
- dns.md / 107-dns.md: document the sync model + why forward-to-Technitium was
abandoned (NetBird self-IP / nameserver-group quirks).
- Cleanup: removed inert Mac secondary; reverted primary AXFR; home-lab-dns ->
[192.168.8.2] (1/1 Available); deleted vestigial Proxmox Names group.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hosts/hubris.md:
- Update At a glance network section: vmbr1 uplink (192.168.178.10),
vmbr0 portless internal bridge with 192.168.8.1 alias
- Remove Phase 1 WiFi failover section (wlp3s0 disabled 2026-06-02)
- Changelog: Slate AX retired, SODOLA added, Proxmox as subnet router
containers/121-caddy.md:
- Changelog: caddy.service unit was missing from hubris1 package,
recreated manually; risk of loss on package reinstall noted
containers/107-dns.md:
- Update Who points here: Technitium DHCP hands out .2 as DNS for
homelab clients; Fritz!Box LAN clients still get Fritz!Box DNS
- Add DHCP section documenting the homelab scope (100-240, gw .1)
- Changelog: DHCP enabled 2026-06-02, replaces Slate AX DHCP
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- plans/: mark Slate AX → SODOLA migration as Done (2026-06-02)
- infrastructure/network.md: rewrite to reflect actual topology —
Fritz!OS 8.x lacks second-IP-network support, so Proxmox routes between
vmbr1 (192.168.178.10, uplink) and vmbr0 (192.168.8.x, internal portless
bridge with 192.168.8.1 alias). Static route on Fritz!Box, no NAT on Proxmox.
Technitium DHCP enabled for 192.168.8.100–240 (gateway .1, DNS .2).
Caddy service unit was missing on CT 121 — recreated and enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- tools/setup-hermes-soul.sh: new auto-setup script (no-op on non-Hermes)
- HERMES.md: canonical persona for all agent types, agent-type table
- AGENTS.md: auto-setup section lists both caveman and hermes-soul
- post-pull.sh runs it automatically on every sync
`bootstrap.sh --with-hermes` installs the Goose CLI, drops a Goose
config pinning the OpenRouter provider + Nous Hermes model + the
homelab MCP extension, symlinks `bin/hermes` and HERMES.md, and links
HERMES.md as `.goosehints` so the persona is injected as the system
prompt every session.
`bin/hermes` decrypts `secrets/openrouter-api-key.yaml` via the existing
`homelab secret` flow and execs `goose session`.
`homelab client add --with-hermes` grants the new sops secret to the
host's age_pubkey at finalize time (parallel to the existing
shared-secrets grant). `client remove` revokes it.
`operations/hermes-agent.md` covers the end-to-end flow, verification,
troubleshooting, and queues one follow-up: the MCP server still runs
SSE-only but Goose 1.x deprecated SSE — the Goose config targets
`streamable_http` and the `homelab` extension won't connect until
`mcp/server.py` migrates. The `developer` extension (shell + edit +
`homelab` CLI) carries the agent in the meantime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Domain-level forward-auth needs its own external_host domain when the IdP core
and outpost are on different hosts. sso.hubris.network -> Caddy -> LAN outpost.
Includes the redirect_uris-regeneration gotcha. Carry the DNS record into
Technitium in DNS Phase 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).
- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Management crash-loop root cause and fix documented.
Phase 6 (Authentik as Netbird IdP) marked complete and live.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
netbird mgmt might reject unknown top-level JSON keys depending on
parser flags. The rendering note stays in turnserver.conf (# comments
are valid INI syntax). After this, the rendered management.json will
be byte-identical to what's currently live on the VPS, so the dry-run
will show /opt/management.json as unchanged.
Lets the operator see exactly what would change on the VPS before
applying. Secret values from the decrypted sops files are masked as
<REDACTED> so the diff is safe to paste into chat/PRs.
The IONOS netbird VPS held two credentials in plaintext that were the last
holdouts from the homelab's sops+age secrets pattern:
- /root/turn-pass.txt (coturn long-term-credential password)
- PKCEAuthorizationFlow.ProviderConfig.ClientSecret inline in
/opt/management.json (Authentik OIDC client secret)
This commit moves both into sops-encrypted YAML in the repo and adds a render
command that recreates the VPS config files from templates + decrypted secrets:
* secrets/turn-shared-secret.yaml — encrypted `password: <coturn pwd>`
* secrets/netbird-authentik-oidc.yaml — encrypted `client_secret: <...>`
Both recipients = hubris + apps + republic-laptop (same 3 as hello.yaml).
* vps/turnserver.conf.tmpl + vps/management.json.tmpl — templates with
{{TURN_PASSWORD}} + {{AUTHENTIK_CLIENT_SECRET}} placeholders.
* bin/homelab new subcommand `render-vps-configs`:
- Decrypts both secrets locally (works on any recipient).
- Substitutes placeholders into templates.
- Diffs against current VPS state via ssh, prompts, applies atomically
(write `.new` then mv), restarts coturn + netbird-mgmt.
- --dry-run + -y flags. Hops through hubris when not running on hubris
itself, since VPS sshd is locked to hubris's pubkey.
* inventory.yaml adds the VPS as `kind: external` (new kind; reserved for
ssh-managed hosts that aren't homelab clients themselves — no age key,
no /opt/homelab-context). hosts/netbird-vps.yaml regenerated.
* SHARED_SECRETS list includes both new secrets so re-keys on enrollment
changes pick them up automatically.
After this lands + the 5-min sync propagates to hubris, run from hubris PVE
shell (or any client; hubris just skips the extra ssh hop):
homelab render-vps-configs --dry-run # see plan, no changes
homelab render-vps-configs -y # apply + restart services
Once verified working, the plaintext `/root/turn-pass.txt` should be deleted
on the VPS (the rendered /etc/turnserver.conf no longer needs it as a
reference).
Three coordinated changes addressing follow-ups from the 2026-05-21 netbird vanilla
migration, plus a related off-LAN ergonomics fix:
bin/homelab:
- New ssh_target(name, force_mesh=False) helper resolves (addr, port, user)
from inventory, honoring ssh.netbird_port (forces mesh path) and ssh.user
(default "root"). Falls back to the netbird FQDN when LAN IP fails a
cached 1.5s TCP probe — helps off-LAN operators on VPN/symmetric-NAT
paths where the netbird subnet route doesn't reach 192.168.8.0/24.
- New ssh_base() builds the full `ssh ... user@addr` invocation; hubris_ssh()
is now a back-compat shim. cmd_ssh, cmd_logs, cmd_restart, cmd_nuke
refactored to use it — no more hardcoded "root@" anywhere.
inventory.yaml:
- New ssh.user convention (root by default, explicit per workstation).
- hubris.ssh.user=root (explicit, documents convention).
- republic-laptop, mac-mini: ssh.user=dtoro. ludo-mini left default (TODO).
- Comment block in the header explains the field + why it exists (netbird-ssh
defaults to LOCAL username; "user not found" on LXCs is the gotcha).
- hosts/*.yaml regenerated from build_host_files.py.
infrastructure/mesh.md:
- Migration changelog entry updated: Device Code Stage is now configured
(was "NOT yet" — landed in d41d73f); --setup-key no longer required.
- New subsection documenting the post-migration JWT-issuer cache bug:
client/internal/engine_ssh.go's updateSSH() bails out when sshServer is
already running, so systemctl restart and netbird down/up don't refresh
the SSH JWT validator. Full daemon stop/start is the fix.
- Companion username gotcha (`netbird ssh` defaulting to local username).
operations/agent-enrollment.md:
- Three new troubleshooting rows: JWT-issuer cache, user-not-found, and
LAN-unreachable-from-mesh-peer (the new homelab CLI behavior).
Verification: ssh_target resolution against the live inventory yields
- hubris → ssh -p 22022 root@proxmox-server.netbird.selfhosted (mesh-forced)
- jellyfin/gitea → ssh root@192.168.8.x (LAN reachable, probe passed)
- republic-laptop/mac-mini → ssh dtoro@<fqdn> (per ssh.user)
- ludo-mini → ssh root@<fqdn> (default)
Two changes that together collapse new-workstation onboarding from ~7 steps
to ~2 commands:
* bootstrap.sh:
- Dep-check now AUTO-INSTALLS missing tools (apt/dnf/brew) instead of
printing instructions and exiting. Re-verifies after install.
- New pre-mesh-check block: if netbird isn't installed, installs it
from the netbird apt/dnf repo (or `brew install --cask netbird` on
Darwin), then if mgmt isn't connected, runs `netbird up
--management-url=https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`.
Operator clicks the device-code URL inline. Waits up to ~30s for
Management: Connected before continuing. Skipped on --no-secrets +
--dry-run.
* containers/124-authentik.md: replaces the "KNOWN MISSING — Device Code
Stage" subsection with a working recipe — Authentik 2026.2 routes
/device via a BRAND-level "Device code flow" field, not a provider
field. Documented stage bindings for a `default-device-code-flow`
flow (identification → password → user-login → consent) and the
brand-level binding step.
* operations/agent-enrollment.md: Path B (interactive `netbird up`) is
now the default; Path A (setup-key) demoted to "headless/scripted"
alternative. "Install dependencies" section collapsed into a note
that bootstrap handles it, with the manual recipes kept in a
collapsible <details> block for air-gapped use.
The flow uniquely available to lab owners (single Authentik user today)
still relies on the first-time-owner sqlite promotion documented in
124-authentik.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Updates to four pages reflecting the combined → vanilla mgmt+signal+relay+coturn
cutover and the IONOS-3478-firewall-exception discovery:
* infrastructure/mesh.md — rewrites the ICE/STUN section to cover the new
TURN endpoint, the IONOS upstream TCP-3478 filtering (load-bearing,
undocumented before today), and the verification probe. New changelog
entry covering the migration outcome + Device Code Stage gap.
* infrastructure/vps-hardening.md — "At a glance" lists the new 6-service
docker stack + host coturn. Firewall section notes the new
`iifname ens6 tcp dport 3478 accept` rule plus the IONOS upstream
exception. New changelog entry.
* containers/124-authentik.md — replaces the "Netbird IdP integration —
DEFERRED" section with the LANDED state: Provider details (Public
client type — Confidential breaks PKCE on the dashboard SPA), the
first-time owner-promotion sqlite recipe, the missing Device Code
Stage gap + workaround (setup-keys), and a note that the old 2026-04-22
pre-work Provider/App is now obsolete and safe to delete. Updated
changelog (Phase 6 landed).
* operations/agent-enrollment.md — new "Getting onto Netbird" subsection
explaining the setup-key path (currently the only working flow until
Device Code Stage lands) and why direct OIDC from the public internet
fails (auth.hubris.network is mesh-only-reachable). Prerequisites table
row updated to point at the new section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundles the documentation slice of the apt-sweep backlog:
* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
recovery) + Path B (PVE web Shell when the netbird mesh broke
alongside the dpkg state, as happened during Wave 6 on hubris).
Closes B2.
* operations/commands.md — new "Fleet apt operations" section
documenting `homelab apt-audit` and `homelab apt-upgrade`
(--status / --safe / --force). Adds the dpkg-interrupted runbook to
Related.
* operations/agent-enrollment.md —
- new "Claude Code permissions for fleet ops" section with the
`permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
`Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
- two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
(G2) and the cosmetic netbird DNS-probe warning.
* infrastructure/auto-deploy.md — new "Custom-built binaries that
overlap apt-managed paths" section describing the two acceptable
patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
apt-mark hold) and the discovery path via `homelab apt-audit`'s
NONAPT column. Closes D3.
Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
G1 from the apt-sweep backlog. After ssh ControlMaster setup, install
the `mcp[cli]` python package via pipx so `homelab mcp <tool>` works
out-of-the-box on new workstations. Skipped on LXCs / VMs.
Idempotent (`command -v mcp` guard), respects --dry-run, falls back
through brew (Darwin) → dnf → apt for pipx itself if not already
installed. Runs the install as $SUDO_USER (not root) so the binary
lands in the user's pipx env.
This closes one of the "discovered missing dep" gaps from the 2026-05-21
fleet sweep: republic-laptop had pipx-installed homelab CLI but no `mcp`
binary, so `homelab mcp <tool>` died with an instructional message
instead of just working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
F3 from the apt-sweep backlog. `homelab apt-upgrade --safe ...` takes a
pre-upgrade snapshot per LXC before launching the apt run, so rollback is
trivial if anything regresses.
Snapshot strategy per target:
- LXC: try `pct snapshot <id> preupgrade_<timestamp>` first (CoW, near
instant). If that refuses ("snapshot feature is not available" — the
failure mode for LXCs with host bind-mounts like `mp0: /mnt/library`),
fall back to `vzdump <id> --mode snapshot --storage local --compress
zstd`. Slower but works on bind-mounted LXCs.
- hubris (PVE host): skipped intentionally; no host-level snapshot in v1.
If any snapshot fails the entire run refuses unless --force is passed.
Snapshot rollback hints are printed after launch so the operator has the
recovery path one copy/paste away.
Validated 2026-05-21: py_compile clean; --status mode works on bind-
mounted LXCs (skips snapshot path). Live snapshot test deferred to next
real fleet sweep to avoid leaving stale artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new subcommands (B1+H1 + C1+C2 from the apt-sweep backlog):
* `homelab apt-audit [--target HOST]`
Per-host pre-flight table reporting dpkg-interrupted state, holds,
upgradable count, non-apt binaries in system paths, and DNS health.
Exits nonzero if any target has unconfigured packages (would catch the
latent dpkg-interrupted state we found on caddy + nextcloud before
touching them).
* `homelab apt-upgrade [--target HOST | --all] [--status] [--force]`
Launches `apt update && apt upgrade` inside a transient systemd unit
(`apt-upgrade-<hostname>`) on each target via `systemd-run --collect`,
so the run survives ssh teardown. Output tee'd to
`/var/log/homelab-apt-upgrade.log`. Apt configured with
`Acquire::Retries=3` + `ForceIPv4=true` to absorb transient mirror
blips (would have prevented the jellyfin ffmpeg7 fetch failure that
rolled back 131 packages mid-Wave-3).
Pre-flight audit gate refuses to run on hosts with dpkg-interrupted
state; override with --force.
`--status` mode prints each target's unit state and tails the upgrade
log; safe to run before, during, and after.
Also fixes a latent bug in `hubris_ssh()`: the function joined hubris's
`lan_ip` (192.168.8.77) with the netbird-only port 22022. Port 22022
listens on hubris's netbird interface ONLY — not on the LAN interface,
not even via the netbird subnet route. Result: every off-LAN client
hitting `homelab pct/logs/restart/status` got "Connection refused".
Now uses the netbird FQDN when netbird_port is in use; falls back to
host_address otherwise.
Targets exclude VMs and workstations (they have their own update flows).
Standard apt targets = hubris + every LXC (kind=lxc with a pve_id).
Validated 2026-05-21 on republic-laptop:
- apt-audit reported clean dpkg + 40 upgradable on hubris (the still-
deferred PVE/kernel packages) + 0 on every LXC, DNS ok across the
fleet
- apt-upgrade --target mule-photos-new launched the unit, ran apt to
completion (0 upgrades pending), log captured with =START= / =END
rc=0= markers, --status reports unit + log correctly
- --all --status fans out across all 16 targets without issue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
