74 Commits

Author SHA1 Message Date
a45e4f6f29 docs(teddycloud): correct IP, port, disable boxCertAuth, ESP32 onboarding
Real IP is 192.168.8.243 (not .214). Web UI is on port 8443 HTTPS,
not 8080. Caddy backend uses tls_insecure_skip_verify. boxCertAuth
disabled so ESP32 Toniebox connects without client cert extraction.
Added SD card onboarding steps for Leon's ESP32 box.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 22:13:11 +02:00
77b3a6f677 docs(teddycloud): provision LXC 131 with TeddyCloud
New privileged LXC (131) running TeddyCloud via Docker Compose.
Content bind-mounted from /mnt/library/cloud/leon. Management UI
at teddy.hubris.network via Caddy; Toniebox device traffic routed
direct to LXC:443 via Technitium prod.de.bb-online.com override.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 21:38:56 +02:00
41df77fb20 docs(runbook): update budget-from-csv with API corrections and Cookie Share calc
- Fix subscription path to /budget/subscriptions (not /subscriptions/)
- Document RRULE rejection; use cycle_interval instead
- Add GET /budget/ vs ?month= behavior note
- Add full category/payment-method ID tables
- Add Cookie Share recommendation: 2,650 €/month based on 6-mo analysis

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 11:45:34 +02:00
b67389b137 docs(operations): add budget-from-csv runbook for Yuvomi
Step-by-step procedure to import N26 CSV transactions into Yuvomi's
Budget and Subscriptions modules using the yuvomi-mcp tools.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 10:55:25 +02:00
45b7813eb7 chore(grimmory): finalize age_pubkey for LXC 130
Fills in the age public key issued by secrets-issuance after
homelab client enrollment completed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 09:17:31 +02:00
8f6f43eeb1 docs(grimmory): fix IP assignment — PVE static config, not DHCP reservation
All LXCs use ip=<addr>/24 in pve net0 config directly. No Fritz!Box
lease needed. Updated container doc and migration runbook accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 08:52:34 +02:00
c2c257aa12 docs(grimmory): correct Authentik OIDC setup notes
OIDC config migrated from Booklore DB dump — Confidential client (not
PKCE), credentials intact. offline_access scope added to Authentik
provider. Backchannel logout URL set to permanent IP 192.168.8.213.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 08:48:52 +02:00
41fe34a3c3 migrate: Booklore → Grimmory on new LXC 130
Grimmory (grimmory-tools/grimmory) is the community fork/successor of
Booklore. Moving it off the shared apps LXC (105) onto a dedicated
container (130, 192.168.8.213) to avoid the Portainer-managed stack
footgun that wiped mariadb in April.

- containers/130-grimmory.md: new container doc with compose layout,
  Authentik OIDC notes (PKCE/Public client), media-GID setup
- plans/2026-06-29-grimmory-migration.md: step-by-step runbook for
  DB dump/restore, LXC provisioning, Caddy cutover, Authentik update
- inventory.yaml: add grimmory LXC 130
- containers/105-apps.md: remove Booklore section + hostname entry
- containers/index.md: update 105 row; add 130 row
- infrastructure/media-permissions.md: add LXC 130; update Booklore→Grimmory refs
- README.md: update 105 row; add 130 row

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-29 01:11:00 +02:00
b6c9184a7b decommission: destroy plato (LXC 126)
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-28 22:32:59 +02:00
root
636c90f99c secrets: populate hermes-house-users with household Signal numbers
+4917622791635 → Yuvomi user 1 (Taco)
+491726924525  → Yuvomi user 3 (Steffi)
2026-06-28 16:50:51 +02:00
root
cef1b836c2 sops: add yuvomi-api-token + hermes-house-users secrets (house + workstations) 2026-06-28 16:02:13 +02:00
root
335498d069 client-add: house (finalize age_pubkey + grant shared + hermes secrets) 2026-06-28 15:59:47 +02:00
root
867998e9c0 hosts: regenerate — add house, mac-mini, ludo-mini; remove stale orphans 2026-06-28 15:58:36 +02:00
cb0c1a5c0d house: add Google Calendar OAuth config to docs 2026-06-27 14:40:29 +02:00
ddf541ebd3 house: switch Paperless to native DMS connector (remove WebDAV bridge) 2026-06-26 19:57:50 +02:00
4472e73ae1 Yuvomi deployment: LXC 129 house.hubris.network, Docker, Caddy, DNS, VPS traefik, Authentik OIDC, Paperless WebDAV bridge 2026-06-26 19:06:49 +02:00
59261c7ef1 hosts: generate trmnl.yaml (prereq for SOPS enrollment of LXC 128) 2026-06-24 23:41:29 +02:00
41f22c2d85 trmnl (128): auto-deploy pipeline + LAN DNS record wired
- auto-deploy.md: dtoro/terminalito pipeline (webhook 12, :9797), ALLOWED_HOST_LIST
  += 192.168.8.211, changelog
- 128-trmnl.md: auto-deploy now wired, Technitium A record done, changelog
- 107-dns.md: A record trmnl -> 192.168.8.175 (changelog)
- 104-gitea.md: webhook id 12 + allowed host (changelog)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 18:06:56 +02:00
b7cfc350b3 trmnl (128): public path verified live; root-cause the mesh outage
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:38:57 +02:00
fffa5560eb trmnl (128): provisioned — container page, inventory, ingress
LXC 128 trmnl hosts the TRMNL plugins middleware (dtoro/terminalito), polled by
TRMNL cloud. trmnl-plugins.service on :9851; Caddy block + LE cert; VPS traefik
router trmnl-public + cert mirror. Public path pending VPS<->home netbird route
recovery (was "No networks available" at provision time, artifacto/blog 504 too).
LAN Technitium record + SOPS enrollment + Google/MVG creds pending. Plan -> In Progress.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 17:24:31 +02:00
d6bfd7b8c9 plans: TRMNL plugins LXC (128) + middleware deploy pipeline
Pre-flight runbook for a dedicated trmnl LXC hosting self-hosted middleware for
TRMNL e-ink plugins (TRMNL cloud polls /<plugin>/dashboard). First consumer:
munich-home. Source repo dtoro/terminalito; this documents the fabric wiring
(provision via homelab client add, Shape-B auto-deploy, SOPS secret, caddy + VPS
traefik exposure). Status: Planned.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 16:59:03 +02:00
b71402aaed add auth-outpost (106) and dns (107) to inventory + regenerate host files 2026-06-22 09:31:04 +02:00
2a7876c711 add caveman compression mode to agent bootstrap
AGENTS.md now instructs every agent to communicate
tersely. CAVEMAN.md holds the full rules. Agents
apply from first message in each session.
2026-06-21 23:36:06 +02:00
a5da521751 docs: Fritz!Box DNSv4 set to Technitium (192.168.8.2) — household LAN now resolves *.hubris.network 2026-06-18 00:41:36 +02:00
4c83960035 docs: sab.hubris.network Authentik auth — update arriman + caddy pages 2026-06-13 11:58:21 +02:00
ddc5e8771a docs: Authentik session lifetime investigation, fix docs, changelog 2026-06-06 23:45:59 +02:00
ebf521007d docs: emphasize Caddyfile must be edited via dtoro/caddy-conf repo, not directly on LXC 2026-06-06 14:17:12 +02:00
ab3d484824 docs: add investigation entry for 2026-06-05 authentik ident stage skip (broken Trust me policy) 2026-06-05 20:08:28 +02:00
35b1a25f08 fix: remaining DHCP drifts + improve validation script
- Fixed apps LXC 105: /etc/network/interfaces was still 'iface eth0 inet
  dhcp' -> changed to static 192.168.8.205/24, killed dhclient, applied
- Fixed mule-images LXC 120: DHCP client had overridden static .136 with
  DHCP lease .108 -> killed dhclient, restored .136
- Fixed sophia docs: inventory said .157, actual Proxmox config is .109
- Updated check-caddy-backends.sh: uses curl with connect-timeout for
  reliable TCP checks, distinguishes 'port open / no HTTP' from 'unreachable'
2026-06-05 17:59:46 +02:00
b3729a941e fix: gitea DHCP drift + add caddy backend validation script 2026-06-05 13:21:58 +02:00
3a2270ba86 fix: update paperless (192.168.8.130→243) and haos (192.168.8.101→241) after DHCP lease changes
- Caddyfile updated on caddy LXC with new backend IPs
- inventory.yaml updated to reflect current DHCP assignments
- hosts/{paperless,haos}.yaml regenerated from inventory
2026-06-05 10:51:51 +02:00
6d808a1fd8 arriman: SABnzbd HTTPS enabled on port 9090 2026-06-05 00:21:56 +02:00
6b5fed040f arriman: all services behind gluetun, SAB port 8082 fix 2026-06-05 00:14:06 +02:00
2f81287eb1 cleanup: destroy LXC 124, remove stale docs, update DNS refs to Technitium 2026-06-04 23:12:06 +02:00
699fd94746 inventory: remove retired LXC 124 (authentik), update dnsmasq→dns service ref 2026-06-04 23:02:31 +02:00
2d3e979d67 docs: note 2026-06-04 Caddy proxy fixes (auth.hubris.network, sso, forward-auth) 2026-06-04 22:57:02 +02:00
b4500cafc7 mac-mini wired to LAN: update IPs and revert WiFi moonlight/sunshine configs
- mac-mini: lan_ip 192.168.8.174 → 192.168.178.182 (wired en0)
- ludo-mini: add lan_ip (192.168.178.181) and ssh user (dtoro)
- Investigation doc: mark Ethernet wiring resolved, add wired-era settings
- Moonlight: bitrate 60→80 Mbps, framepacing off
- Sunshine: bitrate 40→80 Mbps, fec 2→5%, packetsize 1024→1316
2026-06-04 22:05:51 +02:00
728e501435 investigations: add moonlight/sunshine WiFi jitter report + index entry
- New investigation doc: 2026-06-03-moonlight-sunshine-wifi-jitter.md
- Updated investigations/index.md with link and status
- Added .gitignore for .DS_Store
- Saved Hermes planning docs from recent sessions
2026-06-04 21:05:20 +02:00
e32b21072d claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core
- Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only)
- Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor
- Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift
- Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options)
- Wire Matrix (matrix:dtoro) as health alert delivery platform
- Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml
- Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
2026-06-04 10:14:56 +02:00
fbe76c84d4 fix: ZimaOS static IP set to .195 via systemd-networkd overlay injection
- Injected /etc/systemd/network/10-static.network into overlay partition
- Match en*/eth*, address 192.168.8.195/24, gateway .1, DNS .2
- Verified: HTTP 200 at .195, Caddy zimaos.hubris.network returns 200
- Old DHCP lease at .103 expires naturally
- Updated vms/100-zimaos.md: open items resolved, changelog added
2026-06-03 23:01:34 +02:00
edac189aa7 docs: DHCP pool narrowed to .241-.254; ZimaOS IP drift documented
- network.md: updated pool range, fixed 'below .100' claim, added changelog
- 107-dns.md: updated pool range, fixed 'below .100' claim, added changelog
- 100-zimaos.md: documented IP drift (.195→.103) and Caddy 502
- plan: .hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md

DHCP scope changed via Technitium API. No static IPs changed. Old leases
(.101-.110) expire naturally by 2026-06-04.
2026-06-03 22:50:33 +02:00
4c4cfb56cd fix: remove stray merge marker in .sops.yaml; rekey netbird-pat to 4 recipients
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 22:03:36 +02:00
cb1d8b39b9 Merge branch 'main' of git-ssh.hubris.network:dtoro/Homelab-Docs
# Conflicts:
#	.sops.yaml
#	containers/107-dns.md
#	infrastructure/dns.md
2026-06-03 22:02:52 +02:00
f81310ae6b dns: Technitium-master sync to NetBird managed zone + cleanup
- scripts/dns-sync.py: reconcile Technitium named A-records -> NetBird managed
  zone via API (cron */10 on dns LXC 107). Single authoring source; kills the
  manual drift behind the auth/sso/nfs-export saga.
- secrets/netbird-pat.yaml: sops-encrypted NetBird API PAT for the sync.
- dns.md / 107-dns.md: document the sync model + why forward-to-Technitium was
  abandoned (NetBird self-IP / nameserver-group quirks).
- Cleanup: removed inert Mac secondary; reverted primary AXFR; home-lab-dns ->
  [192.168.8.2] (1/1 Available); deleted vestigial Proxmox Names group.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 21:59:54 +02:00
ce1bf4d0d2 docs: gluetun VPN + static IP migration (2026-06-02) 2026-06-02 22:42:46 +02:00
41de395240 docs: post-migration changelog entries on hubris, caddy, dns
hosts/hubris.md:
- Update At a glance network section: vmbr1 uplink (192.168.178.10),
  vmbr0 portless internal bridge with 192.168.8.1 alias
- Remove Phase 1 WiFi failover section (wlp3s0 disabled 2026-06-02)
- Changelog: Slate AX retired, SODOLA added, Proxmox as subnet router

containers/121-caddy.md:
- Changelog: caddy.service unit was missing from hubris1 package,
  recreated manually; risk of loss on package reinstall noted

containers/107-dns.md:
- Update Who points here: Technitium DHCP hands out .2 as DNS for
  homelab clients; Fritz!Box LAN clients still get Fritz!Box DNS
- Add DHCP section documenting the homelab scope (100-240, gw .1)
- Changelog: DHCP enabled 2026-06-02, replaces Slate AX DHCP

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-02 22:02:34 +02:00
99d412370b network: migration complete — Proxmox as subnet router, Technitium DHCP live
- plans/: mark Slate AX → SODOLA migration as Done (2026-06-02)
- infrastructure/network.md: rewrite to reflect actual topology —
  Fritz!OS 8.x lacks second-IP-network support, so Proxmox routes between
  vmbr1 (192.168.178.10, uplink) and vmbr0 (192.168.8.x, internal portless
  bridge with 192.168.8.1 alias). Static route on Fritz!Box, no NAT on Proxmox.
  Technitium DHCP enabled for 192.168.8.100–240 (gateway .1, DNS .2).
  Caddy service unit was missing on CT 121 — recreated and enabled.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-02 21:34:56 +02:00
1e7c5a7153 hermes-soul: auto-provision SOUL.md from HERMES.md via setup-hermes-soul.sh
- tools/setup-hermes-soul.sh: new auto-setup script (no-op on non-Hermes)
- HERMES.md: canonical persona for all agent types, agent-type table
- AGENTS.md: auto-setup section lists both caveman and hermes-soul
- post-pull.sh runs it automatically on every sync
2026-06-02 00:54:03 +02:00
c7f68a095b tools: post-pull auto-setup hook + caveman (RTK-style token optimization) 2026-06-02 00:31:48 +02:00
3154b3ca02 docs: mark MCP streamable-http migration resolved (#4) 2026-06-02 00:30:06 +02:00
520a86c986 fix: agent-enrollment.md doc references to use /mcp path 2026-06-02 00:28:43 +02:00
ec48041b54 fix: correct endpoint path to /mcp for streamable-http transport 2026-06-02 00:25:34 +02:00
c299409431 docs: update ssh-access.md + agent-enrollment.md with universal SSH setup 2026-06-02 00:20:28 +02:00
bebebf486a ssh-config: add gen-config.py + homelab ssh-config CLI subcommand 2026-06-02 00:16:15 +02:00
e7bdf9fca7 fix: streamable-http (hyphen, not underscore) in FastMCP transport 2026-06-02 00:12:59 +02:00
00a891b154 ssh: fix LXC name parsing in deploy-keys.sh (Lock column edge case) 2026-06-02 00:06:58 +02:00
5adb66237e ssh: add authorized_keys dir + deploy-keys.sh for key distribution 2026-06-02 00:05:03 +02:00
0a3e43473e inventory: add lan_ip for mac-mini (192.168.8.174) 2026-06-02 00:00:47 +02:00
daf4747c5c post-mortem: mac-mini onboarding — 12 issues found 2026-06-01 23:45:22 +02:00
e615599a2c revert test 2026-06-01 23:41:19 +02:00
19b2109efc test 2026-06-01 23:41:06 +02:00
10fdb0fe09 revert: test commit 2026-06-01 23:34:32 +02:00
946ded6d08 test credential 2026-06-01 23:34:19 +02:00
95a3ae75c3 hermes-agent: update model to deepseek/deepseek-v4-flash (tool-use support) 2026-06-01 23:28:43 +02:00
9be6f7f31d openrouter-api-key: seed real key 2026-06-01 23:24:36 +02:00
root
5f929ce3ee openrouter-api-key: seed real key 2026-06-01 23:09:41 +02:00
root
76bbe99798 client-add: mac-mini (finalize age_pubkey + grant shared + hermes secrets) 2026-06-01 22:50:36 +02:00
543745cad9 network: retire Slate AX; add SODOLA switch migration plan and network topology doc
- infrastructure/network.md: permanent topology reference — VLAN 1 (main LAN)
  + VLAN 10 (homelab 192.168.8.0/24), Fritz!Box 7590 as VLAN router,
  SODOLA 5-Port 2.5Gbit as homelab switch
- plans/2026-06-01-slate-ax-to-sodola-migration.md: full migration runbook
  (pre-flight, Fritz!Box second-network config, SODOLA VLAN table, cutover
  procedure, verification commands). Gateway 192.168.8.1 is unchanged —
  no LXC/VM config edits needed. Fritz!Box DHCP for VLAN 10 hands out
  192.168.8.2 (Technitium CT 107) as DNS, closing the outstanding item in dns.md.
- plans/index.md: establishes plans/ convention (Planned → In Progress → Done)
- CONTRIBUTING.md: plan page template + linking discipline for plans

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 22:48:02 +02:00
ce6df85bb5 Merge pull request 'hermes-agent: onboard Nous-Hermes-on-Goose to homelab clients' (#1) from hermes-onboarding into main
Reviewed-on: dtoro/Homelab-Docs#1
2026-06-01 22:27:09 +02:00
f25d9e0648 hermes-agent: onboard Nous-Hermes-on-Goose to homelab clients
`bootstrap.sh --with-hermes` installs the Goose CLI, drops a Goose
config pinning the OpenRouter provider + Nous Hermes model + the
homelab MCP extension, symlinks `bin/hermes` and HERMES.md, and links
HERMES.md as `.goosehints` so the persona is injected as the system
prompt every session.

`bin/hermes` decrypts `secrets/openrouter-api-key.yaml` via the existing
`homelab secret` flow and execs `goose session`.

`homelab client add --with-hermes` grants the new sops secret to the
host's age_pubkey at finalize time (parallel to the existing
shared-secrets grant). `client remove` revokes it.

`operations/hermes-agent.md` covers the end-to-end flow, verification,
troubleshooting, and queues one follow-up: the MCP server still runs
SSE-only but Goose 1.x deprecated SSE — the Goose config targets
`streamable_http` and the `homelab` extension won't connect until
`mcp/server.py` migrates. The `developer` extension (shell + edit +
`homelab` CLI) carries the agent in the meantime.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 22:27:09 +02:00
563dbe21b1 arch-migration P2: Technitium dns LXC (107); retire LXC 124
- containers/107-dns.md: new Technitium DNS server page
- dns.md: dnsmasq -> Technitium changelog; flags 3 overlapping DNS sources + router DHCP action
- 124-authentik: marked RETIRED (shut down)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 22:10:18 +02:00
4b713d7805 arch-migration P1: document sso.hubris.network outpost callback domain
Domain-level forward-auth needs its own external_host domain when the IdP core
and outpost are on different hosts. sso.hubris.network -> Caddy -> LAN outpost.
Includes the redirect_uris-regeneration gotcha. Carry the DNS record into
Technitium in DNS Phase 2.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 00:45:25 +02:00
2762dcbdb8 arch-migration P1: dedicated forward-auth outpost (LXC 106)
- containers/106-auth-outpost.md: new LAN outpost page
- 124-authentik: forward-auth cut over, Authentik stopped (DNS-only now)
- investigation: forward-auth outpost marked RESOLVED

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 00:33:26 +02:00
8dbba323c1 authentik: migrate from LXC 124 to VPS; eliminate mesh bootstrap deadlock
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).

- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 23:17:00 +02:00
110 changed files with 6082 additions and 639 deletions

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
.DS_Store

View File

@@ -0,0 +1,277 @@
# Plan: Migrate library SSD to ludo-mini + Proxmox gaming/media server
## Goal
Split the homelab into two Proxmox hosts:
| Host | Role | Storage |
|------|------|---------|
| **hubris** | Core services (reverse-proxy, SSO, Matrix, git, documents, HA) | SSD 1 — boot + LXC rootfs (unchanged) |
| **ludo-mini** | Gaming server + media/library services | SSD 2 — Samsung 990 EVO Plus 4 TB (moved from hubris) |
The library SSD physically moves from hubris to ludo-mini. hubris LXCs that still need `/mnt/library` access it over NFS from ludo-mini.
## Current state
### hubris hardware
- GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS, 12 vCPU, ~28 GiB RAM
- 2× Samsung 990 EVO Plus NVMe:
- nvme0: `local` (95G) + `local-lvm` (856G) — boot, ISOs, LXC rootfs
- nvme1: `library` LVM (3.7T) — `/mnt/library` ext4 via `/dev/mapper/library-library`
### LXCs binding `/mnt/library` (host-level bind-mount)
| ID | Name | Role | I/O profile |
|----|------|------|-------------|
| 101 | jellyfin | Media streaming | Read-heavy, sequential |
| 103 | paperless | Document archive | Mixed, OCR writes |
| 104 | gitea | Git server | Mixed, lots of small files |
| 105 | apps | Docker (booklore, audiobookshelf, artifacto, MCP) | Mixed, depends on container |
| 114 | nextcloud | File sync | Mixed, WebDAV |
| 119 | sophia | Workshop | Low I/O |
| 120 | mule-images | Photo management | Write-heavy (processing), iGPU |
| 122 | arriman | *arr stack + downloads | Write-heavy (downloads) |
| 126 | plato | App (sub-mount: `/mnt/library/documents/plato`) | Light |
### NFS export chain (for VM 100 zimaos)
```
/mnt/library (ext4, host) → bind-mount → LXC 102 (nfs-export) → NFSv4 → VM 100 (zimaos)
```
### ludo-mini current
- Linux workstation, wired Ethernet 2.5 Gbps, `192.168.178.181` (household LAN)
- Runs Sunshine for game streaming
- No Proxmox, no LVM config
- Connected to SODOLA switch (same switch as hubris eno1)
### Network topology
```
Fritz!Box 7590 (192.168.178.1)
└── SODOLA 2.5G switch
├── hubris eno1 → vmbr1 (192.168.178.10)
│ └── routes to vmbr0 (192.168.8.0/24) — all LXCs
└── ludo-mini (192.168.178.181)
```
hubris routes between `192.168.8.0/24` (vmbr0) and `192.168.178.0/24` (vmbr1). So LXCs can reach ludo-mini via hubris as a router.
## Key decisions
### 1. Service split — what moves, what stays
**Move to ludo-mini** (high I/O, benefits from data locality + GPU):
- 101 jellyfin — media streaming, GPU transcoding
- 120 mule-images — photo processing, iGPU passthrough
- 122 arriman — *arr stack, downloads write to library
**Stay on hubris, NFS-mount library from ludo-mini:**
- 103 paperless — documents, moderate I/O
- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
- 105 apps — Docker apps, mixed workloads
- 114 nextcloud — file sync
- 119 sophia — workshop, light use
- 126 plato — app, light use
- 100 zimaos — NAS frontend, already NFS-mounted
### 2. NFS architecture
Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:
```
ludo-mini: /mnt/library (ext4, local NVMe)
└── NFSv4 export to 192.168.8.0/24
└── hubris host: NFS-mount at /mnt/library
└── LXCs: bind-mount /mnt/library (unchanged!)
```
This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.
### 3. Network — ludo-mini reachability from hubris LXCs
LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
- `vmbr0` → hubris kernel routing → `vmbr1` → SODOLA → ludo-mini
- Already works (IP forwarding enabled on hubris)
**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.
### 4. Gaming on ludo-mini with Proxmox
ludo-mini runs Sunshine (game streaming). Under Proxmox:
- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
- **Option B:** LXC with GPU device passthrough (`/dev/dri`) — lighter, shares kernel
- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)
Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.
### 5. What about nfs-export (LXC 102)?
Currently exports `/mnt/library` to zimaos. After migration:
- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
- Better: zimaos NFS-mounts directly from ludo-mini.
- So LXC 102 gets decommissioned (or repurposed).
- zimaos gets a new NFS mount pointing directly at ludo-mini.
## Migration phases
### Phase 1 — Preparation (no downtime)
1. **Document current state** on hubris:
- `pct list` — full container inventory
- `pct config <id>` for every library-mounting LXC
- `cat /etc/fstab` — capture the library mount line
- `df -h /mnt/library` — confirm space usage
- `lsblk -f` — UUID, filesystem
- Identify the exact NVMe device (`nvme1n1`)
2. **Pre-flight on ludo-mini:**
- Confirm hardware: CPU, RAM, available M.2 slots, GPU model
- Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
- Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
- Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly
3. **Install Proxmox on ludo-mini:**
- Download Proxmox VE 9.x ISO
- Install to ludo-mini's system drive (NOT the library SSD)
- Configure networking: bridge for Proxmox, IP on 192.168.178.x
- Test: web UI accessible
4. **Prepare NFS server on ludo-mini Proxmox:**
- Create NFS-export LXC (or serve from host — simpler for now)
- Prepare `/etc/exports`: `192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)`
- Same squash params as current nfs-export LXC 102
### Phase 2 — Physical SSD move (planned downtime)
1. **Graceful shutdown on hubris:**
- Stop all library-mounting LXCs (101, 103, 104, 105, 114, 119, 120, 122, 126)
- Unmount `/mnt/library` on hubris host
- Edit `/etc/fstab` to comment out the library mount line
- Power off hubris
2. **Physical drive swap:**
- Remove Samsung 990 EVO Plus (library SSD) from hubris
- Install into ludo-mini M.2 slot
- Power on ludo-mini
3. **Bring library online on ludo-mini:**
- Detect the new NVMe device
- If it's the whole device with LVM, activate the VG:
```
vgscan && vgchange -ay library
mount /dev/mapper/library-library /mnt/library
```
- Add to `/etc/fstab` for auto-mount
- Verify content: `ls /mnt/library` — same tree as before
4. **Start NFS export on ludo-mini:**
- `exportfs -ra`
- Verify: `showmount -e <ludo-mini-ip>`
### Phase 3 — Reconnect hubris LXCs
1. **Power on hubris** (without library SSD — it'll boot fine, just won't mount library)
2. **Mount NFS on hubris host:**
- Install `nfs-common` if not present
- Add to `/etc/fstab`:
```
192.168.178.181:/mnt/library /mnt/library nfs rw,vers=4,soft,timeo=30,retrans=3 0 0
```
Use `soft` to prevent hangs if ludo-mini is down; `hard` with `intr` is safer for data integrity but can block processes.
- `mount /mnt/library`
- **Verify permissions:** `ls -la /mnt/library` — should show `www-data:media` ownership for shared subtrees (same uid 33, gid 10000). The NFS all_squash guarantees this.
3. **Start LXCs:**
- Start the LXCs that stayed on hubris (103, 104, 105, 114, 119, 126)
- Their bind-mounts should work — `/mnt/library` is populated via NFS
- Verify each service: web UIs, git clone, document access
4. **Update DNS/Caddy** for services that moved:
- If jellyfin, arr services moved to ludo-mini, update Caddyfile to point to ludo-mini IPs
- Update DNS entries if needed
### Phase 4 — Migrate services to ludo-mini
1. **Create LXCs/VM on ludo-mini Proxmox:**
- 101 jellyfin — privileged LXC, mount `/mnt/library`, add `media` group
- 120 mule-images — privileged LXC, mount `/mnt/library` + `/dev/dri` passthrough
- 122 arriman — privileged LXC, mount `/mnt/library`
2. **Migrate configs:**
- Copy LXC configs from hubris (`/etc/pve/lxc/<id>.conf`) as templates
- Adjust network (IPs on household subnet or Proxmox bridge)
- Restore app data from backups or copy over NFS
3. **Gaming VM:**
- Create VM with GPU passthrough
- Pass through the dGPU for gaming performance
- Install Sunshine + game libraries
- Storage: VM disk on Proxmox storage, games on library SSD
4. **Update reverse proxy:**
- Caddy on hubris (121): update backend IPs for jellyfin, jellyseerr, qbit, sab, mule-images → point to ludo-mini
- Test: `media.hubris.network` serves from ludo-mini jellyfin
### Phase 5 — Cleanup
1. **Decommission nfs-export LXC 102** on hubris (no longer needed)
2. **Update zimaos (100)** — change NFS mount from `192.168.8.200` → `192.168.178.181`
3. **Remove old LXCs** from hubris (101, 120, 122) after confirming migration works
4. **Update inventory.yaml:**
- ludo-mini: `kind: proxmox-host`, add mounts/storage, add LXCs
- Move services from hubris to ludo-mini
- Remove nfs-export
5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
6. **Run `homelab sync`** to propagate changes
## Open questions / unknowns
1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.
2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.
3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.
4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.
5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.
6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.
7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.
## Files affected
| File | Change |
|------|--------|
| `/opt/homelab-context/inventory.yaml` | ludo-mini: workstation → proxmox-host; add LXCs, mounts; remove nfs-export; move service backends |
| `/opt/homelab-context/hosts/hubris.md` | Remove library storage, add NFS mount note |
| `/opt/homelab-context/hosts/ludo-mini.yaml` | Complete rewrite — Proxmox host, storage, tenants |
| `/opt/homelab-context/containers/102-nfs-export.md` | Mark decommissioned |
| `/opt/homelab-context/containers/index.md` | Move 101, 120, 122 to ludo-mini; remove 102 |
| `/opt/homelab-context/infrastructure/dns.md` | Update nfs-export entry |
| `/opt/homelab-context/infrastructure/media-permissions.md` | Note NFS squash from ludo-mini, not hubris |
| hubris `/etc/fstab` | Replace ext4 mount with NFS mount |
| ludo-mini `/etc/fstab` | Add library ext4 mount |
| ludo-mini `/etc/exports` | Add NFS export config |
| caddy (LXC 121) Caddyfile | Backend IPs for moved services |
| DNS (LXC 107 Technitium) | Update entries for moved services |
## Validation checklist
- [ ] ludo-mini Proxmox web UI accessible
- [ ] Library SSD detected and mountable on ludo-mini
- [ ] NFS export from ludo-mini: `showmount -e <ip>` shows `/mnt/library`
- [ ] hubris host NFS mount: `df -h /mnt/library` shows NFS, not ext4
- [ ] hubris LXCs start and bind-mount /mnt/library (content visible)
- [ ] gitea: `git clone` over SSH works, repos readable
- [ ] paperless: document ingestion works, OCR processing
- [ ] nextcloud: file sync, WebDAV
- [ ] jellyfin: media plays from ludo-mini, transcoding works
- [ ] arriman: downloads write to library, jellyfin picks up new media
- [ ] mule-images: photo import and processing
- [ ] zimaos: NFS mount from ludo-mini works, Files UI shows library
- [ ] Sunshine: game streaming from ludo-mini VM works
- [ ] All `*.hubris.network` services resolve and load through Caddy

View File

@@ -0,0 +1,271 @@
# Homelab structure revision & improvement plan
## Goal
Identify structural issues in the current hubris homelab topology and propose an
actionable improvement roadmap — DNS consolidation, monitoring gaps, backup
recovery, mesh completion, resource rightsizing, and operational hygiene.
---
## Current state summary
| Dimension | Status |
|-----------|--------|
| Hypervisor | hubris (single-node Proxmox) — also does subnet routing (192.168.8.0/24) |
| LXCs | 14 active, 1 retired (124), 1 new (107 dns) |
| VMs | HAOS (108), ZimaOS (100) |
| Workstations | mac-mini (macOS), republic-laptop, ludo-mini |
| VPS | 1 IONOS box — netbird mgmt+signal+relay, traefik, authentik, coturn |
| Switch | SODOLA 5-Port 2.5Gbit (L2, flat) |
| Networking | 192.168.8.0/24 internal, fr!tz box main LAN via vmbr0→vmbr1 routing |
| Mesh | Netbird + Tailscale (migrating), ~mixed state |
| DNS | 3 sources: Technitium (CT 107), NetBird managed zone, public IONOS |
| Backups | DISABLED since 2026-04-22 |
| Monitoring | claudio-monitor (host health only, no apt/docker checks yet) |
| Agent enrollment | 3/19 hosts enrolled (hubris, apps, republic-laptop) |
---
## Issues identified
### 1. Three overlapping DNS sources (highest risk)
**Problem:** Technitium on CT 107, NetBird managed DNS zone, and the public
IONOS wildcard all answer `*.hubris.network` queries. The dns.md doc still
references the old dnsmasq on LXC 124 (though the change log says it moved).
NetBird's managed DNS bypasses Technitium entirely for some app names — there
is no single source of truth for DNS.
**Risk:** Mismatched answers → services unreachable → "works on some clients
but not others" debugging sessions. Already cost time when `auth.hubris.network`
re-pointed to the VPS.
**Proposal:**
- Phase out NetBird managed DNS zone for `hubris.network` — Technitium is the
authoritative answerer for mesh & LAN clients
- Set Technitium as the sole DNS for all LXCs (remove router DNS / Tailscale
MagicDNS fallbacks)
- Document the full authoritative chain: Technitium → upstream forwarders → public
- Track Technitium config in git (dtoro/technitium-config or equivalent)
### 2. Backups disabled with no alternative (data loss risk)
**Problem:** The only backup was restic to an external USB that caused host
crashes. It was disabled 2026-04-22 as an A/B test — host stability was
confirmed by the SODOLA migration (no more Slate AX double-NAT hangs). The
drive is still removed.
**Risk:** `/mnt/library` (~429 GB of irreplaceable data: photos, documents,
git repos, gitea data) has no off-host copy. Single-disk failure = total loss.
**Proposal:**
- Re-evaluate the USB drive stability with the new SODOLA switch topology
(direct rear USB 3.0 port, no hub chain)
- OR adopt a cloud-backed strategy: `rclone`-to-Hetzner Storage Box or
Backblaze B2 for the irreplaceable subset (docs, photos, gitea data)
- OR use hubris's own `zfs send` to a second host/disk if ZFS is feasible
- Minimum viable: at minimum restore gitea backups + sops-encrypted secrets
via an off-site cron (cheap B2 bucket)
### 3. Mesh migration still incomplete
**Problem:** Most LXCs still use Tailscale. This forces per-LXC DNS workarounds
(/etc/hosts overrides, local dnsmasq) and runs two VPN stacks in parallel.
Mesh migration doc (mesh.md) is comprehensive but execution stalled.
**Proposal:**
- Batch-migrate all LXCs from Tailscale to Netbird in one maintenance window
- Remove Tailscale from the PVE host
- Ensure all LXCs resolve `*.hubris.network` via Technitium (no more /etc/hosts
overrides)
- Document Netbird client on each LXC (netbird version, setup key rotation)
### 4. LXC resource imbalance & disk pressure
**Problem:**
| LXC | Cores | RAM | Rootfs | Disk usage |
|-----|-------|-----|--------|------------|
| mule-images (120) | 6 | 12 GiB | 60 GiB | photo AI, reasonable |
| arriman (122) | 4 | 8 GiB | 24 GiB | media stack, OK |
| nextcloud (114) | 4 | 6 GiB | 25 GiB | file sync, OK |
| apps (105) | 2 | 4 GiB | 30 GiB | 6+ services, tight |
| paperless (103) | 2 | 3 GiB | 8 GiB | 86.9% disk |
| elementsynapse (118) | 1 | 2 GiB | 8 GiB | 86.8% disk |
| gitea (104) | 1 | 1 GiB | 8 GiB | git server, adequate |
| caddy (121) | 1 | 512 MiB | 6 GiB | fine for reverse proxy |
Paperless (103) and elementsynapse (118) are at critical disk levels. apps (105)
is undersized for 6+ services.
**Proposal:**
- Resize rootfs on paperless (8→16 GiB) and elementsynapse (8→16 GiB)
- Bump apps (105) to 4 cores / 6 GiB RAM / 40 GiB rootfs
- Enable claudio-monitor's disk check to alert before next crisis
### 5. VPS is a single point of failure
**Problem:** One IONOS VM runs netbird management (control plane), traefik
(public ingress), authentik (identity), and coturn (TURN relay). If it goes
down: no remote mesh, no public services, no auth.
**Proposal:**
- Document a VPS recovery runbook (how to restore from a known-working backup)
- Consider splitting authentik into a separate host or at minimum having a
standby configuration
- Not a high priority (the VPS has been stable) but worth documenting the
blast radius and recovery path
### 6. No centralized logging
**Problem:** Each LXC has independent journald. Cross-service debugging
involves hopping between `pct exec <id> -- journalctl -u <service>`. There is
no aggregation or retention.
**Proposal:**
- Deploy a lightweight log shipper (Loki + promtail, or vector.dev) on each LXC
- Ship logs to a central Loki instance on apps (105) or a new small LXC
- Grafana dashboard optional — even a simple `logcli` query saves time
### 7. Agent enrollment incomplete
**Problem:** Only hubris, apps, and republic-laptop are enrolled in the
homelab-context system (age keys, sync timers, MCP access). mac-mini,
ludo-mini, claudio-bot, and all other LXCs are not.
**Proposal:**
- Batch-enroll remaining LXCs (jellyfin, paperless, gitea, nextcloud, etc.)
- Enroll mac-mini (macOS — exercises the launchd timer path)
- Enroll ludo-mini (needs SSH user config in inventory first)
- Wire claudio-bot into inventory-aware queries
### 8. Configuration drift on untracked configs
**Problem:** Technitium config, dnsmasq (legacy), and several service-specific
configs are not git-tracked.
**Proposal:**
- Track Technitium zone backup + compose config in a git repo
- Apply the same pattern as caddy-conf: dtoro/technitium-conf with auto-deploy
### 9. No capacity planning / resource monitoring
**Problem:** No trend data on CPU, RAM, or disk growth. The 86% disk alerts
were discovered reactively. Rootfs resize is painful (requires Proxmox stop +
resize + growfs inside).
**Proposal:**
- Enable the missing claudio-monitor checks (disk growth trend, apt upgradable
counts, docker image drift)
- Set up a simple Prometheus + node_exporter on hubris or use the PVE API
directly
- At minimum, surface disk usage in the existing homelab-mcp management tools
### 10. No standard deploy / orchestration for bare-metal LXCs
**Problem:** Some services are bare-metal CLI apps (sophia, claudio-bot),
some are Docker on apps (105), some are Portainer-managed. No consistent
deploy pattern means every new service reinvents the deployment.
**Proposal:**
- Don't over-engineer this — the current pragmatism works
- Just document the decision tree:
- Needs `/mnt/library` mount + heavy I/O → dedicated LXC
- Small stateless web service → Docker on apps (105)
- Media stack → dedicated LXC (arriman, jellyfin)
- Everything else → judge by complexity
---
## Phased implementation plan
### Phase 1 — Critical fixes (this week)
1. Resize rootfs on paperless (103) 8→16 GiB, elementsynapse (118) 8→16 GiB
2. Enable claudio-monitor disk check + disk-growth alerting
3. Pick one backup strategy and implement minimum viable (e.g. nightly
gitea dump + sops-encrypted secrets to B2 via rclone)
4. Verify Technitium is the sole DNS for all LXCs (remove NetBird managed zone
for hubris.network)
### Phase 2 — Mesh consolidation (next week)
5. Batch-migrate remaining LXCs from Tailscale to Netbird
6. Remove Tailscale from PVE host
7. Remove all per-LXC /etc/hosts DNS overrides
8. Update DNS documentation to reflect Technitium as single source
### Phase 3 — Agent enrollment & logging (next 2 weeks)
9. Enroll all LXCs in homelab-context (age keys, sync timers)
10. Enroll mac-mini (macOS launchd path — exercises untested code path)
11. Enroll ludo-mini
12. Deploy log shipper (Loki + promtail) on apps (105) + all LXCs
### Phase 4 — Resource & monitoring hardening (next month)
13. Resize apps (105) rootfs, bump RAM
14. Deploy Prometheus + node_exporter or equivalent for trend data
15. Track Technitium config in git with auto-deploy
16. Write VPS recovery runbook
### Phase 5 — Drive re-evaluation (optional, behind host-stability gate)
17. Re-attach USB backup drive with the new SODOLA topology (direct port)
18. If stable for 7 days, re-enable restic backup schedule (chunked)
19. If not stable, finalize cloud backup as permanent strategy
---
## Files likely to change
| Path | Change |
|------|--------|
| `/opt/homelab-context/inventory.yaml` | LXCs enrolled, resource updates |
| `/opt/homelab-context/secrets/*.yaml` | New recipients for enrolled LXCs |
| `/opt/homelab-context/infrastructure/dns.md` | Reflect Technitium as sole source |
| `/opt/homelab-context/infrastructure/mesh.md` | Remove Tailscale references post-migration |
| `/opt/homelab-context/infrastructure/backups.md` | New strategy |
| `/opt/homelab-context/infrastructure/monitoring.md` | Enable missing checks |
| `/opt/homelab-context/containers/103-paperless.md` | Rootfs resize |
| `/opt/homelab-context/containers/118-elementsynapse.md` | Rootfs resize |
| `/opt/homelab-context/containers/105-apps.md` | Resource bump, logging addition |
| `/opt/homelab-context/containers/index.md` | Updated resource table |
| `.sops.yaml` | New age pubkeys for enrolled LXCs |
## Verification
Each phase ends with a verification milestone:
- Phase 1: `claudio-monitor` triggers on paperless disk → confirmed alert. Backup
of gitea data lands in B2 (or equivalent). DNS query from any LXC returns
Technitium answer.
- Phase 2: `netbird status` shows all LXCs connected. Tailscale not running on
PVE host. `curl auth.hubris.network` from any LXC resolves correctly without
/etc/hosts.
- Phase 3: Every LXC has `/opt/homelab-context/` + `/etc/age/key.txt`. MCP
tools return valid host info for all enrolled LXC names. `journalctl` shows
promtail shipping to Loki.
- Phase 4: `claudio-monitor` shows disk growth trend. apps (105) can run all
6+ services without OOM.
## Risks & tradeoffs
- **Netbird migration window:** All LXCs will briefly lose mesh connectivity
during the Tailscale→Netbird cutover. Schedule in off-hours.
- **Backup cost:** B2/e2 costs ~$5/month for ~500 GB. The USB drive was free
but unstable — trade money for reliability.
- **DNS consolidation:** Removing the NetBird managed DNS zone means any
NetBird-specific names stop resolving for hubris.network — verify nothing
depends on that path.
- **Loki on apps (105):** Adds another container to an already-loaded host.
May need to bump resources before deploying.
- **Agent enrollment on every LXC:** Each enrollment creates an age keypair
and commits a pubkey to inventory. Process is scriptable via `homelab client
add` but still takes ~2 min per host for verification.
## Open questions
1. Is the USB backup drive still physically attached to hubris? If not, the
simplest "re-enable" path requires physically re-attaching it.
2. Authentik is now on the VPS — is LXC 124 (old Authentik) still running or
was it fully decommissioned? The dns.md changelog says "shut down" but
index.md lists it as "running".
3. What's the actual disk layout on hubris? nvme0n1, ZFS pool, mount structure
— needed to plan rootfs resizes safely.
4. Does the user want to keep Tailscale on any host for a specific reason, or
is full Netbird migration the clear goal?

View File

@@ -0,0 +1,246 @@
# Plan: Narrow Technitium DHCP Pool to Avoid Static-IP Conflicts
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
**Goal:** Eliminate the IP conflict risk created by the Technitium DHCP pool (`.100.240`) overlapping with all static LXC/VM IPs (`.101.239`).
**Architecture:** Shrink the DHCP pool range on Technitium so it only covers IPs that no static host uses. No LXC/VM IPs change. Single server-side change (Technitium API), plus documentation updates.
**Tech Stack:** Technitium DNS API (`/api/dhcp/scopes/set`), bash/curl, homelab-context repo for docs.
---
## Problem statement
The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
| Host | IP | Inside pool? |
|---|---|---|
| hubris (Proxmox) | .77 | No — below `.100` |
| haos (VM 108) | .101 | YES |
| gitea (104) | .121 | YES |
| paperless (103) | .130 | YES |
| arriman (122) | .132 | YES |
| mule-images (120) | .136 | YES |
| sophia (119) | .157 | YES |
| mac-mini | .174 | YES |
| caddy (121) | .175 | YES |
| authentik (124) | .180 | YES |
| plato (126) | .190 | YES |
| zimaos (VM 100) | .195 | YES |
| nfs-export (102) | .200 | YES |
| apps (105) | .205 | YES |
| jellyfin (101) | .206 | YES |
| nextcloud (114) | .224 | YES |
| claudio-bot (123) | .230 | YES |
| elementsynapse (118) | .239 | YES |
The docs claim "Static-IP LXCs (below `.100`) are unaffected" — this is **false**. Static IPs span `.101.239`, the DHCP pool spans `.100.240`. They overlap almost entirely.
If the DHCP server hands out `.121/.136/.224` (or any of the above) to a new dynamic client before the static LXC claims it on boot, the static service will fail to bind and the service goes dark.
---
## Proposed approach: Shrink the pool
**Move the DHCP pool start from `.100` to `.241`**, resulting in:
- **New pool:** `192.168.8.241 192.168.8.254` (14 dynamic IPs)
- **Reserved:** `.100.240` stays for static hosts, `.2` for Technitium, `.1` for gateway
- **Zero changes to any LXC, VM, Caddy, or Proxmox config.**
Why `.241.254`:
- Highest static IP is `.239` (elementsynapse) — `.241` gives a 1-IP gap
- `.255` is the broadcast address (unusable)
- 14 IPs is plenty for truly dynamic clients (new transient containers, test VMs)
- If more are ever needed, the pool can easily be widened back down
---
## Tasks
### Task 1: Verify current Technitium DHCP scope from the API
**Objective:** Confirm the active pool range matches what's documented.
**Step 1: Log in to Technitium API and get a token**
```bash
TOKEN=$(curl -sk -X POST http://192.168.8.2:5380/api/user/login \
-H "Content-Type: application/json" \
-d '{"user":"admin","pass":"'$(cat /opt/technitium/admin_password.txt)'","includeInfo":false}' \
| jq -r '.token')
echo "Token: ${TOKEN:0:10}..."
```
**Step 2: Fetch current DHCP scopes**
```bash
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq .
```
**Expected:** One scope named `homelab` with `startingAddress: "192.168.8.100"` and `endingAddress: "192.168.8.240"`.
**Verification:** If the scope is NOT `.100.240`, note the actual range and adjust the plan.
---
### Task 2: Update the DHCP scope to `.241.254`
**Objective:** Shrink the pool so it no longer overlaps static IPs.
**Step 1: Update the scope via API**
```bash
curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/scopes/set?token=$TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "homelab",
"startingAddress": "192.168.8.241",
"endingAddress": "192.168.8.254",
"subnetMask": "255.255.255.0",
"gatewayAddress": "192.168.8.1",
"dnsServerAddresses": ["192.168.8.2"],
"leaseTime": 86400
}'
```
**Step 2: Verify the change took effect**
```bash
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
```
**Expected:**
```json
{
"startingAddress": "192.168.8.241",
"endingAddress": "192.168.8.254"
}
```
**Pitfall:** If the API returns `{"status":"error"}`, the scope name or parameter format may differ. Inspect the response body. Technitium's API might use `rangeStart`/`rangeEnd` instead of `startingAddress`/`endingAddress`. Adjust if needed (check the full scope object from Task 1 step 2 for exact key names).
---
### Task 3: Check for active DHCP leases in the old pool that would be stranded
**Objective:** Ensure no DHCP client is currently holding an IP in `.100.240` that it will lose when its lease expires.
**Step 1: List active DHCP leases**
```bash
curl -sk "http://192.168.8.2:5380/api/dhcp/leases/list?token=$TOKEN" | jq '.response.leases[] | {ip: .ipAddress, client: .clientHostname, mac: .hardwareAddress, expires: .leaseExpires}'
```
**Step 2: Interpret results**
- If the only leases are from static LXCs that configured themselves before the DHCP move (e.g., old leases from before the 2026-06-02 static-IP migration), these leases are stale and harmless.
- If a *dynamic* client (e.g., a test laptop, transient VM) holds `.195` or similar, note it — it will lose its IP on next renew and should be moved to a static assignment or into the `.241+` pool.
- **ZimaOS (VM 100) at `.195` is a DHCP lease, not static** — this is the one host that needs attention. Either:
- Set a static IP inside ZimaOS (preferred), or
- Add a DHCP reservation for MAC in Technitium to pin `.195`
**Verification:** No "surprise" dynamic clients that would break on lease expiry.
---
### Task 4: Fix ZimaOS IP stability (if needed)
**Objective:** Ensure ZimaOS at `.195` won't float or break when the pool shrinks.
**If ZimaOS already has a static IP configured inside the VM:** Nothing to do.
**If ZimaOS is DHCP-only (likely — doc says "DHCP lease, not a reservation"):**
Option A (preferred): Set a static IP inside ZimaOS via its web UI at `http://192.168.8.195` → Settings → Network → Static IP → `192.168.8.195/24`, gateway `192.168.8.1`, DNS `192.168.8.2`.
Option B: Add a DHCP reservation in Technitium for ZimaOS's MAC address:
```bash
ZIMAMAC=$(ssh root@hubris "qm config 100 | grep net0 | grep -oE '([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}'")
curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/reservations/add?token=$TOKEN" \
-H "Content-Type: application/json" \
-d "{\"hardwareAddress\":\"$ZIMAMAC\",\"ipAddress\":\"192.168.8.195\"}"
```
**Pitfall:** The `/api/dhcp/reservations/add` endpoint signature is unverified — confirm the exact endpoint name from Technitium's API docs or the web UI before running it. The web console at `http://192.168.8.2:5380` → DHCP → Reservations can be used as a manual fallback.
---
### Task 5: Update documentation in homelab-context
**Objective:** Fix the now-wrong claims about static IPs being "below .100".
**Files to edit:**
1. **`infrastructure/network.md`** — Line 53
- Old: `Most homelab LXCs use static IPs below \`.100\`. DHCP only covers new/transient containers.`
- New: `Static IPs span \`.101.239\` (all LXCs + VMs + workstations). DHCP pool narrowed to \`.241.254\` to avoid overlap.`
2. **`containers/107-dns.md`** — Lines 37, 42, 55
- Line 37: Update pool range: `192.168.8.241 192.168.8.254`
- Line 42: `Static-IP LXCs (below \`.100\`)` → `Static-IP LXCs (\`.101.239\`) are excluded from the pool.`
- Line 55: Add changelog entry for the pool shrink
3. **`containers/107-dns.md`** — Add changelog entry:
```markdown
### 2026-06-03 — DHCP pool narrowed to `.241.254` to exclude static IPs
Previous pool `.100.240` overlapped with all static LXCs/VMs (\`.101.239\`), creating IP conflict risk. Shrunk pool to `.241.254`. No services re-IP'd. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
```
4. **`infrastructure/network.md`** — Line 51: Update pool range in the DHCP table row.
5. **`plans/2026-06-01-slate-ax-to-sodola-migration.md`** — Line 60: Optionally update the pool range in the config table (or add a post-migration note). This is the historical migration plan, so a footnote rather than an edit may be better.
**Commit:**
```bash
cd /opt/homelab-context
git add infrastructure/network.md containers/107-dns.md plans/
git commit -m "docs: DHCP pool narrowed to .241-.254 to exclude static IPs"
git push
```
---
### Task 6: Verify no regressions
**Objective:** Smoke-test that DNS and key services still work after the scope change.
```bash
# 1. DNS resolution via Technitium
dig @192.168.8.2 +short git.hubris.network
# Expected: 192.168.8.175
# 2. Caddy reverse-proxy chain
curl -sI https://git.hubris.network | head -1
# Expected: HTTP/2 200
# 3. All app names resolve
for name in git cloud media paperless photos matrix auth plato artifacto; do
result=$(dig @192.168.8.2 +short ${name}.hubris.network)
printf "%-20s → %s\n" "${name}.hubris.network" "$result"
done
# 4. Technitium DHCP scope is correct
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
```
---
## Risk assessment
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| API call fails (wrong field names) | Medium | Low | Inspect live scope object first (Task 1); adjust payload |
| ZimaOS loses IP on next boot | Low | Medium | Task 4 makes ZimaOS static or reserved |
| Active DHCP client in `.100.240` gets stranded | Low | Low | Task 3 surfaces this; client just requests a new IP from `.241+` |
| Technitium admin password file missing | Low | Medium | `/opt/technitium/admin_password.txt` was created during setup; verify existence |
## Open questions
1. Is zimaos (VM 100) currently DHCP or static? The doc says DHCP lease, but it's listed as `lan_ip: 192.168.8.195` in inventory. If it's actually DHCP, it's the one host that needs a static assignment before the pool shrinks.
2. Are there any transient DHCP clients (test laptops, phones) on the homelab subnet that hold `.100.240` addresses? Check leases before cutting over.
3. Should we widen the pool slightly (e.g., `.230.254`) for more headroom? Currently 14 IPs. If 3+ transient devices are expected, `.230.254` = 25 IPs — still safe since the highest static is `.239` and `.230.239` could be excluded.
## Execution preference
All changes are on the Technitium API + homelab-context repo. No LXC/VM restarts needed. The pool shrink takes effect immediately for NEW DHCP requests; existing leases in the old range continue until expiry (24h max).

View File

@@ -0,0 +1,129 @@
# Plan: Prevent DHCP IP drift from breaking Caddy backends
**Date:** 2026-06-05
**Slug:** prevent-dhcp-ip-drift
---
## Goal
Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's `reverse_proxy` directives hardcode.
**Triggering incident:** Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from `.130→.243` and `.101→.241` respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.
## Current context
### DHCP vs static IP inventory
| Machine | Type | PVE ID | Current IP | Allocation method | Status |
|---------|------|--------|-----------|-------------------|--------|
| **jellyfin** | LXC | 101 | 192.168.8.206 | Static (`ip=.../24`) | ✅ |
| **paperless** | LXC | 103 | 192.168.8.243 | **DHCP** (`ip=dhcp`) | ❌ broken, hotfixed to .243 |
| **gitea** | LXC | 104 | 192.168.8.121 | Static | ✅ |
| **apps** | LXC | 105 | 192.168.8.205 | Static | ✅ |
| **nextcloud** | LXC | 114 | 192.168.8.224 | Static | ✅ |
| **elementsynapse** | LXC | 118 | 192.168.8.239 | Static | ✅ |
| **mule-images** | LXC | 120 | 192.168.8.136 | Static | ✅ |
| **caddy** | LXC | 121 | 192.168.8.175 | Static | ✅ |
| **arriman** | LXC | 122 | 192.168.8.132 | Static | ✅ |
| **sophia** | LXC | 119 | 192.168.8.157 | Static? | ? (not in 2026-06-02 list) |
| **nfs-export** | LXC | 102 | 192.168.8.200 | Static? | ? |
| **plato** | LXC | 126 | 192.168.8.190 | Static? | ? |
| **HAOS** | VM | 108 | 192.168.8.241 | **DHCP** (VM — OS-managed) | ❌ broken, hotfixed to .241 |
| **zimaos** | VM | 100 | 192.168.8.195 | DHCP (known stale lease, see note) | ⚠️ open issue |
| **authentik** | — | — | — | migrated to VPS (external) | N/A |
### Infrastructure facts
- **DHCP server:** Technitium on CT 107 (192.168.8.2), pool `.241.254`
- **Static IP range:** `.101.239`
- **DNS:** Split-horizon on Technitium — `*.hubris.network → 192.168.8.175` (Caddy itself)
- **Caddyfile:** Has 29 `reverse_proxy` directives, all using **hardcoded IP:port**
- **Caddy reload:** Auto-deployed via webhook on git push to `dtoro/caddy-conf`
- **Documentation:** `inventory.yaml` updated, `hosts/*.yaml` stale-before-regenerate
### Why it happened
1. Paperless LXC 103 was **missed** during the 2026-06-02 static-IP migration (Proxmox config still shows `ip=dhcp`)
2. HAOS VM 108 uses DHCP internally (HAOS manages its own network — can't just `pct set`)
3. Both got new IPs from the Technitium `.241.254` pool after a lease renewal
4. Caddyfile still pointed at the old static-range IPs → connection refused
---
## Proposed approach
Three-layer solution:
### Layer 1: Fix the immediate offenders (static assignment)
**Paperless LXC 103:**
- `pct set 103 --net0 name=eth0,bridge=vmbr0,gw=192.168.8.1,hwaddr=BC:24:11:0A:8D:C2,ip=192.168.8.130/24,ip6=auto,type=veth`
- Inside the LXC, update `/etc/network/interfaces` to match
- Reboot/restart networking
**HAOS VM 108:**
- Set a **DHCP reservation** in Technitium for the VM's MAC address, pinning it to `192.168.8.101`
- This avoids needing to reconfigure HAOS internally (which is tedious)
- Alternatively: use `ha network update` from the HAOS console to set a static IP
### Layer 2: Audit and fix all remaining DHCP hosts
Check every LXC/VM hosted on hubris:
```bash
for ct in $(pct list | awk 'NR>1{print $1}'); do
echo "=== CT $ct ==="
pct config $ct | grep "^net0"
done
```
Any with `ip=dhcp` that Caddy reverse-proxies to → convert to static.
**Known candidates to check:**
- CT 102 (nfs-export) — `.200` but not in Caddy. May not need static.
- CT 119 (sophia) — `.157` — is this static or DHCP? Not sure.
- CT 126 (plato) — `.190` — same question.
- VM 100 (zimaos) — `.195` but known to have a stale lease (see 2026-06-03 changelog)
### Layer 3: Add validation
Create a script that runs periodically (cron or homelab cronjob):
**`/opt/homelab-context/scripts/check-caddy-backends.sh`:**
1. Parse `/etc/caddy/Caddyfile` on CT 121 to extract all `reverse_proxy IP:port` targets
2. For each `IP:port`, attempt a TCP connect (timeout 3s)
3. Report any that fail
Could also run as a homelab cron job that notifies dtoro on Matrix if a backend is unreachable.
This catches any future drift proactively (before a user reports it).
### Files likely to change
| File | Change |
|------|--------|
| `inventory.yaml` | May update paperless/HAOS IPs if we choose different static IPs |
| LXC 103 Proxmux config (via `pct set`) | Set static IP |
| Technitium DHCP reservations | Add HAOS reservation |
| `/etc/caddy/Caddyfile` on CT 121 | Already fixed — only changes again if we re-assign paperless IP to `.130` |
| `scripts/check-caddy-backends.sh` | New validation script (new file in homelab-context) |
### Risks / Tradeoffs
- **Pinning paperless to `.130`** — if the LXC was reinstalled since then, `.130` may already be in use. Verify first with `arp-scan` or `nmap`.
- **HAOS static IP via Technitium reservation** vs **inside HAOS**: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
- **Validation script false positives** — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
- **Caddy reload** — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).
### Verification
1. After setting paperless static: `ssh root@192.168.8.175 "curl -s -o /dev/null -w '%{http_code}' http://192.168.8.130:8000"` → 302
2. After Technitium HAOS reservation: `curl -s -o /dev/null -w '%{http_code}' http://192.168.8.101:8123` → 200
3. Run validation script → all targets reachable
4. Confirm from iPhone: both `paperless.hubris.network` and `home.hubris.network` load
### Open questions
1. Should paperless go back to `.130` (its original), or stay at `.243` (current)? Going back to `.130` means updating the Caddyfile again, but keeps the static range allocation consistent.
2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
4. ZimaOS (VM 100) — should we also pin its IP while we're at it?

View File

@@ -0,0 +1,130 @@
# Plan: Fix Frequent Authentik Login Prompts
## Goal
Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 730 days, or until explicit logout).
## Current Context
Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
```
Browser → Caddy (LXC 121) → VPS Traefik → Authentik
```
Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
## Root Cause Found
### Primary: `SESSION_EXPIRE_AT_BROWSER_CLOSE = True`
The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
### Secondary: `SESSION_COOKIE_AGE = 86400` (24 hours)
Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
### Session configuration (from Docker Python environment):
| Setting | Current Value | Default in Django |
|---------|---------------|-------------------|
| `SESSION_EXPIRE_AT_BROWSER_CLOSE` | `True` | `False` |
| `SESSION_COOKIE_AGE` | `86400` (24h) | `1209600` (14d) |
| `SESSION_SAVE_EVERY_REQUEST` | `False` | `False` |
| `SESSION_COOKIE_SAMESITE` | `Lax` | `Lax` |
### What ISN'T the problem:
- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
- **Caddy configuration** — The forward-auth chain is correctly set up.
- **Outpost health** — All containers healthy, up for 6 days.
## Proposed Approach
Change two Django session settings via Authentik environment variables:
1. **`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
2. **`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
## Step-by-step Plan
### Step 1: Add environment variables to Docker compose
Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
```yaml
authentik-server:
environment:
# ... existing vars ...
AUTHENTIK_SESSION_COOKIE_AGE: "604800" # 7 days (was 86400 / 24h)
AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE: "false" # was true
```
Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
### Step 2: Verify variable naming
Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
### Step 3: Restart Authentik server
```bash
ssh root@82.165.190.79
docker compose -f /opt/docker-compose.yml restart authentik-server
```
### Step 4: Verify the fix
```bash
# Check session settings took effect
ssh root@82.165.190.79 'docker exec -i authentik-server python3 << "PYEOF"
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
import django
django.setup()
from django.conf import settings
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)
PYEOF'
```
### Step 5: Functional test
1. Login to Authentik at `auth.hubris.network`
2. Close the browser completely
3. Re-open browser, navigate to a forward-auth-gated service (e.g., paperless.hubris.network)
4. Verify you're NOT redirected to login
5. Verify OAuth2 services (Gitea) also maintain the session
## Files Likely to Change
| File | Change |
|------|--------|
| `/opt/docker-compose.yml` | Add `AUTHENTIK_SESSION_COOKIE_AGE` and `AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` env vars |
## Tests / Validation
1. **Config verification** — Run Python snippet inside container to confirm Django settings changed
2. **Browser test** — Close/reopen browser, verify session persists (Step 5 above)
3. **24-hour test** — Check session is still alive after 24h of normal use
## Risks, Tradeoffs, and Open Questions
| Risk | Mitigation |
|------|------------|
| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
## Open Questions
1. **What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
2. **Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
3. **Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.

View File

@@ -0,0 +1,149 @@
# Plan: Fix Caddyfile truncation + prevent recurring outages
**Date:** 2026-06-06
**Slug:** caddyfile-truncation-permanent-fix
---
## Goal
Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
## Root cause
The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
**Evidence:**
- `git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
- `origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
**Why "third time this week":**
| Incident | Date | Cause |
|---|---|---|
| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
## Immediate fix
### Step 1: Restore Caddyfile from origin/master and reload
On LXC 121:
```bash
cd /etc/caddy
# Stash any local changes
git stash
# Reset to origin/master
git checkout --force origin/master -- Caddyfile
# Caddyfile now has all 30+ sites
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy
```
This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
### Step 2: Add `photos2.hubris.network` via git (if still needed)
The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
### Step 3: Verify
- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
- Flush mac-mini DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
## Permanent safeguards
### Layer 1: Caddyfile integrity check (deploy hook)
Add a site-count validation to the deploy script (`/etc/caddy/scripts/deploy.sh`):
```bash
# Count site blocks (lines matching *.hubris.network {)
SITE_COUNT=$(grep -c '^[a-z].*hubris.network {' Caddyfile)
if [ "$SITE_COUNT" -lt 20 ]; then
echo "[deploy] ERROR: Only $SITE_COUNT sites found (expected 20+). Refusing to reload."
exit 1
fi
```
This catches any future truncation before `caddy reload` runs.
### Layer 2: Caddyfile backup on deploy
Add to deploy script before git pull:
```bash
cp Caddyfile "Caddyfile.bak.$(date +%s)"
```
Keep last 3 backups, auto-rotate.
### Layer 3: Dirty-tree handling in deploy webhook
The deploy webhook currently hard-fails when the working tree is dirty. Change the receiver script to handle this gracefully:
```bash
cd /etc/caddy
# If dirty, stash local changes
if ! git diff --quiet; then
echo "[deploy] Working tree dirty — stashing"
git stash push -m "auto-stash by deploy webhook $(date)"
fi
git pull --ff-only
```
This prevents the webhook from blocking on future local edits.
### Layer 4: Scheduled Caddyfile health check
Add a homelab cron job that runs `check-caddy-backends.sh` every 10 minutes and notifies if any Caddy backend is unreachable.
```yaml
# In homelab context: cronjob
schedule: "*/10 * * * *"
script: /opt/homelab-context/scripts/check-caddy-backends.sh
```
### Layer 5: DNS sync cron (fix already-deployed sync)
The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
```bash
echo "*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1" > /etc/cron.d/dns-sync
```
## Files likely to change
| File | Change |
|------|--------|
| `/etc/caddy/Caddyfile` on LXC 121 | Restore from origin/master |
| `/etc/caddy/scripts/deploy.sh` on LXC 121 | Add site-count validation + backup + dirty-tree handling |
| `caddy-conf` git repo | PR with deploy.sh improvements + photos2 (if wanted) |
| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
## Verification
1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
2. `check-caddy-backends.sh` exits 0 on hubris
3. `systemctl status caddy` shows active on LXC 121
4. `dns-sync` runs and writes to `/var/log/dns-sync.log`
## Risks / Tradeoffs
- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
## Open questions
1. Keep `photos2.hubris.network`? If yes, add via proper git push.
2. `prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?

View File

@@ -23,7 +23,9 @@ creation_rules:
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/gitea-pat\.yaml$
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
@@ -32,7 +34,9 @@ creation_rules:
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/gitea-tokens\.yaml$
# Workstations only.
@@ -42,7 +46,7 @@ creation_rules:
- path_regex: ^secrets/webhook-hmacs\.yaml$
# LXCs that run a webhook receiver.
age: >-
# placeholder — fill with age_pubkey of: apps, caddy, claudio-bot, claudio-monitor host
# placeholder — fill with age_pubkey of: apps, caddy
- path_regex: ^secrets/turn-shared-secret\.yaml$
# coturn TURN long-term-credential password. Consumed by hubris (which
@@ -52,7 +56,9 @@ creation_rules:
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
# Authentik OIDC client secret for the netbird-dashboard provider.
@@ -61,5 +67,46 @@ creation_rules:
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/netbird-pat\.yaml$
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
- path_regex: ^secrets/openrouter-api-key\.yaml$
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
# spawning a Goose session. Recipients are any host that should run a
# Nous-Hermes agent. Add a host's age_pubkey here, then
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
# See operations/hermes-agent.md.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/yuvomi-api-token\.yaml$
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
# LXC 129 (house).
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
- path_regex: ^secrets/hermes-house-users\.yaml$
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
age: >-
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
# webhook noop 2026-05-20T18:16:57+02:00

File diff suppressed because one or more lines are too long

View File

@@ -83,7 +83,33 @@ Grep is fine for browsing or when MCP is unreachable.
- **Wiki updates**: same-session rule applies to any meaningful state change
this client makes.
## 6. When in doubt
## 6. Communication mode
Read and apply `/opt/homelab-context/CAVEMAN.md` (if present). It defines the lab's
terse-communication standard — drop filler, keep substance, use fragments.
## 7. Auto-setup mechanism
The homelab-context repo ships tooling that gets automatically installed
on every client after `git pull`. This is handled by `tools/post-pull.sh`
(replaces the raw git pull in the sync timer) which runs any script matching
`tools/*.setup.sh` after pull.
Currently auto-setup:
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
package, wrapper scripts, and compact output templates for token-efficient
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
Hermes agent follows the canonical homelab persona (token efficiency, source
of truth hierarchy). No-op on non-Hermes agents.
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
commit and push. All enrolled clients pick it up within 5 minutes.
To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
## 8. When in doubt
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
The clone is the fallback; MCP is the index.

33
CAVEMAN.md Normal file
View File

@@ -0,0 +1,33 @@
# CAVEMAN.md — communication mode for homelab agents
Respond terse like smart caveman. All technical substance stay. Only fluff die.
## Rules
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
Pattern: `[thing] [action] [reason]. [next step].`
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
## Levels
- **lite** — no filler/hedging. Keep articles + full sentences. Professional but tight.
- **full** (default) — drop articles, fragments OK, short synonyms. Classic caveman.
- **ultra** — abbreviate prose words (DB/auth/config/req/res), strip conjunctions, arrows (X → Y). Code symbols/API names/errors: never abbreviate.
Switch: `/caveman lite|full|ultra`. Stop: "normal mode".
## Auto-Clarity
Drop caveman for: security warnings, irreversible actions, multi-step sequences where fragments risk misread, user confused/repeating. Resume after clear part done.
## Boundaries
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
---
Source: https://github.com/JuliusBrussee/caveman
Copy to `~/.hermes/skills/` for Hermes Agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.

View File

@@ -67,6 +67,30 @@ Links to nodes that host or depend on this.
## Changelog
```
### Plan (`plans/YYYY-MM-DD-slug.md`)
```markdown
# YYYY-MM-DD — <title>
## Goal
What this change achieves and why.
## Current topology / state
Diagram or description of what exists now.
## Target topology / state
What it looks like after.
## Pre-flight checklist
## Step-by-step procedure
## Verification
## Post-migration
Changelog entries to write, index status to update.
```
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
```markdown
@@ -89,6 +113,7 @@ Links to nodes that host or depend on this.
- Every container page links to every cross-cutting page it participates in.
- Every cross-cutting page lists the nodes that participate.
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
## Changelog hygiene

83
HERMES.md Normal file
View File

@@ -0,0 +1,83 @@
# HERMES.md — Agent persona for homelab clients
This file is the canonical agent persona for **all** AI agents running on
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
conventions, and the source-of-truth hierarchy.
## Source of truth
The homelab-context repo at `/opt/homelab-context/` is the single source of
truth for:
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
- Service endpoints and credentials (via `homelab secret`)
- Agent behaviour and conventions
- Everything in this file
When in doubt, check `/opt/homelab-context/` first.
## Agent type — how this file gets loaded
| Agent | Loading mechanism |
|-------|------------------|
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints``/opt/homelab-context/HERMES.md` |
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
homelab-context repo instead. Changes propagate to all clients on the next
sync (`sudo homelab sync`).
---
## Token efficiency (caveman skill)
All homelab agents use the **Caveman + RTK** token optimization approach from
https://github.com/adityahimaone/hermes-agent-rtk-caveman.
### Before running any CLI command, ask:
1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
- `~/bin/caveman_wrapper.sh git-status` — compact git status
- `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
- `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
- `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
```
rtk <command>
```
RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
deduplicates repeated lines. This reduces token usage by 60-90% on CLI
operations.
3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
raw SSH/shell — they're already token-optimized.
### Templates
Caveman templates live at `~/templates/`:
- `git_status.txt` — compact git status format
- `git_log.txt` — compact git log format
- `lint_results.txt` — compact ESLint format
- `test_results.txt` — compact vitest/jest format
### When to skip caveman/rtk
- Interactive commands (editors, prompts) — let human-readable output pass
- Commands with no output — skip entirely
- When you need the exact raw output for post-processing
### Verification
```bash
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
```
## Important note for Hermes agents
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
can verify the content matches or re-provision by running:
bash /opt/homelab-context/tools/setup-hermes-soul.sh

View File

@@ -22,22 +22,21 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links:
| 102 | [nfs-export](containers/102-nfs-export.md) | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS |
| 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt |
| 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server |
| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) |
| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / PlantUML / Portainer / WriteFreely) |
| 114 | [nextcloud](containers/114-nextcloud.md) | 192.168.8.224 | Personal cloud |
| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse |
| 119 | [sophia](containers/119-sophia.md) | 192.168.8.157 | Sophia |
| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos |
| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy |
| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) |
| 123 | [claudio-bot](containers/123-claudio-bot.md) | 192.168.8.230 | Matrix control plane |
| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS |
| 126 | [plato](containers/126-plato.md) | 192.168.8.190 | Plato (notes/discovery workspace) |
| 130 | [grimmory](containers/130-grimmory.md) | 192.168.8.213 | Digital library (Grimmory — fork of Booklore) |
### Cross-cutting infrastructure
- [DNS — split-horizon](infrastructure/dns.md)
- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
- [Monitoring — claudio-monitor](infrastructure/monitoring.md)
- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
- [SSH access](infrastructure/ssh-access.md)
- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
@@ -61,7 +60,7 @@ Time-stamped incident notes / experiments in [`investigations/`](investigations/
```
- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image, claudio-bot) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
## Maintaining this wiki

36
bin/hermes Executable file
View File

@@ -0,0 +1,36 @@
#!/usr/bin/env bash
#
# hermes — launch a Goose session pre-wired with the homelab persona,
# OpenRouter (Nous Hermes) provider, and the homelab MCP server.
#
# See operations/hermes-agent.md for the full onboarding flow.
set -euo pipefail
die() { echo "hermes: $*" >&2; exit 1; }
command -v goose >/dev/null \
|| die "goose binary not found — re-run bootstrap.sh with --with-hermes"
command -v homelab >/dev/null \
|| die "homelab CLI not found — is this client bootstrapped?"
# Decrypt OpenRouter API key.
# `homelab secret` re-execs via sudo for non-root users (age key is 0600 root).
SECRET_YAML=$(homelab secret openrouter-api-key 2>&1) || \
die "could not decrypt secrets/openrouter-api-key.yaml — this host probably
isn't a recipient yet. See operations/hermes-agent.md ('Granting the OpenRouter
key to a new host'). sops output:
${SECRET_YAML}"
API_KEY=$(printf '%s' "$SECRET_YAML" | python3 -c \
'import sys, yaml; print(yaml.safe_load(sys.stdin)["api_key"])')
case "$API_KEY" in
PLACEHOLDER_*|"")
die "openrouter-api-key.yaml still contains the placeholder; operator
must run \`sops secrets/openrouter-api-key.yaml\` on hubris to insert a real
\`sk-or-...\` key and push the change." ;;
esac
export OPENROUTER_API_KEY="$API_KEY"
exec goose session "$@"

View File

@@ -199,6 +199,12 @@ SHARED_SECRETS = [
("secrets/netbird-authentik-oidc.yaml", "^secrets/netbird-authentik-oidc\\.yaml$"),
]
# Secrets granted only to hosts that opt into running the Hermes agent
# (via `homelab client add --finalize-pubkey ... --with-hermes`).
HERMES_SECRETS = [
("secrets/openrouter-api-key.yaml", "^secrets/openrouter-api-key\\.yaml$"),
]
# -------- comment-preserving inventory.yaml edits --------
# yaml.safe_load + safe_dump round-trips strip every comment, which is fine
@@ -360,11 +366,15 @@ def _add_recipient_to_sops_policy(sops_path: Path, path_regex_pattern: str, pubk
return True
def _grant_shared_secrets(pubkey: str) -> None:
"""Add `pubkey` to the recipient list of every shared secret + re-key."""
def _grant_shared_secrets(pubkey: str, secrets: list[tuple[str, str]] = SHARED_SECRETS) -> None:
"""Add `pubkey` to the recipient list of every listed secret + re-key.
`secrets` defaults to SHARED_SECRETS; pass HERMES_SECRETS to grant the
Hermes-only set.
"""
sops_path = CONTEXT / ".sops.yaml"
env = {**os.environ, "SOPS_AGE_KEY_FILE": str(AGE_KEY)}
for rel_path, pattern in SHARED_SECRETS:
for rel_path, pattern in secrets:
target = CONTEXT / rel_path
if not target.exists():
print(f" skipping {rel_path}: file does not exist yet")
@@ -440,11 +450,14 @@ def _remove_recipient_from_sops_policy(sops_path: Path, path_regex_pattern: str,
return True
def _revoke_shared_secrets(pubkey: str) -> None:
"""Remove `pubkey` from every shared-secret rule + re-key the files."""
def _revoke_shared_secrets(pubkey: str, secrets: list[tuple[str, str]] = SHARED_SECRETS) -> None:
"""Remove `pubkey` from every listed secret rule + re-key the files.
Defaults to SHARED_SECRETS; pass HERMES_SECRETS to revoke the Hermes-only set.
"""
sops_path = CONTEXT / ".sops.yaml"
env = {**os.environ, "SOPS_AGE_KEY_FILE": str(AGE_KEY)}
for rel_path, pattern in SHARED_SECRETS:
for rel_path, pattern in secrets:
target = CONTEXT / rel_path
if not target.exists():
continue
@@ -504,6 +517,19 @@ def cmd_ssh(args: argparse.Namespace) -> int:
os.execvp(cmd[0], cmd)
def cmd_ssh_config(args: argparse.Namespace) -> int:
"""Generate SSH config from inventory.yaml."""
script = CONTEXT / "ssh" / "gen-config.py"
if not script.exists():
die(f"ssh-config generator not found: {script}")
cmd = [sys.executable or "python3", str(script)]
if args.install:
cmd.append("--install")
return subprocess.call(cmd)
def cmd_pct(args: argparse.Namespace) -> int:
lxc = args.lxc
inv = inventory()
@@ -1033,7 +1059,8 @@ def cmd_client_add(args: argparse.Namespace) -> int:
return subprocess.call(["sudo", "-E", sys.argv[0], "client", "add"]
+ ([args.name] if args.name else [])
+ (["--finalize-pubkey", args.finalize_pubkey]
if args.finalize_pubkey else []))
if args.finalize_pubkey else [])
+ (["--with-hermes"] if args.with_hermes else []))
name = args.name
inv = inventory()
if not args.finalize_pubkey:
@@ -1054,6 +1081,7 @@ def cmd_client_add(args: argparse.Namespace) -> int:
print(f" 2. On {name}: curl -fsSL <gitea>/dtoro/Homelab-Docs/raw/main/bootstrap.sh | sudo bash")
print(f" 3. bootstrap prints an age pubkey — bring it back here and run:")
print(f" homelab client add {name} --finalize-pubkey <age1...>")
print(f" (append --with-hermes to also grant the Hermes agent's OpenRouter key.)")
return 0
# finalize_pubkey path
@@ -1065,8 +1093,13 @@ def cmd_client_add(args: argparse.Namespace) -> int:
print(f"set age_pubkey for {name}")
print("granting shared secrets...")
_grant_shared_secrets(pubkey)
commit_subject = f"client-add: {name} (finalize age_pubkey + grant shared secrets)"
if args.with_hermes:
print("granting hermes-only secrets...")
_grant_shared_secrets(pubkey, HERMES_SECRETS)
commit_subject = f"client-add: {name} (finalize age_pubkey + grant shared + hermes secrets)"
push_inventory(
f"client-add: {name} (finalize age_pubkey + grant shared secrets)",
commit_subject,
extra_paths=[".sops.yaml", "secrets/"],
)
print(f"finalized {name}.")
@@ -1103,6 +1136,9 @@ def cmd_client_remove(args: argparse.Namespace) -> int:
if pubkey:
print("revoking shared secrets...")
_revoke_shared_secrets(pubkey)
# Also revoke from hermes-only secrets; idempotent if the pubkey
# was never on those rules (logs a "not present" warning, no harm).
_revoke_shared_secrets(pubkey, HERMES_SECRETS)
else:
print(f" note: no age_pubkey recorded for {name} — skipping sops re-key")
@@ -1436,6 +1472,12 @@ def main() -> int:
sp.add_argument("command", nargs=argparse.REMAINDER)
sp.set_defaults(func=cmd_ssh)
sp = sub.add_parser("ssh-config",
help="generate ~/.ssh/config.d/homelab from inventory.yaml")
sp.add_argument("--install", "-i", action="store_true",
help=f"write to ~/.ssh/config.d/homelab and wire Include into main config")
sp.set_defaults(func=cmd_ssh_config)
sp = sub.add_parser("pct", help="proxy pct commands via ssh to hubris")
sp.add_argument("lxc")
sp.add_argument("action")
@@ -1518,6 +1560,10 @@ def main() -> int:
csub_add.add_argument("name")
csub_add.add_argument("--finalize-pubkey", default=None,
help="set/update age_pubkey for an already-added client")
csub_add.add_argument("--with-hermes", action="store_true",
help="also grant secrets/openrouter-api-key.yaml so this "
"host can run the Hermes agent (see "
"operations/hermes-agent.md). Combine with --finalize-pubkey.")
csub_add.set_defaults(func=cmd_client_add)
csub_rm = csub.add_parser("remove")
csub_rm.add_argument("name")

View File

@@ -5,6 +5,7 @@
# curl -fsSL https://git.hubris.network/dtoro/Homelab-Docs/raw/main/bootstrap.sh \
# | sudo bash
# curl ... | sudo bash -s -- --with-mcp # also wire Claude's .mcp.json
# curl ... | sudo bash -s -- --with-hermes # also install Goose + Hermes wrapper
# curl ... | sudo bash -s -- --dry-run # show what would happen
# curl ... | sudo bash -s -- --no-secrets # skip age-key issuance
#
@@ -22,9 +23,12 @@ REPO_HTTPS="${HOMELAB_REPO_URL:-https://git.hubris.network/dtoro/Homelab-Docs.gi
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab-context}"
ISSUANCE_URL_NETBIRD="${HOMELAB_ISSUANCE_NETBIRD:-https://secrets.hubris.network/issue}"
ISSUANCE_URL_TAILSCALE="${HOMELAB_ISSUANCE_TAILSCALE:-https://secrets.hubris.network/issue}"
MCP_URL="${HOMELAB_MCP_URL:-https://mcp.hubris.network/sse}"
MCP_URL="${HOMELAB_MCP_URL:-https://mcp.hubris.network/mcp}"
HERMES_MCP_URI="${HOMELAB_HERMES_MCP_URI:-https://mcp.hubris.network/mcp}"
HERMES_MODEL="${HOMELAB_HERMES_MODEL:-nousresearch/hermes-4-405b}"
WITH_MCP=0
WITH_HERMES=0
DRY_RUN=0
NO_SECRETS=0
GITEA_TOKEN="${HOMELAB_GITEA_TOKEN:-}"
@@ -34,6 +38,7 @@ GITEA_USER="${HOMELAB_GITEA_USER:-dtoro}"
while [ $# -gt 0 ]; do
case "$1" in
--with-mcp) WITH_MCP=1; shift ;;
--with-hermes) WITH_HERMES=1; shift ;;
--dry-run) DRY_RUN=1; shift ;;
--no-secrets) NO_SECRETS=1; shift ;;
--gitea-token) GITEA_TOKEN="$2"; shift 2 ;;
@@ -403,6 +408,116 @@ PYEOF
fi
fi
# -------- Hermes (Goose + Nous Hermes) wiring --------
# Installs the Goose CLI binary system-wide, symlinks the `hermes` wrapper
# and HERMES.md persona, and drops a Goose config that pins the OpenRouter
# provider, the Nous Hermes model, and the homelab MCP extension.
# See operations/hermes-agent.md.
if [ "$WITH_HERMES" -eq 1 ]; then
# Resolve the operator's home (SUDO_USER under `sudo bash`).
if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then
H_USER="$SUDO_USER"
H_HOME=$(eval echo "~$SUDO_USER")
else
H_USER="root"
H_HOME="$HOME"
fi
# 1. Install Goose binary at /usr/local/bin/goose (idempotent).
if ! command -v goose >/dev/null 2>&1; then
echo "[bootstrap] installing Goose CLI"
if [ "$DRY_RUN" -eq 1 ]; then
echo "+ would run upstream goose installer and symlink to /usr/local/bin/goose"
else
# Upstream installer drops the binary at ~/.local/bin/goose for the
# invoking user. We run it as $H_USER then symlink system-wide.
sudo -u "$H_USER" \
env CONFIGURE=false \
bash -c 'curl -fsSL https://github.com/aaif-goose/goose/releases/download/stable/download_cli.sh | bash'
if [ -x "$H_HOME/.local/bin/goose" ]; then
ln -sfn "$H_HOME/.local/bin/goose" /usr/local/bin/goose
else
echo "[bootstrap] WARNING: goose binary not found at $H_HOME/.local/bin/goose after install" >&2
fi
fi
else
echo "[bootstrap] goose already installed: $(command -v goose)"
fi
# 2. Symlink hermes wrapper.
echo "[bootstrap] linking hermes CLI to /usr/local/bin/hermes"
run "ln -sfn '$CLONE_DIR/bin/hermes' /usr/local/bin/hermes"
# 3. Symlink HERMES.md persona. The hermes wrapper does not need it — the
# Goose config below references the canonical clone path — but operators
# frequently `cat /root/HERMES.md` to inspect the persona, mirroring the
# AGENTS.md convention above.
case "$OS" in
Linux) HERMES_LINK=/root/HERMES.md ;;
Darwin) HERMES_LINK=/etc/HERMES.md ;;
esac
run "ln -sfn '$CLONE_DIR/HERMES.md' '$HERMES_LINK'"
echo "[bootstrap] linked HERMES.md → $HERMES_LINK"
# 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the
# operator added by hand, overwrites only the keys we manage.
GOOSE_DIR="$H_HOME/.config/goose"
GOOSE_CONFIG="$GOOSE_DIR/config.yaml"
GOOSEHINTS="$GOOSE_DIR/.goosehints"
run "mkdir -p '$GOOSE_DIR'"
PY_GOOSE_MERGE=$(cat <<PYEOF
import os, sys
try:
import yaml
except ImportError:
print("PyYAML required", file=sys.stderr); sys.exit(2)
path = "$GOOSE_CONFIG"
mcp_uri = "$HERMES_MCP_URI"
model = "$HERMES_MODEL"
cfg = {}
if os.path.exists(path):
with open(path) as f:
try:
cfg = yaml.safe_load(f) or {}
except Exception:
cfg = {}
cfg["GOOSE_PROVIDER"] = "openrouter"
cfg["GOOSE_MODEL"] = model
cfg.setdefault("GOOSE_MODE", "smart_approve")
cfg.setdefault("extensions", {})
cfg["extensions"]["developer"] = {
"bundled": True, "enabled": True, "name": "developer",
"timeout": 300, "type": "builtin",
}
cfg["extensions"]["homelab"] = {
"enabled": True, "name": "homelab",
"description": "Read-only homelab context tools (FastMCP).",
"type": "streamable_http", "uri": mcp_uri, "timeout": 60,
}
with open(path, "w") as f:
yaml.safe_dump(cfg, f, sort_keys=False)
print("[bootstrap] merged Goose config at", path)
PYEOF
)
if [ "$DRY_RUN" -eq 1 ]; then
echo "+ would merge Goose config at $GOOSE_CONFIG"
else
python3 -c "$PY_GOOSE_MERGE"
chown -R "$H_USER" "$GOOSE_DIR"
fi
# 5. Symlink HERMES.md as the global .goosehints — Goose injects it into
# the system prompt on every session start.
run "ln -sfn '$CLONE_DIR/HERMES.md' '$GOOSEHINTS'"
if [ "$DRY_RUN" -eq 0 ]; then
chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true
fi
fi
# -------- netbird tuning (skip per-session SSO for ssh into mesh peers) --------
# Apply the SSH JWT cache TTL so `ssh ... .netbird.selfhosted` doesn't trigger
# device-code SSO on every connection. Flag added in netbird 0.71.x

View File

@@ -8,7 +8,7 @@ Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clien
- **LAN DNS:** `nfs-export.hubris.network``192.168.8.200` (direct, no Caddy)
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, plato, apps)
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
## What it does

View File

@@ -24,7 +24,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
## Known issues
- Disk usage was 86.9% at last claudio-monitor reading on 2026-04-21. Monitor or grow rootfs.
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](124-authentik.md)

View File

@@ -45,6 +45,9 @@ LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (r
## Changelog
### 2026-06-24 — terminalito deploy webhook (id 12)
Push webhook on `dtoro/terminalito``http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -1,6 +1,6 @@
# 105 — `apps`
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20.
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
## At a glance
- **Hostname:** `apps`
@@ -15,7 +15,6 @@ Docker host for everything that doesn't justify its own LXC. Currently runs Arti
| Hostname | Container | Backend port | Notes |
| --------------------------------- | ---------------- | ------------ | ----- |
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
| `books.hubris.network` | Booklore | `:6060` | Native OIDC. Redirect URI `/oauth2-callback`. |
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
@@ -46,11 +45,6 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### Booklore
Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (NOT `/api/oidc`). Container needs `extra_hosts: auth.hubris.network:192.168.8.175`. **Edit via Portainer UI** if it's a Portainer-managed stack.
> ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning.
### homelab-mcp (`/opt/homelab-mcp/`)
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
@@ -60,7 +54,7 @@ is `dtoro/Homelab-Docs/mcp/server.py`; service unit
disabled at the FastMCP layer because mesh+LAN gating is the actual
trust boundary.
- Endpoint: `https://mcp.hubris.network/sse` (Caddy → `:9810`).
- Endpoint: `https://mcp.hubris.network/mcp` (Caddy → `:9810`). StreamableHTTP transport (POST `/mcp`).
- 14 tools registered: `get_host`, `list_services`, `find_service`,
`get_topology`, `search_docs`, `get_page`, `get_changelog`, `whoami`,
`list_my_secrets` (context); `get_service_status`, `tail_log`,
@@ -113,6 +107,9 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h
## Changelog
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
### 2026-05-20 — homelab-mcp + secrets-issuance live
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
`homelab-mcp.service` on `:9810` (MCP read+management surface) and

View File

@@ -0,0 +1,56 @@
# 106 — `auth-outpost`
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
## At a glance
- **Hostname:** `auth-outpost`
- **IP:** `192.168.8.6` (static)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
- **Mounts:** none
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md).
## Service / port map
| Service | Listen | Notes |
| --- | --- | --- |
| authentik proxy outpost | `192.168.8.6:9000` | `/outpost.goauthentik.io/*` (ping, auth/caddy) |
## Config paths
- `/opt/authentik-outpost/docker-compose.yml``ghcr.io/goauthentik/proxy:2026.5.2`, `AUTHENTIK_HOST=https://auth.hubris.network`, `AUTHENTIK_INSECURE=false`, port bound `192.168.8.6:9000`.
- `/opt/authentik-outpost/.env` (mode 600, **untracked**) — `AUTHENTIK_TOKEN` for outpost `hubris-lan-outpost` (sops-encrypt into `secrets/` — TODO Phase 5).
## The outpost object (VPS Authentik)
- Outpost **`hubris-lan-outpost`** (type proxy), providers: `hubris-forward-auth` (domain, Paperless + domain-level apps), `Provider for Torrent` (qBittorrent), `Provider for Artifacto` (single).
- Health: VPS admin → Applications → Outposts → last-seen current; `curl http://192.168.8.6:9000/outpost.goauthentik.io/ping``204`; container log `Successfully connected websocket`.
## Caddy wiring
[Caddy (121)](121-caddy.md) `(authentik)` snippet `reverse_proxy`/`forward_auth``http://192.168.8.6:9000` (was `192.168.8.180:9000` on LXC 124). Tracked in `dtoro/caddy-conf`.
## `sso.hubris.network` — the callback domain (critical)
Domain-level (`forward_domain`) providers redirect the browser to `{external_host}/outpost.goauthentik.io/callback` after login. On LXC 124 that `external_host` was `auth.hubris.network`, which resolved (LAN) to Caddy → the 124 outpost — same box, so it worked. Post-migration `auth.hubris.network` points to the **VPS core**, so the callback hit the wrong outpost → **400 / redirect-uri mismatch**.
Fix: the LAN outpost gets its **own** domain.
- DNS: `sso.hubris.network → 192.168.8.175` (Caddy). *(in dnsmasq today — must be carried into Technitium in DNS Phase 2.)*
- Caddy: site `sso.hubris.network { tls dns ionos; reverse_proxy 192.168.8.6:9000 }` (in `caddy-conf`).
- Authentik: `hubris-forward-auth` and `Provider for Torrent` have `external_host=https://sso.hubris.network`; their `redirect_uris` must match (`set_oauth_defaults()` regenerates them from `external_host` — changing `external_host` alone does NOT update them).
- `Provider for Artifacto` is `forward_single` on its own domain, so it was unaffected.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
- [Caddy (121)](121-caddy.md) — forward-auth consumer
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md)
## Changelog
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000``192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).

67
containers/107-dns.md Normal file
View File

@@ -0,0 +1,67 @@
# 107 — `dns`
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
## At a glance
- **Hostname:** `dns`
- **IP:** `192.168.8.2` (static — stable, decoupled from any app)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 1 GiB / 8 GiB rootfs
- **Created:** 2026-06-01, Debian 13. Its own resolver is `1.1.1.1` (no circular dependency on the DNS it serves).
## Role
Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recursive forwarding (`1.1.1.1`, `8.8.8.8`) for everything else. Technitium runs in Docker (`technitium/dns-server:latest`, `network_mode: host`), web console on `:5380`.
## The `hubris.network` zone
- Specific A overrides: app names → `192.168.8.175` (Caddy), `nfs-export → 192.168.8.200`, `auth/sso/...` as needed.
- `auth.hubris.network → 82.165.190.79` (VPS Authentik), `sso.hubris.network → 192.168.8.175` (LAN forward-auth outpost).
- **Wildcard** `*.hubris.network → 82.165.190.79` — mirrors the public IONOS wildcard so undefined names (e.g. `netbird`) resolve to the VPS, matching public behaviour.
- **MX / SPF-TXT / CAA replicated from public** so an authoritative zone doesn't shadow `hubris.network` email/cert records.
## Config / access
- `/opt/technitium/docker-compose.yml`; admin password in `/opt/technitium/admin_password.txt` (mode 600 — sops-encrypt in Phase 5).
- Console: `http://192.168.8.2:5380` (user `admin`).
- API: `http://192.168.8.2:5380/api/...` (token via `/api/user/login`). Zone was built via the API.
## Who points here
- **NetBird mesh peers:** resolve via the **NetBird managed DNS zone**, kept in sync *from* this Technitium (see dns-sync below). The `home-lab-dns` nameserver group (`→ 192.168.8.2`) is a thin fallback forwarder.
- **Homelab DHCP clients:** Technitium's own DHCP scope hands out `192.168.8.2` as the DNS server for `192.168.8.x` leases (see DHCP section below).
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
- **Scope:** `homelab``192.168.8.241 192.168.8.254`
- **Gateway:** `192.168.8.1` (Proxmox `vmbr0` alias)
- **DNS:** `192.168.8.2` (self)
- **Lease time:** 24 h
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101.239`) are excluded from the pool. Pool narrowed from `.100.240` to `.241.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
## Changelog
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241.254`
Previous pool `.100.240` overlapped with all static LXCs/VMs (`.101.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
### 2026-06-02 — DHCP server enabled; replaces Slate AX DHCP
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).

View File

@@ -1,6 +1,6 @@
# 118 — `elementsynapse`
Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
## At a glance
- **Hostname:** `elementsynapse`
@@ -33,16 +33,29 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
## Known issues
- ~~Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL)
- ~~[claudio-bot (123)](123-claudio-bot.md)~~decommissioned 2026-06-04, replaced by Hermes Agent
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
**Fix:**
- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
- `ifdown eth0 && ifup eth0` applied the static IP
- Killed lingering dhclient process
- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
### 2026-05-15 — phantom-notification cleanup for `@admin`
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.

View File

@@ -28,7 +28,7 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8081`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
@@ -65,6 +65,28 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
## Changelog
### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
**Fix:**
- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
- `systemctl reload caddy`
**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
### 2026-04-28 — wiki entry created
Initial documentation. 16 active sites at this date.

View File

@@ -22,19 +22,27 @@ Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr
## Service / port map
All services route through gluetun's network namespace. Ports are exposed via
the gluetun container:
| Service | Host:Container | Public hostname |
| ------------- | -------------- | ------------------------------------ |
| sonarr | `8989:8989` | direct only |
| radarr | `7878:7878` | direct only |
| lidarr | `8686:8686` | direct only |
| prowlarr | `9696:9696` | direct only |
| bazarr | `6767:6767` | direct only |
| gluetun (VPN) | — | |
| sonarr | `8989:8989` | direct only (via gluetun) |
| radarr | `7878:7878` | direct only (via gluetun) |
| lidarr | `8686:8686` | direct only (via gluetun) |
| prowlarr | `9696:9696` | direct only (via gluetun) |
| bazarr | `6767:6767` | direct only (via gluetun) |
| jellyseerr | `5056:5055` | [`jellyseerr.hubris.network`](../infrastructure/dns.md) |
| qbittorrent | `8080:8080` | [`qbit.hubris.network`](../infrastructure/dns.md) |
| sabnzbd | `8081:8080` | [`sab.hubris.network`](../infrastructure/dns.md) |
| sabnzbd | `8082:8082` HTTP, `9090:9090` HTTPS | [`sab.hubris.network`](../infrastructure/dns.md) |
| flaresolverr | `8191:8191` | internal only |
| homarr | `7575:7575` | internal only |
Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_default` using docker service names.
Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `localhost:<port>`
(services share gluetun's shared network namespace). External services reach them
via `gluetun:<port>` (e.g. Sonarr → qBittorrent at `localhost:8080` or
`gluetun:8080`).
## Categories (qBit + SAB + *arr)
@@ -48,18 +56,30 @@ Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_defa
Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`.
## Auth (qBit reverse-proxy + Authentik forward-auth)
## Auth (reverse-proxy + Authentik forward-auth)
qBit auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
- `WebUI\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
- `WebUI\ReverseProxySupportEnabled=true`
- `WebUI\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
### qBit
Auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
- `WebUI\\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
- `WebUI\\ReverseProxySupportEnabled=true`
- `WebUI\\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
SABnzbd `host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
### SABnzbd
Gated with Authentik forward-auth (applied 2026-06-13). Caddy `sab.hubris.network` block uses `import authentik` inside a `handle` block. SABnzbd's own web auth is disabled:
- `html_login = 0` → no HTML login form
- `username` / `password` cleared → CherryPy Basic Auth not activated
- `local_ranges = 172.18.0.0/16, 192.168.8.0/24, 127.0.0.0/8` → proxied requests from Caddy (192.168.8.x) and Docker-proxy (172.18.x) pass without auth
**API key** (`67ef5a45e4e04157994e977005a33878`) still works for internal service-to-service calls (Sonarr/Radarr/Lidarr via Docker internal networking — they talk to SAB at `localhost:8082`, not through Caddy).
`host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
## Credentials
@@ -101,6 +121,32 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has
## Changelog
### 2026-06-13 — SABnzbd gated with Authentik forward-auth
SABnzbd now uses Authentik forward-auth (same `import authentik` Caddy pattern as qBit). SABnzbd's own web auth disabled: `html_login=0`, credentials cleared, `local_ranges` extended to cover Docker bridge + homelab LAN. API key still works for internal *arr service calls. See [Auth section](#auth-reverse-proxy--authentik-forward-auth) above.
### 2026-06-04 — all arr services moved behind gluetun VPN; SAB port conflict fixed
- All services (sonarr, radarr, lidarr, bazarr, prowlarr, jellyseerr, homarr,
flaresolverr) now use `network_mode: service:gluetun` — whole stack routes
through the VPN
- Port mappings moved from individual services to gluetun container
- **Fixed SABnzbd port conflict**: was crashing in a restart loop because
qBittorrent held port 8080 inside the shared gluetun namespace. Changed
SAB internal port to 8082 (config at `/config/sabnzbd-config/sabnzbd.ini`)
- Caddy `sab.hubris.network` updated to point to `:8082`
- Jellyseerr's `extra_hosts` (auth.hubris.network) moved to gluetun since
`extra_hosts` conflicts with `network_mode`
### 2026-06-02 — ProtonVPN added (gluetun); LXC IP set static
- Added `gluetun` container to compose as a WireGuard VPN sidecar (ProtonVPN, server AL#57, located in Tirana, Albania)
- **qbittorrent** and **sabnzbd** now use `network_mode: service:gluetun` — all traffic routes through the VPN
- Ports 8080 (qBit WebUI), 6881 tcp/udp (qBit BT), 8081 (SAB WebUI) exposed through gluetun
- gluetun config at `gluetun-config/wireguard/wg0.conf` (read-only mount)
- Healthcheck on gluetun; qBit/SAB wait for `service_healthy` before starting
- LXC IP changed from DHCP to static (`192.168.8.132`) via `pct set` + `/etc/network/interfaces`
- **After first start:** Sonarr/Radarr/Lidarr download client host needs updating from `qbittorrent``gluetun` (SAB similarly `sabnzbd``gluetun`)
- **Also fixed:** 7 other DHCP LXCs (101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy) set to static IPs to prevent floating on reboot. See infrastructure/dns.md.
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -1,4 +1,9 @@
# 123 — `claudio-bot`
# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04)
> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini.
> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob.
> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea.
> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details.
Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
@@ -66,6 +71,12 @@ Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/dep
## Changelog
### 2026-06-04 — LXC destroyed; replaced by Hermes Agent
LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd
units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on
Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job.
`@claudio:avispero` Matrix account decommissioned.
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -1,186 +0,0 @@
# 124 — `authentik`
Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box".
## At a glance
- **Hostname:** `authentik`
- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP)
- **Privilege:** privileged
- **Resources:** 2 cores / 4 GiB RAM / 20 GiB rootfs
- **Mounts:** none from `/mnt/library`
- **Public hostname:** [`auth.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `:9000`
- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream).
## Authentik stack (`/opt/authentik/`)
Upstream `docker-compose.yml` + `.env`. Services: `postgresql` (16-alpine), `server`, `worker`. Authentik 2026.x dropped the Redis dependency.
- `.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`.
- `AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled.
- Ports: 9000 (http), 9443 (https) on the LXC.
- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at.
- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created.
## Forward-auth pattern (every gated app)
- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless).
- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint).
- **Authorization flow:** `default-provider-authorization-implicit-consent` (or explicit).
- **Application Launch URL** MUST be the full public URL `https://<sub>.hubris.network/` — outpost matches incoming `X-Forwarded-Host` against it.
- Each Application MUST have at least one **policy/group/user binding** — zero bindings = outpost returns 404 on access.
- **Restart Authentik after binding new apps to the outpost:**
```
pct exec 124 -- docker compose -f /opt/authentik/docker-compose.yml restart server worker
```
Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`.
### Per-app username override pattern (Authentik)
Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`).
1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes).
2. Customization → Property Mappings → Create → **Scope Mapping** (not SAML):
- Name: `<app>-uid-override`, Scope name: `<app>-uid`, Expression:
```python
return {"nc_uid": user.attributes.get("<app>_uid", user.username)}
```
- **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering.
3. Attach the new scope to the provider (Providers → app → Scopes).
4. On the app side, point its OIDC UID-mapping setting at the custom claim.
For Nextcloud:
```
occ user_oidc:provider <name> --mapping-uid=nc_uid
occ user_oidc:provider <name> --scope="openid profile email <app>-uid"
```
### Bypass forward-auth for API paths (mobile apps)
If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block:
```
paperless.hubris.network {
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
@api path /api/*
handle @api {
reverse_proxy 192.168.8.130:8000
}
handle {
import authentik
reverse_proxy 192.168.8.130:8000
}
}
```
API paths to bypass per app:
- [Paperless](103-paperless.md): `/api/*` (Bearer)
- [Sonarr / Radarr / Lidarr / etc.](122-arriman.md): `/api/*` (X-Api-Key)
- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`)
- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs
- Homarr: no mobile client
- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed
### Backend trust of Authentik headers (skip the app's own login after SSO)
- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI.
- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX.
## Per-app integration map
| App | Type | Notes |
| ----------------------------------------- | ---------------- | ----- |
| [Paperless (103)](103-paperless.md) | Forward-auth + REMOTE_USER | `/api/*` bypass |
| [Nextcloud (114)](114-nextcloud.md) | Native OIDC | `nc_uid` override; local dnsmasq required (Guzzle bypasses `/etc/hosts`) |
| [mulita (120)](120-mule-images.md) | Native OIDC | `extra_hosts` override in compose |
| [Booklore (105)](105-apps.md) | Native OIDC | Redirect URI `/oauth2-callback`; `extra_hosts` |
| [Portainer (105)](105-apps.md) | Native OAuth2 | `portainer_uid` custom claim; `--trusted-origins` flag |
| [WriteFreely (105)](105-apps.md) | Native OIDC | `[oauth.generic]` block; `extra_hosts` |
| [qBittorrent (122)](122-arriman.md) | Forward-auth via IP whitelist | Reverse-proxy support enabled in qBit |
| [Artifacto (105)](105-apps.md) | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass |
| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
## Netbird IdP integration — LANDED 2026-05-21
The combined `netbirdio/netbird-server` image was replaced with the canonical vanilla stack (`mgmt + signal + relay + dashboard` 0.71.3) on the VPS so external OIDC actually works. Authentik is now the netbird dashboard's IdP. Full migration context in [mesh.md changelog](../infrastructure/mesh.md#changelog).
**Active provider & app:**
- Provider `NetBird` (OAuth2/OpenID), **Client type: `Public`** (PKCE-only — `Confidential` would break the dashboard SPA's token exchange).
- Client ID: `netbird-dashboard`. Client Secret is in `/opt/management.json` `PKCEAuthorizationFlow.ProviderConfig.ClientSecret` on the VPS (TODO: sops-encrypt as `secrets/netbird-authentik-oidc.yaml`).
- Application `NetBird`, slug `netbird`, launch URL `https://netbird.hubris.network/`.
- Redirect URIs: `https://netbird.hubris.network/peers`, `/nb-auth`, `/nb-silent-auth`, plus `https://netbird.hubris.network/` for post-logout.
- Scopes enabled on the provider: `openid`, `profile`, `email`.
- Discovery URL: `https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration` — netbird mgmt fetches this on startup; logs `loaded OIDC configuration from the provided IDP configuration endpoint`.
**Login flow:** netbird dashboard PKCE → Authentik authorize → redirect back to `/nb-auth` → JS token exchange at Authentik's `/token` endpoint → mgmt validates the bearer against Authentik's JWKS.
**First-time owner promotion gotcha** (write-down for future operators):
When a new Authentik user logs in for the first time against an account that already has peers, netbird mgmt adds them as `role=user, blocked=1, pending_approval=1`. The OLD account-owner (the one in `store.db` from before the IdP swap) can't be reached anymore, so there's no admin to approve. Recovery is a direct sqlite update on `mgmt_data`:
```
docker stop netbird-mgmt
sqlite3 /var/lib/docker/volumes/opt_mgmt_data/_data/store.db \
"UPDATE users SET role='owner', blocked=0, pending_approval=0 WHERE id='<new authentik sub>';"
docker start netbird-mgmt
```
The Authentik sub-claim is the value of the `id` column on the newly-created user row (look for `role=user, blocked=1, pending_approval=1`).
### Device Code grant — configured (2026-05-21)
`netbird up` (interactive, without `--setup-key`) works against Authentik. The recipe:
1. **Flow** `default-device-code-flow` (designation: `Stage Configuration`) with 4 stage bindings in order:
- 10: `default-authentication-identification` (username/email lookup)
- 20: `default-authentication-password` (password validation)
- 30: `default-authentication-login` (attach authenticated user to session)
- 40: `default-provider-authorization-explicit-consent`'s Consent Stage (`default-provider-authorization-consent`) — the "Authorize NetBird?" approval
2. **Brand** (System → Brands → edit the brand serving `auth.hubris.network`): set **Device code flow** field to `default-device-code-flow`.
3. No provider-side change is required — Authentik 2026.x routes `/device` via the brand's device-code flow, not via the OAuth2/OpenID provider's `Authorization flow`.
**Why this matters for the lab**: Authentik 2026.x doesn't ship a default device-code flow. Without this configuration, the URL `https://auth.hubris.network/device` renders blank (the `/device` endpoint is unrouted), so `netbird up` device-codes expire without consent → only `--setup-key` works for onboarding. The above unblocks interactive onboarding.
**Verifying** from a browser tab: visit `https://auth.hubris.network/device`. You should see a form with one **Code** input + Continue button. Then `netbird up` (no setup-key) end-to-end:
- CLI prints `verification_uri_complete: https://auth.hubris.network/device?code=...`
- Open URL → identification (skipped if logged in) → password (re-auth check) → consent ("Authorize NetBird?") → Continue
- CLI completes registration with `Connected`
### Self-service onboarding (not yet — future-session)
`auth.hubris.network` is only reachable from inside the netbird mesh (split-horizon DNS). A brand-new client that isn't on the mesh yet can't OIDC-login → setup-key is the only path. To enable self-service onboarding via Authentik from the public internet:
- Add a Traefik route on the VPS for `auth.hubris.network` that forwards via the netbird-routed `192.168.8.0/24` to LXC 124.
- DNS already points `auth.hubris.network → 82.165.190.79` (IONOS wildcard).
Tracked in homelab memory as a queued follow-up.
### Old pre-work to remove
The previous `Provider for Netbird` + `netbird` app from 2026-04-22 (client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`, service account `netbird-service`) is now obsolete — replaced by `netbird-dashboard` above. Safe to delete from Authentik admin UI; nothing currently uses the old client ID. The service account + API token can also be removed unless we wire IdpManagerConfig in mgmt later (currently `ManagerType: none`).
## DNS responsibility
dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md).
## Related
- [DNS split-horizon](../infrastructure/dns.md)
- [Caddy (121)](121-caddy.md)
- [Mesh migration](../infrastructure/mesh.md)
- Every gated app under [containers/index](index.md)
## Changelog
### 2026-05-21 — Netbird IdP swap landed (Phase 6 done)
VPS migrated from combined netbird-server to vanilla mgmt+signal+relay+dashboard 0.71.3 (see [mesh.md](../infrastructure/mesh.md)), enabling Authentik as the dashboard IdP via PKCE. New Provider/App = `netbird-dashboard`, replacing the deferred pre-work. Device Code Stage still missing — interactive `netbird up` fails consent; setup-keys are the workaround until that's added.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred
Combined netbird-server image couldn't take an external IdP. Pre-work in Authentik kept for later (now superseded by 2026-05-21 above). Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired
Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.
### 2026-04-21 — deployed; Phases 15 complete
LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC.

View File

@@ -1,108 +0,0 @@
# 126 — `plato`
Docker host for [Plato](https://git.hubris.network/dtoro/Plato) — a cross-linked notes workspace (SvelteKit SPA embedded into a Go HTTP server, SQLite-backed). LAN+mesh only, no public ingress.
## At a glance
- **Hostname:** `plato`
- **IP:** `192.168.8.190`
- **Privilege:** privileged
- **Resources:** 2 cores / 2 GiB RAM / 8 GiB rootfs / 1 GiB swap
- **Mounts:** host `/mnt/library/documents/plato` ↔ container `/opt/plato/data`
- **Public hostname:** [`plato.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.190:8080`
## Stack
Single-container deploy. The repo's `Dockerfile` is a three-stage build (Node → Go → distroless/static-debian12:nonroot, ~23 MiB final image). The container exposes `:8080` and writes its SQLite db to `/data`.
- **Checkout:** `/opt/plato/app` (clone of `http://192.168.8.121:3000/dtoro/Plato.git`, using the cached gitea PAT in `/root/.git-credentials` — same pattern as [caddy (121)](121-caddy.md)).
- **Data:** host `/mnt/library/documents/plato` (owned `65532:65532` to match the distroless nonroot UID) bind-mounted into the LXC at `/opt/plato/data`, then bound into the container at `/data` via a `docker-compose.override.yml`:
```yaml
services:
plato:
volumes: !override
- /opt/plato/data:/data
restart: unless-stopped
```
- **`.env`** at `/opt/plato/app/.env` (optional, untracked) — LLM provider keys (`OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`, etc.) and `PLANTUML_BASE_URL` override. Absent by default; LLM features stay greyed out, PlantUML defaults to the public service.
- **Run / update:** push to `dtoro/Plato` (auto-deploys, see below) or `cd /opt/plato/app && git pull && docker compose up -d --build` for a manual rebuild.
## Auto-deploy
Push to `dtoro/Plato` `main` triggers a rebuild — same Shape B pattern as [Artifacto / mule-image](../infrastructure/auto-deploy.md). Webhook receiver at `/opt/plato-deploy/`, systemd unit `plato-deploy-webhook.service`, port `9799`, gitea hook id 8.
- Receiver: `http://192.168.8.190:9799/deploy`, signed payload (HMAC-SHA256, secret in `/etc/plato-deploy/secret`).
- Logs: `journalctl -u plato-deploy-webhook -f`.
- Health: `curl http://127.0.0.1:9799/health`.
- Manual deploy: `/opt/plato-deploy/deploy.sh`.
- Gitea's `app.ini` `ALLOWED_HOST_LIST` was extended with `192.168.8.190` to allow this delivery.
## Fresh-DB bootstrap workaround
The `schema` constant in `backend/internal/views/store.go` (as of commit `e0542c0`) creates the `views` table without `project_id`, then immediately runs `CREATE UNIQUE INDEX … ON views(project_id, lower(title))`. On a fresh DB this fails (no such column) and Plato crash-loops with `open views store: SQL logic error: no such column: project_id`. `ensureProjectIDColumn()` adds the column on subsequent migrations, but the schema apply happens first.
Until the upstream fix lands, pre-seed the DB before first start:
```
docker compose stop
rm -f /opt/plato/data/plato.db
python3 - <<'PY'
import sqlite3
c = sqlite3.connect('/opt/plato/data/plato.db')
c.executescript("""
CREATE TABLE views (
id TEXT PRIMARY KEY,
type TEXT NOT NULL DEFAULT 'document',
title TEXT NOT NULL,
aliases TEXT NOT NULL DEFAULT '[]',
content TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
project_id TEXT NOT NULL DEFAULT ''
);
CREATE UNIQUE INDEX views_project_title_lower ON views(project_id, lower(title));
CREATE INDEX views_project_id ON views(project_id);
""")
c.commit()
PY
chown 65532:65532 /opt/plato/data/plato.db
docker compose up -d
```
Once the column exists, every subsequent boot's `IF NOT EXISTS` clauses no-op. Once Plato is fixed upstream (remove the `CREATE UNIQUE INDEX` line from the boot `schema` constant — `ensureTitleIndexPerProject()` already re-creates it after the migration), this preseed becomes unnecessary.
## Why privileged
Matches the docker-host convention used by [120 mule-images](120-mule-images.md) and [122 arriman](122-arriman.md). Distroless nonroot's UID `65532` on the host bind mount maps directly through; unprivileged would shift the UID by the idmap offset and the container couldn't write `/data` without extra plumbing.
## Caddy
```
plato.hubris.network {
tls {
dns ionos {env.IONOS_AUTH_API_TOKEN}
}
reverse_proxy 192.168.8.190:8080
}
```
No Authentik forward-auth — Plato has no auth model yet; access control is "be on the LAN or the mesh".
## DNS
dnsmasq entry on [124-authentik](124-authentik.md):
```
address=/plato.hubris.network/192.168.8.175
```
## Related
- [Caddy (121)](121-caddy.md)
- [DNS (split-horizon)](../infrastructure/dns.md)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-05-13 — auto-deploy wired
Shape B pipeline added (`/opt/plato-deploy/`, port `9799`, gitea hook id 8). `ALLOWED_HOST_LIST` in gitea `app.ini` extended with `192.168.8.190`. See [auto-deploy](../infrastructure/auto-deploy.md#plato).
### 2026-05-13 — container created, Plato deployed
LXC 126 stood up on Debian 12 standard, privileged, docker-ce installed. Plato cloned from `dtoro/Plato`, built and started. Caddy site and dnsmasq split-horizon entry added. Recycled the IP/ID slot freed earlier the same day by the [decommissioned Seafile experiment (LXC 125)](index.md#recently-destroyed-kept-for-archaeology). Hit the [fresh-DB bootstrap bug](#fresh-db-bootstrap-workaround) on first boot; worked around by pre-seeding the SQLite schema.

48
containers/128-trmnl.md Normal file
View File

@@ -0,0 +1,48 @@
# 128 — `trmnl`
Self-hosted middleware for TRMNL e-ink plugins. TRMNL cloud polls it; it fetches/shapes live data into JSON the plugin's Liquid template renders.
## At a glance
- **Hostname:** `trmnl`
- **IP:** `192.168.8.211`
- **Privilege:** unprivileged
- **Resources:** 1 core / 768 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** `trmnl.hubris.network` (via [VPS ingress](../infrastructure/ingress.md))
## Role
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `trmnl-plugins` | `0.0.0.0:9851` | uvicorn aggregator; `EnvironmentFile=/etc/trmnl-plugins/env` |
## Storage / config paths
- `/opt/terminalito` — git checkout (origin = internal gitea `http://192.168.8.121:3000/dtoro/terminalito.git`)
- `/opt/terminalito/server/.venv` — venv
- `/etc/trmnl-plugins/env``TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
- `/etc/systemd/system/trmnl-plugins.service`
## Auto-deploy
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito``http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main``server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
## Secrets
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
- [Plan: 2026-06-24 TRMNL plugins LXC](../plans/2026-06-24-trmnl-plugins-lxc.md)
## Changelog
### 2026-06-24 — auto-deploy + LAN DNS wired
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-06-24 — public path live
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
### 2026-06-24 — provisioned
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public``192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.

51
containers/129-house.md Normal file
View File

@@ -0,0 +1,51 @@
# 129 — `house`
Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modules: calendar, tasks, meals, groceries, budget, documents, notes, contacts, birthdays, housekeeping, recipes, reminders.
## At a glance
- **Hostname:** `house`
- **IP:** `192.168.8.212` (static)
- **Privilege:** unprivileged
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** [`house.hubris.network`](../infrastructure/ingress.md) → VPS traefik → Caddy
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `oikos` (Yuvomi) | `0.0.0.0:3000` | Docker Compose at `/opt/yuvomi/`, image `ghcr.io/ulsklyc/yuvomi` |
## Integrations
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
## Config paths
- `/opt/yuvomi/docker-compose.yml` — downloaded from upstream
- `/opt/yuvomi/.env` — config including secrets (untracked)
- `/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
- `/opt/yuvomi/backups/` — auto backups
- `/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
- [Deployment plan](../plans/2026-06-25-yuvomi-deployment.md)
## Changelog
### 2026-06-27 — Google Calendar OAuth env vars configured
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
### 2026-06-26 — provisioned
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.

View File

@@ -0,0 +1,65 @@
# 130 — `grimmory`
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.213` (static, set in PVE `net0` config — same pattern as all other LXCs)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
- **Mounts:** `/mnt/library`
- **Public hostname:** `books.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| Grimmory | `192.168.8.213:6060` | Docker Compose at `/opt/grimmory/` |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/grimmory/docker-compose.yml`. Key points:
- Image: `ghcr.io/grimmory-tools/grimmory:latest`
- Books library: `/mnt/library/books``/books` (read-write; media GID 10000 via `GROUP_ID=10000`)
- Bookdrop (watched folder for auto-import): `/opt/grimmory/bookdrop``/bookdrop`
- App data (covers, DB config): `/opt/grimmory/data``/app/data`
- MariaDB config: `/opt/grimmory/mariadb/config``/config` (linuxserver/mariadb image)
- `extra_hosts: auth.hubris.network:192.168.8.175` — routes Authentik OIDC discovery to Caddy from inside the container
- `FORCE_DISABLE_OIDC=false` — OIDC stays enabled; provider configured via Grimmory admin UI
Credentials live in `/opt/grimmory/.env` (untracked):
- `DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
- `MYSQL_ROOT_PASSWORD` — MariaDB root password
## Authentik OIDC
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
- **Client ID:** `L1u0eFsNhbKgiIvvFeIr2mvZdbtFyzidCq2h6thL`
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
- **Scopes:** openid, profile, email, offline_access
- **Back-channel logout:** `http://192.168.8.213:6060/api/v1/auth/oidc/backchannel-logout`
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
## Media permissions
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
## Related
- [apps (105)](105-apps.md) — previous host (Booklore)
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.213:6060`
- [Authentik (124)](124-authentik.md) — OIDC provider `Grimmory`
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-06-29 — provisioned; Booklore migrated
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.

View File

@@ -0,0 +1,112 @@
# 131 — `teddycloud`
Open-source replacement server for Toniebox smart audio devices (Tonieboxes). Serves device content and API on port 443 and exposes a management web UI at `teddy.hubris.network`.
## At a glance
- **Hostname:** `teddycloud`
- **IP:** `192.168.8.243` (DHCP reservation; MAC `bc:24:11:11:7a:df`)
- **Privilege:** privileged
- **Resources:** 1 core / 1 GiB RAM / 16 GiB rootfs (Debian 12)
- **Mounts:** `/mnt/library` (`mp0`) — TeddyCloud content at `/mnt/library/cloud/leon`
- **Public hostname:** none (LAN-only)
## Role
Replaces the Boxine cloud (`prod.de.bb-online.com`) as the backend for Leon's Toniebox. Tonieboxes connect on port 443 using a custom CA cert issued by TeddyCloud. Content (Tonies) is stored on the NAS at `/mnt/library/cloud/leon` and is accessible from the management UI.
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| TeddyCloud device API | `0.0.0.0:443` | HTTPS, TeddyCloud self-signed CA, Toniebox connects here |
| TeddyCloud HTTP | `0.0.0.0:80` | Redirects to 443 |
| TeddyCloud web UI | `0.0.0.0:8443` | HTTPS management UI — fronted by Caddy at `teddy.hubris.network` (backend uses `tls_insecure_skip_verify` for self-signed cert on LAN hop) |
## Docker Compose
`/opt/teddycloud/docker-compose.yml`:
```yaml
services:
teddycloud:
image: ghcr.io/toniebox-reverse-engineering/teddycloud:latest
ports:
- "80:80"
- "443:443"
- "8443:8443"
volumes:
- certs:/teddycloud/certs
- config:/teddycloud/config
- /mnt/library/cloud/leon:/teddycloud/content
- /mnt/library/cloud/leon:/teddycloud/library
restart: unless-stopped
volumes:
certs:
config:
```
`certs` and `config` are Docker named volumes (runtime state). `content` and `library` are bind-mounted from `/mnt/library/cloud/leon` so audio content persists across container rebuilds and is browsable from the host.
## Storage / config paths
- `/opt/teddycloud/docker-compose.yml` — compose file
- Docker volume `teddycloud_certs` — TeddyCloud CA + server certs (generated on first boot)
- Docker volume `teddycloud_config` — TeddyCloud config
- `/mnt/library/cloud/leon/` — Tonie content + library (NAS bind mount)
## Networking
Two separate traffic paths — different IPs, no port 443 conflict:
**Management UI (browser):**
```
teddy.hubris.network → Technitium → 192.168.8.175 (Caddy) → 192.168.8.243:8443
```
**Toniebox device traffic:**
```
prod.de.bb-online.com → Technitium override → 192.168.8.243:443 (TeddyCloud direct)
```
Caddy terminates TLS for the management UI (IONOS DNS-01 wildcard cert). TeddyCloud terminates TLS for device traffic with its own self-signed CA — the Toniebox must have this CA installed.
### DNS overrides in Technitium
| Record | Type | Value | Purpose |
|--------|------|-------|---------|
| `teddy.hubris.network` | A | `192.168.8.175` | Management UI → Caddy (standard pattern) |
| `prod.de.bb-online.com` | A | `192.168.8.243` | Toniebox device traffic → TeddyCloud direct |
The `prod.de.bb-online.com` override is Technitium-only — it intercepts Toniebox DNS locally without touching public DNS. The `dns-sync.py` cron on LXC 107 skips non-`hubris.network` records, so it stays local.
## Config notes
- `core.boxCertAuth=false` — client cert validation disabled. The box connects without presenting its unique client cert. Set in `/var/lib/docker/volumes/teddycloud_config/_data/config.ini` (TeddyCloud hot-reloads on change).
- If you ever want per-box auth, flip to `true` and supply `certs/client/ca.der`, `client.der`, `private.der` extracted from the box flash.
## Toniebox onboarding — ESP32 SD card method
Leon's box is ESP32 generation. No hardware mod required.
1. Download the TeddyCloud CA cert from the web UI: **Security → CA Certificate → Download CA** (`ca.der`).
2. Power off the Toniebox, remove the SD card.
3. On the SD card, create folder `cert/` at the root.
4. Copy the downloaded `ca.der` into `cert/ca.der` on the SD card.
5. Reinsert SD card, power on the box.
6. The box patches itself to trust TeddyCloud's CA, then resolves `prod.de.bb-online.com` via Technitium's override (`192.168.8.243`) and connects on port 443.
Reference: [upstream wiki — ESP32 SD card method](https://github.com/toniebox-reverse-engineering/teddycloud/wiki).
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`teddy.hubris.network → 192.168.8.243:8443`)
- [DNS (107)](../infrastructure/dns.md) — Technitium A records for `teddy.hubris.network` and `prod.de.bb-online.com`
- [Media permissions](../infrastructure/media-permissions.md) — NAS `/mnt/library` mount pattern
## Changelog
### 2026-06-29 — provisioned
LXC 131 created (Debian 12, privileged, nesting=1). Docker installed. TeddyCloud running via Docker Compose at `/opt/teddycloud/`. Content bind-mounted from `/mnt/library/cloud/leon`. Caddy block added at `teddy.hubris.network → :8443`. Technitium A records: `teddy.hubris.network → 192.168.8.175` (Caddy), `prod.de.bb-online.com → 192.168.8.243` (device traffic direct).

View File

@@ -7,16 +7,18 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
| 101 | [jellyfin](101-jellyfin.md) | 192.168.8.206 | unpriv (idmap) | 2 | 4 GiB | 16 GiB | `/mnt/library` | `media.hubris.network` | running |
| 103 | [paperless](103-paperless.md) | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
| 104 | [gitea](104-gitea.md) | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
| 105 | [apps](105-apps.md) | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `books` / `artifacto` / `blog` | running |
| 105 | [apps](105-apps.md) | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
| 114 | [nextcloud](114-nextcloud.md) | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
| 118 | [elementsynapse](118-elementsynapse.md) | 192.168.8.239 | unpriv | 1 | 2 GiB | 8 GiB | — | `matrix.hubris.network` | running |
| 119 | [sophia](119-sophia.md) | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
| 120 | [mule-images](120-mule-images.md) | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU passthrough) | `photos.hubris.network` | running |
| 121 | [caddy](121-caddy.md) | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
| 122 | [arriman](122-arriman.md) | 192.168.8.132 | priv | 4 | 8 GiB | 24 GiB | `/mnt/library` | `jellyseerr` / `qbit` / `sab` | running |
| 123 | [claudio-bot](123-claudio-bot.md) | 192.168.8.230 | unpriv | 1 | 512 MiB | 8 GiB | — | — | running |
| 124 | [authentik](124-authentik.md) | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
| 126 | [plato](126-plato.md) | 192.168.8.190 | priv | 2 | 2 GiB | 8 GiB | `/mnt/library/documents/plato` | `plato.hubris.network` | running |
| 128 | [trmnl](128-trmnl.md) | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
| 129 | [house](129-house.md) | 192.168.8.212 | unpriv | 1 | 1344 MiB | 8 GiB | — | `house.hubris.network` | running |
| 130 | [grimmory](130-grimmory.md) | 192.168.8.213 | priv | 1 | 2 GiB | 16 GiB | `/mnt/library` | `books.hubris.network` | running |
| 131 | [teddycloud](131-teddycloud.md) | 192.168.8.243 | priv | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (LAN only) | running |
## Recently destroyed (kept for archaeology)
@@ -26,6 +28,8 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
@@ -41,7 +45,7 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. IPs are stable via the LAN router's DHCP reservations.
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library`. Containers that don't mount `/mnt/library` don't need it.
- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`, `123`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
## Related
- [Hubris host](../hosts/hubris.md)

View File

@@ -38,7 +38,7 @@ services_hosted:
port: 9810
systemd_unit: homelab-mcp
public_host: mcp.hubris.network
endpoint: https://mcp.hubris.network/sse
endpoint: https://mcp.hubris.network/mcp
note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via
Netbird (192.168.8.0/24 is a network resource routed through hubris).
- name: secrets_issuance
@@ -51,5 +51,5 @@ services_hosted:
age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
see_also:
- containers/105-apps.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -30,5 +30,5 @@ services_hosted:
note: jellyseerr / qbit / sab on docker compose
see_also:
- containers/122-arriman.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -1,22 +1,21 @@
# Generated by mcp/build_host_files.py from inventory.yaml.
# Do NOT edit by hand — your changes will be overwritten.
# Source of truth: ../inventory.yaml
name: claudio-bot
name: auth-outpost
kind: lxc
os: linux
role: matrix-agent
role: authentik-gateway
host: hubris
pve_id: 123
lan_ip: 192.168.8.230
pve_id: 106
lan_ip: 192.168.8.6
mesh_globals:
primary: netbird
accepted:
- netbird
- tailscale
notes:
- Reads /opt/homelab-context/ on startup
age_pubkey: age1xmkeq968areza2necqyq0065dpeegngzyr6dhagh0n6pl33lccfqe5mqn9
- Runs Authentik outpost (reverse-proxy/SSO enforcement) for protected services
see_also:
- containers/123-claudio-bot.md
mcp_endpoint: https://mcp.hubris.network/sse
- containers/106-auth-outpost.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -28,5 +28,5 @@ notes:
- /etc/caddy is a git checkout of dtoro/caddy-conf
see_also:
- containers/121-caddy.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -1,32 +1,30 @@
# Generated by mcp/build_host_files.py from inventory.yaml.
# Do NOT edit by hand — your changes will be overwritten.
# Source of truth: ../inventory.yaml
name: authentik
name: dns
kind: lxc
os: linux
role: idp
role: dns-server
host: hubris
pve_id: 124
lan_ip: 192.168.8.180
pve_id: 107
lan_ip: 192.168.8.2
mesh_globals:
primary: netbird
accepted:
- netbird
- tailscale
public_host: auth.hubris.network
runs:
- authentik
- dnsmasq
services_hosted:
- name: authentik
url: https://auth.hubris.network
backend: authentik
- name: dnsmasq
backend: authentik
note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf
backend: dns
dns: null
note: Technitium DNS, split-horizon zone
notes:
- 'Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf'
- Technitium DNS, split-horizon zone for *.hubris.network
- Primary DNS for 192.168.8.0/24 LAN (inventory.services.dns references this)
see_also:
- containers/124-authentik.md
mcp_endpoint: https://mcp.hubris.network/sse
- containers/107-dns.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -25,5 +25,5 @@ services_hosted:
backend: elementsynapse
see_also:
- containers/118-elementsynapse.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -30,5 +30,5 @@ notes:
- Bare repos live at /mnt/library/repos/dtoro/*.git
see_also:
- containers/104-gitea.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -23,5 +23,5 @@ services_hosted:
backend: haos
see_also:
- vms/108-haos.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

24
hosts/house.yaml Normal file
View File

@@ -0,0 +1,24 @@
# Generated by mcp/build_host_files.py from inventory.yaml.
# Do NOT edit by hand — your changes will be overwritten.
# Source of truth: ../inventory.yaml
name: house
kind: lxc
os: linux
role: family-planner
host: hubris
pve_id: 129
lan_ip: 192.168.8.212
mesh_globals:
primary: netbird
accepted:
- netbird
- tailscale
public_host: house.hubris.network
notes:
- Docker host for Yuvomi (family planner). Created 2026-06-26.
- Runs Yuvomi container + WebDAV doc bridge to paperless
age_pubkey: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
see_also:
- containers/129-house.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -6,8 +6,9 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
- **LAN (primary):** `192.168.8.77/24` on bridge `vmbr0` (slave: `eno1`), gateway `192.168.8.1`. Default route metric 0.
- **WiFi (failover):** `192.168.8.141/24` on `wlp3s0` (MediaTek MT7922, AX), DHCP from the same router. Default route metric 200. See [Phase 1 WiFi failover](#phase-1-wifi-failover) below.
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
@@ -36,17 +37,6 @@ See [containers/index](../containers/index.md). 13 active (109 syncthing destroy
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
## Phase 1 WiFi failover
Host is dual-homed on LAN (`eno1`/`vmbr0`) and WiFi (`wlp3s0`) so management/SSH stay reachable when LAN drops. **Guests are not yet failed over** — the LXC fleet remains on `vmbr0`/`eno1`. Phase 2 will migrate guest networking off the bridge so the homelab survives full LAN loss.
- WiFi creds in `/etc/wpa_supplicant/wpa_supplicant-wlp3s0.conf` (hashed PSK, mode 600). SSID lives in `/etc/network/interfaces` as `wpa-conf`.
- Both interfaces sit on the same `192.168.8.0/24`; cross-talk avoided with `arp_ignore=1` + `arp_announce=2` on `eno1`/`vmbr0`/`wlp3s0` (set via `post-up` in `/etc/network/interfaces`).
- A second default route at metric 200 is added on `wlp3s0` (post-up). LAN wins while up.
- **Carrier-based failover:** `vmbr0`'s carrier follows the LXC veth members, so it stays `1` even when `eno1` loses link. `ignore_routes_with_linkdown` is therefore not enough on its own. `wan-failover.service` (`/usr/local/sbin/wan-failover.sh`) watches `/sys/class/net/eno1/carrier` via `ip monitor link` and removes/restores the `vmbr0` default route on transitions. Logs to `journalctl -t wan-failover`.
- Failover verified 2026-04-28: `ip link set eno1 down` → outbound HTTP keeps working via WiFi; `ip link set eno1 up` → vmbr0 default restored.
- Reachable on `192.168.8.77` (LAN) and `192.168.8.141` (WiFi); SSH works on either.
## Network performance baseline (2026-05-14)
| Path | Throughput | Notes |
@@ -99,6 +89,9 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
## Changelog
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).

View File

@@ -31,5 +31,5 @@ services_hosted:
age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
see_also:
- hosts/hubris.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -27,5 +27,5 @@ services_hosted:
backend: jellyfin
see_also:
- containers/101-jellyfin.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -5,6 +5,7 @@ name: ludo-mini
kind: workstation
os: linux
role: dev
lan_ip: 192.168.178.181
mesh:
netbird:
fqdn: ludo-mini.netbird.selfhosted
@@ -13,5 +14,7 @@ mesh_globals:
accepted:
- netbird
- tailscale
mcp_endpoint: https://mcp.hubris.network/sse
ssh:
user: dtoro
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -5,6 +5,7 @@ name: mac-mini
kind: workstation
os: macos
role: dev
lan_ip: 192.168.178.182
mesh:
netbird:
fqdn: mac-mini-234-17.netbird.selfhosted
@@ -17,5 +18,6 @@ ssh:
user: dtoro
notes:
- Only macOS in the fleet. Bootstrap uses launchd.
mcp_endpoint: https://mcp.hubris.network/sse
age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -27,5 +27,5 @@ services_hosted:
backend: mule-images
see_also:
- containers/120-mule-images.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -25,5 +25,5 @@ notes:
- Configs rendered by `homelab render-vps-configs` from vps/turnserver.conf.tmpl + vps/management.json.tmpl,
with secrets decrypted from secrets/turn-shared-secret.yaml + secrets/netbird-authentik-oidc.yaml on
hubris.
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -27,5 +27,5 @@ services_hosted:
backend: nextcloud
see_also:
- containers/114-nextcloud.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -15,5 +15,5 @@ mesh_globals:
- tailscale
see_also:
- containers/102-nfs-export.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -27,5 +27,5 @@ services_hosted:
backend: paperless
see_also:
- containers/103-paperless.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -15,5 +15,5 @@ mesh_globals:
- tailscale
ssh:
user: dtoro
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -7,7 +7,7 @@ os: linux
role: workshop
host: hubris
pve_id: 119
lan_ip: 192.168.8.157
lan_ip: 192.168.8.109
mesh:
tailscale:
fqdn: sophia
@@ -20,5 +20,5 @@ mounts:
- /mnt/library
see_also:
- containers/119-sophia.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -1,28 +1,27 @@
# Generated by mcp/build_host_files.py from inventory.yaml.
# Do NOT edit by hand — your changes will be overwritten.
# Source of truth: ../inventory.yaml
name: plato
name: trmnl
kind: lxc
os: linux
role: app
role: trmnl-middleware
host: hubris
pve_id: 126
lan_ip: 192.168.8.190
pve_id: 128
lan_ip: 192.168.8.211
mesh_globals:
primary: netbird
accepted:
- netbird
- tailscale
mounts:
- /mnt/library/documents/plato
public_host: plato.hubris.network
public_host: trmnl.hubris.network
runs:
- plato
- trmnl
services_hosted:
- name: plato
url: https://plato.hubris.network
backend: plato
- name: trmnl
backend: trmnl
url: https://trmnl.hubris.network
note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud)
see_also:
- containers/126-plato.md
mcp_endpoint: https://mcp.hubris.network/sse
- containers/128-trmnl.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -22,5 +22,5 @@ services_hosted:
backend: zimaos
see_also:
- vms/100-zimaos.md
mcp_endpoint: https://mcp.hubris.network/sse
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -20,10 +20,11 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
- `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
- ~~`192.168.8.190` ([plato (126)](../containers/126-plato.md))~~ (destroyed 2026-06-28)
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
**Don't strip these when editing app.ini.**
@@ -37,17 +38,18 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/126-plato.md) `/opt/plato/app/`~~ (destroyed 2026-06-28) | | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
## When you change a tracked config
@@ -57,8 +59,8 @@ Always commit + push. Local-only edits drift. Common ones:
- `/var/lib/gitea/custom/``dtoro/gitea-customizations` (auto-deploys)
- `/opt/artifacto/``dtoro/Artifacto` (auto-deploys)
- `/opt/mule-image/``dtoro/mule-image` (auto-deploys)
- `/opt/plato/app/``dtoro/Plato` (auto-deploys)
- `/opt/claudio-bot/``dtoro/claudio-bot` (auto-deploys)
- ~~`/opt/plato/app/``dtoro/Plato`~~ (destroyed 2026-06-28)
- ~~`/opt/claudio-bot/``dtoro/claudio-bot`~~ (destroyed 2026-06-04)
- `/opt/backup-library/``dtoro/backup-library` (auto-deploys)
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/``dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
@@ -72,11 +74,6 @@ Always commit + push. Local-only edits drift. Common ones:
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
### Plato
- Shape B (`/opt/plato-deploy/{webhook.py,deploy.sh}`, port `9799`).
- The in-LXC checkout's `origin` is `http://192.168.8.121:3000/dtoro/Plato.git` (internal gitea), and git creds are at `/root/.git-credentials` rather than the `/etc/plato-deploy/git-credentials` pattern — the unit doesn't set `ProtectHome` so root's home is reachable.
- `/data` is a host bind (`/mnt/library/documents/plato`), so `docker compose up -d --build` rebuilds the image + restarts the container without touching the SQLite db. The [fresh-DB bootstrap workaround](../containers/126-plato.md#fresh-db-bootstrap-workaround) only matters if you blow `plato.db` away.
### mule-image / Artifacto
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
@@ -117,12 +114,18 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
## Related
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
## Changelog
### 2026-06-28 — Plato pipeline decommissioned
LXC 126 destroyed, webhook id 8 on `dtoro/Plato` removed. `192.168.8.190` removed from gitea `app.ini` `ALLOWED_HOST_LIST`.
### 2026-06-24 — terminalito pipeline added
Webhook id 12 on `dtoro/terminalito``http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config).
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
@@ -135,6 +138,9 @@ Initial documentation. Six active pipelines.
### 2026-04-22 — Artifacto pipeline added
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
### 2026-06-04 — claudio-bot pipeline decommissioned
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
### 2026-04-21 — mule-image + claudio-bot pipelines added
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.

View File

@@ -71,7 +71,7 @@ Baked into the systemd units:
## Notifications
POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
@@ -95,7 +95,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
## Related
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)

View File

@@ -7,14 +7,15 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-04-28)
## Live entries (as of 2026-06-04)
```
address=/auth.hubris.network/192.168.8.175
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/sso.hubris.network/192.168.8.175 # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
@@ -31,33 +32,37 @@ address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/plato.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/teddy.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200
```
**Non-`hubris.network` override (Toniebox device traffic):**
```
address=/prod.de.bb-online.com/192.168.8.243 # → TeddyCloud (131) direct on :443
```
This intercepts Toniebox DNS locally without touching public DNS. The `dns-sync.py` cron on LXC 107 skips non-`hubris.network` records — this entry is Technitium-only.
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
## Why split-horizon
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
## The gotcha that cost a debug session (2026-04-22)
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
## Recipe — adding a new subdomain
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
```
address=/<new>.hubris.network/192.168.8.175
```
3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`.
5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network``192.168.8.175`.
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
## Public path — what does and doesn't follow the LAN map
@@ -75,10 +80,50 @@ Either:
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
## Changelog
### 2026-06-29 — `teddy.hubris.network` added; `prod.de.bb-online.com` override added
TeddyCloud (LXC 131) provisioned. `teddy.hubris.network → 192.168.8.175` (Caddy → TeddyCloud web UI at :8443). Non-hubris override `prod.de.bb-online.com → 192.168.8.243` routes Toniebox device HTTPS traffic directly to TeddyCloud port 443 — this bypasses Caddy and is Technitium-only (dns-sync cron does not replicate non-hubris.network records to the NetBird managed zone).
### 2026-06-28 — `plato.hubris.network` removed
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17``Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
### 2026-06-02 — 8 LXCs moved from DHCP to static IP
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180``192.168.8.2` for any plain-LAN (non-mesh) clients.
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

View File

@@ -1,7 +1,7 @@
# Homelab context distribution
The cross-client context-and-secrets system that makes every agent (Claude
Code, claudio-bot, future MCP-capable clients) on every machine in the lab
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
self-locating and able to read the same source of truth.
Operational walkthrough for enrolling a new client lives in
@@ -13,10 +13,10 @@ page is the architecture reference.
| Piece | Host | Path | Role |
| --- | --- | --- | --- |
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, claudio-bot |
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab``/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/sse) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` |
| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min |
| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` |
@@ -140,4 +140,4 @@ wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
`client add --finalize-pubkey` grant flow live so new clients are one
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
claudio-bot integration so the matrix agent uses inventory at chat-time.
Hermes Agent integration so the agent uses inventory at chat-time.

View File

@@ -15,7 +15,7 @@ Public client
[Home backend on 192.168.8.x]
```
Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
## Why this shape
@@ -42,9 +42,22 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
| `trmnl.hubris.network` | whole host | `192.168.8.211:9851` ([trmnl 128](../containers/128-trmnl.md)) | `trmnl-ratelimit` (20 rps / 40 burst) | `trmnl.fullchain.crt` / `trmnl.privkey.key` |
| `house.hubris.network` | whole host | `192.168.8.212:3000` ([house 129](../containers/129-house.md)) | `house-ratelimit` (30 rps / 60 burst) | `house.fullchain.crt` / `house.privkey.key` |
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
## Recipe — exposing another service
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
@@ -74,6 +87,12 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
## Changelog
### 2026-06-24 — `trmnl.hubris.network` exposed
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.

View File

@@ -37,7 +37,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / grimmory
- `downloads` — \*arr stack output
- `images` — photoprism / immich / mulita
- `roms` — emu frontends
@@ -61,6 +61,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
@@ -68,7 +69,7 @@ Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*` (109 destroyed 20
## Gotchas
- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container.
- **[apps (105)](../containers/105-apps.md) and [grimmory (130)](../containers/130-grimmory.md) are Docker hosts.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000`, `user: "<uid>:10000"`, or `GROUP_ID=10000` (linuxserver images) in compose. Grimmory, audiobookshelf-in-docker, etc. need this per-container.
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.

View File

@@ -116,6 +116,9 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik``192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.

View File

@@ -1,75 +1,60 @@
# Monitoring — `claudio-monitor`
# Monitoring — Hermes health watchdog
Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
## Why
## Current approach
After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
Two layers:
## Components
1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
service reachability, and apt/docker drift across all hosts.
- **Repo:** `dtoro/claudio-monitor`
- **Checkout:** `/opt/claudio-monitor` on the PVE host
- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
cron. Silent when healthy. When thresholds breach, sends an actionable alert
to Matrix (`@dtoro:avispero`) with options the user can reply to directly
(e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
the selected option via SSH.
## Per-LXC data
Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
apt >10/>50 upgradable, services down.
A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
Home Assistant pulls PVE metrics independently via its Proxmox VE integration
(unaffected by this change).
## Matrix-side plugin
## Legacy: claudio-monitor (deprecated 2026-06-04)
`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
- `/monitor status`
- `/monitor history [N]`
- `/monitor clear <key>`
The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
on LXC 123:9090. All components decommissioned:
Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
| Component | Fate |
|-----------|------|
| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
| `claudio-monitor.timer` | Disabled on hubris |
| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
Events posted to claudio-bot `/notify`:
```json
{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
```
`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
## Current thresholds & active alerts
Active warnings at the time monitor was deployed (2026-04-21):
- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
## Testing
- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
## Not yet deployed (tracked in plan file)
- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
## Related repos
- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
## Related pages
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- [HAOS VM (108)](../vms/108-haos.md)
- [Backups (disabled)](backups.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [Homelab context distribution](homelab-context.md)
## Changelog
### 2026-06-04 — migrated to Hermes health watchdog
claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
Matrix alerts. LXC 123 destroyed, repos archived.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
General health monitor with per-LXC checks. MQTT/REST push paths ripped out.

88
infrastructure/network.md Normal file
View File

@@ -0,0 +1,88 @@
# Network
Physical and logical network topology for the homelab.
## Why
The homelab runs on a dedicated internal subnet (`192.168.8.0/24`) isolated from the main household LAN (`192.168.178.0/24`). Isolation is enforced at Proxmox: LXC/VM traffic is bridged only on the internal `vmbr0` bridge; Proxmox routes packets out to Fritz!Box via `vmbr1`. The main LAN cannot reach homelab services directly without a Fritz!Box static route (which is configured to allow inbound).
Fritz!OS 8.x does not support second IP networks on LAN ports, so Proxmox (`hubris`) acts as the subnet router rather than the Fritz!Box.
## Hardware
| Device | Role |
|---|---|
| Fritz!Box 7590 | Main router / ISP gateway (`192.168.178.1`) |
| SODOLA 5-Port 2.5Gbit Managed | Homelab switch — flat L2, all ports native |
| hubris (Proxmox) | Subnet router — routes between `192.168.8.0/24` and `192.168.178.0/24` |
## Topology
```
ISP
└── Fritz!Box 7590 (192.168.178.1)
│ static route: 192.168.8.0/24 → 192.168.178.10
└── SODOLA 5-Port 2.5Gbit
├── Port 1 uplink → Fritz!Box LAN
├── Port 2 hubris eno1 → vmbr1 (192.168.178.10)
├── Port 3 [device]
├── Port 4 [device]
└── Port 5 spare
hubris internal bridges:
vmbr1 192.168.178.10/24 eno1 (uplink, DHCP-reserved) gateway 192.168.178.1
vmbr0 192.168.8.77/24 no physical port (internal)
192.168.8.1/24 alias — LXC default gateway
├── all 16 LXCs
└── HAOS VM
```
## Subnets
| Subnet | Gateway | Purpose |
|---|---|---|
| `192.168.178.0/24` | `192.168.178.1` | Household LAN — laptops, phones, Fritz!Box DHCP |
| `192.168.8.0/24` | `192.168.8.1` (Proxmox `vmbr0` alias) | Homelab — all LXCs and VMs |
## DHCP
- **Household (`192.168.178.x`)**: Fritz!Box built-in DHCP. Proxmox `vmbr1` has a reservation: MAC `84:47:09:6b:e7:58``192.168.178.10`.
- **Homelab (`192.168.8.x`)**: Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2`. Range `192.168.8.241192.168.8.254`, gateway `192.168.8.1`, DNS `192.168.8.2`.
Static IPs span `.101.239` (all LXCs, VMs, and workstations). DHCP pool narrowed to `.241.254` (2026-06-03) to avoid overlap and IP conflicts.
## DNS
Split-horizon DNS for `*.hubris.network` served by Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2:53`. See [dns.md](dns.md) for full detail.
## Routing
Proxmox has `net.ipv4.ip_forward=1` (already enabled by PVE). Packets from LXCs on `vmbr0` destined for the internet exit via `vmbr1` → Fritz!Box. Fritz!Box masquerades all outbound WAN traffic. Fritz!Box has a static route (`192.168.8.0/24 → 192.168.178.10`) so return traffic reaches the LXCs.
No NAT on Proxmox — traffic flows without double-NAT.
## Remote access
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/).
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
## Related
- [DNS](dns.md) — split-horizon config and entry list
- [Ingress](ingress.md) — public entry points via VPS traefik
- [Mesh](mesh.md) — NetBird / Tailscale VPN overlay
- [hosts/hubris.md](../hosts/hubris.md) — Proxmox host (vmbr0/vmbr1 config)
- [CT 107 — dns](../containers/107-dns.md) — Technitium DNS + DHCP server
## Changelog
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
Previous pool `.100.240` overlapped with all static LXCs/VMs (` .101.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
### 2026-06-02 — Executed migration; Proxmox as subnet router
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).

View File

@@ -1,45 +1,198 @@
# SSH access
How to reach hubris and the VPS over SSH, and the dual-server gotcha.
How to reach every host in the fleet from any workstation, with LAN as
the primary path and Netbird as the automatic backup.
## Hubris
## Architecture
Two SSH endpoints — easy to hit the wrong one.
SSH access relies on three layers:
| Server | Listen | Auth | Notes |
| -------------- | ---------------------------- | --------------------------------- | ----- |
| OpenSSH | `0.0.0.0:22` | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. |
| Netbird SSH | `100.122.38.109:22022` | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. |
1. **Homelab inventory (`inventory.yaml`)** — the single source of truth
for every host's LAN IP, Netbird addresses, SSH user, and port.
2. **Key distribution (`ssh/deploy-keys.sh`)** — deploys workstation SSH
public keys to hubris and every running LXC, so any key-authorized
workstation can log in anywhere.
3. **Config generation (`homelab ssh-config --install`)** — generates
`~/.ssh/config.d/homelab` with short hostname aliases for every host,
using LAN IPs (routed via Netbird's `192.168.8.0/24` subnet route when
off-LAN) with Netbird FQDN fallbacks (`<name>-mesh`) for roaming
workstations.
### Authorized root keys
- `root@hubris` (self, RSA) — original.
- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22.
### How it works
### Notes
- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind.
- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical.
- **From on-LAN:** `ssh gitea` resolves to `192.168.8.121` directly.
- **From off-LAN (Netbird):** The same `192.168.8.121` works because
hubris routes the `192.168.8.0/24` subnet through Netbird.
- **Roaming workstations:** `ssh mac-mini-mesh` or `ssh republic-laptop-mesh`
uses the Netbird FQDN as a fallback when the workstation is off its
home subnet.
## VPS (`82.165.190.79` / `100.122.165.149`)
The `homelab ssh <host>` CLI command also has built-in LAN probing:
it tries a 1.5s TCP connect to the LAN IP, and if that fails, falls
back to the Netbird FQDN.
- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`.
- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
- `ListenAddress` is still 0.0.0.0; gating is firewall-layer.
- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths.
## Key distribution
See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths.
Each workstation's SSH public key lives in the repo at:
`ssh/authorized_keys/<hostname>.pub`
To deploy or re-deploy all workstation keys to hubris + every running LXC:
```bash
# From hubris (or via homelab pct):
sudo bash /opt/homelab-context/ssh/deploy-keys.sh
# Or from any workstation:
ssh root@192.168.8.77 "bash /opt/homelab-context/ssh/deploy-keys.sh"
```
This script:
- Reads all `.pub` files from `ssh/authorized_keys/`
- Adds any missing keys to `/etc/pve/priv/authorized_keys` on hubris
- For each running LXC, appends keys to `/root/.ssh/authorized_keys`
- Is idempotent — skips keys already present
## Config generation
To generate the SSH config on any workstation:
```bash
homelab ssh-config --install
```
This writes to `~/.ssh/config.d/homelab` and ensures
`Include ~/.ssh/config.d/homelab` is present in `~/.ssh/config`.
The config is regenerated automatically on every `homelab sync` (which
kicks the 5-minute context sync timer).
## Adding a new workstation
When onboarding a new machine:
1. Hostname must match an entry in `inventory.yaml`.
2. If the workstation will be on the LAN, add its `lan_ip` to
`inventory.yaml` and push. This gives it a primary LAN entry in the
generated SSH config.
3. Enable SSH Remote Login:
- **macOS:** `sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist`
- **Linux:** `sudo systemctl enable --now sshd`
4. Generate an SSH keypair if one doesn't exist:
```bash
ssh-keygen -t ed25519 -a 100
```
5. Publish the public key to the repo:
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/<hostname>.pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add <hostname> pubkey' && git push
```
6. Deploy the key to all hosts:
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
7. Generate the local SSH config:
```bash
homelab ssh-config --install
```
## Hosts
### Hubris (PVE host)
| Detail | Value |
|--------|-------|
| LAN IP | `192.168.8.77` |
| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
| SSH user | `root` |
| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
Authorized root keys currently deployed:
- `root@hubris` (self, RSA)
- `d.toro.v@pm.me` (ed25519) — mac-mini
### LXCs
Every LXC at `192.168.8.x` accepts root SSH via authorized_keys. Keys
are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
| LXC | Name | LAN IP | Role |
|-----|------|--------|------|
| 101 | jellyfin | `192.168.8.206` | media-server |
| 102 | nfs-export | `192.168.8.200` | storage-export |
| 103 | paperless | `192.168.8.130` | document-archive |
| 104 | gitea | `192.168.8.121` | git-server |
| 105 | apps | `192.168.8.205` | docker-apps |
| 106 | auth-outpost | `192.168.8.184` | authentik-outpost |
| 107 | dns | `192.168.8.185` | dns-helper |
| 114 | nextcloud | `192.168.8.224` | file-sync |
| 118 | elementsynapse | `192.168.8.239` | matrix-server |
| 119 | sophia | `192.168.8.157` | workshop |
| 120 | mule-images | `192.168.8.136` | photo-management |
| 121 | caddy | `192.168.8.175` | reverse-proxy |
| 122 | arriman | `192.168.8.132` | arr-stack |
### Workstations
| Name | OS | LAN IP | Netbird FQDN | SSH user |
|------|----|--------|--------------|----------|
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
### VPS (external)
| Detail | Value |
|--------|-------|
| Public IP | `82.165.190.79` |
| Netbird | `100.122.165.149` (FQDN: `netbird-ionos.netbird.selfhosted`) |
| SSH user | `root` |
| Access | Mesh-only — public port 22 is blocked by nftables. Key-only auth. |
## VPS
Access is mesh-only. From a mesh-connected peer:
```bash
ssh root@100.122.165.149
ssh root@netbird-ionos.netbird.selfhosted
# or via homelab:
homelab ssh netbird-vps
```
## Verification
```bash
# From any workstation after running homelab ssh-config --install:
for name in hubris gitea apps sophia paperless caddy jellyfin nextcloud; do
ssh -o BatchMode=yes "$name" "hostname" && echo "$name OK"
done
```
## Related
- [Hubris host](../hosts/hubris.md)
- [Mesh migration](mesh.md)
- [VPS hardening](vps-hardening.md)
- [Agent enrollment](../operations/agent-enrollment.md)
- [Homelab CLI](../bin/homelab)
## Changelog
### 2026-06-02 — universal SSH reachability
Replaced ad-hoc per-workstation SSH configs with inventory-generated
configs (`ssh/gen-config.py`, `homelab ssh-config`). Added centralized
key distribution (`ssh/deploy-keys.sh`, `ssh/authorized_keys/`). All
LXCs now accept root SSH from any workstation whose pubkey is in the
repo. mac-mini Remote Login enabled. Netbird subnet route
(192.168.8.0/24 via hubris) provides off-LAN reachability for all LAN
IPs.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — VPS SSH hardened to mesh-only
Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md).
Public `:22` blocked at nftables. Key-only sshd.
### 2026-04-22 — iMac key authorized on hubris
`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.
`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.

View File

@@ -51,9 +51,9 @@ services:
authentik:
url: https://auth.hubris.network
backend: authentik
dnsmasq:
backend: authentik
note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf
dns:
backend: dns
note: Technitium DNS, split-horizon zone
jellyfin:
url: https://media.hubris.network
backend: jellyfin
@@ -69,15 +69,16 @@ services:
photos:
url: https://photos.hubris.network
backend: mule-images
plato:
url: https://plato.hubris.network
backend: plato
arr_stack:
backend: arriman
note: jellyseerr / qbit / sab on docker compose
artifacto:
backend: apps
url: https://artifacto.hubris.network
trmnl:
backend: trmnl
url: https://trmnl.hubris.network
note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud)
zimaos:
url: https://zimaos.hubris.network
backend: zimaos
@@ -88,7 +89,7 @@ services:
port: 9810
systemd_unit: homelab-mcp
public_host: mcp.hubris.network
endpoint: https://mcp.hubris.network/sse
endpoint: https://mcp.hubris.network/mcp
note: MCP server. Read-only context + management. Reachable on the LAN via Caddy
and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through
hubris).
@@ -117,6 +118,27 @@ hosts:
mounts:
- /mnt/library
age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
trmnl:
kind: lxc
pve_id: 128
host: hubris
os: linux
role: trmnl-middleware
lan_ip: 192.168.8.211
public_host: trmnl.hubris.network
# not yet mesh/SOPS-enrolled — see containers/128-trmnl.md
house:
kind: lxc
pve_id: 129
host: hubris
os: linux
role: family-planner
lan_ip: 192.168.8.212
public_host: house.hubris.network
notes:
- Docker host for Yuvomi (family planner). Created 2026-06-26.
- Runs Yuvomi container + WebDAV doc bridge to paperless
age_pubkey: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
jellyfin:
kind: lxc
pve_id: 101
@@ -189,7 +211,27 @@ hosts:
- plantuml
- homelab-mcp
- secrets-issuance
# booklore removed 2026-06-29 → migrated to grimmory (LXC 130)
age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
auth-outpost:
kind: lxc
pve_id: 106
host: hubris
os: linux
role: authentik-gateway
lan_ip: 192.168.8.6
notes:
- Runs Authentik outpost (reverse-proxy/SSO enforcement) for protected services
dns:
kind: lxc
pve_id: 107
host: hubris
os: linux
role: dns-server
lan_ip: 192.168.8.2
notes:
- Technitium DNS, split-horizon zone for *.hubris.network
- Primary DNS for 192.168.8.0/24 LAN (inventory.services.dns references this)
nextcloud:
kind: lxc
pve_id: 114
@@ -221,7 +263,7 @@ hosts:
host: hubris
os: linux
role: workshop
lan_ip: 192.168.8.157
lan_ip: 192.168.8.109
mesh:
tailscale:
fqdn: sophia
@@ -272,37 +314,22 @@ hosts:
mounts:
- /mnt/library
age_pubkey: ''
claudio-bot:
grimmory:
kind: lxc
pve_id: 123
pve_id: 130
host: hubris
os: linux
role: matrix-agent
lan_ip: 192.168.8.230
notes:
- Reads /opt/homelab-context/ on startup
age_pubkey: age1xmkeq968areza2necqyq0065dpeegngzyr6dhagh0n6pl33lccfqe5mqn9
authentik:
kind: lxc
pve_id: 124
host: hubris
os: linux
role: idp
lan_ip: 192.168.8.180
public_host: auth.hubris.network
notes:
- 'Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf'
plato:
kind: lxc
pve_id: 126
host: hubris
os: linux
role: app
lan_ip: 192.168.8.190
public_host: plato.hubris.network
role: book-library
lan_ip: 192.168.8.213
public_host: books.hubris.network
mounts:
- /mnt/library/documents/plato
age_pubkey: ''
- /mnt/library
notes:
- Docker host for Grimmory (community fork of Booklore). Created 2026-06-29.
- Migrated from apps LXC 105; MariaDB data carried over (schema-compatible fork).
age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet
# 123 (claudio-bot) — destroyed 2026-06-04, replaced by Hermes Agent
# 126 (plato) — destroyed 2026-06-28, notes workspace decommissioned
zimaos:
kind: vm
pve_id: 100
@@ -334,6 +361,7 @@ hosts:
kind: workstation
os: macos
role: dev
lan_ip: 192.168.178.182
mesh:
netbird:
fqdn: mac-mini-234-17.netbird.selfhosted
@@ -341,17 +369,17 @@ hosts:
user: dtoro
notes:
- Only macOS in the fleet. Bootstrap uses launchd.
age_pubkey: ''
age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
ludo-mini:
kind: workstation
os: linux
role: dev
lan_ip: 192.168.178.181
mesh:
netbird:
fqdn: ludo-mini.netbird.selfhosted
# ssh.user defaults to root; uncomment + set to the actual login user
# before relying on `homelab ssh ludo-mini` or netbird-ssh INTO this host.
# ssh: { user: ludo }
ssh:
user: dtoro
netbird-vps:
kind: external
os: linux

View File

@@ -0,0 +1,109 @@
# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
## Summary
The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
## Timeline
### Trigger
`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
```
Error: failed reading provided config file: /etc/netbird/management.json:
OIDC configuration request returned status 504 with response: Gateway Timeout
```
The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
### Fix — Phase A (restore the mesh)
1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
2. Removed the `auth-authentik``192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
### Fix — Phase B (full data migration)
Once the mesh was back, the homelab DB was reachable:
1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
4. Restarted Authentik. All users / apps / passwords / groups present.
Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
### Fix — DNS cutover (the long tail)
The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
## Issues hit during the migration (and fixes)
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
## Architecture delta
| | Before | After |
|---|---|---|
| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Version | `2026.2.2` | `2026.5.2` |
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
## Lessons learned
1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
## Forward-auth outpost — RESOLVED 2026-06-01
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
**Fix:** Deleted from Postgres:
```sql
DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';
```
The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
## Outstanding
- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
## Related
- [124 — authentik](../containers/124-authentik.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
- [Mesh migration](../infrastructure/mesh.md)

View File

@@ -0,0 +1,191 @@
# mac-mini onboarding — post-mortem & lessons learned
Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
homelab context system with the `--with-hermes` profile. What follows is a
chronological recap of every hitch, the fix, and the systemic improvements
needed so the next workstation takes 5 min instead of an hour.
## Session log
### Step 1 — clone + symlink
- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
- Created `/opt/homelab-context``/Users/dtoro/Homelab-Docs` symlink.
- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
already cloned by hand. The bootstrap's `clone exists; pulling` path handled
it gracefully.
### Step 2 — hostname mismatch
- `scutil --get LocalHostName``Davids-Mac-mini`
- `hostname -s``Mac`
- Inventory file: `hosts/mac-mini.yaml`
- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
- **Lesson:** The bootstrap and `homelab whoami` use different hostname
resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
but the `homelab` CLI binary uses `hostname -s`. Both need to match the
inventory key. On a fresh macOS machine, neither does.
### Step 3 — bootstrap dependencies
- pyyaml was missing → `pip install pyyaml`
- age and sops were missing → `brew install age sops`
- Netbird was already installed and connected ✓
- **Lesson:** The bootstrap preflight handles these, but only if running
`bootstrap.sh` from the start. Since we ran it after manual setup, some
steps (netbird install) were correctly skipped as already-present.
### Step 4 — full bootstrap with `--with-mcp --with-hermes`
- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
- Age key issued ✓
- Launchd sync timer installed ✓
- Goose binary installed ✓
- Hermes CLI linked ✓
- MCP config merged ✓
- `refresh-creds` skipped (not yet a recipient) ⚠️
- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
group-naming quirk)
### Step 5 — finalize from hubris
- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
hubris
- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
stale, bootstrap had already pushed from mac-mini
- **Fix:** `git pull --rebase && git push` on hubris
- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
creating a race. The `homelab client add --finalize-pubkey` command should
pull before committing/pushing.
### Step 6 — sops couldn't find the age key
- `homelab secret hello` failed because sops looks in
`/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
but this requires passwordless sudo and the correct env var passthrough
- **Fix:**
1. Added NOPASSWD sudo rules
2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
agent-enrollment troubleshooting. New clients can't assume `homelab secret`
works out of the gate — the sudo re-exec chain has permission pitfall.
### Step 7 — OpenRouter key was a placeholder
- `secrets/openrouter-api-key.yaml` contained
`api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
- User ran `sops` on hubris, but got the same error (age key not found on
hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
Later: the user pasted the real key, but the sops file showed
`sk-or-...5c55` — the literal content was truncated with ellipsis.
### Step 8 — editor loaded the wrong data
- Neovim on the system is configured with `clipboard+=unnamedplus`, which
points `*` and `+` registers to the macOS clipboard manager rather than
X11. When editing SOPS files, this caused the **system clipboard** to be
pasted instead of the actual ciphertext.
- This wasn't diagnosed during the session — the sops file would load empty
or show the wrong content because the editor's idea of "paste" was
disconnected from what sops expected.
- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
hijack OS clipboards:
```bash
EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
```
- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
because the paste register reads from the system clipboard instead of the
sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
interactively. Alternatively, override the clipboard option with
`EDITOR='nvim -c "set clipboard="'`.
- Also useful for the troubleshooting table in `agent-enrollment.md` under a
new row: "sops file loads empty / wrong content on macOS"
### Step 9 — model doesn't support tool use
- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
- Error: `No endpoints found that support tool use`
- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
`~/.config/goose/config.yaml`
- Also updated `operations/hermes-agent.md` with the correct model
- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
never validated on OpenRouter for tool-use capability. Need to either:
(a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
(b) Or have the bootstrap probe OpenRouter at install time
### Step 10 — MCP homelab extension returns 404
- Goose's homelab extension configured as `streamable_http` at
`https://mcp.hubris.network/mcp` returns HTTP 404
- The actual MCP server runs on SSE (`/sse`), not streamable_http
- **Known follow-up** from `hermes-agent.md` — server needs migration
- **Workaround:** The agent works fine via the built-in `developer` extension
(shell + file) and the `homelab` CLI
### Step 11 — git credential dance
Several layers of issues with git authentication:
1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
couldn't access the credential file, and git's `store` helper couldn't
create its `.lock` file.
- **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
- **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`
2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
as a default credential helper. Even after configuring the `store` helper,
git calls osxkeychain after every successful `get`, which prompts for
macOS keychain access.
- `credential.helper = ""` does NOT disable the compiled-in default
- The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
silently, causing fallback behaviour
- **Fix:** Embed the credential in the remote URL directly:
`https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
- This bypasses all credential helpers entirely
3. **refresh-creds:** The credential is now managed by the remote URL.
If the PAT is rotated, `homelab refresh-creds` won't update the URL.
- **Workaround:** Run `git remote set-url origin` after `refresh-creds`
### Step 12 — `homelab` CLI hostname detection
- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
`hostname -s` (returning `Mac`) while the bootstrap uses
`scutil --get LocalHostName` (which we fixed to `mac-mini`)
- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
LocalHostName` like the bootstrap does, or at least try both and pick the
one that matches an inventory key. This is a CLI bug.
## Improvement backlog
### High priority
| # | Issue | Fix |
|---|-------|-----|
| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
### Medium priority
| # | Issue | Fix |
|---|-------|-----|
| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
### Low priority
| # | Issue | Fix |
|---|-------|-----|
| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
## Changelog
### 2026-06-01 — initial post-mortem
Captured the full onboarding of mac-mini into the hubris homelab context
system, including the hostname mismatch, sudo/sops credential chain, model
selection, git credential gymnastics, and editor clipboard interference.
### 2026-06-02 — MCP server migrated to streamable-http
Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
re-deployed on apps LXC (105).

View File

@@ -0,0 +1,84 @@
# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
## Summary
[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
## Timeline
### Pre-existing state
- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
- Both on the same subnet via a consumer router at 192.168.178.1
- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
### 2026-06-03 — investigation
**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
- TCP throughput: 4986 Mbps, average ~65 Mbps — highly variable
- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
**Additional findings:**
- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
## Root cause
mac-mini is on WiFi. WiFi introduces:
1. **Throughput variance** — 4986 Mbps TCP fluctuation
2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
3. **Jitter** — 28 ms stddev with spikes to 138 ms
Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
## Mitigations applied
### mac-mini — Moonlight (via `defaults write`)
| Setting | Before | After | Rationale |
||---------|--------|-------|-----------|
|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
|| fps | 30 | 60 | Restored to game-appropriate value |
### mac-mini — Moonlight (current, wired — 2026-06-04)
| Setting | WiFi value | Wired value | Rationale |
||---------|-----------|-------------|-----------|
|| bitrate | 60 Mbps | 80 Mbps | Wired 1 GbE can sustain comfortably |
|| framepacing | on (1) | off (0) | Wired latency is consistent, no smoothing needed |
|| fps | 60 | 60 | Unchanged |
### ludo-mini — Sunshine (via SSH)
| Setting | WiFi value | Wired value | Rationale |
||---------|-----------|-------------|-----------|
|| bitrate | 40 Mbps | 80 Mbps | Wired can handle full bandwidth |
|| fec_percentage | 2 | 5 | Restored to default — wired has no dropout concern |
|| packetsize | 1024 | 1316 | Restored to default (Ethernet MTU) |
Sunshine restarted after changes; confirmed active.
## Resolution
### 2026-06-04 — mac-mini wired to LAN
mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
## Open questions
~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~ **RESOLVED** — mac-mini now wired via en0.
2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
3. **Netbird route**`192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
## Changelog
### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
- Moonlight: bitrate 60→80 Mbps, framepacing disabled
- Sunshine: bitrate 40→80 Mbps, fec_percentage 2→5, packetsize restored to default (1316)
- First root cause in investigation (Open questions #1) resolved

View File

@@ -0,0 +1,101 @@
# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
## Summary
User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
## Timeline
- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)``_session_expire_at_browser_close=True`
- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage``session_duration='days=30'`
- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
- **~23:45** — Verified both changes took effect
## Root cause
Two independent but compounding issues:
1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
### Why forward-auth worked but OAuth2 didn't
- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
## Changes applied
### 1. Database — `authentik_stages_user_login_userloginstage`
```sql
UPDATE authentik_stages_user_login_userloginstage
SET session_duration = 'days=30'
WHERE stage_ptr_id = (
SELECT stage_uuid FROM authentik_flows_stage
WHERE name = 'default-authentication-login'
);
```
This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
- Sets `_session_expire_at_browser_close=False` for that session
- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
- Sets the server-side session expiry to 30 days
### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
Added to `/opt/authentik.env`:
```
AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
```
This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
```python
SESSION_COOKIE_AGE = timedelta_from_string(
CONFIG.get("sessions.unauthenticated_age", "days=1")
).total_seconds()
```
Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
## Verification
```python
# Inside authentik-server container:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
import django
django.setup()
from django.conf import settings
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE) # → 2592000.0
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE) # → True
from authentik.stages.user_login.models import UserLoginStage
stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
print("session_duration:", stage.session_duration) # → "days=30"
```
## Open questions
- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
## Related
- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
## Changelog
### 2026-06-06 — created
Documented the session lifetime investigation, root cause, and applied fixes.

View File

@@ -0,0 +1,61 @@
# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
**Date:** 2026-06-06
**Status:** resolved
**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
## Symptom
All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
## Root cause
The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
### Timeline
| Time (UTC+2) | Event |
|---|---|
| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
| Jun 06 22:34 | Investigation began |
| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
### Evidence
- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
- `git stash list` shows one auto-stash entry
- `origin/master` at `1b977aa`: 260 lines, all site blocks present
### Secondary root cause found during investigation
**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
## Fix applied
1. **Caddyfile**`git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
## Permanent safeguards (all deployed)
| Safeguard | Location | What it does |
|---|---|---|
| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
## Related
- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
- [Caddy (121)](../containers/121-caddy.md)
- [elementsynapse (118)](../containers/118-elementsynapse.md)
- [dns-sync script](../scripts/dns-sync.py)
- [check-caddy-backends script](../scripts/check-caddy-backends.sh)

View File

@@ -6,6 +6,9 @@ Time-stamped incident reports and experiments. One entry per incident; the entry
| Date | Title | Status |
| ------------ | ------------------------------------------------------------------ | ------------- |
|| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
|| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |
## Conventions

View File

@@ -333,4 +333,4 @@ def ping_service(service: str) -> dict:
if __name__ == "__main__":
mcp.run(transport="sse")
mcp.run(transport="streamable-http")

View File

@@ -5,6 +5,10 @@ joins the cross-client context system: a `/opt/homelab-context/` clone of
this repo that auto-syncs every 5 min, a per-client age key for SOPS
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh`
> with an additional `--with-hermes` flag.
Architecture in [project_homelab_context_plan](https://… memory link); the
operational reference is here.
@@ -16,7 +20,7 @@ operational reference is here.
| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/sse` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
### Hostname mismatch is the most common bootstrap failure
@@ -79,7 +83,7 @@ If DNS isn't an option at all, override the URLs at bootstrap time:
sudo HOMELAB_GITEA_TOKEN=... \
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
HOMELAB_MCP_URL=http://192.168.8.205:9810/sse \
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
bash /tmp/bootstrap.sh --with-mcp
```
@@ -159,6 +163,77 @@ For Claude Code: start a new session — the `homelab` MCP server appears
in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
1 secrets-metadata).
## Post-bootstrap: SSH reachability
A new workstation must be reachable from other workstations and must be
able to reach every host by short hostname. Run these steps after the
bootstrap verify passes:
### 1. Enable SSH server
```bash
# macOS:
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
# Linux:
sudo systemctl enable --now sshd
```
### 2. Generate SSH key (if missing)
```bash
ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
```
### 3. Publish pubkey to the repo
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
```
### 4. Deploy keys to all hosts
From any existing enrolled machine (hubris or another workstation):
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
This adds the new workstation's pubkey to hubris and every running LXC.
### 5. Generate SSH config
```bash
homelab ssh-config --install
```
Verify:
```bash
ssh hubris hostname # should return "hubris" without password
ssh gitea hostname # should return "gitea" without password
ssh mac-mini hostname # should return "mac-mini" without password (workstation-to-workstation)
```
### 6. Add LAN IP to inventory (if on LAN)
If the workstation has a static or reserved LAN IP, add it to
`inventory.yaml`:
```yaml
hosts:
your-hostname:
lan_ip: 192.168.8.xxx
```
This gives it a primary LAN entry in the generated SSH config (faster
than the Netbird fallback). Commit + push, then:
```bash
cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
```
## Claude Code permissions for fleet ops
By default Claude Code's auto-mode classifier asks for confirmation on every
@@ -274,6 +349,15 @@ The CLI prints a follow-up checklist that the operator must do manually:
## Changelog
### 2026-06-02 — SSH reachability post-bootstrap steps
Added a new "Post-bootstrap: SSH reachability" section covering SSH key
generation, pubkey publication, deployment to hosts, SSH config generation,
and LAN IP registration. New workstations enrolled via this doc will
automatically join the universal SSH mesh.
### 2026-05-31 — cross-link to hermes-agent.md
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.

View File

@@ -26,9 +26,11 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root
## Reverse proxy
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). Tracked in `dtoro/caddy-conf`. Edits should be committed + pushed — see [auto-deploy](../infrastructure/auto-deploy.md).
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md).
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
## DNS

210
operations/hermes-agent.md Normal file
View File

@@ -0,0 +1,210 @@
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
Llama variant) as a working terminal agent on a homelab client. Builds on top
of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md))
— this page covers only the Hermes-specific additions.
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
- The chat loop, multi-turn history, and streaming
- The OpenRouter provider that routes to Nous Hermes
- The built-in `developer` extension (shell + file editor — same surface Claude
Code has)
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
`.goosehints` so it's injected into the system prompt on every session.
## Prerequisites
| Requirement | How |
| --- | --- |
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) |
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
## Onboarding flow
```bash
# 1. On hubris (or any enrolled client): reserve the inventory entry.
homelab client add new-machine
# 2. Join new-machine to Netbird (setup-key or OIDC).
# 3. On new-machine: bootstrap with --with-hermes.
TOKEN=... # gitea PAT, read:repository
curl -fsSL -u "dtoro:$TOKEN" \
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
-o /tmp/bootstrap.sh
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
homelab client add new-machine \
--finalize-pubkey age1... \
--with-hermes
# 5. Wait ≤5 min for sync, then on new-machine:
hermes "what LXCs are running?"
```
The bootstrap `--with-hermes` flag does five things, all idempotent:
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
2. Symlinks `/opt/homelab-context/bin/hermes``/usr/local/bin/hermes`.
3. Symlinks `/opt/homelab-context/HERMES.md``/root/HERMES.md` (Linux) or
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
extensions (preserves any keys the operator added by hand).
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
injected as the system prompt on every session.
## Seeding the OpenRouter key
The first time anyone enrolls with `--with-hermes`, the encrypted file
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
existing recipient):
```bash
sops secrets/openrouter-api-key.yaml
# editor opens; replace api_key value with the real sk-or-... key, save, close.
git -C /opt/homelab-context add secrets/openrouter-api-key.yaml
git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
git -C /opt/homelab-context push
```
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
contains the placeholder`. Subsequent enrollees get the real key automatically
via `--with-hermes` (which adds them as a sops recipient on
`secrets/openrouter-api-key.yaml`).
## Granting the OpenRouter key to an already-enrolled host
If a host was enrolled without `--with-hermes` and you want to add it later:
```bash
# On hubris:
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
```
`--finalize-pubkey` is required by the existing flow even when the pubkey is
unchanged — it's also the trigger that runs the sops grant.
After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to
re-run; only the secret recipient list changed.
## Verifying
```bash
homelab whoami # standard enrollment OK
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
which goose && which hermes # binaries present
goose info -v # provider/model wiring sane
hermes "what LXCs are running?" # interactive Goose session
# Non-interactive smoke test:
echo "List the homelab MCP tools you have available" | hermes
```
## Configuration
The bootstrap-managed keys in `~/.config/goose/config.yaml`:
```yaml
GOOSE_PROVIDER: openrouter
GOOSE_MODEL: deepseek/deepseek-v4-flash
GOOSE_MODE: smart_approve # asks before destructive tool calls
extensions:
developer:
type: builtin
bundled: true
enabled: true
name: developer
timeout: 300
homelab:
type: streamable_http
enabled: true
name: homelab
uri: https://mcp.hubris.network/mcp
timeout: 60
```
Override via env on a single bootstrap run:
```bash
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
sudo bash /tmp/bootstrap.sh --with-hermes
```
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
preserved across re-bootstraps — the merge only overwrites the keys it manages.
## Tool permissions
`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only
shell commands without prompting and asks for confirmation before destructive
ones. To make the agent fully unattended (e.g. for scheduled jobs), set
`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on
every tool call, use `approve`. See
[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/).
## Troubleshooting
| Symptom | Cause | Fix |
| --- | --- | --- |
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py``mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
## Cross-references
- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the
Hermes flow assumes is done.
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
session start (via `~/.config/goose/.goosehints`).
- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key
and execs `goose session`.
- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block.
## Follow-ups
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
(`"SSE transport is no longer supported - kept only for config file
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
that's changed, the `homelab` MCP extension in Goose will fail to connect.
The developer extension (shell + edit) covers most ops without it; this is
a polish item, not a blocker.
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
hosts share one key.
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
directly — OpenRouter periodically rotates the underlying weights.
4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU
node. The wrapper, persona, and MCP wiring stay unchanged; only
`GOOSE_PROVIDER`/`GOOSE_MODEL` change.
7. **Caveman auto-setup via post-pull hook.** The sync timer now calls
`tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull.
Currently this auto-installs the Caveman npm package, wrapper scripts, and
compact output templates on all agent hosts (*token efficiency*).
## Changelog
### 2026-06-01 — caveman + post-pull auto-setup
Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh`
after every git pull. First user: `tools/setup-caveman.sh` installed Caveman
templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
Also created `tools/caveman/` with the wrapper script, JS renderer, and
templates — the canonical source for all agent hosts.
Captures the Hermes-on-Goose onboarding flow added in the same commit as
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
extension. MCP streamable_http migration is queued as follow-up #1.

View File

@@ -0,0 +1,202 @@
# Runbook: Budget import from N26 CSV → Yuvomi
Distil a bank-export CSV into Yuvomi's Budget and Subscriptions modules using
the `yuvomi-mcp` tools. Run this whenever a new CSV period needs to be
summarised into targets and fixed costs.
---
## Prerequisites
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client).
- Direct API base: `https://house.hubris.network/api/v1`
---
## API quirks (Yuvomi ≤ 0.77.x)
- **Subscriptions live under `/budget/subscriptions`**, NOT `/subscriptions/`.
A top-level `/subscriptions` route returns 404.
- `GET /budget/subscriptions``{ data: { subscriptions: [...], summary: {...} } }`
- `GET /budget/subscriptions/meta``{ data: { categories: [...], payment_methods: [...] } }`
- `POST /budget/subscriptions` → create a subscription (name, amount, billing_cycle,
cycle_interval, next_payment_date, currency, category_id, payment_method_id required)
- `GET /budget/` (no month) → returns only **non-recurring** base entries.
Use `GET /budget/?month=YYYY-MM` to get all entries (recurring + one-time) for a month.
- `GET /budget/categories` → expense category keys + income category names (German keys
like `"Erwerbseinkommen"`, `"Sozialleistungen"`, `"Geschenke & Transfers"`).
- Budget entries: `amount` positive = income, negative = expense.
- Recurring entries: set `is_recurring: 1` + `recurrence_interval: "monthly"`.
The `date` field sets the start month.
- `recurrence_virtual: 1` smooths non-monthly amounts across all months in the summary
(e.g. 55.08 € quarterly → shows as ~18.36 €/month).
- Custom RRULE strings (`recurrence_rule`) are **not accepted** by the API — use
`cycle_interval` on the subscription instead, or `recurrence_interval` on budget entries.
---
## Subscription category IDs (as of 2026-06-26)
| id | name | budget_subcategory_key |
|---|---|---|
| 1 | Entertainment | subscription_entertainment |
| 2 | Productivity | subscription_productivity |
| 3 | Utilities | subscription_utilities |
| 4 | Health | subscription_health |
| 5 | Education | subscription_education |
| 6 | Other | subscription_other |
## Payment method IDs
| id | name |
|---|---|
| 1 | Credit Card |
| 2 | Debit Card |
| 3 | PayPal |
| 6 | Bank Transfer / SEPA |
| 7 | Other |
---
## Step 1 — Categorise the transactions
Skip these as internal/already-covered:
- Fixed costs you'll enter as **subscriptions** (Miete, SWM, SYNVIA, Hundefutter,
Netflix, Grover, Rundfunk ARD, KuKita, Lillydoo)
- Internal transfers (The Joy Pot ↔ Cookie, Hauptkonto, Tagesgeldkonto splits)
- Identified income (Cookie Share, Kindergeld, Pocket Money credits, Distributor)
- Fun Money pass-throughs (in and out same month → net zero)
**Variable expense taxonomy:**
| Category key | Subcategory key | Examples |
|---|---|---|
| `food` | `groceries` | E-Center, Knuspr, EDEKA, Tegut, VollCorner, Lidl, Netto, REWE, KoRo, Roast Market |
| `food` | `restaurants_bars` | Restaurants, Lieferando, Cafes, Zeit für Brot, Baobab, Höflinger |
| `personal_health` | `beauty_cosmetics` | DM, Rossmann |
| `personal_health` | `pharmacy` | Apotheke, MVZ Dermatologie |
| `transport` | `apps_taxi` | Uber, RYD GMBH, MVG, Handyparken |
| `shopping_clothing` | `gifts` | Children products: Schlummersack, Catchy Kids, SP EVERY., Dukal, Berger-Lernwelt |
| `shopping_clothing` | `clothes_shoes` | Zalando, Ernsting's, Schuhmair, Thalia, Vinted, Airbnb, Hotel at Booking.com |
| `shopping_clothing` | `electronics` | Amazon, AMZN Mktp DE |
| `housing` | `renovation_maintenance` | IKEA, Markus Festl, Granit, Sostrene Grene, Mol* tischdecken, Gaertnerei, Dehner |
| `education` | `courses_college` | Kathrin Orlob (PEKiP), Nerina Aupperle |
| `leisure` | `streaming` | WOW wowtv.de |
| `financial_other` | `bank_fees` | Unidentified PayPal, Ratepay, N26 fees |
| `Geschenke & Transfers` | *(income)* | One-off incoming transfers |
---
## Step 2 — Create subscriptions
```
get_subscriptions_meta() ← get category_id and payment_method_id
```
**Standard Cookie household subscriptions (as of 2026-07):**
| Name | Amount | billing_cycle | cycle_interval | category_id | payment_method_id |
|---|---|---|---|---|---|
| Miete | 1080.00 | monthly | 1 | 6 (Other) | 6 (Bank Transfer) |
| Strom (SWM) | 79.00 | monthly | 1 | 3 (Utilities) | 6 |
| Internet / TV / Telefon | 29.99 | monthly | 1 | 3 (Utilities) | 6 |
| Hundefutter | 75.00 | monthly | 1 | 6 (Other) | 6 |
| Netflix | 8.00 | monthly | 1 | 1 (Entertainment) | 6 |
| Grover | 16.90 | monthly | 1 | 6 (Other) | 2 (Debit Card) |
| Rundfunk ARD / ZDF | 55.08 | monthly | 3 | 1 (Entertainment) | 6 |
| KuKita Daycare (Leon) | 503.00 | monthly | 1 | 5 (Education) | 6 |
| Lillydoo diapers | 56.70 | monthly | 2 | 4 (Health) | 3 (PayPal) |
Monthly equivalent total: **1,838.60 €** (Yuvomi applies cycle_interval to prorate).
---
## Step 3 — Add recurring income entries
```
stage_add_budget_entry(
title="Kindergeld",
amount=55.00,
category="Sozialleistungen",
date="YYYY-MM-01",
is_recurring=True,
recurrence_interval="monthly",
)
commit_pending(pending_id)
```
**Standard recurring income:**
| Title | Amount | category |
|---|---|---|
| Kindergeld | +55.00 | Sozialleistungen |
| Cookie Share | +2650.00 | Erwerbseinkommen *(see recommended amount below)* |
---
## Step 4 — Post variable transactions
For each non-skipped CSV row, call `stage_add_budget_entry` with the mapped
category/subcategory and the actual transaction amount and date. Use the Partner
Name + Payment Reference as the title (truncate to 100 chars).
---
## Step 5 — Verify
```
get_budget_summary("YYYY-MM")
list_subscriptions()
```
Expected for a full month with KuKita:
- Fixed expenses ≥ 1,838 € (subscriptions)
- Variable expenses ≥ 500 € (groceries alone)
---
## Cookie Share: how much to transfer monthly
Calculated from JanJun 2026 data (Cookie account, one-offs stripped):
| | €/month |
|---|---|
| **Fixed costs (subscriptions)** | **1,839** |
| Miete | 1,080 |
| KuKita *(permanent from Jul 2026)* | 503 |
| Strom + SYNVIA + Rundfunk + Netflix + Grover + Hundefutter + Lillydoo | 256 |
| **Variable (6-month averages)** | **1,032** |
| Groceries | 595 |
| Children products | 142 |
| Dining & cafes | 100 |
| Transport | 66 |
| Drugstore | 52 |
| Clothing, Amazon, Pharmacy | 77 |
| **Total monthly spend** | **≈ 2,871** |
| Minus Kindergeld (fixed income) | 55 |
| Minus Pocket Money (conservative ~600 €) | 600 |
| **→ Recommended Cookie Share** | **≈ 2,650 €** |
| With 200 € buffer | **≈ 2,850 €** |
**Current Cookie Share (Jun 2026): 1,995 € — shortfall ~655 €.**
The gap was covered by irregular Pocket Money top-ups (avg 962 €/mo over 6 months, but
highly variable: 121 €3,000 €). KuKita starting in June is the biggest step-up; raising
Cookie Share to **2,650 €** makes the budget self-sufficient without relying on top-ups.
---
## Changelog
### 2026-06-29 — Corrections from first real import
- Subscriptions endpoint is `/budget/subscriptions`, NOT `/subscriptions/` (404).
- `recurrence_rule` RRULE strings are rejected by the API; use `cycle_interval` instead.
- `GET /budget/` (no filter) returns only non-recurring entries; use `?month=` for full view.
- Added Cookie Share recommendation (2,650 €/month) based on 6-month expense analysis.
- Added full category taxonomy table.
### 2026-06-29 — Initial runbook
Created from JanJun 2026 N26 Cookie account analysis.

View File

@@ -0,0 +1,131 @@
# 2026-06-01 — Slate AX → SODOLA Migration
**Status:** Done — 2026-06-02
**Hardware:** SODOLA 5-Port 2.5Gbit Managed Switch replacing GL.iNet Slate AX
**Router:** Fritz!Box 7590
## Goal
Remove the Slate AX sub-router. It adds double-NAT, double port-forward complexity, and a second DHCP/gateway hop. Replace with the SODOLA switch so Proxmox and homelab devices attach directly to the Fritz!Box—without changing any of their IPs.
## Current topology
```
ISP
└── Fritz!Box 7590 (192.168.178.1)
└── GL.iNet Slate AX WAN: 192.168.178.x LAN: 192.168.8.1 [NAT]
├── hubris (Proxmox) 192.168.8.77
├── + all 16 LXCs on vmbr0 bridged to the same subnet
└── other homelab devices
```
## Target topology
```
ISP
└── Fritz!Box 7590
├── VLAN 1 192.168.178.0/24 (main LAN — laptops, phones)
└── VLAN 10 192.168.8.0/24 gateway 192.168.8.1 (homelab, no NAT)
└── SODOLA 5-Port 2.5Gbit
├── Port 1 uplink → Fritz!Box LAN port
├── Port 2 hubris (Proxmox) 192.168.8.77
├── Port 3 [homelab device]
├── Port 4 [homelab device]
└── Port 5 spare
```
Fritz!Box takes over `192.168.8.1` — the same gateway IP the Slate AX used. No static IPs or gateway entries change on any LXC or VM.
See [network architecture](../infrastructure/network.md) for the permanent topology reference.
## Pre-flight checklist
- [ ] Note Slate AX gateway IP on the homelab subnet (expected `192.168.8.1`)
- [ ] Note Slate AX upstream DNS setting (what it forwards to)
- [ ] List all port forwards currently on the Slate AX → move these to Fritz!Box
- [ ] List all port forwards on Fritz!Box that point at the Slate AX WAN IP → collapse these into direct homelab-IP rules
- [ ] Confirm Fritz!OS version supports "second IP network" (Fritz!OS 7.x required)
- [ ] SSH into hubris: verify `grep gateway /etc/network/interfaces``192.168.8.1`
- [ ] SODOLA web UI accessible and ports configurable before physical move
## Fritz!Box configuration
**Path:** Home Network → Network → Network Settings → IPv4 Addresses → Add IP network
| Field | Value |
|---|---|
| IP address | `192.168.8.1` |
| Subnet mask | `255.255.255.0` |
| DHCP range | `192.168.8.100 192.168.8.240` |
| Assign to | LAN port that connects to SODOLA |
| Network isolation | Enabled (blocks main LAN from initiating into homelab) |
| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../containers/107-dns.md)) |
After creating the network, move any port forwards from the Slate AX into Fritz!Box → Internet → Permits (target IPs are now directly reachable on `192.168.8.x`).
## SODOLA configuration
All ports in VLAN 10, untagged (simple access-port setup):
| Port | PVID | Role |
|---|---|---|
| 1 | 10 | Uplink → Fritz!Box |
| 2 | 10 | hubris (Proxmox) |
| 3 | 10 | homelab device |
| 4 | 10 | homelab device |
| 5 | 10 | spare |
## Proxmox / LXC changes
If the Slate AX gateway was `192.168.8.1`: **no changes needed**. Fritz!Box takes the same IP.
If Slate AX used a different gateway (e.g. `.254`), update:
```bash
# on hubris
nano /etc/network/interfaces # change gateway line
ifreload -a
# for each LXC with a static gateway
pct config <id> # check net0: line for gw=
pct set <id> --net0 name=eth0,bridge=vmbr0,ip=<ip>/24,gw=192.168.8.1
```
## DNS after migration
Technitium ([CT 107](../containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../infrastructure/dns.md).
## Cutover procedure
1. Configure Fritz!Box second IP network (VLAN 10 / `192.168.8.1`) — verify DHCP works via a test laptop on the target port.
2. Configure SODOLA port VLAN assignments.
3. Connect SODOLA port 1 → Fritz!Box target LAN port.
4. Move hubris ethernet → SODOLA port 2. Proxmox should stay reachable immediately (same gateway IP).
5. Move remaining homelab devices to SODOLA ports 35.
6. Decommission Slate AX.
## Verification
```bash
# internet from Proxmox
ssh proxmox "ping -c3 1.1.1.1"
# Fritz!Box main LAN reachable (routing between VLANs works)
ssh proxmox "ping -c3 192.168.178.1"
# split-horizon DNS via Technitium still resolves
ssh proxmox "dig @192.168.8.2 +short git.hubris.network"
# expected: 192.168.8.175
# Caddy reverse proxy responds
curl -sk https://git.hubris.network | head -5
# Authentik on VPS still reachable
curl -sk https://auth.hubris.network/if/flow/default-authentication-flow/ | head -5
```
## Post-migration
- Update [network.md](../infrastructure/network.md) topology to reflect new state.
- Add changelog entries to [hosts/hubris.md](../hosts/hubris.md) and any affected container pages.
- Update status in [plans/index.md](index.md) to `Done`.
- If anything went sideways, open an investigation in `investigations/`.

View File

@@ -0,0 +1,318 @@
# Deprecate claudio-bot (LXC 123) — Hermes Agent now serves as control plane
## Goal
Phase out the **claudio-bot ecosystem** (LXC 123, claudio-monitor, IPC server)
now that Hermes Agent is configured and running on mac-mini. Hermes replaces
claudio-bot's role as the homelab control plane — accepting commands, relaying
alerts, and answering queries about infrastructure state.
## What is claudio-bot
A three-component ecosystem deployed 2026-04-20:
| Component | Location | Purpose |
|-----------|----------|---------|
| **claudio-bot** | LXC 123 (192.168.8.230) | Matrix bot `@claudio:avispero` — slash commands, NL queries, infra notifications |
| **claudio-monitor** | hubris (PVE host) | 5-min watchdog → sends alerts to claudio-bot's IPC server |
| **IPC server** | LXC 123:9090 | `/notify`, `/propose`, `/status` endpoints — used by monitor + backup wrapper |
Dependencies:
- Matrix synapse on LXC 118 (elementsynapse) — claudio-bot connects to `192.168.8.239:8008`
- Gitea auto-deploy webhook → LXC 123:9797 (pipeline `dtoro/claudio-bot`)
- LLM backend: LM Studio on mac-mini (192.168.8.174:1234) or Anthropic API
- Secrets at `/etc/claudio-bot/` (config.yaml, matrix.token, anthropic.key, lmstudio.key, ipc.token)
## What Hermes replaces
| claudio-bot function | Hermes replacement |
|---------------------|-------------------|
| Matrix slash commands | Hermes CLI + `send_message` for outbound notifications |
| NL infrastructure queries | Hermes with `homelab` CLI + MCP tools (already working) |
| Monitor alert relay | Hermes-native: (a) on-demand `homelab health` via extended skill, (b) cronjob every 15 min with actionable alerts (not just passive notifications) |
| IPC server (port 9090) | **Retired** — Hermes doesn't need a separate IPC layer; monitoring runs directly via SSH + `send_message` |
| LLM backend | Already handled by Hermes (OpenRouter → deepseek/deepseek-v4-pro) |
| Auto-deploy webhook (9797) | **Retired** — repo archived |
## Full blast radius (files referencing claudio)
### Homelab-context repo (`/opt/homelab-context/`)
#### Core topology
- `inventory.yaml:275``claudio-bot:` host entry (LXC 123, matrix-agent)
- `hosts/claudio-bot.yaml` — auto-generated host file (delete)
- `.sops.yaml:47` — placeholder comment listing claudio-bot/claudio-monitor
#### Documentation (containers)
- `containers/123-claudio-bot.md` — full container doc
- `containers/index.md:17` — LXC table row
- `containers/118-elementsynapse.md:3,36,39``@claudio:avispero` reference, disk reading, direct connection note
- `containers/103-paperless.md:27` — disk reading from claudio-monitor
#### Documentation (infrastructure)
- `infrastructure/monitoring.md`**ENTIRE FILE** is about claudio-monitor → claudio-bot pipeline
- `infrastructure/backups.md:74,94,98` — IPC notify, thermal monitoring moved out, related link
- `infrastructure/auto-deploy.md:23,41,50,61,120,138-139` — Gitea webhook target, pipeline table, port 9797
- `infrastructure/ssh-access.md:134` — LXC table row
- `infrastructure/homelab-context.md:4,16,143` — client list, integration note
#### Other
- `operations/commands.md:17``pvesh` bulk query note (passive reference, can stay)
- `README.md:32,40,64` — LXC table, monitoring link, tracked configs mention
- `mcp/deploy/webhook/webhook.py:5-6` — port comment (9797, 9799)
- `hosts/hubris.md:57,96``/opt/claudio-monitor` checkout, LXC 109 cleanup note (passive)
- `vms/108-haos.md:30` — claudio-monitor MQTT removal note (historical, can stay)
- `investigations/2026-04-21-hubris-crash-loop.md:27` — historical reference (keep)
### External repos (Gitea)
- `dtoro/claudio-bot` — the bot itself
- `dtoro/claudio-monitor` — the watchdog scripts
### External state (hubris)
- `/opt/claudio-bot/` — bot checkout (LXC 123)
- `/opt/claudio-monitor/` — monitor checkout (hubris host)
- `/etc/claudio-bot/` — secrets directory
- `/etc/claudio-deploy/` — deploy git credentials
- `/etc/claudio-monitor/` — thresholds + bot.token
- `/var/lib/claudio-monitor/` — state files (.alert, .counter)
- `claudio-bot.service` — systemd unit (LXC 123)
- `claudio-monitor.timer` + `.service` — systemd timer (hubris)
- Gitea webhook id 6 (claudio-bot-deploy) + webhook for claudio-monitor (9799) on apps LXC 105
### External state (workstations)
- Nothing on mac-mini, republic-laptop, or ludo-mini references claudio directly
## Phased deprecation plan
### Phase 1 — Stop & verify (low-risk, reversible)
**Goal**: Stop claudio services, verify nothing breaks, keep LXC 123 paused (not destroyed).
1. **Stop claudio-monitor on hubris**
```bash
ssh hubris systemctl disable --now claudio-monitor.timer
```
The 5-minute watchdog stops. Home Assistant still gets PVE metrics independently via its own integration — no alerting gap there.
2. **Stop claudio-bot on LXC 123**
```bash
ssh claudio-bot systemctl stop claudio-bot
```
The Matrix bot goes offline. Nobody can send slash commands to `@claudio:avispero`.
3. **Pause LXC 123** (not destroy yet)
```bash
ssh hubris pct shutdown 123
```
IPC server (port 9090) and deploy webhook (port 9797) become unreachable.
4. **Verify no breakage** (run for 24-48h)
- Check Gitea webhook logs — the `dtoro/claudio-bot` webhook will 404 on pushes (non-fatal)
- Check Caddy logs — no upstream errors for claudio-related paths (there shouldn't be any — no public hostname)
- Check DNS — no queries for claudio-bot.hubris.network (doesn't have a public hostname)
- Confirm Hermes works normally for all homelab operations
### Phase 2 — Hermes-native health monitoring (replaces claudio-monitor)
**Goal**: Two capabilities that together exceed what claudio-monitor did:
1. **On-demand health check** — user asks "how's the homelab?" → Hermes runs a
comprehensive check across all hosts/LXCs and reports results conversationally.
2. **Cron-based watchdog** — runs every 15 minutes, checks thresholds, raises
alarms with actionable options (not just passive alerts). User can respond
and Hermes takes action directly.
**Key difference from claudio-monitor**: The old system was a bash script that
POSTed JSON to an IPC server that a Matrix bot reformatted into a flat message.
The Hermes-native approach gives the LLM the raw metrics and lets it reason about
what's important before alerting — and crucially, the user can reply to an alert
and Hermes can SSH into hubris to resize a rootfs, restart a service, or
investigate deeper.
#### 2a. Extend `homelab-hardware-health` skill
The existing skill covers CPU/NVMe temps. Expand it to be the **single
homelab health check skill** covering:
- **Hardware**: CPU temps (sysfs), NVMe/SATA SSD temps (smartctl), block device inventory
- **LXC resources**: `pvesh get /cluster/resources --type vm --output-format json` → disk %, CPU %, RAM usage per container
- **Service health**: Check key services respond (gitea:3000, caddy:80/443, synapse:8008, paperless, nextcloud, jellyfin)
- **System drift**: `apt list --upgradable` counts per host, docker image pull digests on apps LXC 105
Thresholds (from old `/etc/claudio-monitor/thresholds.env`):
| Metric | Warning | Critical |
|--------|---------|----------|
| LXC disk usage | > 80% | > 90% |
| NVMe temp | > 60°C | > 70°C |
| CPU temp | > 70°C | > 80°C |
| Apt upgradable | > 10 | > 50 |
| Service down | any | 2+ services |
Update the skill file at `devops/homelab-hardware-health/SKILL.md` with:
- Expanded trigger phrases ("check homelab health", "how's the lab", "health report")
- LXC resource check commands
- Service health check commands
- Thresholds table
- "If you find issues, present options to the user" instruction
#### 2b. Create Hermes cronjob `homelab-health-watchdog`
```bash
hermes cron create \
--schedule 'every 15m' \
--name 'homelab-health-watchdog' \
--prompt 'Load skill homelab-hardware-health. Run a full health check across all hosts (hubris + all LXCs). Check: (1) hardware temps, (2) LXC disk/CPU/RAM via pvesh, (3) service reachability for gitea, caddy, synapse, paperless, nextcloud, jellyfin, (4) apt upgradable counts. Compare against thresholds in the skill. If nothing is critical, say nothing (no message = healthy). If anything breaches thresholds, send a concise alert with: what broke, current value vs threshold, and 2-3 actionable options the user can reply to. DO NOT send routine "all clear" messages.'
```
This runs every 15 minutes (Hermes cron is LLM-driven, heavier than a bash
script — 15 min is appropriate vs claudio-monitor's 5 min). When thresholds
are clean, the agent stays silent. When something breaches, the user gets a
message like:
> ⚠️ LXC 103 (paperless) disk at 91% (> 90% critical)
>
> Options: [resize rootfs to 16 GiB] [investigate what's using space] [snooze 24h]
User picks an option → Hermes takes action (SSH into hubris, run `pct resize`,
grow the filesystem inside the container).
#### 2c. Add `homelab-health` alias to `homelab` CLI
For quick on-demand checks from the terminal:
```bash
homelab health # runs the skill inline via Hermes
# or equivalently:
hermes --prompt "Load homelab-hardware-health, run full health check"
```
### Phase 3 — Documentation & inventory cleanup
**Goal**: Update homelab-context to reflect claudio deprecation, without destroying history.
#### Files to edit
| File | Action |
|------|--------|
| `inventory.yaml` | Mark `claudio-bot` entry as deprecated (comment + `status: deprecated`). Do NOT remove — keeps historical ID mapping. |
| `containers/123-claudio-bot.md` | Add deprecation banner at top, changelog entry |
| `containers/index.md` | Change status from "running" to "stopped (deprecated)" |
| `infrastructure/monitoring.md` | Rewrite to document new Hermes-based monitoring. Keep claudio-monitor section as "legacy (deprecated 2026-06-04)". |
| `infrastructure/auto-deploy.md` | Mark claudio-bot and claudio-monitor webhook entries as deprecated |
| `infrastructure/backups.md` | Update notification section (IPC path no longer available). Note: backups themselves are still disabled — this is forward-looking. |
| `infrastructure/ssh-access.md` | Mark LXC 123 as "offline (deprecated)" |
| `infrastructure/homelab-context.md` | Remove claudio-bot from client list |
| `containers/118-elementsynapse.md` | Note `@claudio:avispero` is decommissioned |
| `containers/103-paperless.md` | Remove stale disk reading reference (already resolved — rootfs grew to 16 GiB 2026-05-15 per 118 doc) |
| `README.md` | Update LXC table, remove monitoring link |
| `hosts/claudio-bot.yaml` | Add deprecation banner (or delete if auto-generated — it is, so just update inventory) |
| `.sops.yaml` | Remove claudio-bot from placeholder comment |
#### Files to leave untouched (historical)
| File | Reason |
|------|--------|
| `investigations/2026-04-21-hubris-crash-loop.md` | Historical investigation — claudio-monitor is factual context |
| `vms/108-haos.md:30` | Historical note about MQTT removal — factual |
| `mcp/deploy/webhook/webhook.py:5-6` | Comment only — harmless, keep for port mapping context |
| `operations/commands.md:17` | Passive reference — harmless |
### Phase 4 — Purge (after confirmation window)
**Goal**: Full removal after confirming Phase 1-3 is stable for ≥2 weeks.
1. **Destroy LXC 123**
```bash
ssh hubris pct destroy 123 --purge
```
Frees 8 GiB rootfs, 512 MiB RAM, 1 core.
2. **Remove claudio-monitor from hubris**
```bash
ssh hubris rm -rf /opt/claudio-monitor /etc/claudio-monitor /var/lib/claudio-monitor
ssh hubris systemctl disable claudio-monitor.timer # already done in phase 1
```
3. **Archive Gitea repos**
- Archive `dtoro/claudio-bot` (mark read-only in Gitea)
- Archive `dtoro/claudio-monitor` (mark read-only in Gitea)
4. **Remove Gitea webhooks**
- Remove claudio-bot-deploy webhook (id 6) from Gitea
- Remove claudio-monitor webhook (9799) from apps LXC 105
- Remove `192.168.8.230` from Gitea `app.ini` `ALLOWED_HOST_LIST`
5. **Final documentation cleanup**
- Remove `containers/123-claudio-bot.md` (or move to `containers/_destroyed/`)
- Remove `hosts/claudio-bot.yaml`
- Remove claudio-bot row from `containers/index.md`
- Update all cross-references (see blast radius above)
## Files changed by phase
### Phase 3 (documentation)
- `/opt/homelab-context/inventory.yaml`
- `/opt/homelab-context/containers/123-claudio-bot.md`
- `/opt/homelab-context/containers/index.md`
- `/opt/homelab-context/infrastructure/monitoring.md`
- `/opt/homelab-context/infrastructure/auto-deploy.md`
- `/opt/homelab-context/infrastructure/backups.md`
- `/opt/homelab-context/infrastructure/ssh-access.md`
- `/opt/homelab-context/infrastructure/homelab-context.md`
- `/opt/homelab-context/containers/118-elementsynapse.md`
- `/opt/homelab-context/containers/103-paperless.md`
- `/opt/homelab-context/README.md`
- `/opt/homelab-context/hosts/claudio-bot.yaml`
- `/opt/homelab-context/.sops.yaml`
### Phase 4 (destruction)
- Proxmox: LXC 123 destroyed
- Gitea: `dtoro/claudio-bot` archived, `dtoro/claudio-monitor` archived
- Gitea webhooks removed (apps LXC 105)
- hubris: `/opt/claudio-monitor/`, `/etc/claudio-monitor/`, `/var/lib/claudio-monitor/` removed
- `containers/123-claudio-bot.md` → moved to `_destroyed/` or deleted
## Risks & tradeoffs
| Risk | Mitigation |
|------|-----------|
| **Monitoring gap** — no alerts during Phase 1-2 transition | Phase 1 stops the old timer but Phase 2 builds the Hermes-native replacement immediately. If there's a gap, keep claudio-monitor.timer running until the Hermes cronjob is verified with a test alert. |
| **Backup notifications** — the backup wrapper's IPC notify path is dead | Backups are already DISABLED. When re-enabled, wire them to the new Hermes monitoring pipeline instead. |
| **Matrix `@claudio:avispero`** — users may still try to interact | The bot just won't respond. Could leave a notice in the Matrix room. |
| **Gitea webhook failures** — pushes to `dtoro/claudio-bot` will 404 | Non-fatal. Archive the repo in Phase 4 to stop pushes entirely. |
| **Losing monitoring history** — state files on hubris | Keep `/var/lib/claudio-monitor/` until Phase 4. No rush. |
| **LM Studio dependency** — claudio-bot was using mac-mini's LM Studio | Hermes already uses OpenRouter. LM Studio on mac-mini can stay for other uses or be reclaimed. |
## Open questions
1. **Should the Matrix `@claudio:avispero` account be deactivated or left as-is?** It has no public hostname and was invite-only to a private room.
2. **Are you OK archiving `dtoro/claudio-bot` and `dtoro/claudio-monitor` on Gitea (read-only, preserved), or do you want them fully deleted?**
3. **Should LXC 123 be paused (reversible) or destroyed immediately?** Recommend pause for at least 2 weeks as a safety net.
4. **Which messaging platform should the health watchdog deliver alerts to?** Currently Hermes is connected via CLI (mac-mini). To get push alerts, we need a `send_message` target — Telegram, Discord, or Matrix are available. Which do you prefer?
## Verification
After Phase 1:
- `ssh hubris systemctl is-active claudio-monitor.timer` → `inactive`
- `ssh claudio-bot systemctl is-active claudio-bot` → `inactive` (or unreachable if LXC stopped)
- `ssh hubris pct status 123` → `stopped`
- `curl -s http://192.168.8.230:9090/status` → connection refused (IPC dead)
After Phase 2:
- `hermes cron list` shows `homelab-health-watchdog` running every 15 min
- "check homelab health" → Hermes runs comprehensive check, reports temps + LXC resources + service status + apt drift
- Force an alert: `NVME_WARN_TEMP_C=1` on hubris → watchdog catches it on next 15-min tick → alert arrives via Hermes messaging with actionable options
- Alert format includes 2-3 options user can reply to, not just passive notification
After Phase 3:
- All cross-references in homelab-context point to deprecated/removed status
- `grep -r 'claudio' /opt/homelab-context/ --include='*.md' --include='*.yaml' | grep -v '_destroyed\|investigations\|changelog'` → only historical references remain
After Phase 4:
- `ssh hubris pct list | grep 123` → no output
- `ssh hubris ls /opt/claudio-monitor` → no such file
- Gitea `dtoro/claudio-bot` → archived badge

View File

@@ -0,0 +1,107 @@
# 2026-06-24 — TRMNL plugins LXC (128) + middleware deploy pipeline
## Goal
Stand up a dedicated LXC to host self-hosted **middleware for TRMNL e-ink plugins**.
TRMNL cloud polls `https://trmnl.hubris.network/<plugin>/dashboard` every 15 min; the
middleware fetches/shapes live data and returns JSON that TRMNL merges into the plugin's
Liquid template. First consumer: the Munich Home Dashboard (`/munich-home/dashboard`).
One LXC + one FastAPI service hosts all current and future plugins (router per plugin).
Source repo: gitea `dtoro/terminalito` (app code). This repo only documents the fabric
wiring, same split as Artifacto/Plato.
## Current state
- No TRMNL middleware in the lab. Highest LXC id is 127 (see `containers/index.md`).
- Public hostnames terminate at the [VPS netbird traefik](../hosts/netbird-vps.md) → netbird
mesh → [caddy (121)](../containers/121-caddy.md) → backend LXC. Cert obtained by Caddy
(IONOS DNS-01) and mirrored to the VPS by the daily cert-sync timer on the host.
- Auto-deploy pipelines are gitea-webhook driven, two shapes (see [auto-deploy](../infrastructure/auto-deploy.md)).
## Target state
```
TRMNL cloud --GET 15m, Bearer token--> https://trmnl.hubris.network/munich-home/dashboard
VPS traefik (public TLS) --netbird--> caddy (121) --> trmnl (128) :9851 trmnl-plugins.service
├ Open-Meteo (weather)
├ MVG departures (transit)
└ Google Calendar (OAuth, SOPS)
```
- **LXC 128 `trmnl`**: Debian, unprivileged, ~1 core / 512 MiB1 GiB / 8 GiB rootfs. No mounts.
- **Service** `trmnl-plugins.service``uvicorn server.app:app --host 0.0.0.0 --port 9851`,
`EnvironmentFile=/etc/trmnl-plugins/env`. Auth: every path except `/health` requires
`Authorization: Bearer $TRMNL_POLL_TOKEN`.
- **Auto-deploy** (Shape B): `/opt/terminalito` working tree, sibling `/opt/terminalito-deploy/`.
- Public hostname `trmnl.hubris.network`.
## Pre-flight checklist
- [ ] Confirm next free LXC id is 128 (`homelab list`, `containers/index.md`).
- [ ] Decide IP on `192.168.8.0/16` LAN (e.g. `192.168.8.211`) — pick a free one.
- [ ] Have Google OAuth client + refresh token, MVG stop globalIds, and a generated
`trmnl_poll_token` ready for the secret (see `dtoro/terminalito` README).
## Step-by-step procedure
1. **Provision + enroll**
```bash
# create LXC 128 trmnl on hubris (Debian), then enroll it:
homelab client add trmnl # joins netbird, provisions /etc/age/key.txt, edits inventory.yaml
homelab client add --finalize-pubkey <age_pubkey> # commits the age pubkey
ssh trmnl 'apt-get install -y python3-venv git'
```
2. **Secret** (`trmnl-oauth`): create `secrets/trmnl-oauth.yaml` with
`google_client_id/secret/refresh_token`, MVG stop ids, and `trmnl_poll_token`; add a
`path_regex` rule in `.sops.yaml` granting **trmnl**'s age pubkey; `sops updatekeys`.
The service reads it at deploy time via `homelab secret trmnl-oauth` → `/etc/trmnl-plugins/env`.
3. **App + service** on LXC 128 (clone uses the **internal** gitea URL — `git.hubris.network`
resets from inside LXCs):
```bash
git clone http://192.168.8.121:3000/dtoro/terminalito.git /opt/terminalito
python3 -m venv /opt/terminalito/server/.venv
/opt/terminalito/server/.venv/bin/pip install -r /opt/terminalito/server/requirements.txt
# install /etc/systemd/system/trmnl-plugins.service, enable --now
```
4. **Deploy pipeline** (Shape B, mirrors homelab-mcp): create `/opt/terminalito-deploy/`
`{webhook.py,deploy.sh}` (HMAC vs `/etc/terminalito-deploy/secret`, filter `refs/heads/main`,
`git pull` → `pip install -r server/requirements.txt` → rebuild env from `homelab secret` →
`systemctl restart trmnl-plugins`). Receiver `:9797`. Git creds at
`/etc/terminalito-deploy/git-credentials` (mode 600). Register a gitea webhook on
`dtoro/terminalito`; add `192.168.8.<128-ip>` to gitea `app.ini` `ALLOWED_HOST_LIST`.
5. **DNS**: add `trmnl.hubris.network` A → `192.168.8.175` (caddy) on [Technitium (107)](../containers/107-dns.md).
6. **Caddy** (`dtoro/caddy-conf`, commit+push auto-deploys):
```
trmnl.hubris.network { reverse_proxy 192.168.8.<128-ip>:9851 }
```
7. **Public exposure** on the [VPS](../hosts/netbird-vps.md): add traefik router+service for
`Host(\`trmnl.hubris.network\`)` → `http://192.168.8.<128-ip>:9851`; add the host to the
cert-sync map so the LE cert mirrors over.
8. **TRMNL cloud**: create a Polling private plugin, URL `…/munich-home/dashboard`, header
`Authorization: Bearer <trmnl_poll_token>`, refresh 15 min; paste `full.liquid`; add to a playlist.
## Verification
- `ssh trmnl systemctl is-active trmnl-plugins` → `active`; `curl -s localhost:9851/health` → `ok`.
- LAN: `curl -s -H "Authorization: Bearer <tok>" https://trmnl.hubris.network/munich-home/dashboard`
→ 200 JSON; without the header → 401; `/health` → 200.
- Public: same curl from off-mesh resolves via VPS, 200.
- Push a no-op commit → `/opt/terminalito-deploy` logs show pull+restart; webhook 202.
- `homelab mcp get_host trmnl` and `search_docs trmnl` agree with `containers/128-trmnl.md`.
## Post-migration
When executed, write changelog entries (same date) on:
`containers/128-trmnl.md` (new page), `containers/index.md` (row), `inventory.yaml`
(`services.trmnl`), `infrastructure/auto-deploy.md` (pipeline row + `ALLOWED_HOST_LIST` +
changelog), `containers/104-gitea.md` (webhook + allowed host), `containers/121-caddy.md`
(new proxied host), `containers/107-dns.md` (A record), `hosts/netbird-vps.md` (public route +
cert-sync). Then set this plan's status to `Done` in `plans/index.md`.

View File

@@ -0,0 +1,443 @@
# Yuvomi deployment — `house.hubris.network`
Deploy [Yuvomi](https://yuvomi.cloud/) (previously Oikos) — a self-hosted
family planner with 14 modules (calendar, tasks, meals, groceries, budget,
documents, notes, etc). Single Docker container (Express.js + SQLCipher
SQLite), 256 MB RAM min.
**Target hostname:** `house.hubris.network` — publicly reachable via VPS
traefik, LAN reachable via Caddy.
**Integrations:**
- Authentik SSO (OIDC)
- Google Calendar (tokens exist on trmnl LXC 128)
- Paperless (Yuvomi's Documents module / clarification needed — see Phase 4)
---
## Phase 0 — Clarifications needed
### 0.1 Paperless connection
Yuvomi's "Documents" module stores documents inside its encrypted SQLite DB or
optionally on WebDAV. There is **no direct Paperless-ngx API connector** in
Yuvomi. Options:
a) **Keep as-is** — Yuvomi's docs are separate from Paperless, no integration
b) **WebDAV bridge** — Mount Paperless's consumption dir as WebDAV, point
Yuvomi doc storage there (Yuvomi stores newly uploaded docs directly in the
Paperless consume folder)
c) **Custom module** — Write a Yuvomi module that fetches from Paperless API
Decision needed before Phase 3 config.
**Decision:** WebDAV bridge (Phase 6.2).
### 0.2 Deployment target
Two options:
| Option | Pros | Cons |
|--------|------|------|
| **apps LXC (105)** — Docker already there, 4GB RAM, 2 cores | Zero provisioning, existing compose pattern | Shared with artifacto, MCP, secrets-issuance; Portainer-managed stacks can be tricky |
| **New LXC (~129)** — dedicated, clean | Isolated, no side-effects | Need to create, install Docker, wire into everything |
**Decision:** New LXC (129).
---
## Phase 1 — Provision new LXC (129) for Yuvomi
### 1.1 Create the LXC on hubris
```
ssh root@192.168.8.77 << 'EOF'
# Check available templates
pveam list local | grep debian
# Create unprivileged Debian 13 LXC (follows trmnl's unpriv pattern)
pct create 129 local:vztmpl/debian-13-standard_13.7-1_amd64.tar.zst \
--hostname house \
--description "Yuvomi family planner — house.hubris.network" \
--cores 1 \
--memory 1024 \
--swap 512 \
--rootfs local:8 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp,type=veth \
--unprivileged 1 \
--features nesting=1 \
--onboot 1 \
--start 1
EOF
```
Resources: 1 core / 1 GiB RAM / 8 GiB rootfs (generous for a single Express.js
container; can downsize later).
### 1.2 Set static IP and install Docker
After the LXC boots, find its DHCP lease, then set a static IP:
```
# Find actual IP
ssh root@192.168.8.77 'lxc-attach 129 -- ip addr show eth0 | grep inet'
# Reserve 192.168.8.212 (or whatever is free) via Technitium DHCP,
# or set static IP in PVE config:
ssh root@192.168.8.77 'pct set 129 --net0 name=eth0,bridge=vmbr0,ip=192.168.8.212/24,gw=192.168.8.1,type=veth'
ssh root@192.168.8.77 'lxc-attach 129 -- reboot'
```
### 1.3 Install Docker inside the LXC
```
ssh root@192.168.8.77 << 'DOCKER'
lxc-attach 129 -- bash -c '
apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
systemctl enable --now docker
docker --version
docker compose version
'
DOCKER
```
### 1.4 Download Yuvomi and start
```
ssh root@192.168.8.77 'lxc-attach 129 -- bash -c "
mkdir -p /opt/yuvomi /opt/yuvomi/data /opt/yuvomi/backups /opt/yuvomi/modules
cd /opt/yuvomi
curl -O https://raw.githubusercontent.com/ulsklyc/yuvomi/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/ulsklyc/yuvomi/main/.env.example
cp .env.example .env
"'
```
### 1.5 Generate keys and configure .env
```
ssh root@192.168.8.77 'lxc-attach 129 -- bash -c "
SESSION_SECRET=\$(openssl rand -hex 32)
DB_KEY=\$(openssl rand -hex 32)
cd /opt/yuvomi
sed -i \"s/SESSION_SECRET=.*/SESSION_SECRET=\$SESSION_SECRET/\" .env
sed -i \"s/DB_ENCRYPTION_KEY=.*/DB_ENCRYPTION_KEY=\$DB_KEY/\" .env
sed -i \"s/OIKOS_HTTP_PORT=3000/OIKOS_HTTP_PORT=3000/\" .env
sed -i \"s/# TZ=.*/TZ=Europe\\/Berlin/\" .env
echo \"SESSION_SECURE=true\" >> .env
echo \"TRUST_PROXY=1\" >> .env
echo \"BASE_URL=https://house.hubris.network\" >> .env
"'
```
### 1.6 Start Yuvomi
```
ssh root@192.168.8.77 'lxc-attach 129 -- bash -c "cd /opt/yuvomi && docker compose up -d"'
```
### 1.7 Verify
```
ssh root@192.168.8.77 'lxc-attach 129 -- curl -s http://127.0.0.1:3000/health'
# Expected: 200 OK
---
## Phase 2
### 2.1 Caddy — add `house.hubris.network`
Edit `/etc/caddy/Caddyfile` on LXC 121 (via `dtoro/caddy-conf` repo):
```
house.hubris.network {
tls {
dns ionos
}
reverse_proxy 192.168.8.212:3000
}
```
- Commit to `dtoro/caddy-conf` → auto-deploy via webhook
- If not yet deployed, push manually: `cd /etc/caddy && git add Caddyfile && git commit -m 'add house.hubris.network → yuvomi' && git push`
### 2.2 Verify LAN access
```
curl -sI https://house.hubris.network/
# Expected: 200 or 302 (redirect to /login or the setup wizard)
```
### 2.3 DNS — add Technitium record
Add A record `house.hubris.network → 192.168.8.175` (Caddy) on DNS LXC (107).
If using the DNS web UI: http://192.168.8.2/ → Zones → hubris.network → Add A record.
### 2.4 DNS mesh sync
If mesh DNS (Netbird managed zone) is in use, add the same record there or
verify dns-sync picks it up.
---
## Phase 3 — Public exposure (VPS traefik)
### 3.1 Add cert sync entry
On hubris (PVE host), edit `/usr/local/bin/hubris-public-cert-sync.sh`, add:
```bash
[house.hubris.network]="house.fullchain.crt house.privkey.key"
```
Run once:
```
systemctl start hubris-public-cert-sync.service
```
Verify certs landed on VPS:
```
ssh root@100.122.165.149 "ls -la /var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data/house.*"
```
### 3.2 Add traefik router
On the VPS, edit `/opt/traefik-dynamic.yaml`:
```yaml
http:
routers:
house-public:
rule: 'Host(`house.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls: {}
middlewares:
- house-ratelimit
service: house-public
middlewares:
house-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
services:
house-public:
loadBalancer:
servers:
- url: 'http://192.168.8.212:3000'
tls:
certificates:
- certFile: /letsencrypt/house.fullchain.crt
keyFile: /letsencrypt/house.privkey.key
```
Restart traefik:
```
docker restart netbird-traefik
```
### 3.3 Verify public access
From outside the homelab LAN (or with `--resolve`):
```
curl -sI --resolve house.hubris.network:443:82.165.190.79 https://house.hubris.network/
# Expected: 200 or 302
echo | openssl s_client -connect 82.165.190.79:443 -servername house.hubris.network 2>&1 | openssl x509 -noout -subject
# Expected: CN=house.hubris.network (not TRAEFIK DEFAULT CERT)
```
---
## Phase 4 — Authentik SSO (OIDC)
### 4.1 Create OIDC provider in Authentik
Via VPS admin UI (`https://auth.hubris.network/if/admin/`):
- Applications → Providers → Create → OAuth2/OpenID Provider
- Name: `yuvomi`
- Client ID: auto-generated
- Client Secret: auto-generated (save this)
- Redirect URIs: `https://house.hubris.network/oauth2/callback`
- Signing Key: auto-generated
- Subject Mode: Based on User ID (or Based on Username — pick what Yuvomi expects)
### 4.2 Create application in Authentik
- Applications → Applications → Create
- Name: `Yuvomi`
- Slug: `yuvomi`
- Provider: select the one created above
- Launch URL: `https://house.hubris.network`
### 4.3 Set env vars in Yuvomi `.env`
On apps LXC (105), edit `/opt/yuvomi/.env`:
```
OIDC_ISSUER=https://auth.hubris.network/application/o/yuvomi/
OIDC_CLIENT_ID=<from Authentik>
OIDC_CLIENT_SECRET=<from Authentik>
# OIDC_TRUST_EMAIL_WITHOUT_VERIFIED_CLAIM=true # if Authentik doesn't send email_verified
```
### 4.4 Restart Yuvomi
```
pct exec 105 -- bash -c 'cd /opt/yuvomi && docker compose restart'
```
### 4.5 Verify SSO flow
Open `https://house.hubris.network/` — should redirect to Authentik login,
then back to Yuvomi.
---
## Phase 5 — Google Calendar
### 5.1 Extract tokens from trmnl LXC
On trmnl (LXC 128), the env file at `/etc/trmnl-plugins/env` contains:
```
GOOGLE_CLIENT_ID=119823214387-32f20ed3imesiv7uh5si7p3rou9fros4.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=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
GOOGLE_REFRESH_TOKEN=1//03CS0rkuf7XVQCgYIARAAGAMSNwF-L9Irr5_b4kLhkx-dtf9EGQ1eJ1OnvxkaV_P1_4TDPwUN2lFb7nsbrZklN7qbjpkHpQyYlBY
```
### 5.2 Add Google Account redirect URI
In the Google Cloud Console (OAuth 2.0 Client IDs), add:
```
https://house.hubris.network/auth/google/callback
```
to the authorized redirect URIs for the existing client ID.
### 5.3 Set env vars in Yuvomi `.env`
```
GOOGLE_CLIENT_ID=119823214387-32f20ed3imesiv7uh5si7p3rou9fros4.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=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
```
Note: Yuvomi's Google Calendar integration uses the OAuth flow to get its own
refresh token — it doesn't reuse the trmnl refresh token. The first-time setup
in Yuvomi Settings → Calendar → Google Calendar will prompt for authorization.
### 5.4 Restart and verify
```
pct exec 105 -- bash -c 'cd /opt/yuvomi && docker compose restart'
```
Then in Yuvomi UI: Settings → Calendar → Connect Google Calendar → authorize.
---
## Phase 6 — Paperless integration (decide approach first)
### 6.1 If using as standalone documents module (no Paperless bridge)
No action needed. Yuvomi's Documents module works out of the box — docs stored
in encrypted SQLite.
### 6.2 If using WebDAV bridge to Paperless consumption
- Paperless consumes documents from `/mnt/library/documents/consume/`
- Point Yuvomi's WebDAV document storage at a WebDAV server serving that dir
- Options: run a lightweight WebDAV container on paperless LXC (103), or use
Nextcloud's WebDAV if documents are already in `/mnt/library`
Set env vars:
```
DOCUMENT_STORAGE_WEBDAV_ENABLED=true
DOCUMENT_STORAGE_WEBDAV_URL=http://192.168.8.130:8000/... # or WebDAV server
DOCUMENT_STORAGE_WEBDAV_USERNAME=...
DOCUMENT_STORAGE_WEBDAV_PASSWORD=...
DOCUMENT_STORAGE_WEBDAV_ALLOW_PRIVATE_NETWORK=true
```
### 6.3 If building a custom module
Write a Yuvomi module (client-side JS + module.json) that reads from
Paperless API at `https://paperless.hubris.network/api/` using a Paperless
API token. See `modules/MODULES.md` in the Yuvomi repo for the module format.
---
## Phase 7 — Backup & maintenance
### 7.1 Data persistence
Yuvomi stores everything in a single SQLCipher-encrypted SQLite file at
`/opt/yuvomi/data/oikos.db`. This is the only file needed for backup.
### 7.2 Add to homelab context
- Create `/opt/homelab-context/containers/129-yuvomi.md` (or `.../house.md`)
- Update `inventory.yaml` if using a new LXC
- Add changelog entries to caddy (121) and ingress docs
- Update `plans/index.md` → mark this plan `Done`
### 7.3 Schedule backup
Add a cron (or existing backup system) for `/opt/yuvomi/data/` if not already
covered by the host-level backup scheme.
---
## Summary of steps
| Phase | What | Who/Where |
|-------|------|-----------|
| 0 | Clarify Paperless approach + deployment target | dtoro |
| 1 | Docker Compose on apps LXC, start container | Hermes |
| 2 | Caddy block + DNS record for `house.hubris.network` | Hermes |
| 3 | VPS traefik router + cert sync for public exposure | Hermes |
| 4 | Authentik OIDC provider + env vars | Hermes (needs admin UI) |
| 5 | Google Calendar tokens + redirect URI | Hermes + dtoro (Google Cloud Console) |
| 6 | Paperless integration (depends on Phase 0 decision) | Hermes |
| 7 | Documentation, backup, inventory updates | Hermes |
---
## Duration estimate
| Phase | Time | Notes |
|-------|------|-------|
| Phase 1 | ~15 min | Download, config, startup |
| Phase 2 | ~10 min | Caddy + DNS |
| Phase 3 | ~15 min | VPS traefik + cert sync |
| Phase 4 | ~20 min | Authentik provider setup + env |
| Phase 5 | ~10 min + Google UI | Redirect URI takes 1 min in console |
| Phase 6 | TBD | Depends on chosen approach |
| Phase 7 | ~10 min | Doc + inventory updates |
| **Total** | **~1.5h + Phase 6** | |
## Rollback
If anything goes wrong:
```bash
# Stop and remove container
pct exec 105 -- bash -c 'cd /opt/yuvomi && docker compose down'
# Remove Caddy block, commit, push — auto-deploys
# Remove VPS traefik router, restart netbird-traefik
# Remove cert sync entry
# Remove DNS record
```

View File

@@ -0,0 +1,246 @@
# Plan: Migrate Booklore → Grimmory (LXC 130)
**Status:** in-progress
**Date:** 2026-06-29
**Goal:** Replace Booklore on shared apps LXC 105 with Grimmory on a dedicated LXC 130. Grimmory is the community fork/successor of Booklore with the same database schema and port, so the migration is a near-drop-in swap.
---
## Pre-flight checklist
- [ ] Note Booklore MariaDB credentials from Portainer compose on LXC 105 (`DATABASE_PASSWORD`, `MYSQL_ROOT_PASSWORD`)
- [ ] Confirm `/mnt/library/books` is readable on LXC 105 (`ls /mnt/library/books | head`)
---
## Step 1 — Dump Booklore MariaDB
```bash
# On hubris — find the MariaDB container name
pct exec 105 -- docker ps --format '{{.Names}}' | grep -i maria
# Dump (replace <CONTAINER> and <PASSWORD> from Portainer compose)
pct exec 105 -- docker exec <CONTAINER> \
mysqldump -u grimmory -p<PASSWORD> grimmory \
> /tmp/booklore-$(date +%Y%m%d).sql
# Pull to hubris root for safekeeping
pct pull 105 /tmp/booklore-$(date +%Y%m%d).sql /root/
```
---
## Step 2 — Create LXC 130
```bash
# On hubris — list available Debian 13 templates
pveam list local | grep debian-13
# Create LXC
pct create 130 local:vztmpl/debian-13-standard_13.0-1_amd64.tar.zst \
--hostname grimmory \
--ostype debian \
--unprivileged 0 \
--cores 1 --memory 2048 --rootfs local-lvm:16 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
--onboot 1 \
--mp0 /mnt/library,mp=/mnt/library \
--features nesting=1
pct start 130
pct exec 130 -- apt-get update -qq
```
Set the static IP directly in PVE (same pattern as all other LXCs — no Fritz!Box reservation needed):
```bash
pct set 130 --net0 name=eth0,bridge=vmbr0,ip=192.168.8.213/24,gw=192.168.8.1
pct reboot 130
```
---
## Step 3 — Bootstrap LXC 130
```bash
pct exec 130 -- bash -c '
# Media group
groupadd -g 10000 media
# Docker
apt-get install -y ca-certificates curl
curl -fsSL https://get.docker.com | sh
systemctl enable --now docker
'
```
---
## Step 4 — Deploy Grimmory compose
```bash
pct exec 130 -- mkdir -p /opt/grimmory/mariadb/config /opt/grimmory/data /opt/grimmory/bookdrop
```
Write `/opt/grimmory/docker-compose.yml` on LXC 130:
```yaml
services:
grimmory:
image: ghcr.io/grimmory-tools/grimmory:latest
container_name: grimmory
restart: unless-stopped
ports:
- "192.168.8.213:6060:6060"
volumes:
- ./data:/app/data
- /mnt/library/books:/books
- ./bookdrop:/bookdrop
environment:
- DATABASE_URL=jdbc:mariadb://mariadb:3306/grimmory
- DATABASE_USERNAME=grimmory
- DATABASE_PASSWORD=${GRIMMORY_DB_PASSWORD}
- USER_ID=0
- GROUP_ID=10000
- TZ=Europe/Berlin
- FORCE_DISABLE_OIDC=false
extra_hosts:
- "auth.hubris.network:192.168.8.175"
depends_on:
mariadb:
condition: service_healthy
mariadb:
image: lscr.io/linuxserver/mariadb:11.4.8
container_name: grimmory-mariadb
restart: unless-stopped
volumes:
- ./mariadb/config:/config
environment:
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- MYSQL_DATABASE=grimmory
- MYSQL_USER=grimmory
- MYSQL_PASSWORD=${GRIMMORY_DB_PASSWORD}
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
interval: 10s
timeout: 5s
retries: 5
```
Write `/opt/grimmory/.env` on LXC 130 (fill real passwords):
```
GRIMMORY_DB_PASSWORD=<same_password_as_booklore>
MYSQL_ROOT_PASSWORD=<root_password>
```
Start:
```bash
pct exec 130 -- bash -c 'cd /opt/grimmory && docker compose up -d mariadb'
# wait ~15s for MariaDB to init, then start grimmory
pct exec 130 -- bash -c 'cd /opt/grimmory && docker compose up -d'
```
Verify Grimmory responds (before DB restore — will show setup wizard):
```bash
curl -s -o /dev/null -w '%{http_code}' http://192.168.8.213:6060
# expect 200 or 302
```
---
## Step 5 — Restore Booklore DB
```bash
# Stop Grimmory (keep MariaDB running)
pct exec 130 -- docker stop grimmory
# Copy dump to LXC 130
pct push 130 /root/booklore-$(date +%Y%m%d).sql /tmp/booklore.sql
# Restore (replace <PASSWORD>)
pct exec 130 -- docker exec -i grimmory-mariadb \
mysql -u grimmory -p<GRIMMORY_DB_PASSWORD> grimmory \
< /tmp/booklore.sql
# Restart Grimmory
pct exec 130 -- docker start grimmory
```
Verify books appear:
```bash
curl -s http://192.168.8.213:6060 | grep -i grimmory
```
---
## Step 6 — Authentik OIDC update
In Authentik Admin UI (`https://auth.hubris.network`):
1. Providers → find `Booklore` provider
2. Edit:
- Name: `Grimmory`
- Client Type: **Public** (Grimmory uses PKCE — no secret needed)
- Redirect URIs: `https://books.hubris.network/oauth2-callback`
- Scopes: openid, profile, email, offline_access
- Back-channel logout URL: `http://192.168.8.213:6060/api/v1/auth/oidc/backchannel-logout`
3. Note the **Client ID** and **Application slug** for Grimmory's OIDC settings
In Grimmory Admin UI (`http://192.168.8.213:6060` → Settings → Authentication → OIDC):
- Issuer URI: `https://auth.hubris.network/application/o/<slug>/` (trailing slash required!)
- Client ID: (from Authentik)
- Client Secret: leave blank (PKCE)
- Click **Test Connection** — all checks should pass (container reaches Authentik via extra_hosts)
---
## Step 7 — Caddy cutover
In the `dtoro/caddy-conf` repo, update `books.hubris.network`:
```caddy
books.hubris.network {
reverse_proxy 192.168.8.213:6060
}
```
Git push → Caddy webhook auto-reloads (see [caddy (121)](../containers/121-caddy.md)).
Test:
```bash
curl -s -o /dev/null -w '%{http_code}\n' https://books.hubris.network
```
---
## Step 8 — Enroll as homelab client
```bash
homelab client add grimmory --lan-ip 192.168.8.213
```
Commits the `age_pubkey` back to `inventory.yaml`.
---
## Step 9 — Verify end-to-end
- [ ] `https://books.hubris.network` loads Grimmory
- [ ] OIDC login via Authentik works
- [ ] Library books from `/mnt/library/books` are visible
- [ ] Reading progress / metadata from Booklore is present
---
## Step 10 — Decommission Booklore on LXC 105
1. Portainer → navigate to the Booklore stack → Stop → Remove
2. Keep the dump at `/root/booklore-<date>.sql` on hubris (or archive to `/mnt/library/documents/`)
---
## Rollback
If something goes wrong before Caddy cutover: no user-visible impact, just shut down LXC 130.
If Caddy already cut over: revert the `books.hubris.network` block to `192.168.8.205:6060` and push. Booklore still running on LXC 105 until Portainer stack is removed.

19
plans/index.md Normal file
View File

@@ -0,0 +1,19 @@
# Plans
Pre-flight runbooks for planned changes that haven't happened yet. Once executed, move the outcome to `investigations/` (if anything interesting happened) or just a changelog entry on the affected node pages.
## Index
| Date | Title | Status |
| ---- | ----- | ------ |
| 2026-06-24 | [TRMNL plugins LXC (128) + middleware deploy pipeline](2026-06-24-trmnl-plugins-lxc.md) | In Progress |
| 2026-06-25 | [Yuvomi deployment — house.hubris.network](2026-06-25-yuvomi-deployment.md) | Done |
| 2026-06-24 | [TRMNL plugins LXC (128) + middleware deploy pipeline](2026-06-24-trmnl-plugins-lxc.md) | In Progress |
| 2026-06-01 | [Slate AX → SODOLA managed switch migration](2026-06-01-slate-ax-to-sodola-migration.md) | Done |
## Conventions
- File name: `YYYY-MM-DD-<slug>.md`. Use the *target* date if known, otherwise the planning date.
- Status: `Planned``In Progress``Done` (update index entry as it moves).
- When done: add a changelog entry on every affected node page, then update status to `Done` here. If things went sideways, open an investigation.
- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.

41
scripts/check-caddy-backends.sh Executable file
View File

@@ -0,0 +1,41 @@
#!/bin/bash
# check-caddy-backends.sh — validate all Caddy reverse_proxy targets are reachable
# Run this on hubris (192.168.8.77) or any host on the homelab LAN.
# Returns non-zero if any backend is unreachable.
# Uses curl with a short timeout for reliable TCP checks.
set -o pipefail
CADDY_HOST="192.168.8.175"
echo "=== Caddy backend connectivity check ==="
echo "Date: $(date)"
echo ""
TMPFILE=$(mktemp /tmp/caddy-checks.XXXXXX)
trap "rm -f $TMPFILE" EXIT
# Extract unique IP:port targets from Caddyfile
ssh root@"$CADDY_HOST" "grep reverse_proxy /etc/caddy/Caddyfile \
| grep -oP '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d+' \
| sort -u" > "$TMPFILE"
TOTAL=0
FAILED=0
while read -r target; do
[ -z "$target" ] && continue
TOTAL=$((TOTAL + 1))
# Use curl with 3s connect timeout for reliable TCP check
if timeout 3 curl -s -o /dev/null --connect-timeout 2 "$target" 2>/dev/null; then
echo "$target"
elif timeout 2 bash -c "echo >/dev/tcp/${target/:/\/}" 2>/dev/null; then
echo " ⚠️ $target — port open, no HTTP response"
else
echo "$target — unreachable"
FAILED=$((FAILED + 1))
fi
done < "$TMPFILE"
echo ""
echo "Checked $TOTAL targets, $FAILED failures"
exit $FAILED

49
scripts/dns-sync.py Normal file
View File

@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""Sync Technitium hubris.network A-records -> NetBird managed DNS zone.
Technitium is the single authoring source; this reconciles NetBird to match."""
import json, urllib.request, urllib.parse, ssl, sys
NB_API = "https://netbird.hubris.network/api"
ZONE_ID = "d7gaad00qfrc73er3n1g"
NB_TOKEN = open("/opt/dns-sync/netbird-token").read().strip()
TECH = "http://127.0.0.1:5380/api"
TECH_PW = open("/opt/technitium/admin_password.txt").read().strip()
ctx = ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
def jget(url):
with urllib.request.urlopen(url, context=ctx, timeout=15) as r: return json.load(r)
def nb(method, path, data=None):
req = urllib.request.Request(NB_API+path, method=method)
req.add_header("Authorization", "Token "+NB_TOKEN)
body=None
if data is not None:
req.add_header("Content-Type","application/json"); body=json.dumps(data).encode()
with urllib.request.urlopen(req, data=body, context=ctx, timeout=15) as r:
t=r.read(); return json.loads(t) if t else {}
# 1. source of truth: Technitium named A-records
tok = jget(f"{TECH}/user/login?user=admin&pass={urllib.parse.quote(TECH_PW)}&includeInfo=false")["token"]
recs = jget(f"{TECH}/zones/records/get?token={tok}&zone=hubris.network&domain=hubris.network&listZone=true")["response"]["records"]
source = { r["name"]: r["rData"]["ipAddress"]
for r in recs
if r.get("type")=="A" and not r["name"].startswith("*") and r["name"]!="hubris.network" }
# 2. current NetBird records
cur = { r["name"]: (r["id"], r["content"]) for r in nb("GET", f"/dns/zones/{ZONE_ID}/records") if r.get("type")=="A" }
# 3. reconcile
c=u=d=0; changes=[]
for name, ip in source.items():
if name not in cur:
nb("POST", f"/dns/zones/{ZONE_ID}/records", {"name":name,"type":"A","content":ip,"ttl":300}); c+=1; changes.append(f"+ {name} {ip}")
elif cur[name][1] != ip:
nb("DELETE", f"/dns/zones/{ZONE_ID}/records/{cur[name][0]}")
nb("POST", f"/dns/zones/{ZONE_ID}/records", {"name":name,"type":"A","content":ip,"ttl":300}); u+=1; changes.append(f"~ {name} {cur[name][1]}->{ip}")
for name,(rid,ip) in cur.items():
if name not in source:
nb("DELETE", f"/dns/zones/{ZONE_ID}/records/{rid}"); d+=1; changes.append(f"- {name} {ip}")
print(f"dns-sync: {len(source)} source records | +{c} ~{u} -{d}")
for ch in changes: print(" "+ch)

View File

@@ -1,15 +1,14 @@
[Unit]
Description=Pull /opt/homelab-context from Gitea
Description=Pull /opt/homelab-context from Gitea + auto-setup tooling
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/git -C /opt/homelab-context pull --ff-only --quiet
ExecStart=/usr/bin/env bash /opt/homelab-context/tools/post-pull.sh
TimeoutStartSec=60
# Don't fail aggressively — a missed pull just retries next tick.
SuccessExitStatus=0 1
Nice=10
[Install]
WantedBy=multi-user.target
WantedBy=multi-user.target

View File

@@ -7,12 +7,9 @@
<key>ProgramArguments</key>
<array>
<string>/usr/bin/git</string>
<string>-C</string>
<string>/opt/homelab-context</string>
<string>pull</string>
<string>--ff-only</string>
<string>--quiet</string>
<string>/usr/bin/env</string>
<string>bash</string>
<string>/opt/homelab-context/tools/post-pull.sh</string>
</array>
<key>StartInterval</key>
@@ -29,4 +26,4 @@
<key>Nice</key>
<integer>10</integer>
</dict>
</plist>
</plist>

View File

@@ -13,29 +13,47 @@ sops:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQL1FueldvT09lV011Q0hz
MlJ1YkxIYXZDNDVQQWo1MFQrRlNBcXNIYVNRCllXYVZrczdBNHdPemJybUVlQkVl
d0hKUTVsTyszOUFrY1BQQllSM2JWa2sKLS0tIDFISUw3T1lLYm10YXZaeFlTMWVj
ZHFWN0ZQTlhIc0pGMThoWWViN2lmVUUKsWRBHqoRVmY2Clcro5aKhQj9yX5W4Kh3
arC5FKBoZ/8KlcS7zxC9vIISOqlQu7MCKA8L//HX/gkfcz77aS6t1w==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaRWN5ZzIrNCtYeVRVNWdB
b0M3SHF1WVRTZzhhZ3NqK0llaVQ4SWxvSndNClprREJjWm80aGViTktSbGV0ZjFJ
R2thaXNXLzM5UmY2UHhuYXNITys2eGcKLS0tIGJkZ0RJS1lFTUhCZFlQTmd6M0NH
d0N1RWRRNTkzcGR2Zit3cnN2TXd0dm8KpvHCBO1gejHD0okrivBzC0qmfcFDQgHY
8ZLi83Mv7PflCZpcv67d7mai1F89DGDCsoo7TMGHh6rNU+Mpy/g5+A==
-----END AGE ENCRYPTED FILE-----
- recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFTVB1eWh5M3RUVFRubUFE
OTRrbWRnUDhsVGlLZkFWckRXZ08xTkxCMlVNCnFHcU9aMklma3RJZnZ5OHUxQVYx
SWFZMEJSNHExT0lsdW9FNkx0T3ZNQ0UKLS0tIEpseGdCbTJudUNWUUJGYUg1STJV
Qm9QU3Z2NHhiRWh6VnY5cDRRQVk2ZUEKktovOMBi2eT4Hxwm8i8gy4GYPpOZ3nIy
cDM6yBpUQMybiY3jMji41ELg12vF2kQ286W6lQKURdl9yYoGz0SfqQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2YVQ0WnVDUEl3eU1kdGw1
aE81eU9iUExJZkg2ZElNR3Z6b3VCWkZSNHk4CkVGZFVZUnpxZEZRd2t3eDBhcmd3
VWIwSGhWS2JKSXQ2K0xva1NTYVB5bDAKLS0tIEFEc0Q4TXBXeWV6eW93eWxSMkdM
TllXK2xIL0R6T3dHOGdDNVdDTzdycHMKntYg1r5tOHWxpkce89ixirQBOBpIdAuN
PBAdd7vY9zL4tq+AB5Goz7gj2I56Fw3tM970YX/JaThCCiyquk2OPQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHd0NhOC9FWmVSL0ExKzIz
L1hiU3RhKzh4SzFCNURhWFAxZTF4QWJIdmhBClQ4SzBEY2VweUR5UnJJSkk1OUVj
MWp5WUtQcE0yNFh5R0JoTGpUaWJiU0UKLS0tIDFQbDZoUXlPSURndTNEN2d0TjJS
djVvR2hGSllUNlZJNURrMG43bmZlU0EKl7SfN6/JGbzDA3YVrRnua5QK8xEdkWoB
ey5Yf4Zg0obqYIS5SX8QC/YtoeMANVkKtqVTOVE2PYLJtrC6l7vKxQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDTnlKU2J0RFU0YU43TzJt
MGpQdy9Gb1J5cnd2K2Nvc0FnTUVCSzZudHpvCndmUThtaStCTWFqUzhMVlZ3dmZZ
ZHpsTUFmWEVCYWRpT29RNHRVSk5iQlkKLS0tIERpY1RUalA4UkQvSFR4SXNlZ3Z0
VG1xSDJCV0ZZR3gwczcxYlNsLzArOVkKvnPBHgk24SBvAsQWw+2FxCVLUdKMNyb3
D/Zk0EnwdT6JitDTHcQ2PlDoKBSK6BpW2Bk1gnaC9doiXSz6ykziWw==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Y2c3Z1NUT0NxQlpabTlP
SGtwK3JwcHorUDRXKzlFTjBhUno0SUsrK0VzCkdVQzAzSHJuRG9BRmJ1QW9aWFk0
MHcxL044b2VxYnpOTlJtNFN3NjNsek0KLS0tIGovbGtSOWdmclVKMytHZU5tUmhl
WFJLbFVlQjQ5U2pQenhWN1dUd1dWcUUKxb2yYZFys0AqY7+M0/gTIDK+1Bl0FIEj
DSqbAb2UYfNXKVEScJl4QeRLEYuTUKejD0WT4cYJzyjRAZzc7PnykA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRU0RhSWZpTVdFelVTeXF1
aXhPSGFLdWxMQ2J6UkVEQlN5aUJLYm0xN21ZCnhTYWpvQ1B1Z0dnamQ5SWk3NjVR
bGs5TGdZb2NkVkROS0hYQ2ZDVXpIWVEKLS0tIENyblo2L203OUNISUVKMkhXWmg1
NllKUW1mUVRldU03RUM1dThUeDJ5WjAKiy7aGBpQImoMdmFkNwR33Aksx8YDrjOg
DfOpNOiGHWWNfmLHngvGKzCwjqV0nsl1rM2khXcZ21w/9L8USNCceQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-05-20T16:24:18Z"
mac: ENC[AES256_GCM,data:d3qcjEqpPIFeut634ImvISJLdit0MQWcdFYomrsqriqJ0NUKdyq3XCk85+vvPvDWikB4WEApk6HMXssSY8mhDci95q5Ssmr+JRAhLebMZjs9yXUf9A7vXpNFswmdldu3CKBiDrhm2GE08qUfsCkcf3jjihEqsfHhxMuWSd5fZpU=,iv:VZgvZ3bBArRChgRX38U6/43XoX5aTnts7Kd4S9spBPk=,tag:/5oOw5yvulvFT6KXv2zj0g==,type:str]

View File

@@ -9,29 +9,47 @@ sops:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCbWhQa2RreXl3Ni9zZExQ
dzB5Q1hSZzJSb1NlaEx3bEFCK2x6Ty82ZEJ3Cng3Q1NmKzhpdmpvTWFOL0lUb3F6
emtvNCt3elQwTndoNzRIM2FoZTlpTVUKLS0tIER6STdwVUtnL0lreFh3Z09FYlBo
azJuWjVuNGJURTNPYU1aWWZGM0dpc2cKFM8AjHFg68fedY0Xrg2U69+Qxc5Pgn6f
QipSsaXrCohzNKieqCG9maKuLcGrgMQ1OJqZQm+hfit6CRKUmVHRkw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFMWU0K3V5NmZBNUk2TWNO
NmdRWjlBV3p5dWNhUWhTOFlEdCtkMVh5STJBCkwzbUNkenJjei9TVWVUMi81d0ZX
b3dTTnJudVZiV0YzajNPcXlUWUtxdWMKLS0tIEo1eUR3dmJQRWM0bHRtMFc5Q290
QmwyMk44SVBXQ0k5QjN3ZWtNS1FSWTgKcc/F1HfMfnwun+mIUq1Ds+DZPk7F1ohe
OIVt5aJliApDYesRJ14K1ZEdi4YqCqO+1cLi69iWKHNT3PcOYrepUA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2TVpWeC9FcmUxckNadmtK
d3BwZnExR3VqeXBQSGhNdStPVXM2MkdjQXhBCkpHMmxPZ3JWaVVvUGwrVjZadXlR
V3VaR2l5Zm9zb1VrREphM09RMU5mR0UKLS0tIGlYZm5XbTA2WUcvWU9FWlZYSWNZ
L1JYYmJzLyt6bzJqdlk2TjYvNUdVVTgKlj1b9zQ9cBG3YQI2t4fjz3t9ySBxQ15w
qxcyGspEQ9F+4OtoRQP0FCfmJIBNxvAoT2I5TjspH6dl2o1ID+332Q==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFNVUyajV1NFo5VFRyMDlG
dWFlRlVJZDlJTHdYOFJLRi81OU9BdkI2RG13ClMwLzFHQ0VxSXJzQ3V2S2pjVFRS
dEQvK1VMZ3FDZDkxSHdYb3NHeHE4QTgKLS0tIDNyZnV5OFVQNXVEUUlzK3RYMytt
Nnl5bTZuMWFLY28vZnBPWlFEYmFyK00KtwcrF0W91FIi+9nn8rF2G8xP2/ca2h3R
DagqhSmpaEC7QHtYGP9SAChhBrDuWdVtK0Gdbc4m/31gW7Fh815zfA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFa1NwSGlhUzgxdjI2V0Ew
VVNRS1o2L00wM0QvdTFXelFLVnBrQ2ZHRHprCks0Z2ZQbzdQNG5hVFlCZXFrSG5B
cG5Qb2JUbnk4ekdJSER2MU5NRGZ6NEUKLS0tIDF4cjNzWnBLaXJNS1RHbVozVUtp
bTkzM21jNndEdDV0RmtxSVZ5SkdQcGMK+MZNNAfnnXq7VjIvaDtJEaImk0mecErp
gzV3b0MOKc2eTiuaWRNH1yAhJOFTK9iQfC7dDUI8npE+zLOgpLEZ9w==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrTlFZYVFHZHNCUkllZC94
ZUFaS3JZTmdPclA4dlR0QklxMmVxNWNyZUhvCnVoY0YvWGlsbm9CaURkQzZUMTRs
ZVNmcDJxZWN2WG9nUUNtT0I4S1FIem8KLS0tIHBRWFhLN3p2Z0ZaWDEyeEw4WUhs
Z0RwQ290OWloQzh1eXVLenlrOTZyRzQKQEnY4KC8ReKGFDzklK/A6uIGMRIhMfLW
IzogPQiWYcDerMDd14kktBOcbjKRkI9gPSDZdmieS9z/wA5J37DNNA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCSUlxdTFhR1dxaDU2MGdL
dVJoYVQwZm9DbEVOYXd4Ymd0dk56R283WjJFCi9MS05CVkxYY2NPc0pjeGNacm8w
ZDZGYUMrdDdKS05BckhoRDl4OTVxYjgKLS0tIHlPaUg4Yk96UVJacDFhSm1FaGZo
RVVSWVRteGcwN2dHOVhuWFpmU0p5aEkKXetFyc9PKB6dCljl0c/+JJrEyDvYJP0Y
1fbPf7WBJsLTaQgrjsGHU4wqlg0Se5GHMeriOPYaRjiL9locKW2SiQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwcWxaenJybWpqQ0RpM3ov
cEszZDI4YkswY1hpUmZDcmZ1Ky84N0xUUlh3CjJLcm5nVE0rSHhyeEhNSmFIaE00
RkpqVHFEdVR5VmdETnhnQ0JiSlFyNTQKLS0tIGNXdkMzaHBtTWU3azdDVm15NkpZ
TWlYZXdBYTROcHBqMCtFcEtJRUJtOEEKdBbB5a5teUnvIYaLJaLp3KZkEXhnkFor
pbnL/WrHQViLdu8pJ5nscIO9ryQt7dS6aAu6gP3Tbs7XdREOTFgKmg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-05-20T14:39:14Z"
mac: ENC[AES256_GCM,data:CAoIW1sJkIbNoXMaVlXwfHdREzATNqDKpSN5ecaSiRARThB0/tcLNbkazXKyV1Rx5b+b3s+yCpBzwmNnH6ytaCzG5pKPcnfd6YJq7C5rm/EQHyaVGrPdBsSfqiz4yBXKClej//v3+qkD5Ls2PMppv5KmqigOKNQC/ot3ht2c7lM=,iv:eOmDDclksr9f9CDu6dJprSFwlWuoggzVFjyfqWf5Uhg=,tag:IU3sMlIi92ohM86zOIJdAg==,type:str]

View File

@@ -0,0 +1,50 @@
members:
"+491726924525": ENC[AES256_GCM,data:RA==,iv:FLOi3HmBPFtS8BAqPR9Epihs868OwAWs1t2LGjg9kGU=,tag:bf8PGNipOeepfnzHUE5KMA==,type:str]
"+4917622791635": ENC[AES256_GCM,data:Mw==,iv:onlvwM9nlAlXnL6f+BTRZlOi4+C6pryyiay421GY9WA=,tag:KiYrTcRlBNbB5mYYlb7GHg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmYW9VS2FLb1BTVTk2OFZZ
Wnd1U2tnanR5N2d6SmNqdFYrekpzZTRMRGpZCjZhREcrY29qc2pFZUhnQnc3U1dE
aU5Kd1Robkk4OVd2M0ZkalBWMlQzR0EKLS0tIDJOTjkwR2htaDhDR0hqN0R4cCtB
dldLSHBOVjZxNVFZVUtkZm1INkdvM1kK2iaqr75MZ+4QlWaNOjH1X11zHbi4Ahy0
ely7Fakx6wSoQtl803q6UyNJyIqCOavJKoLQw48FhAl2yGmv9KsvrQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4eXFvMjZXVWkvZnNLSGhS
eWJ3Q0M0QzlnYnJ2dTVhZGRtZ3Y4Ylh5NHc0CkVTRnNLa0NzK2czcjFSSmgxQ081
ZVo2MzR4S2ZlY1AxZC8rWTJ4d25vekUKLS0tIEFnK2gvTVBUWk5jOW1NaUtXTmRL
U1FCaHZjaTUwZ1RGVlZkZ0JzT1NOek0KceYGFDlpcpAQte0yqebyuQBxr00/Rurh
McQ607wCQWbGKkwkHCDi1VGqqokvbU7MRT9iOBLLBlxEF+KS9UiYuA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6aXRkd3d4ZTV0RmltR1JN
bFg4VU0vMDVuR0cxc2I0b0NRRUQyQlE0RHk4CnpPaUpGMEh0Unk4d3BDeE84Q0px
RnZnUHRQa3BJQUpGWUFZd2dLczhmc3MKLS0tIE9NOUNacDM5dktuWWZ2WWRxajZu
Uk5Gb216dmdvUkJzYkoyWE5yQ044eWMKSGR5dDya0gyBWDMB+NSudXK38VQYP88y
M6lyeZUAJ3cFydjyE1PjyllBehX9tYh/rh9RS67y/zRSmkXH5e7eAg==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkM2I2MkV6NlhydURwYk8w
ajB1cEtiQjlLYkRGbDZXTHkzTWxuNW53NTJVCmJuMzJZSlNtU1BxVFNpOWpSVWp3
TmxOeTRhRmZlanp4QmVmRjRSOElHL3cKLS0tIHp1MzZQUlUxNXFlS1B6TWhQdzhE
ekFibGF3ZE95K2FjZmZmVGkzek54R1UK7Au0Op4P+JZjHIiJJ5effjxjeAJItXw1
Wrq8Z6D9NZTAZ2TBUGpdqIZcZhBiOuqxdNV88ZHOu8sjT1SLwm17GQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-06-28T14:50:45Z"
mac: ENC[AES256_GCM,data:7gWwrXY9q7zpiDZgFidEMa4VbxJ0uM7HdG6Ea8P4fbISgRnOx+aoAwznEQ+K/fIBsa1qtiR82lIxsNr6b44W4jmFF3+rPAm9DMs6ImX1KIVCW3GVptM9fMqOxnCI7RGMKKwWYxkvfE2lwl5EsBljqqlsuxDfX26etfqukIhbOIY=,iv:6iI7Q8wFKguICPtFvZWshok7BkEUUITp5XcSYt9hwFM=,tag:jIOzsy4jiSvuHh5GNQsjpQ==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4

View File

@@ -8,29 +8,47 @@ sops:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzVVZIQzlPak4xTVVPMm9w
M2tFbTFoQmpRUGFETjF1ZTQ3ajlEdGxWQmtZCkdkZzdhblFybGV3OEUrWERBWFUv
bTdXSnZFSU8vY1JJVTJjL3dLdWpiNGsKLS0tIHVlVmcrcTNZUGs4MmJtWkR0aTh3
Z0pTOWhsZWYyVW5TeXk5ZHhaQnZvZ3MKrnBt5T7WjSxGYvRc1olfhuMN6nOEJbbX
xoliPcKkGsBExXVgkpood+OdlH8dNAaT0z1+INzNiBAZ8SazZA4p9g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxZjBPQ3M3NkJwZ2dDZlA3
MGR2cTM1b0NBN2ZETFJZUW95M2lITXo4dlNJCjY5NEMrc2xNaEpCVUtkVHJYa1Rr
citGL3ZQQnlMR3BpVFpmU3d0eEhiZG8KLS0tIFZGejF0UDJZTUVUMU9KL2hySm9L
aEx2ZWRuTldDdFJrNUlmMHZVay9vRnMK0XMd/S3VSdD71eh2MUJGs8c7Jdxmfpua
p4ZN4Y4MARz2XMA8xqDpw1SrMevaceD5+p+R+zhunb99aFfhtV7WxQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjYk1SOVZxUTBWYUw4aUhy
WG93N2xVWDlEWnBqQ0JTRWcxc2tXUnppSFF3Cmp6L1g4OUliVlNFYUo2bEtNOHhm
aytHK0RFWmNpSWFMaGc1cnNXNlEwbTQKLS0tIHMxU3VXYTZHNWk3cWZFNXBNRElD
bE1CUHc1TFlsbGU2aTJjSmhwZENuUDgKvGHg2Df/eBw5akRPYFLvXhzh7P6jTOgj
E56n29EJ+p4kTkC8yVBci01qpMioL+Wx2Rt+X+0LWrGBzu5fic2U5g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3aXdya1RpSGdsQVNMYXlQ
cFhCZDUvTnFUMWdpQzAreGprTm9ZOE8vZUI4CmF4K2ZGQ29QOU1kMlhUV3N6RUNk
bzJwZDNmR2NITUlzY2hSekozdDBsQzAKLS0tIHI4cy9IR1gvWVo1emcxbzFURWVM
YzJvRjlNeE5KancrUUtSZndQNWxhYkEKAedJLVpc47R8rgm4YRyT7Z9G4tfbBnqQ
c1UObcIa551wR79n7vJvVb2cSLz3VEURe6sNi4OqJmg017qrMOS5Wg==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwSGdBYzlsYmc4dVJ5ZE8r
Q1FMR1NQL2lzZ1lmaTg1bGROa0hOcVk2ZTJvCllFMUtJdG5JVFhseWc5eXFsYk9p
Q2tIVm5yelAreVRrVkQrTnJ1N3FLclEKLS0tIGQ5dTdqZ3lKWU1KcHdHK1UwajIw
Znh5aXkwNmdyVi9EZlBzMUpTTW1VdjgKVVNGjiiKw3nLxq4YsCWoTS4R8wUD8wqD
awXSIOvZj5xiz6NvFk1X5T3H4XeEm6tKKzOpBvoVn5yl3zgn/bh5pA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrazAvYnovQ3ZFQ3YvcTh3
eGhYdFFTZ2ZkTElzVmFta3ROMGpCMlE3VEVvCjNXd2x2QTREQ2NVMEk5OU1JY2Z6
NHQzQVVaMVR0bWM4R0lxVVpwTEY4YXMKLS0tIHFTT1dRejNyQlVoUEREUnhlM09V
ZEF5WFlnYmgyVDRUYXBTVTgyV2laWkkKdE7hAUxtDZJt8P4LrfOomK8rMTlXeXd0
fyY29wRnXqn8s0IkpibWkByWLgyXHe1nNHHgsiPsxYMFbCRW1iwZoQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNdzIxYUlkY1YvTVdrREgz
bEhrTlpXRDFST2MxTUFpSWwrMzF6ZTVsUFRNCnB5WVJKaHNFL0hQak9ZbTEwNjF6
YmFEdVJ2YkEyS3BDQkUwZjBJNXh4UUUKLS0tIEpIcGV4N1R6aUU5S0l5dGZOdnNV
Njh1WlAwR2tsTXZoUS9JMmlYZm1ZUzQKcqXYn1KgSzAXS+m13/wLmeriNf+fTzY6
TSBcB3OxgqvYAWwZutXr6UScVauZPuatSjwE1va4HJQuhSVoCGwqcQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4blY0TEdsU3VoSmR3M0FK
dlg4S1JCRVpZRnc4SURNaWdSUjVQdnpCazJnCjhobHN2dmpGZzhrMWs4UUFFdVF6
RmtJb0VaQlFIeGhOMFA3Ui9iVmI1K2cKLS0tIHk3M3RhaUtSSm5vajFJUUprRzNa
c3JiWC9WNVRYOXJIQmMzUHZxbUZtS0EKsE5cG85lsDFABMbR/A80TQMlurf8qHDR
tVxZuMQPIXBdg6Ov72xnQeMGC//apWO40gnfjGn+P3oZGy7m8XJBuw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-05-21T20:13:20Z"
mac: ENC[AES256_GCM,data:bi4o17skIOtZoxJGzLFJ7IAv+X265qIhgnQz6wr4bch2xofwPZZfzECck0I0xDhcwxcDtx3VZ5E2VBdvHWk+iSan+clRq7735k9+DIpjdWaGxIHuylmelZl6aE5kE36UO7fQdnwmiqCPHxiVq/sKuqknFXkl1+FvO1ryrtSHQsU=,iv:xxm+5rK7o2c43iGS1j2Q274QCRsbDn+k+wskpp5tzeU=,tag:sZHk9eth/TwqM8tIWSrv3w==,type:str]

36
secrets/netbird-pat.yaml Normal file
View File

@@ -0,0 +1,36 @@
#ENC[AES256_GCM,data:yB1EyrABkGl4G+Mq67djNXCuySBS1TDZjKwtko8hfZi15br6+L9dlwhVTHrzvHAdGYOvkZu/dCssii0kKXC+3QeJws+SfwUV,iv:r7bb8QjkyfeaATPKN3VzO4H+ykQTyUKRMxoE9bmGo+E=,tag:TA0actRKQmYsgCMAcvQkjA==,type:comment]
#ENC[AES256_GCM,data:oVH2uanu0pyv7B0SKiaSKfAXWmKAimXKR2mHKi/+hvFIQGzf3mYTZ5zGMQdBCLoy,iv:pgQrEqMZmynEHfIIHHi52xSNwn0/uQ2ccnK0T9T/Z7U=,tag:8hdflKb6l7HlqHTto8eEyQ==,type:comment]
netbird_pat: ENC[AES256_GCM,data:cz9pdu7aLR+vKV9QbQ2PtKWf5j5PWxcHEWZ8IEseHS/qPdR4M8282g==,iv:3MCOU0q/7nqQCD6+7qWq++XR9uHi5QHZoenCIT2aYIk=,tag:m4ZE4pDycjwGxoLi1Li1bg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpYVlocHFacWxtTnhPeTV1
SnZqVUUwbXVId0dKNTdmaVQzSUcrT3gxYTA0CloyYm1oQlU3QXVUNVIxTXpGQkN6
cTBxTHViZ0E5bEF0dmx4S3l5WVJZOW8KLS0tIEQ5UmJGZG5aVDhGblFDREc5M1dR
ZlJ1UEVLeWFsQm0zQ0Q1Z2pTbEpFenMKsw+gfYZ+sby1crIyfoy+yGIaQ2WZ2KYU
YgsXY4Vt5b5DC0u/sHKaqXH3bw8ooBQ7tR4WX3JvU6emIAJU0rAuwg==
-----END AGE ENCRYPTED FILE-----
recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaM3NXTExiRzZ0VWdRR1RH
RUlBVVBTWFpUQWtCTnc2T0VqMDJITVFiOFNnClZRbElxRFlGb1orbThmdGI3dFRM
V1Jqck9pRzVjWGhVQlIvVUFLMXkyT1EKLS0tIGY5K3g3VDJjUHpjSTNrOTRBUEpz
dHBmQ0FpeHRSc1ZyMjJHdU1pM1FzbFEKqIJ0qC0SY0aSceh0ZhGVDMb/WCkh1JZZ
E3cP7r4Sa6rMZXRR4u23wTHnrckdHgwxSSXYYymy4ilXhgiFQ3ZTBA==
-----END AGE ENCRYPTED FILE-----
recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhSElYV3dkMU5hR1Vnb0tl
L1R0SG0zeFRuZUd2aTVVaU1leU14bHJjVWl3CjhibVp4NWFsazRYWkl4R1NWTUdk
TFdrOXc2OWY0aWxUb3NqM3R1T2xVVEkKLS0tIHlUZGUrRmM5VVVTOEZNYTBSZ2s0
VWVTdXlZcmJqeU4xc2NLSDY3MWNUUVkKix4IZEbRI7icZciI7bYzIvDiW4M3Nb7O
uopkLc9bC/E+qK9H7jfySkk/zIeW1s/s7fBEKB5utUVpvmKzUB24pA==
-----END AGE ENCRYPTED FILE-----
recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
lastmodified: "2026-06-03T19:57:46Z"
mac: ENC[AES256_GCM,data:zez2CaP1NPJJnUbEWT58If6jKxFiAudRRPKVJCzbjtp0loFohvOHt1nFY3g0XldqkNngsSZ83GOeJ9g+CvdD7/plkmjz0C04uHWahnNhjAwNRiN32xt31XFqEejScOHW4dbOunX8wXuQ0Q4SrSLTl5G9O/qbLwlwEiqp8KxwRWA=,iv:ctqQ9TDUM65POxn4v1iILubygU2lloCLz9u0bAP6gsY=,tag:05FpkqIhPRMX5hYopFNK2Q==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1

View File

@@ -0,0 +1,48 @@
api_key: ENC[AES256_GCM,data:TbPkuLCidS7cg52DYb0MOCui3TsaiwkOWyYLhp6gcvXI8TqvXej1tNsCF0TNAIr9vvnFaYno/UTEPiOKe/EQqR8uQMyWuvsmMg==,iv:bahy9ae4Qxvkv1OAjx6LwzK5ggqDh2GVHDj95PVP1Mc=,tag:J7PajLaeqFOdz3FESYZmSQ==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4TFVsb3VYWnVFcnhRL2Vz
SVp6cG9CYzdqSFNkTGhHSmtlRDJjVWwyK0RNCkdEcEJtNjB0aHBZQVZBbTlHMTla
cHduT1lyTWlESmQrVHB6NU9LTmU2MTQKLS0tIHkwQXM5TzZCRXFsOHdjYThZQ0Z2
bS9wOTBDenRMb2hGcmgrcTl5R3dnTG8KZfSI1pnfpcqtD0Z5N6hONSeuAaggCkZI
VJgh6TF7soOHbvddcNc5RINHQI6XLOZAeSOzvKm3zT8y5xe48Ff+3g==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkb3ZmZmpPZDAzZzB5SmNx
SHFKT3JENnVwZUttcFNYd3RoTHI2MCttQUNFCllZc2Z5Y1dSSnV0bW1LeGJvaURh
TlZGWjhseUdjU1FRWE41aTlMczI0U28KLS0tIHpzajRWWmlUU3FCaEFWMlBsVjBT
bGVRMGF2YjVwcVNCYkNEZVM1TmsyUncKYLscCS6BD3fwVmK3JnkISsEhXvhuRBnr
tJ+v7mcEXj4pz8FjIyY6MG3T+EXx6+HwuWtPyxbmEfqnxu2Ocfg5Dg==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUSCtGaUdaR2pVekZCUUkz
am1pRTJSdzgvUjJsTlpXVE0rVlczOFZ3aGpzClJNVXRJV2JqbHB5ZEUvUFJXR2cy
MUpad0ZMYUI2MmtoUm8zcG1RS01WcmMKLS0tIGsvVGhITGRsQkJXcUdmQllOVEVr
QnN5QVZXcFR0cDVab09hcEFiWjRoMmsKYo5ThhNgdp9GUXZkyF6JGGdWnmHmZK5c
QIqk5L5fYq+Wif8bKUJBEv2fGyJJZD74AOCFjQUIDgBnLx2tK3Kf4Q==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiVHRnMTZXK1FNWmFJZlF5
MWk5cjRHQ2NuRWJrMlhQVXhPL094elF1MW5VCkx5TklkR1VQSXc1OTljaXBFR2FQ
WEtudU5DNUxTWGNpbUFESjBoSkh6bVUKLS0tIFFiV2NkcktOdnF1djY2UC93OVdF
Unhpa0h5RU9xUHNic0hXVGhqSUthN3MKMAOF3gxkJVFU9aKqeUaViDs+Ka8NI4YQ
Bb8wPf9BcQe0NPjVMrxxP7L9c5oACiiNuA/46YxMYu7K1dOj0KapGw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-06-01T21:23:34Z"
mac: ENC[AES256_GCM,data:DXvUZUOMKQA0aDpN1foKy+aHhBh6daGSRJM1FagTh76qutk4XF9Een9iyy9sQ7olabCjf3oUezvU7XWB83bSDD3e23CFuweOl0RreRFqIsbXBi55Dbcxw8+9fvMyOK4PXu6mkT/PRao/XR5sSMqiHt3FoJbjazSaJlI5teeVXbM=,iv:IICO5ay+KqkTikfxW4kqB2qcSRpbsOP1NjGWtCB9uqI=,tag:MX2TqUxre+ff3RszCfXeFg==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4

View File

@@ -8,29 +8,47 @@ sops:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiaFhuUlZJMkxERlR1T2xL
dnVKMXNMNVBIdDFuYzJtTjJWZE9JZnlaQ1dJCnNNbzJKc3VXOUZ1QUFzRWhSUkdF
WVJlNUtJSytSS2h5NGQyMHA4d2U0c28KLS0tIFVhSGJQOWlEQlp6SlRmS0ZMZzJ4
Y2ZoVVVkeVNiYUNwWFFLd2VGaTR1VUkK9Dpk0kjuKoUh3zfVQV7qs/YTTg2BaOkg
kNYY01k9MftEwdtvpKk1ogzdHyhGFJ1yEepwK7se6W5KHDJXyVfA8w==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVZnc3SXVxWlJHV2FlZU91
Q2pVRXEvbFNOODVaQ2F3dzVva3o5RTI0TTFnCkI4MWN2TDR1dnR4OEhmZzhZMW1Z
a2wxaGR5MlVNWmh5YkZwb2dTbGJJZncKLS0tIEUyVkhYZWJiQUhNNTNtS095ellm
UnJBZGVPNC9lY0xKTGNldWs1aFJGSk0KvKBYSxJ0jwddf9OycUfrfYYmH8MoxIbC
oO8nk2sfY9enGziu+A0GKEmCIzhBkD+Cj/8jyqTH56NKFykKNpFQyg==
-----END AGE ENCRYPTED FILE-----
- recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4N2xmZEM3R2tMQjA4a1Fr
cWZ5a2lwUWlJZDdVSExnaHA3cEpubTVMWG1BCjNjTEJzL01PWm9xRWlEWUlWend5
TUdMMjUzNE1RN0h5bzJuQnl5QkJXVzAKLS0tIDZ6aVh6dXlxTTVPZDRKQUkyZmlk
a2FLZ2s1NTA1ZUljeFBhSDZEcGZLNlkKcX5b3dXcJZejeSP4TLr1cOXQj7YEjj0G
znXBdDt6c42qE5XS+LGciunf3MmYxt0xXDb11cnSSEVgn0VyqE2x6w==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZMXpiYlVadUgwei8xU0tv
bkpoK1RzS0tnYlJFN3hqRmJiK0Q2VVhyN1NjClZLRVZMcldNQ01yOVhQdnV2QkRE
bUl3YTQwSHJPSmlpeEpqUG5tSnZRZ0UKLS0tIFVodEZtaVFoYzFkdGd6VkdIeW5P
bHQvbnhYNnFuR0JGbkljdVRjVEZ6a3cKXccTdlgR94QmaeLGYnXXKOYFuJYkJ08Q
JYkpwRlYzYMfqlYNuANi2LYTZePQITIYZ+A9sHajCXQlUGjs9PCWsA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBybWtTZEw3NTdwMWZhOUMv
L1RNWGduU1BSM0FodWMwUEtHNkxSY3MvdGhzCm1qTGlnM2ZRM2RoekhZbjFzZ2tE
Qk1DNjlnMk01R1ZzUDBubTMvR2hPNmcKLS0tIGsxZ2lQMi9vY0Zpb3pwRHBSSm5t
OWtkeCtNNnpTVXV6S1p2Y2E0V2toL1EK3RI0RgM0SudRguOpOimke7niuX3cIIVi
X+zK75QbvLN7pe1pIIC2UOEmRB9BhgGMVssHxnLzTJv0LUKjDMGdbw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0bkppcWFWZkVxWGQzRzli
bkMvemZ4SGJodEUycGVxNlZFYzJqQlpnZml3Cm9UNFBjd29MTm1NTXI2OWljNFRZ
bGErNC8ydVFjTHNRcWpWTFFvNFQ1V2MKLS0tIHpXUzR1WGpPRnJNa3JEWFhFQ0N0
ZWhnMzNEaTJ5RHRaRi9lMDZyVkxFd00K/zh7XlSjvO/we8GNDhKvhmPBAPZTi1o9
kBruJ1OAFIJU4h/6dOmgGsj1Jxx/KZwMMAcHNzUBzm8x282MlFzM4g==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqUjZXMmpXTitCam1HbkIz
Y1UxY0xLR25GbnJucTMxV2x5Wk9SZFVVUVFrCkJzQmE4cUkxSGV3VjAyVlZEQkda
aW9SRmdGbWwxd0o3OTYvV2pmUHJxN2sKLS0tIG44cXZRTC9kSmJLcE1JSVVqWjYw
QkQzTFZZWno5dHhmN2pEWXpycDdvUjQKmcbWAZh85vSozzsX4CndtrxneA3em3JG
nhESflBPc/E2KckyTHOEJkPBq48UXiAvM3pY0s6FGgkhXHriTVJ7xA==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNdWNGZi9JWkZnSjF5eUZN
OUhtZ3B5aDVaV2dpU0liOHdtZitQSUcwbGpZCmttMEI3Mm94SnZwNEd4bUYrRHlD
aGJBa3pKZ0VRVXlkL1JHVG41U24rU1kKLS0tIFEvYlliOE9iSHhIbHR3RVZNc0dx
VDVkN3dYVEJXb2dPYjAzOFU5Z1VaU2MKzVcJ9/z4+9phsdwyowwK//Tpb0Ga8rwo
nTcLYw50wOKKG4ju1ISB5RuuUckU34dmUV7k4se9/oxnnectFwYVgQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-05-21T20:13:20Z"
mac: ENC[AES256_GCM,data:PXXEpDbJIoIwWuNwMetpALtxrcB7yJhDYqp6LSKV8WFQI2TJSDgDHiNQ2cngfB9PXKmUNNhlvhmB52Hi6WQSWQdXjiy79T1fOjMhbUAKmykTaZDrwvlrDGkPgiZ1cFSb8+hx/5aZw7YhdQO9+7LAmymeIldZIFEpEO2pvuPZ6MM=,iv:mJb3K8TnsnY183OX7o9pF7oVMDcwVU/rOgX74KTRpr4=,tag:f+M+FT0Vjozm7a8LFDI8lA==,type:str]

View File

@@ -0,0 +1,48 @@
token: ENC[AES256_GCM,data:LgiLFEsli412rVxSo9YU69wx9Lh7eDGBw6nmROh6Qbw7bYN8lylNjSDWkjNJ/UJvDg==,iv:vNmiV/D+SQaMtgwTK/mFDQIwSQ/L0JRatmoj5AWfBiw=,tag:W5P98TlmevQ0j051044LPQ==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQYVBwS250T0ZFa2VGbGow
R0MwMzNsY0t2N1ZBVnlnbkF6UEIyUStaUEhJCnBuNWNZT1VxRFlKNFBjOTZHNEd0
cCtxN1p3TjkzWUZNZkdZeXRJVEhaQzgKLS0tIFVSWUpVQ2k3VE83eHU4VmR4SnVH
RVgxdDNnKzMwM2c2bm1YS1ZSRDdsMVkK8PVncSpesaxIEz7a39rVBzB0V3RphmAA
ANV8fY4W4tLuxo+EhB90ciy2dnAHaKEbxHBWMMjN4u22yZRT4lPWQw==
-----END AGE ENCRYPTED FILE-----
- recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4Sk1hdENFQWIvZXhIK0U0
TmhCWkw5TWhSVXdIbDN5SmVNdHkwYm1aOGxvCmlnQmZYWitxUmdURzNYRDBFUUVM
TjdIN2FEd2ptWG5UVlFCb1IyVzFjMmsKLS0tIDJMZHlXVHFNUlY2bnhlS3hFT3FY
U2szdVB4MGxxUnpJN1BBaUhKVE5WQkEKz7BkFIfs2RSyAww73dH5+PyDNjo4Ocor
mX5pLhSRGGf13bfA5lMQiGvqtd3jw1c4zKVZD9BcthIk5H/oqallNQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmVWI2L05GbjdLR1k3a280
N2RRWmZsZWZPeWdmd3I3aThrV1ppOEw4VTJRCit6aXgwTVIwQldkSmpRNkM4UTh4
UEcyQjkrMFk2UGVBL1JZdGg2WDI3c0EKLS0tIGV5VkRWYWpXZXZGN1FXNmhTSmR5
cjQ0TWg0TkFmMnIyMDNBNVZXa1l0REUKVHdvbXFwLgMECi2JLg8aoYIkOWHwYyZh
v3mVVaaki5KceuyUhcGsdthNadrM0RDi89uUw8O2cfADdDgLlha/iQ==
-----END AGE ENCRYPTED FILE-----
- recipient: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnemkvTUxHZDNxYmJ4eGRK
Y09laWNnN0pXN1VHQzUzSWlJZGFqcmkwTWdJCjkxZG9pVVVzWXBTU0lpL0JFR2w3
TWhPTDNFSXhPTkVMVUpOY0xHMlN6RkkKLS0tIDB3d0ZyRVpGT3YzSXIrM1pPQ1Uz
QkpQYXRWaGtabnU5c1RpeE9vb0MwMWcKVRnpepDJ8a2ECD5uaK6O3fMKwWcLJP8a
uFWds5rvwDfrM1aNLXYbPIfXtqLx4fbxkoRZvHcuZ2/lYLDBFBrFlQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-06-28T14:02:13Z"
mac: ENC[AES256_GCM,data:NDjBNtSzGdm0+SqAvaUlltljXcyXzLQx3b9RUWZIlS/D3ox1BTO6t3t4WT8szbSnKcT51mJGJ8EIqppNWkmOpEaMNGdMw70DmGnf/IJAzScTpRyvhfsGlohCVaekMBjF+wg1uoB91vSKcTFq/wfUW79KICmxMF6woavRQ94yfQs=,iv:1f9sGKyTITSslqDXB3Khk/OGiPaJM2BreSEG6aTbZDs=,tag:UulDl/JQamg/p/q4esvtIA==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.9.4

View File

@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMtgjgpljZcu98NOW1ibry8ZBQSoUg0IaxK8iW10w2+ d.toro.v@pm.me

91
ssh/deploy-keys.sh Normal file
View File

@@ -0,0 +1,91 @@
#!/bin/bash
# deploy-keys.sh — deploy workstation SSH pubkeys to hubris + all LXCs
#
# Run from hubris (PVE host) as root.
# Reads keys from /opt/homelab-context/ssh/authorized_keys/*.pub
# and appends them to /root/.ssh/authorized_keys in each running LXC,
# and to /etc/pve/priv/authorized_keys on hubris.
#
# Idempotent — skips keys already present.
set -euo pipefail
KEYS_DIR="/opt/homelab-context/ssh/authorized_keys"
if ! [ -d "$KEYS_DIR" ]; then
echo "ERROR: $KEYS_DIR not found. Is homelab context synced?"
exit 1
fi
# Collect all pubkeys into a single variable, one per line
ALL_KEYS=""
for f in "$KEYS_DIR"/*.pub; do
[ -f "$f" ] || continue
key=$(cat "$f" | head -1)
ALL_KEYS="${ALL_KEYS}${key}
"
done
if [ -z "$ALL_KEYS" ]; then
echo "ERROR: no .pub files found in $KEYS_DIR"
exit 1
fi
echo "= Deploying SSH keys to hubris ="
AUTH_FILE="/etc/pve/priv/authorized_keys"
touch "$AUTH_FILE"
added=0
while IFS= read -r key; do
[ -z "$key" ] && continue
if ! grep -qF "$key" "$AUTH_FILE" 2>/dev/null; then
echo " + Adding key to hubris: ${key:0:40}..."
echo "$key" >> "$AUTH_FILE"
added=$((added + 1))
fi
done <<< "$ALL_KEYS"
echo " hubris: $added key(s) added"
# Deploy to LXCs
echo ""
echo "= Deploying SSH keys to LXCs ="
# pct list output: VMID Status Lock Name (Lock may be empty)
# Use awk to grab VMID, Status, and Name (last field)
pct list | tail -n +2 | while read -r line; do
vmid=$(echo "$line" | awk '{print $1}')
status=$(echo "$line" | awk '{print $2}')
name=$(echo "$line" | awk '{print $NF}')
# Skip header if any
[ "$vmid" = "VMID" ] && continue
[ -z "$vmid" ] && continue
if [ "$status" != "running" ]; then
echo " SKIP $name ($vmid): status=$status"
continue
fi
echo " -> $name ($vmid)"
# Ensure .ssh directory exists
pct exec "$vmid" -- mkdir -p /root/.ssh 2>/dev/null
# For each key, check if already present, append if not
while IFS= read -r key; do
[ -z "$key" ] && continue
if ! pct exec "$vmid" -- grep -qF "$key" /root/.ssh/authorized_keys 2>/dev/null; then
echo " + ${key:0:40}..."
# Use tee to append (pct exec preserves stdin)
echo "$key" | pct exec "$vmid" -- tee -a /root/.ssh/authorized_keys >/dev/null 2>&1
fi
done <<< "$ALL_KEYS"
# Fix permissions
pct exec "$vmid" -- chmod 600 /root/.ssh/authorized_keys 2>/dev/null || true
pct exec "$vmid" -- chmod 700 /root/.ssh 2>/dev/null || true
echo " <- $name done"
done
echo ""
echo "=== Deploy complete ==="
echo "Keys deployed. Test from any workstation with:"
echo " ssh root@<lxc-ip>"

208
ssh/gen-config.py Normal file
View File

@@ -0,0 +1,208 @@
#!/usr/bin/env python3
"""Generate ~/.ssh/config.d/homelab from inventory.yaml.
Usage:
python3 ssh/gen-config.py # print to stdout
python3 ssh/gen-config.py --install # write to ~/.ssh/config.d/homelab
The generated config provides short hostname aliases for every host in
the homelab inventory. LAN IPs are preferred (they work directly on-LAN
and are routed via Netbird 192.168.8.0/24 off-LAN); mesh FQDNs are
available as <name>-mesh fallbacks for roaming workstations.
"""
from __future__ import annotations
import argparse
import os
import sys
from pathlib import Path
try:
import yaml
except ImportError:
print("PyYAML required (pip install pyyaml)", file=sys.stderr)
sys.exit(2)
INVENTORY_PATH = Path(os.environ.get(
"HOMELAB_CONTEXT_DIR", "/opt/homelab-context"
)) / "inventory.yaml"
SSH_CONFIG_DIR = Path.home() / ".ssh" / "config.d"
SSH_CONFIG_FILE = SSH_CONFIG_DIR / "homelab"
SSH_MAIN_CONFIG = Path.home() / ".ssh" / "config"
INCLUDE_LINE = f"Include ~/.ssh/config.d/homelab"
# Hosts to skip in the generated config
SKIP_HOSTS = {"ludo-mini", "authentik"} # offline / not enrolled
def inventory() -> dict:
if not INVENTORY_PATH.exists():
print(f"ERROR: no inventory at {INVENTORY_PATH}", file=sys.stderr)
sys.exit(1)
return yaml.safe_load(INVENTORY_PATH.read_text())
def gen_config() -> str:
inv = inventory()
hosts = inv.get("hosts", {})
mesh_globals = inv.get("mesh", {})
lines: list[str] = []
lines.append("# Homelab SSH config — generated from inventory.yaml")
lines.append(f"# Source: {INVENTORY_PATH}")
lines.append("# Do not edit by hand. Run: homelab ssh-config")
lines.append("")
# --- Common defaults (before any Host block) ---
lines.append("# --- Defaults ---")
lines.append("Host *")
lines.append(" IdentityFile ~/.ssh/id_ed25519")
lines.append(" IdentitiesOnly yes")
lines.append(" ServerAliveInterval 30")
lines.append(" StrictHostKeyChecking accept-new")
lines.append("")
def add_entry(
tag: str,
hostname: str,
user: str = "root",
port: int = 22,
extra_lines: list[str] | None = None,
) -> None:
lines.append(f"Host {tag}")
lines.append(f" HostName {hostname}")
lines.append(f" User {user}")
if port != 22:
lines.append(f" Port {port}")
if extra_lines:
lines.extend(f" {el}" for el in extra_lines)
lines.append("")
# Sort hosts by kind for a logical output order
def sort_key(item):
name, h = item
kind = h.get("kind", "")
order = {
"workstation": 0,
"proxmox-host": 1,
"lxc": 2,
"vm": 3,
"external": 4,
}
return (order.get(kind, 9), name)
sorted_hosts = sorted(
[(n, h) for n, h in hosts.items() if n not in SKIP_HOSTS],
key=sort_key,
)
for name, h in sorted_hosts:
kind = h.get("kind", "")
ssh_config = h.get("ssh", {}) or {}
user = ssh_config.get("user", "root")
port = ssh_config.get("port", 22)
lan_ip = h.get("lan_ip")
mesh = h.get("mesh", {}) or {}
nb = mesh.get("netbird", {}) or {}
ts = mesh.get("tailscale", {}) or {}
nb_fqdn = nb.get("fqdn") or nb.get("ip") or ""
nb_port = ssh_config.get("netbird_port")
# Determine primary address
if nb_port and not lan_ip:
# Hosts with netbird SSH but no LAN IP: mesh-only
primary = nb_fqdn
elif lan_ip:
primary = lan_ip
elif nb_fqdn:
primary = nb_fqdn
elif ts.get("fqdn"):
primary = ts["fqdn"]
else:
continue # no address found
extra = []
if kind == "proxmox-host":
extra = [
"ControlMaster auto",
"ControlPath ~/.ssh/cm/%C",
"ControlPersist 2h",
]
add_entry(name, primary, user, port, extra_lines=extra)
# Mesh fallback for workstations and hubris
if nb_fqdn and (kind == "workstation" or nb_port):
mesh_tag = f"{name}-mesh"
mesh_port = nb_port or port
mesh_extra = []
if kind == "proxmox-host":
mesh_extra = [
"ControlMaster auto",
"ControlPath ~/.ssh/cm/%C",
"ControlPersist 2h",
]
add_entry(mesh_tag, nb_fqdn, user, mesh_port, extra_lines=mesh_extra)
# --- *.hubris.network ControlMaster ---
lines.append("# --- Mesh ControlMaster (speeds up repeated mesh ops) ---")
lines.append("Host *.netbird.selfhosted")
lines.append(" ControlMaster auto")
lines.append(" ControlPath ~/.ssh/cm/%C")
lines.append(" ControlPersist 2h")
lines.append("")
return "\n".join(lines)
def ensure_include() -> bool:
"""Add 'Include ~/.ssh/config.d/homelab' to the main SSH config if missing."""
if not SSH_MAIN_CONFIG.exists():
SSH_MAIN_CONFIG.parent.mkdir(parents=True, exist_ok=True)
with open(SSH_MAIN_CONFIG, "w") as f:
f.write(f"{INCLUDE_LINE}\n")
return True
content = SSH_MAIN_CONFIG.read_text()
for line in content.splitlines():
stripped = line.strip()
if stripped.startswith("Include") and "homelab" in stripped:
return False # already present
# Prepend to existing config
updated = f"{INCLUDE_LINE}\n\n{content}"
SSH_MAIN_CONFIG.write_text(updated)
return True
def install() -> None:
"""Write config to ~/.ssh/config.d/homelab and ensure Include is set."""
SSH_CONFIG_DIR.mkdir(parents=True, exist_ok=True)
config = gen_config()
SSH_CONFIG_FILE.write_text(config)
SSH_CONFIG_FILE.chmod(0o644)
changed = ensure_include()
print(f"Wrote {SSH_CONFIG_FILE} ({len(config.splitlines())} lines)")
if changed:
print(f"Added '{INCLUDE_LINE}' to {SSH_MAIN_CONFIG}")
print("Done. Run: ssh <hostname> (e.g. ssh gitea)")
def main() -> int:
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("--install", "-i", action="store_true",
help=f"write to {SSH_CONFIG_FILE} and wire Include into main config")
args = p.parse_args()
if args.install:
install()
else:
print(gen_config())
return 0
if __name__ == "__main__":
sys.exit(main())

Some files were not shown because too many files have changed in this diff Show More