Created via the Gitea API (POST /repos/dtoro/Homelab-Docs/hooks) rather
than the UI, since the existing PAT turned out to have sufficient scope.
Webhook id 14: http://192.168.8.205:9831/deploy, push events, main branch
filter, active.
The shared secret was generated and registered with Gitea before the
apps-side bootstrap ran (order reversed from the usual install.sh-first
flow, since direct SSH deploy to apps is still pending operator
execution — see oikos/console/deploy/README.md). Stored as
secrets/oikos-console-deploy-secret.yaml (SOPS, recipient: apps only)
rather than left as a local plaintext file, with explicit operator
sign-off. When the apps-side install runs, skip webhook/install.sh's
random-secret generation and write this exact value into
/etc/oikos-console-deploy/secret instead.
infrastructure/auto-deploy.md updated with the real webhook id (was
"not yet registered").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
teddycloud was live on hubris (LXC 131, docker compose, TeddyCloud —
a Toniebox cloud reimplementation) but never made it into inventory.yaml.
Already referenced in passing by containers/132-rclone.md ("131 was
already taken by an undocumented teddycloud container") and
hosts/strong.md's migration changelog (a DHCP conflict fix), but no
inventory entry or doc page existed until oikos/drift.py's inventory-
vs-live check caught it.
Verified live via read-only SSH (pct config 131, pct exec 131 -- ...,
docker ps): hostname, static IP 192.168.8.150, 1 core/1GiB/16GiB rootfs,
Debian 12, runs via docker compose at /opt/teddycloud. No changes made
to the running container.
Also fixed: house's inventory notes claimed 192.168.8.212 is teddycloud's
current IP via DHCP — stale, teddycloud has a static IP now.
Flagged in the new container page: teddycloud has no Caddy forward-auth
gate, unlike sab.hubris.network on the same Caddyfile.
`python3 oikos/drift.py` no longer reports an inventory-vs-live finding
for pve_id 131. (A separate, pre-existing gap surfaced while verifying
this: rclone's own inventory.yaml block is missing pve_id/host/lan_ip —
out of scope here, flagging for a follow-up.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.
Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
(dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
(/opt/oikos-console) rather than /opt/homelab-context, which would have
made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
etc. than the scheduler and CLI use — silently forking signal/approval
data into two locations in production. Fixed to match mcp/server.py's
CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
(approval replies, signal ack/resolve) don't sit uncommitted against
the 5-min-synced clone.
Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.
oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.
Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).
Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.
60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).
Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.
Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.
- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
register the new container.
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LXC 128 trmnl hosts the TRMNL plugins middleware (dtoro/terminalito), polled by
TRMNL cloud. trmnl-plugins.service on :9851; Caddy block + LE cert; VPS traefik
router trmnl-public + cert mirror. Public path pending VPS<->home netbird route
recovery (was "No networks available" at provision time, artifacto/blog 504 too).
LAN Technitium record + SOPS enrollment + Google/MVG creds pending. Plan -> In Progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- scripts/dns-sync.py: reconcile Technitium named A-records -> NetBird managed
zone via API (cron */10 on dns LXC 107). Single authoring source; kills the
manual drift behind the auth/sso/nfs-export saga.
- secrets/netbird-pat.yaml: sops-encrypted NetBird API PAT for the sync.
- dns.md / 107-dns.md: document the sync model + why forward-to-Technitium was
abandoned (NetBird self-IP / nameserver-group quirks).
- Cleanup: removed inert Mac secondary; reverted primary AXFR; home-lab-dns ->
[192.168.8.2] (1/1 Available); deleted vestigial Proxmox Names group.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- plans/: mark Slate AX → SODOLA migration as Done (2026-06-02)
- infrastructure/network.md: rewrite to reflect actual topology —
Fritz!OS 8.x lacks second-IP-network support, so Proxmox routes between
vmbr1 (192.168.178.10, uplink) and vmbr0 (192.168.8.x, internal portless
bridge with 192.168.8.1 alias). Static route on Fritz!Box, no NAT on Proxmox.
Technitium DHCP enabled for 192.168.8.100–240 (gateway .1, DNS .2).
Caddy service unit was missing on CT 121 — recreated and enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Domain-level forward-auth needs its own external_host domain when the IdP core
and outpost are on different hosts. sso.hubris.network -> Caddy -> LAN outpost.
Includes the redirect_uris-regeneration gotcha. Carry the DNS record into
Technitium in DNS Phase 2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).
- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Management crash-loop root cause and fix documented.
Phase 6 (Authentik as Netbird IdP) marked complete and live.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The IONOS netbird VPS held two credentials in plaintext that were the last
holdouts from the homelab's sops+age secrets pattern:
- /root/turn-pass.txt (coturn long-term-credential password)
- PKCEAuthorizationFlow.ProviderConfig.ClientSecret inline in
/opt/management.json (Authentik OIDC client secret)
This commit moves both into sops-encrypted YAML in the repo and adds a render
command that recreates the VPS config files from templates + decrypted secrets:
* secrets/turn-shared-secret.yaml — encrypted `password: <coturn pwd>`
* secrets/netbird-authentik-oidc.yaml — encrypted `client_secret: <...>`
Both recipients = hubris + apps + republic-laptop (same 3 as hello.yaml).
* vps/turnserver.conf.tmpl + vps/management.json.tmpl — templates with
{{TURN_PASSWORD}} + {{AUTHENTIK_CLIENT_SECRET}} placeholders.
* bin/homelab new subcommand `render-vps-configs`:
- Decrypts both secrets locally (works on any recipient).
- Substitutes placeholders into templates.
- Diffs against current VPS state via ssh, prompts, applies atomically
(write `.new` then mv), restarts coturn + netbird-mgmt.
- --dry-run + -y flags. Hops through hubris when not running on hubris
itself, since VPS sshd is locked to hubris's pubkey.
* inventory.yaml adds the VPS as `kind: external` (new kind; reserved for
ssh-managed hosts that aren't homelab clients themselves — no age key,
no /opt/homelab-context). hosts/netbird-vps.yaml regenerated.
* SHARED_SECRETS list includes both new secrets so re-keys on enrollment
changes pick them up automatically.
After this lands + the 5-min sync propagates to hubris, run from hubris PVE
shell (or any client; hubris just skips the extra ssh hop):
homelab render-vps-configs --dry-run # see plan, no changes
homelab render-vps-configs -y # apply + restart services
Once verified working, the plaintext `/root/turn-pass.txt` should be deleted
on the VPS (the rendered /etc/turnserver.conf no longer needs it as a
reference).
Three coordinated changes addressing follow-ups from the 2026-05-21 netbird vanilla
migration, plus a related off-LAN ergonomics fix:
bin/homelab:
- New ssh_target(name, force_mesh=False) helper resolves (addr, port, user)
from inventory, honoring ssh.netbird_port (forces mesh path) and ssh.user
(default "root"). Falls back to the netbird FQDN when LAN IP fails a
cached 1.5s TCP probe — helps off-LAN operators on VPN/symmetric-NAT
paths where the netbird subnet route doesn't reach 192.168.8.0/24.
- New ssh_base() builds the full `ssh ... user@addr` invocation; hubris_ssh()
is now a back-compat shim. cmd_ssh, cmd_logs, cmd_restart, cmd_nuke
refactored to use it — no more hardcoded "root@" anywhere.
inventory.yaml:
- New ssh.user convention (root by default, explicit per workstation).
- hubris.ssh.user=root (explicit, documents convention).
- republic-laptop, mac-mini: ssh.user=dtoro. ludo-mini left default (TODO).
- Comment block in the header explains the field + why it exists (netbird-ssh
defaults to LOCAL username; "user not found" on LXCs is the gotcha).
- hosts/*.yaml regenerated from build_host_files.py.
infrastructure/mesh.md:
- Migration changelog entry updated: Device Code Stage is now configured
(was "NOT yet" — landed in d41d73f); --setup-key no longer required.
- New subsection documenting the post-migration JWT-issuer cache bug:
client/internal/engine_ssh.go's updateSSH() bails out when sshServer is
already running, so systemctl restart and netbird down/up don't refresh
the SSH JWT validator. Full daemon stop/start is the fix.
- Companion username gotcha (`netbird ssh` defaulting to local username).
operations/agent-enrollment.md:
- Three new troubleshooting rows: JWT-issuer cache, user-not-found, and
LAN-unreachable-from-mesh-peer (the new homelab CLI behavior).
Verification: ssh_target resolution against the live inventory yields
- hubris → ssh -p 22022 root@proxmox-server.netbird.selfhosted (mesh-forced)
- jellyfin/gitea → ssh root@192.168.8.x (LAN reachable, probe passed)
- republic-laptop/mac-mini → ssh dtoro@<fqdn> (per ssh.user)
- ludo-mini → ssh root@<fqdn> (default)
Updates to four pages reflecting the combined → vanilla mgmt+signal+relay+coturn
cutover and the IONOS-3478-firewall-exception discovery:
* infrastructure/mesh.md — rewrites the ICE/STUN section to cover the new
TURN endpoint, the IONOS upstream TCP-3478 filtering (load-bearing,
undocumented before today), and the verification probe. New changelog
entry covering the migration outcome + Device Code Stage gap.
* infrastructure/vps-hardening.md — "At a glance" lists the new 6-service
docker stack + host coturn. Firewall section notes the new
`iifname ens6 tcp dport 3478 accept` rule plus the IONOS upstream
exception. New changelog entry.
* containers/124-authentik.md — replaces the "Netbird IdP integration —
DEFERRED" section with the LANDED state: Provider details (Public
client type — Confidential breaks PKCE on the dashboard SPA), the
first-time owner-promotion sqlite recipe, the missing Device Code
Stage gap + workaround (setup-keys), and a note that the old 2026-04-22
pre-work Provider/App is now obsolete and safe to delete. Updated
changelog (Phase 6 landed).
* operations/agent-enrollment.md — new "Getting onto Netbird" subsection
explaining the setup-key path (currently the only working flow until
Device Code Stage lands) and why direct OIDC from the public internet
fails (auth.hubris.network is mesh-only-reachable). Prerequisites table
row updated to point at the new section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundles the documentation slice of the apt-sweep backlog:
* operations/runbook-dpkg-interrupted.md (NEW) — Path A (ssh-reachable
recovery) + Path B (PVE web Shell when the netbird mesh broke
alongside the dpkg state, as happened during Wave 6 on hubris).
Closes B2.
* operations/commands.md — new "Fleet apt operations" section
documenting `homelab apt-audit` and `homelab apt-upgrade`
(--status / --safe / --force). Adds the dpkg-interrupted runbook to
Related.
* operations/agent-enrollment.md —
- new "Claude Code permissions for fleet ops" section with the
`permissions.allow` snippet (`Bash(ssh -p 22022 *)`,
`Bash(homelab *)`) for `~/.claude/settings.json`. Closes A3.
- two new Troubleshooting rows: chat-mode `!` sudo no-tty gotcha
(G2) and the cosmetic netbird DNS-probe warning.
* infrastructure/auto-deploy.md — new "Custom-built binaries that
overlap apt-managed paths" section describing the two acceptable
patterns (epoch-versioned .deb à la caddy 1:2.11.3-hubris1; or
apt-mark hold) and the discovery path via `homelab apt-audit`'s
NONAPT column. Closes D3.
Remaining backlog after this commit: A4 (upstream OpenSSH/netbird mux
bug), D1 (apt-mark hold caddy in caddy-conf bootstrap — superseded
in practice by the epoch .deb), E1/E2 (LXC DNS fallback for
tailscale-managed resolv.conf), F1/F2 (vzdump fallback doc; F3 already
shipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds infrastructure/homelab-context.md as the architecture reference for
the cross-client context + MCP + secrets-issuance system. Updates:
- 105-apps.md: two new ## Stacks sections (homelab-mcp, secrets-issuance)
with their deploy pipelines + a row each in the public-hostname table;
changelog entry.
- auto-deploy.md: both new pipelines added to the table (one repo, two
webhooks, same push); per-pipeline notes covering the clone-per-service
pattern and the deploy.sh self-restart caveat; changelog entry.
- README.md: link to the new infrastructure page.
Operational walkthrough already lives at operations/agent-enrollment.md;
this commit is the architecture side of the same story.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container had been stopped since 2026-04-21 and was never re-enabled.
pct destroy 109 --purge cleaned up vm-109-disk-0 on local-lvm and the
config file. /mnt/library/syncthing subtree was already empty at the
time of destruction and is retained as an empty dir (no real data to
migrate or back up).
- README.md, containers/index.md: removed row, moved to "recently
destroyed" table
- hosts/hubris.md: dropped from /mnt/library subtree list, updated
containers/index summary line, added changelog entry
- infrastructure/media-permissions.md: dropped from membership table
and onboarding example, generalised pct-exec gotcha hostname,
added changelog
- vms/100-zimaos.md: dropped from "existing fleet" enumeration
- containers/102-nfs-export.md: dropped from bind-mount sibling list
(7 LXCs now, not 8)
- containers/109-syncthing.md: deleted
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After Files-UI evaluation passed (library renders as folder, thumbnails
work), flipped /etc/exports from ro to rw on LXC 102. Tested: write from
ZimaOS appears on /mnt/library as www-data:media, confirming the
all_squash,anonuid=33,anongid=10000 design works.
Documented two architectural findings discovered this session:
- ZimaOS Drives panel sources from GET /v2/local_storage/storages (read-only
API). Network shares cannot become Drives — Files-as-folder is supported.
- Mesh peers reach ZimaOS via hubris's existing 192.168.8.0/24 netbird subnet
advertisement; no new infra needed, just DNS (Management nameserver group
for hubris.network or per-device /etc/hosts override).
User destroyed the heaper LXC on 2026-05-14. Removed it from the
container index, README quicktable, hubris host doc, and
media-permissions membership table; moved to the "recently destroyed"
archaeology list. /mnt/library/heaper retained (224 MiB).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LXC 125 stood up as a Seafile CE 13.0 docker-compose deployment, behind
files.hubris.network. Authentik OAuth wired up via ak shell. No data
migration — exploration alongside Nextcloud (114).
Initial documentation of the hubris Proxmox homelab as a cross-linked
markdown wiki. Per-node pages, cross-cutting infrastructure pages, an
investigation log, and an operations cheatsheet. Each node and topic
ends with a Changelog section so changes can be tracked in-place going
forward.
Refreshed against live state on 2026-04-28 — 14 active LXCs (109
syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test
state of the 2026-04-21 hubris crash-loop investigation.
