New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.
homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).
mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.
oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.
runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`bootstrap.sh --with-hermes` installs the Goose CLI, drops a Goose
config pinning the OpenRouter provider + Nous Hermes model + the
homelab MCP extension, symlinks `bin/hermes` and HERMES.md, and links
HERMES.md as `.goosehints` so the persona is injected as the system
prompt every session.
`bin/hermes` decrypts `secrets/openrouter-api-key.yaml` via the existing
`homelab secret` flow and execs `goose session`.
`homelab client add --with-hermes` grants the new sops secret to the
host's age_pubkey at finalize time (parallel to the existing
shared-secrets grant). `client remove` revokes it.
`operations/hermes-agent.md` covers the end-to-end flow, verification,
troubleshooting, and queues one follow-up: the MCP server still runs
SSE-only but Goose 1.x deprecated SSE — the Goose config targets
`streamable_http` and the `homelab` extension won't connect until
`mcp/server.py` migrates. The `developer` extension (shell + edit +
`homelab` CLI) carries the agent in the meantime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets the operator see exactly what would change on the VPS before
applying. Secret values from the decrypted sops files are masked as
<REDACTED> so the diff is safe to paste into chat/PRs.
The IONOS netbird VPS held two credentials in plaintext that were the last
holdouts from the homelab's sops+age secrets pattern:
- /root/turn-pass.txt (coturn long-term-credential password)
- PKCEAuthorizationFlow.ProviderConfig.ClientSecret inline in
/opt/management.json (Authentik OIDC client secret)
This commit moves both into sops-encrypted YAML in the repo and adds a render
command that recreates the VPS config files from templates + decrypted secrets:
* secrets/turn-shared-secret.yaml — encrypted `password: <coturn pwd>`
* secrets/netbird-authentik-oidc.yaml — encrypted `client_secret: <...>`
Both recipients = hubris + apps + republic-laptop (same 3 as hello.yaml).
* vps/turnserver.conf.tmpl + vps/management.json.tmpl — templates with
{{TURN_PASSWORD}} + {{AUTHENTIK_CLIENT_SECRET}} placeholders.
* bin/homelab new subcommand `render-vps-configs`:
- Decrypts both secrets locally (works on any recipient).
- Substitutes placeholders into templates.
- Diffs against current VPS state via ssh, prompts, applies atomically
(write `.new` then mv), restarts coturn + netbird-mgmt.
- --dry-run + -y flags. Hops through hubris when not running on hubris
itself, since VPS sshd is locked to hubris's pubkey.
* inventory.yaml adds the VPS as `kind: external` (new kind; reserved for
ssh-managed hosts that aren't homelab clients themselves — no age key,
no /opt/homelab-context). hosts/netbird-vps.yaml regenerated.
* SHARED_SECRETS list includes both new secrets so re-keys on enrollment
changes pick them up automatically.
After this lands + the 5-min sync propagates to hubris, run from hubris PVE
shell (or any client; hubris just skips the extra ssh hop):
homelab render-vps-configs --dry-run # see plan, no changes
homelab render-vps-configs -y # apply + restart services
Once verified working, the plaintext `/root/turn-pass.txt` should be deleted
on the VPS (the rendered /etc/turnserver.conf no longer needs it as a
reference).
Three coordinated changes addressing follow-ups from the 2026-05-21 netbird vanilla
migration, plus a related off-LAN ergonomics fix:
bin/homelab:
- New ssh_target(name, force_mesh=False) helper resolves (addr, port, user)
from inventory, honoring ssh.netbird_port (forces mesh path) and ssh.user
(default "root"). Falls back to the netbird FQDN when LAN IP fails a
cached 1.5s TCP probe — helps off-LAN operators on VPN/symmetric-NAT
paths where the netbird subnet route doesn't reach 192.168.8.0/24.
- New ssh_base() builds the full `ssh ... user@addr` invocation; hubris_ssh()
is now a back-compat shim. cmd_ssh, cmd_logs, cmd_restart, cmd_nuke
refactored to use it — no more hardcoded "root@" anywhere.
inventory.yaml:
- New ssh.user convention (root by default, explicit per workstation).
- hubris.ssh.user=root (explicit, documents convention).
- republic-laptop, mac-mini: ssh.user=dtoro. ludo-mini left default (TODO).
- Comment block in the header explains the field + why it exists (netbird-ssh
defaults to LOCAL username; "user not found" on LXCs is the gotcha).
- hosts/*.yaml regenerated from build_host_files.py.
infrastructure/mesh.md:
- Migration changelog entry updated: Device Code Stage is now configured
(was "NOT yet" — landed in d41d73f); --setup-key no longer required.
- New subsection documenting the post-migration JWT-issuer cache bug:
client/internal/engine_ssh.go's updateSSH() bails out when sshServer is
already running, so systemctl restart and netbird down/up don't refresh
the SSH JWT validator. Full daemon stop/start is the fix.
- Companion username gotcha (`netbird ssh` defaulting to local username).
operations/agent-enrollment.md:
- Three new troubleshooting rows: JWT-issuer cache, user-not-found, and
LAN-unreachable-from-mesh-peer (the new homelab CLI behavior).
Verification: ssh_target resolution against the live inventory yields
- hubris → ssh -p 22022 root@proxmox-server.netbird.selfhosted (mesh-forced)
- jellyfin/gitea → ssh root@192.168.8.x (LAN reachable, probe passed)
- republic-laptop/mac-mini → ssh dtoro@<fqdn> (per ssh.user)
- ludo-mini → ssh root@<fqdn> (default)
F3 from the apt-sweep backlog. `homelab apt-upgrade --safe ...` takes a
pre-upgrade snapshot per LXC before launching the apt run, so rollback is
trivial if anything regresses.
Snapshot strategy per target:
- LXC: try `pct snapshot <id> preupgrade_<timestamp>` first (CoW, near
instant). If that refuses ("snapshot feature is not available" — the
failure mode for LXCs with host bind-mounts like `mp0: /mnt/library`),
fall back to `vzdump <id> --mode snapshot --storage local --compress
zstd`. Slower but works on bind-mounted LXCs.
- hubris (PVE host): skipped intentionally; no host-level snapshot in v1.
If any snapshot fails the entire run refuses unless --force is passed.
Snapshot rollback hints are printed after launch so the operator has the
recovery path one copy/paste away.
Validated 2026-05-21: py_compile clean; --status mode works on bind-
mounted LXCs (skips snapshot path). Live snapshot test deferred to next
real fleet sweep to avoid leaving stale artifacts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new subcommands (B1+H1 + C1+C2 from the apt-sweep backlog):
* `homelab apt-audit [--target HOST]`
Per-host pre-flight table reporting dpkg-interrupted state, holds,
upgradable count, non-apt binaries in system paths, and DNS health.
Exits nonzero if any target has unconfigured packages (would catch the
latent dpkg-interrupted state we found on caddy + nextcloud before
touching them).
* `homelab apt-upgrade [--target HOST | --all] [--status] [--force]`
Launches `apt update && apt upgrade` inside a transient systemd unit
(`apt-upgrade-<hostname>`) on each target via `systemd-run --collect`,
so the run survives ssh teardown. Output tee'd to
`/var/log/homelab-apt-upgrade.log`. Apt configured with
`Acquire::Retries=3` + `ForceIPv4=true` to absorb transient mirror
blips (would have prevented the jellyfin ffmpeg7 fetch failure that
rolled back 131 packages mid-Wave-3).
Pre-flight audit gate refuses to run on hosts with dpkg-interrupted
state; override with --force.
`--status` mode prints each target's unit state and tails the upgrade
log; safe to run before, during, and after.
Also fixes a latent bug in `hubris_ssh()`: the function joined hubris's
`lan_ip` (192.168.8.77) with the netbird-only port 22022. Port 22022
listens on hubris's netbird interface ONLY — not on the LAN interface,
not even via the netbird subnet route. Result: every off-LAN client
hitting `homelab pct/logs/restart/status` got "Connection refused".
Now uses the netbird FQDN when netbird_port is in use; falls back to
host_address otherwise.
Targets exclude VMs and workstations (they have their own update flows).
Standard apt targets = hubris + every LXC (kind=lxc with a pve_id).
Validated 2026-05-21 on republic-laptop:
- apt-audit reported clean dpkg + 40 upgradable on hubris (the still-
deferred PVE/kernel packages) + 0 on every LXC, DNS ok across the
fleet
- apt-upgrade --target mule-photos-new launched the unit, ran apt to
completion (0 upgrades pending), log captured with =START= / =END
rc=0= markers, --status reports unit + log correctly
- --all --status fans out across all 16 targets without issue
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two additions:
1. secrets-issuance backup: daily timer snapshots /var/lib/secrets-issuance
to /mnt/library/.secrets-issuance-backup/ as a date-stamped tar.gz,
keeping the last 14 days. Closes the catastrophic-fail-mode where an
LXC 105 loss wipes every client's age key with no recovery path.
Caveat: privileged LXCs that mount /mnt/library can read the backup
(root-uid maps to host root); encrypted-tarball variant is a future
refinement.
2. homelab doctor: 10 invariant checks for an enrolled client — clone
present, sync timer/launchd job active, age key perms, CLI symlinked,
AGENTS.md linked, inventory entry exists, MCP reachable, secrets
/health responds, sops canary decrypts, git creds present. Returns
nonzero on any 'fail'. Useful after enrollment or whenever something
smells off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit against actual netbird+tailscale peer lists:
- hubris is the only LXC-host on Netbird; only workstations + hubris
have netbird entries
- 10 LXCs+VMs have real Tailscale FQDNs: apps, jellyfin, paperless,
gitea, nextcloud, elementsynapse, sophia, mule-images→muleimage,
arriman→arr, haos→homeassistant
- 7 hosts are LAN-only (no mesh block): nfs-export, caddy, claudio-bot,
authentik, plato, mule-photos-new, zimaos
- mac-mini's netbird FQDN corrected to the actual peer name
(mac-mini-234-17.netbird.selfhosted)
Also: bin/homelab host_address() now prefers lan_ip first — universally
reachable from any LAN client and from any Netbird peer via the
192.168.8.0/24 network resource routed through hubris. Mesh FQDNs are
fallbacks for roaming workstations without a fixed lan_ip.
This makes 'homelab status' from republic show all backends 'ok' instead
of falsely reporting them 'down' against unresolvable netbird FQDNs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yaml.safe_load + safe_dump stripped every comment from inventory.yaml
on each enrollment, eroding the file's documentation value. New
_inventory_set_age_pubkey / _inventory_remove_host / _inventory_append_host
helpers do line-based edits so comments outside the modified region
survive. inventory.yaml gets its top-of-file conventions block back.
build_host_files.py round-trips cleanly (--check returns 0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Critical bug — the loop reset age_block_last_idx unconditionally on
every '- path_regex:' line. So after finding the target rule's age
block, encountering the NEXT rule wiped the result and the function
returned False. _grant_shared_secrets / _revoke_shared_secrets both
silently no-op'd because of this.
Fix: break out of the scan once we've collected the target rule's
data. Refactored the remove helper to share a find_target_age_lines
inner so the dangling-trailing-comma fixup uses the same logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The remove flow ran 'sops updatekeys' but never edited .sops.yaml first,
so the removed client stayed a recipient on every shared secret —
exactly the opposite of what 'remove' should do. Adds the
_remove_recipient_from_sops_policy / _revoke_shared_secrets helpers
(inverse of the grant-side ones from the previous commit); cmd_client_remove
now resolves the pubkey from inventory before deletion and feeds it
through that pipeline.
Also adds the sudo re-exec pattern so 'homelab client remove' works
from a non-root user (matching cmd_secret / cmd_refresh_creds /
cmd_client_add).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Setting the age_pubkey is half the enrollment; the new client also needs
to be a recipient on shared secrets (hello.yaml, gitea-pat.yaml) to
actually use them. Now --finalize-pubkey:
1. writes hosts.<name>.age_pubkey
2. appends the pubkey to each shared-secret rule in .sops.yaml
(preserving comments via line-by-line edit, not yaml round-trip)
3. runs sops updatekeys -y on each shared file
4. commits inventory + hosts/ + .sops.yaml + secrets/ as one commit
Also: cmd_client_add now re-execs via sudo when invoked as a regular
user (matches the pattern in cmd_secret + cmd_refresh_creds).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds secrets/gitea-pat.yaml (SOPS-encrypted, dtoro PAT with read+write
scopes) so any enrolled client can push to dtoro/Homelab-Docs — not just
where I have SSH. Recipient set = hello.yaml's (hubris, apps, republic);
expand alongside hello.yaml when enrolling new clients.
bin/homelab gains 'refresh-creds': decrypts gitea-pat.yaml, rewrites
/etc/homelab-context/git-credentials with the write token, repoints
git's --system credential helper. Re-execs via sudo for non-root callers
(same pattern as 'homelab secret').
After this lands, 'homelab client add/remove' and wiki edits can run
from any client. The initial bootstrap still needs an operator-supplied
read-only PAT (chicken-and-egg); 'refresh-creds' upgrades the client
to write afterwards.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/etc/age/key.txt is 0600 root and /etc/age is 0700 root, so the CLI's
Path.exists() check was returning False under regular users — making the
subcommand look broken when bootstrap had actually written the key fine.
Re-exec via sudo preserves the existing UX (one password prompt, then
plaintext) without loosening the key's perms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the foundation for distributing homelab context to every client
(LXCs, VMs, workstations including republic-laptop, mac-mini, ludo-mini)
with a single source of truth, structured query layer (MCP), and per-client
age-key issuance for secrets:
- inventory.yaml — canonical topology (hosts, services, mesh addresses)
- hosts/*.yaml — per-host identity files generated from inventory by
mcp/build_host_files.py; do not edit by hand
- AGENTS.md — orientation doc symlinked to /root/AGENTS.md on every client
- bootstrap.sh — one-shot enroll (Linux + macOS), clones repo, fetches age
key from issuance, installs sync timer/launchd job, drops the homelab CLI
- bin/homelab — single-binary Python CLI: whoami, list, ssh, pct, logs,
restart, open, status, secret, sync, mcp, client add/remove, nuke
- mcp/server.py — FastMCP server: context tools + read-only management
tools (no mutations exposed); shell-outs use mcp-reader restricted ssh key
- mcp/deploy/ — claudio-monitor-style gitea webhook deploy scaffold for the
MCP service on LXC 105 (ports 9810 mcp, 9811 webhook)
- secrets-issuance/ — per-client age key auto-provisioning over the mesh;
source-IP gated against inventory, with denylist for revoked clients
(ports 9820 issue, 9821 webhook)
- secrets/, .sops.yaml — SOPS recipient scaffolding; the operator fills in
age public keys after Phase 3a generates them
- scripts/sync/ — systemd timer (Linux) + launchd plist (macOS) pulling
/opt/homelab-context every 5 min
Mesh: both Netbird (preferred, 100.122.0.0/16) and Tailscale accepted
during the in-flight migration; no client is gated on completing the move.
Plan reference: /root/.claude/plans/lets-make-a-plan-fluttering-trinket.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
