New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only off-LAN-reachable workstations (e.g. republic-laptop, mac-mini)
need to join Netbird. LAN-reachable LXCs/VMs on 192.168.8.0/24 don't —
they're already directly reachable, and off-LAN clients reach them via
hubris's routed 192.168.8.0/24 Netbird network resource. Brings the
runbook in line with oikos/ontology.yaml's lifecycle transition, which
already says "mesh-joined-if-needed".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.
homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).
mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.
oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.
runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).
Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.
Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.
Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.
- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
register the new container.
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standard baseline for an enrolled workstation-class client, matching
mac-mini/republic-laptop/etc: the bootstrap decrypt-test secret plus
the write-scoped Gitea PAT so strong can push to the wiki repo on its
own (homelab client add/remove, wiki edits from that host).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
strong now has its own age key (issued over LAN via --no-mesh),
pubkey recorded in inventory.yaml. Not yet granted to any secrets
file - that's a separate decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Running bootstrap without --no-secrets always tried to install and
connect Netbird, even when the mesh-check right after it already knows
how to fall back to plain LAN reachability. On a host nobody's watching
interactively (e.g. driven over SSH), this hangs forever at the
device-code prompt — hit live on strong, had to kill the stuck
`netbird up` process manually. --no-mesh skips netbird install/up while
still allowing the existing LAN-fallback path to satisfy secrets
issuance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered live re-running bootstrap on strong for secrets issuance:
apt-get install sops fails outright (no such Debian package — matches
what agent-enrollment.md's manual-install recipe already does, fetching
the binary from GitHub releases instead of a package manager). dnf would
have the same problem. Added install_sops_binary(), used on both the
dnf and apt paths; Darwin still installs via brew, which does carry sops.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pipx/mcp-CLI step and the Hermes goose installer both called
`sudo -u <user> ...` unconditionally. On minimal Linux images reached
via `ssh root@host` (no SUDO_USER, and often no `sudo` binary at all —
seen live on strong), this failed with "sudo: command not found" and
silently no-opped the mcp CLI install. Added a run_as() helper that
only shells out to sudo when there's a real invoking user distinct
from root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bootstrap.sh --no-secrets ran clean: sync timer, homelab CLI, and
AGENTS.md are live on strong. Noted two follow-ups: secrets issuance
is reachable over plain LAN (mesh: lan) so age-key enrollment doesn't
actually need Netbird, and bootstrap's pipx/mcp-CLI step silently no-ops
when run as root over SSH (missing `sudo` binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix subscription path to /budget/subscriptions (not /subscriptions/)
- Document RRULE rejection; use cycle_interval instead
- Add GET /budget/ vs ?month= behavior note
- Add full category/payment-method ID tables
- Add Cookie Share recommendation: 2,650 €/month based on 6-mo analysis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Step-by-step procedure to import N26 CSV transactions into Yuvomi's
Budget and Subscriptions modules using the yuvomi-mcp tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills in the age public key issued by secrets-issuance after
homelab client enrollment completed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All LXCs use ip=<addr>/24 in pve net0 config directly. No Fritz!Box
lease needed. Updated container doc and migration runbook accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OIDC config migrated from Booklore DB dump — Confidential client (not
PKCE), credentials intact. offline_access scope added to Authentik
provider. Backchannel logout URL set to permanent IP 192.168.8.213.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LXC 128 trmnl hosts the TRMNL plugins middleware (dtoro/terminalito), polled by
TRMNL cloud. trmnl-plugins.service on :9851; Caddy block + LE cert; VPS traefik
router trmnl-public + cert mirror. Public path pending VPS<->home netbird route
recovery (was "No networks available" at provision time, artifacto/blog 504 too).
LAN Technitium record + SOPS enrollment + Google/MVG creds pending. Plan -> In Progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
