The value written to apps' /etc/oikos-console-deploy/secret didn't
match what Gitea webhook 14 had configured, causing every deploy
attempt to 403 with a signature mismatch — likely drift introduced by
the earlier two-step PATCH sequence (secret set in one call,
branch_filter/active restored in a second call without re-including
the config object). Rotated cleanly this time: fresh secret set on
Gitea and re-encrypted here in one pass, single atomic PATCH covering
config+events+branch_filter+active together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Console is live: cloned to /opt/oikos-console, deploy.sh ran clean,
webhook secret written to /etc/oikos-console-deploy/secret from the
pre-registered SOPS secret (never printed — decrypted and piped
straight into the target file in one command), both systemd units
enabled and active. Verified locally (127.0.0.1:8091 -> 200) and
end-to-end (https://oikos.hubris.network/ -> 302, the Authentik gate
firing correctly).
Found a real bug during first boot: oikos-console.service's
ReadWritePaths listed /opt/homelab-context/signals and .../approvals,
but neither existed yet on apps' clone — git doesn't track empty
directories, and nothing had ever written a signal/approval from that
host. ProtectSystem=strict + a missing ReadWritePaths target is a hard
226/NAMESPACE crash, not a graceful degradation. Fixed two ways:
the unit now marks those paths optional (`-` prefix) so a fresh deploy
never crash-loops on this again, and deploy.sh now mkdir -p's them
explicitly so the console has real write access from the first boot,
not just a non-crashing-but-broken start.
This is also the first real exercise of the auto-deploy pipeline: this
push should land via Gitea webhook 14 -> oikos-console-deploy.service
on apps, same as homelab-mcp/secrets-issuance already work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A record -> 192.168.8.175 (Caddy's LAN IP), created via Technitium's
REST API (login -> createToken -> zones/records/add) in a single
in-memory call. Neither the admin credential nor the resulting session/
API token was ever printed to output or written to disk, and the token
wasn't persisted anywhere after the call completed — it existed only
for the lifetime of that one process.
Verified: dig @192.168.8.2 +short oikos.hubris.network -> 192.168.8.175.
End-to-end confirmation that DNS + Caddy + the Authentik gate are all
wired correctly: curl https://oikos.hubris.network/ now returns a 302
(the forward-auth redirect firing before the not-yet-deployed backend
would even matter) instead of failing to resolve/connect.
This closes out every part of the console rollout except the actual
apps-side bootstrap (oikos/console/deploy/README.md "One-time setup"),
which remains pending direct operator execution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pushed dtoro/caddy-conf@c195142: oikos.hubris.network -> 192.168.8.205:8091,
Authentik-gated (matches paperless.hubris.network's live pattern —
confirmed exact snippet syntax against the real Caddyfile rather than
trusting the paraphrase in the original README, which turned out to
have the wrong forward_auth target: the live snippet points at
127.0.0.1:8099 on Caddy's own LXC, not 192.168.8.6:9000 as
containers/106-auth-outpost.md's older text suggested). Reload verified
clean — an unrelated existing route stayed healthy through it.
Found and fixed a real deploy-blocking bug in the process:
oikos-console.service bound 127.0.0.1 only, but Caddy runs on a
different host (121) and can only reach apps (105) over the LAN — the
console would have been completely unreachable once deployed. Now binds
0.0.0.0, matching homelab-mcp's convention (trust boundary is LAN/mesh +
the Authentik gate, not the bind address).
Encountered and deliberately left alone: a pre-existing local clone at
/tmp/caddy-conf with an unpushed commit + uncommitted diff about
jellyfin's auth gating, from before this clone fell 12 commits behind
origin. That work turned out to be superseded (origin already reached
the same conclusion — SSO plugin handles jellyfin auth, no forward-auth
gate — via a different, already-merged path). Didn't touch it; used a
fresh clone instead to avoid any risk of losing or corrupting that state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Created via the Gitea API (POST /repos/dtoro/Homelab-Docs/hooks) rather
than the UI, since the existing PAT turned out to have sufficient scope.
Webhook id 14: http://192.168.8.205:9831/deploy, push events, main branch
filter, active.
The shared secret was generated and registered with Gitea before the
apps-side bootstrap ran (order reversed from the usual install.sh-first
flow, since direct SSH deploy to apps is still pending operator
execution — see oikos/console/deploy/README.md). Stored as
secrets/oikos-console-deploy-secret.yaml (SOPS, recipient: apps only)
rather than left as a local plaintext file, with explicit operator
sign-off. When the apps-side install runs, skip webhook/install.sh's
random-secret generation and write this exact value into
/etc/oikos-console-deploy/secret instead.
infrastructure/auto-deploy.md updated with the real webhook id (was
"not yet registered").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
teddycloud was live on hubris (LXC 131, docker compose, TeddyCloud —
a Toniebox cloud reimplementation) but never made it into inventory.yaml.
Already referenced in passing by containers/132-rclone.md ("131 was
already taken by an undocumented teddycloud container") and
hosts/strong.md's migration changelog (a DHCP conflict fix), but no
inventory entry or doc page existed until oikos/drift.py's inventory-
vs-live check caught it.
Verified live via read-only SSH (pct config 131, pct exec 131 -- ...,
docker ps): hostname, static IP 192.168.8.150, 1 core/1GiB/16GiB rootfs,
Debian 12, runs via docker compose at /opt/teddycloud. No changes made
to the running container.
Also fixed: house's inventory notes claimed 192.168.8.212 is teddycloud's
current IP via DHCP — stale, teddycloud has a static IP now.
Flagged in the new container page: teddycloud has no Caddy forward-auth
gate, unlike sab.hubris.network on the same Caddyfile.
`python3 oikos/drift.py` no longer reports an inventory-vs-live finding
for pve_id 131. (A separate, pre-existing gap surfaced while verifying
this: rclone's own inventory.yaml block is missing pve_id/host/lan_ip —
out of scope here, flagging for a follow-up.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.
Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
(dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
(/opt/oikos-console) rather than /opt/homelab-context, which would have
made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
etc. than the scheduler and CLI use — silently forking signal/approval
data into two locations in production. Fixed to match mcp/server.py's
CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
(approval replies, signal ack/resolve) don't sit uncommitted against
the 5-min-synced clone.
Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.
oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.
Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).
Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.
60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only off-LAN-reachable workstations (e.g. republic-laptop, mac-mini)
need to join Netbird. LAN-reachable LXCs/VMs on 192.168.8.0/24 don't —
they're already directly reachable, and off-LAN clients reach them via
hubris's routed 192.168.8.0/24 Netbird network resource. Brings the
runbook in line with oikos/ontology.yaml's lifecycle transition, which
already says "mesh-joined-if-needed".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.
homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).
mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.
oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.
runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).
Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.
Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.
Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.
- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
register the new container.
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standard baseline for an enrolled workstation-class client, matching
mac-mini/republic-laptop/etc: the bootstrap decrypt-test secret plus
the write-scoped Gitea PAT so strong can push to the wiki repo on its
own (homelab client add/remove, wiki edits from that host).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
strong now has its own age key (issued over LAN via --no-mesh),
pubkey recorded in inventory.yaml. Not yet granted to any secrets
file - that's a separate decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Running bootstrap without --no-secrets always tried to install and
connect Netbird, even when the mesh-check right after it already knows
how to fall back to plain LAN reachability. On a host nobody's watching
interactively (e.g. driven over SSH), this hangs forever at the
device-code prompt — hit live on strong, had to kill the stuck
`netbird up` process manually. --no-mesh skips netbird install/up while
still allowing the existing LAN-fallback path to satisfy secrets
issuance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered live re-running bootstrap on strong for secrets issuance:
apt-get install sops fails outright (no such Debian package — matches
what agent-enrollment.md's manual-install recipe already does, fetching
the binary from GitHub releases instead of a package manager). dnf would
have the same problem. Added install_sops_binary(), used on both the
dnf and apt paths; Darwin still installs via brew, which does carry sops.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pipx/mcp-CLI step and the Hermes goose installer both called
`sudo -u <user> ...` unconditionally. On minimal Linux images reached
via `ssh root@host` (no SUDO_USER, and often no `sudo` binary at all —
seen live on strong), this failed with "sudo: command not found" and
silently no-opped the mcp CLI install. Added a run_as() helper that
only shells out to sudo when there's a real invoking user distinct
from root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bootstrap.sh --no-secrets ran clean: sync timer, homelab CLI, and
AGENTS.md are live on strong. Noted two follow-ups: secrets issuance
is reachable over plain LAN (mesh: lan) so age-key enrollment doesn't
actually need Netbird, and bootstrap's pipx/mcp-CLI step silently no-ops
when run as root over SSH (missing `sudo` binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix subscription path to /budget/subscriptions (not /subscriptions/)
- Document RRULE rejection; use cycle_interval instead
- Add GET /budget/ vs ?month= behavior note
- Add full category/payment-method ID tables
- Add Cookie Share recommendation: 2,650 €/month based on 6-mo analysis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Step-by-step procedure to import N26 CSV transactions into Yuvomi's
Budget and Subscriptions modules using the yuvomi-mcp tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills in the age public key issued by secrets-issuance after
homelab client enrollment completed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All LXCs use ip=<addr>/24 in pve net0 config directly. No Fritz!Box
lease needed. Updated container doc and migration runbook accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OIDC config migrated from Booklore DB dump — Confidential client (not
PKCE), credentials intact. offline_access scope added to Authentik
provider. Backchannel logout URL set to permanent IP 192.168.8.213.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
