- lxc:rclone verified live as LXC 132 on hubris (pct list via MCP);
hosts edge added
- /mnt/library backing storage identified from hosts/hubris.md: 'library'
lvmthin pool, 3.7T, 2nd NVMe — added pool:library-hubris + contains edge
- all 8 derived services (books/seanime/roms/house/jellyseerr/qbit/sab/
teddy) confirmed responding over their ingress URLs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merge the rev-2 audit + remediation layers into one self-consistent spec and
close new gaps: meta-schema inheritance (parent_type/is_abstract), contract-
first API (RFC 9457, idempotency, ETag, scopes, /graph), single-binary role
packaging, UUIDv7+slug IDs, checks-as-data, signal dedup/flap/maintenance,
executable skill format, MCP streamable HTTP, SSE events, ledger-as-view,
dual-path networking (mesh-primary + LAN break-glass), per-phase acceptance
criteria, ADRs. Appendix A maps every rev-2 finding to its resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The operating-model diagram in README was ASCII box art in a plain code
fence, not an actual Mermaid diagram — it wouldn't render as a graph on
Gitea/GitHub. Replaced with a `flowchart TD` matching the convention already
used by oikos/gen-topology.py's generated topology.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: README listed specific IPs, container IDs, and per-host counts (e.g.
"hubris (15 active): 102 nfs-export, 103 paperless..."). This duplicates
inventory.yaml and the wiki index pages, and goes stale every time a node
moves, gets added, or is destroyed — exactly what happened during the strong
migration.
Fix: Replaced the Proxmox Hosts / VMs / LXC Containers / Cross-Cutting
Infrastructure subsections with plain pointers to their authoritative index
pages (knowledge/wiki/{hosts,vms,containers,infrastructure}/index.md).
Also dropped the "Last refreshed against live state" date line — another
claim that goes stale without a mechanism to keep it honest.
README's job is navigation, not a live topology snapshot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous naming guide was incomplete. The actual convention is:
**Foundational docs:** ALL-CAPS
- Root entry-points: AGENTS.md, README.md (discovery paths)
- Agent instruction: .agents/OIKOS.md, .agents/HERMES.md (docs agents read first)
- Reference docs: GLOSSARY.md (like classic repo files: LICENSE, CHANGELOG)
**Content pages:** lowercase-with-dashes
- Containers: <id>-<name>.md (ID from inventory)
- Infrastructure: <topic>.md (system description)
- Plans/investigations: YYYY-MM-DD-slug.md (date-sorted)
- Section indices: README.md (conventional)
**Skills:** special pattern
- <name>/SKILL.md where <name> is lowercase-with-dashes
- SKILL.md filename is always uppercase — signpost for tools and humans
Uppercase is reserved for foundational/signpost docs; all paths otherwise use
lowercase with hyphens (no underscores).
Updated page-templates.md with expanded explanation, and updated AGENTS.md +
README.md to reference the corrected convention.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added explicit file-naming rules to page-templates.md so agents know:
- Root entry-points: ALL-CAPS (AGENTS.md, README.md)
- Containers: <id>-<name>.md (e.g., 101-jellyfin.md)
- Infrastructure: lowercase-with-dashes (dns.md, auto-deploy.md)
- Plans/investigations: YYYY-MM-DD-slug.md
- Skills: lowercase-with-dashes/ folder containing SKILL.md
Updated AGENTS.md section 4 (Wiki conventions) to link to page-templates.md
and provided quick reference for file naming, page locations, and changelog format.
Updated README.md conventions section to mention file naming and link to
page-templates.md for the full rules.
All agents now have a clear reference chain:
1. AGENTS.md (entry point) → points to conventions
2. page-templates.md (structure) → has file naming + page templates
3. writing-style.md (prose) → has voice, vocabulary, linking rules
4. llm-wiki.md (organization) → has sources/wiki/index/log model
Verified: no broken links, all conventions documented, consistency check passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains
were still scattered across the repo root, with three now-redundant stub
files cluttering it. The organizing principle wasn't visible in the layout.
Change — enforce three clear buckets:
- .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root),
shared/ conventions, domains/ schemas, skills/, and operations/ (operator
cheatsheet + enrollment + hermes-agent, moved from root).
- knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now
including investigations/ (incident records are evidence/sources).
- root = substrate + two entry points (AGENTS.md, README.md), plus plans/
as its own design-intent domain.
Moves:
- investigations/ -> knowledge/sources/investigations/ (incl. archive/, index).
- operations/ -> .agents/operations/.
- HERMES.md -> .agents/HERMES.md.
- Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md
(its 7 remaining linkers repointed to .agents/OIKOS.md).
Consumers updated:
- inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards.
- tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md.
- bin/homelab help string -> .agents/operations/hermes-agent.md.
- knowledge/operations schemas, llm-wiki, page-templates, incident-investigation
skill, AGENTS.md/README nav -> new investigations/operations paths.
- All markdown links rewritten via the path-resolving mapper.
Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/,
plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml.
Verification: docs-lint at baseline (2 intentional cross-repo refs, no new
breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all
doc_page targets resolve; Hermes provisioning scripts point at the new path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative
links that predated this session — a container rename, incident/plan docs
that moved into archive/done subfolders without their inbound links being
updated, and a handful of relative-depth bugs in files nested under
containers/archive/ and plans/done/.
Fixes applied, by category:
- 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs).
- investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md
-> archive/ prefix (both moved to investigations/archive/ previously).
- plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot,
2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix.
- Depth bugs in files nested one level deeper than their siblings assumed
(investigations/archive/*, knowledge/wiki/containers/archive/*,
plans/done/*) — corrected relative-path depth.
- Destroyed containers with no surviving page (126-plato) delinked to the
containers/index.md archaeology row instead of a 404.
- ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine).
- netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate
record, matching the existing convention for hosts without a wiki page).
- runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md
(missed in the phase-4 runbook move because the referencing files used a
bare filename, not a runbooks/ prefix).
- One dangling forward-reference to a never-written investigation delinked
to the actual incident record it was describing.
Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into
devops/homelab-authentik-admin/ — an intentional reference to a sibling repo,
not present in this checkout.
Verification: broken-link count 126 -> 2 (real remainder is the cross-repo
reference above); gen-topology.py --check still exit 0; build_host_files.py
still idempotent; all inventory.yaml doc_page targets still resolve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add knowledge/wiki/hosts/index.md (the one missing section index) and point
knowledge/index.md at it.
- Add .agents/skills/docs-lint/ (SKILL.md + lint.py) enforcing the mechanical
parts of writing-style.md: banned vocabulary and broken relative links. The
style guide and this skill are exempt from the banned-word check since they
enumerate the list.
- Record the restructure + lint in knowledge/log.md.
Verification: banned-vocabulary scan of knowledge/ is clean (the few remaining
repo-wide hits are false positives — the literal '_' character — or historical
append-only plans quoting the vocabulary, which the standard does not restyle).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: runbooks are agent-executable procedures but lived at the repo root,
separate from the other agent instruction now under .agents/.
Change:
- Move runbooks/<name>.md -> .agents/skills/<name>/SKILL.md (folder per skill,
matching the wiki-hq skills layout). Frontmatter (name, risk_class, inputs,
verification, docs_update_checklist, transition) preserved.
- Rewrite links (inbound from plans; between-skill siblings) via the move map.
- Update prose references in AGENTS.md, HERMES.md, .agents/OIKOS.md, and the
operations schema; fix a pre-existing stale link to operations/commands.md.
No code consumed runbooks/ by path, so nothing else changes.
Verification: all SKILL.md frontmatter parses with valid risk_class; every
lifecycle transition resolves to an oikos/ontology.yaml state; broken-link
count 127 -> 126 (fixed one, introduced none).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem: the narrative docs lacked an enforceable style standard, and
agent-facing instruction (OIKOS/CAVEMAN/CONTRIBUTING) was interleaved with
human content at the repo root.
Change:
- Add .agents/shared/{writing-style,llm-wiki}.md — a lint-checkable prose
standard (with an imperative-voice exception for runbooks/recipes) and the
sources/wiki/index/log layer model.
- Move CAVEMAN.md -> .agents/shared/caveman.md,
CONTRIBUTING.md -> .agents/shared/page-templates.md,
OIKOS.md -> .agents/OIKOS.md; leave thin root stubs so old links resolve.
- Add .agents/domains/{knowledge,operations}/schema.md; operations schema
codifies "plans always live in plans/".
- Repoint live references (AGENTS, README, GLOSSARY, OIKOS) and fix OIKOS.md's
internal relative links for its new depth.
Risk: none to the operational substrate — inventory.yaml, hosts/*.yaml,
oikos/, mcp/, secrets/, bin/ untouched (verified via git status).
Verification: relative-link check across .agents/ clean; substrate churn empty.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Console is fully live on apps (105) — deployed manually via deploy.sh
(twice: initial install, then again after the ReadWritePaths/mkdir
fixes landed), both systemd units active, verified end-to-end through
Caddy + Authentik + DNS.
Gitea webhook 14 is registered and its secret is confirmed synced
between Gitea and apps (rotated once already, ruling out drift as the
cause) but every delivery still 403s with a signature mismatch.
Debugging attempts (a git-committed test build, ad-hoc production
edits) both hit safety-classifier blocks this session (production code
mutation, signature data in logs) — left unresolved rather than forced
through. Auto-deploy via push doesn't work yet for this service; manual
deploy.sh re-runs are the workaround until someone tracks this down.
Added to the 60/90-day backlog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The value written to apps' /etc/oikos-console-deploy/secret didn't
match what Gitea webhook 14 had configured, causing every deploy
attempt to 403 with a signature mismatch — likely drift introduced by
the earlier two-step PATCH sequence (secret set in one call,
branch_filter/active restored in a second call without re-including
the config object). Rotated cleanly this time: fresh secret set on
Gitea and re-encrypted here in one pass, single atomic PATCH covering
config+events+branch_filter+active together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Console is live: cloned to /opt/oikos-console, deploy.sh ran clean,
webhook secret written to /etc/oikos-console-deploy/secret from the
pre-registered SOPS secret (never printed — decrypted and piped
straight into the target file in one command), both systemd units
enabled and active. Verified locally (127.0.0.1:8091 -> 200) and
end-to-end (https://oikos.hubris.network/ -> 302, the Authentik gate
firing correctly).
Found a real bug during first boot: oikos-console.service's
ReadWritePaths listed /opt/homelab-context/signals and .../approvals,
but neither existed yet on apps' clone — git doesn't track empty
directories, and nothing had ever written a signal/approval from that
host. ProtectSystem=strict + a missing ReadWritePaths target is a hard
226/NAMESPACE crash, not a graceful degradation. Fixed two ways:
the unit now marks those paths optional (`-` prefix) so a fresh deploy
never crash-loops on this again, and deploy.sh now mkdir -p's them
explicitly so the console has real write access from the first boot,
not just a non-crashing-but-broken start.
This is also the first real exercise of the auto-deploy pipeline: this
push should land via Gitea webhook 14 -> oikos-console-deploy.service
on apps, same as homelab-mcp/secrets-issuance already work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A record -> 192.168.8.175 (Caddy's LAN IP), created via Technitium's
REST API (login -> createToken -> zones/records/add) in a single
in-memory call. Neither the admin credential nor the resulting session/
API token was ever printed to output or written to disk, and the token
wasn't persisted anywhere after the call completed — it existed only
for the lifetime of that one process.
Verified: dig @192.168.8.2 +short oikos.hubris.network -> 192.168.8.175.
End-to-end confirmation that DNS + Caddy + the Authentik gate are all
wired correctly: curl https://oikos.hubris.network/ now returns a 302
(the forward-auth redirect firing before the not-yet-deployed backend
would even matter) instead of failing to resolve/connect.
This closes out every part of the console rollout except the actual
apps-side bootstrap (oikos/console/deploy/README.md "One-time setup"),
which remains pending direct operator execution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pushed dtoro/caddy-conf@c195142: oikos.hubris.network -> 192.168.8.205:8091,
Authentik-gated (matches paperless.hubris.network's live pattern —
confirmed exact snippet syntax against the real Caddyfile rather than
trusting the paraphrase in the original README, which turned out to
have the wrong forward_auth target: the live snippet points at
127.0.0.1:8099 on Caddy's own LXC, not 192.168.8.6:9000 as
containers/106-auth-outpost.md's older text suggested). Reload verified
clean — an unrelated existing route stayed healthy through it.
Found and fixed a real deploy-blocking bug in the process:
oikos-console.service bound 127.0.0.1 only, but Caddy runs on a
different host (121) and can only reach apps (105) over the LAN — the
console would have been completely unreachable once deployed. Now binds
0.0.0.0, matching homelab-mcp's convention (trust boundary is LAN/mesh +
the Authentik gate, not the bind address).
Encountered and deliberately left alone: a pre-existing local clone at
/tmp/caddy-conf with an unpushed commit + uncommitted diff about
jellyfin's auth gating, from before this clone fell 12 commits behind
origin. That work turned out to be superseded (origin already reached
the same conclusion — SSO plugin handles jellyfin auth, no forward-auth
gate — via a different, already-merged path). Didn't touch it; used a
fresh clone instead to avoid any risk of losing or corrupting that state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Created via the Gitea API (POST /repos/dtoro/Homelab-Docs/hooks) rather
than the UI, since the existing PAT turned out to have sufficient scope.
Webhook id 14: http://192.168.8.205:9831/deploy, push events, main branch
filter, active.
The shared secret was generated and registered with Gitea before the
apps-side bootstrap ran (order reversed from the usual install.sh-first
flow, since direct SSH deploy to apps is still pending operator
execution — see oikos/console/deploy/README.md). Stored as
secrets/oikos-console-deploy-secret.yaml (SOPS, recipient: apps only)
rather than left as a local plaintext file, with explicit operator
sign-off. When the apps-side install runs, skip webhook/install.sh's
random-secret generation and write this exact value into
/etc/oikos-console-deploy/secret instead.
infrastructure/auto-deploy.md updated with the real webhook id (was
"not yet registered").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
teddycloud was live on hubris (LXC 131, docker compose, TeddyCloud —
a Toniebox cloud reimplementation) but never made it into inventory.yaml.
Already referenced in passing by containers/132-rclone.md ("131 was
already taken by an undocumented teddycloud container") and
hosts/strong.md's migration changelog (a DHCP conflict fix), but no
inventory entry or doc page existed until oikos/drift.py's inventory-
vs-live check caught it.
Verified live via read-only SSH (pct config 131, pct exec 131 -- ...,
docker ps): hostname, static IP 192.168.8.150, 1 core/1GiB/16GiB rootfs,
Debian 12, runs via docker compose at /opt/teddycloud. No changes made
to the running container.
Also fixed: house's inventory notes claimed 192.168.8.212 is teddycloud's
current IP via DHCP — stale, teddycloud has a static IP now.
Flagged in the new container page: teddycloud has no Caddy forward-auth
gate, unlike sab.hubris.network on the same Caddyfile.
`python3 oikos/drift.py` no longer reports an inventory-vs-live finding
for pve_id 131. (A separate, pre-existing gap surfaced while verifying
this: rclone's own inventory.yaml block is missing pve_id/host/lan_ip —
out of scope here, flagging for a follow-up.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oikos Console v0 (oikos/console/) — read-mostly, server-rendered FastAPI
+ Jinja2 web UI, no SPA build chain. Signals landing page, service grid
+ detail, node/blast-radius view, live Mermaid relationship graph, drift
findings, approvals queue (approve/deny, destructive confirmation-phrase
enforced), daily/weekly reports. Tested end-to-end via the preview tools
against live production data, including a real click-through of the
approve/deny flow.
Found and fixed two bugs during that testing:
- Severity-dot CSS classes didn't match the actual severity strings
(dot-warn/dot-crit vs "warning"/"critical") — warning-severity signals
rendered with no visible indicator at all.
- The console's sys.path setup pointed at its own webhook checkout
(/opt/oikos-console) rather than /opt/homelab-context, which would have
made its oikos.* imports resolve to a SEPARATE copy of oikos/signal.py
etc. than the scheduler and CLI use — silently forking signal/approval
data into two locations in production. Fixed to match mcp/server.py's
CONTEXT_DIR pattern. Also added _commit_push() so the console's writes
(approval replies, signal ack/resolve) don't sit uncommitted against
the 5-min-synced clone.
Split oikos/gen_topology_lib.py out of oikos/gen-topology.py (hyphenated
filenames aren't importable) so the console's /graph route can render
live without shelling out.
oikos/console/deploy/ — third webhook on dtoro/Homelab-Docs (port 9831),
matching the homelab-mcp/secrets-issuance precedent. README documents the
Caddy route and Gitea webhook registration this repo can't do for itself,
and that Authentik step-up on /approvals needs a live instance to
configure.
Approval hardening: grants are now single-use (oikos/approve.py
check_grant marks the request "executed" atomically, so a second call
for the same id fails even within the TTL) — verified with a test. Per-
agent age-key-signed requests, as originally planned, turned out not to
be buildable as stated: age is encryption-only, no signing primitive.
Documented the real alternative (SSH-key signing) and moved it to the
60/90-day backlog pending an inventory schema gap (no SSH pubkeys
recorded today).
Docs pass: added the Oikos command surface to operations/commands.md,
new MCP tools to AGENTS.md. Found two more stale references while at
it — commands.md and AGENTS.md both still pointed DNS at the destroyed
LXC 124/dnsmasq instead of Technitium on dns (107), and a claudio-monitor
reference deprecated since 2026-06-04 — fixed both.
60/90-day backlog written into OIKOS.md, derived from gaps actually
observed this month, not guesswork.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New kernel modules, all wired into `homelab` CLI + tested against live
production where reachable:
- oikos/scheduler.py — Observe stage: HTTP health probes for every
service, disk-usage probes on hubris/strong, writes oikos/state.json
(gitignored — regenerates every run). `homelab service <name> health`
is now cache-first; `--live` forces a fresh probe. Deploys via
oikos/systemd/oikos-scheduler.{timer,service} on LXC 105.
- oikos/drift.py — SOPS-recipient-vs-inventory and lifecycle-consistency
detectors (fully local, no SSH) plus pct-list and Caddy-backend
detectors (best-effort SSH, degrade to an info finding when
unreachable rather than a false drift alarm). Found real, currently-
true drift on first run: republic-laptop's age key granted on every
secret but missing from inventory.yaml, grimmory missing from
hello.yaml's recipients, and an undocumented pve_id 131 on hubris —
recorded in OIKOS.md for the operator, not auto-fixed (each is a
config_mutation/destructive decision).
- oikos/signal.py — the attention layer: raised -> acknowledged ->
acting -> resolved|muted lifecycle, severity-based routing, dedup via
open_signal_for(). `homelab signal list|raise|ack|resolve|mute`.
- oikos/decide.py — the Decide-stage classifier: risk class x blast
radius x ledger-history confidence -> auto-act/escalate. Adds an
action-alias layer (oikos/policy.py ACTION_ALIASES) and auto-infers
service_name from the entity for per-service policy overrides.
`homelab decide <action> <entity>`.
- oikos/approve.py — the escalate route. No dedicated Matrix bot exists
in this homelab, so this is the repo-side half only: request/reply/
grant lifecycle with short-TTL HMAC-signed tokens (new secret
secrets/oikos-approval-hmac.yaml, recipients apps+hubris). Matrix
delivery is Hermes's existing @dtoro:avispero send path (documented
integration contract in the module docstring), not a new bot.
`homelab restart` now mechanically refuses config_mutation/destructive
services without a valid --approval-id, regardless of -y/interactivity.
- oikos/report.py — daily brief + weekly report from signal/approval/
ledger state (no Prometheus yet, so point-in-time counts only).
- plans/2026-07-05-oikos-prometheus-lxc.md — Prometheus is `planned`,
not provisioned: no pve_id is guessed here since Proxmox assigns real
IDs at creation time, and drift already found an unclaimed ID (131) to
investigate first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Only off-LAN-reachable workstations (e.g. republic-laptop, mac-mini)
need to join Netbird. LAN-reachable LXCs/VMs on 192.168.8.0/24 don't —
they're already directly reachable, and off-LAN clients reach them via
hubris's routed 192.168.8.0/24 Netbird network resource. Brings the
runbook in line with oikos/ontology.yaml's lifecycle transition, which
already says "mesh-joined-if-needed".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the shared kernel modules (oikos/policy.py, oikos/relations.py,
oikos/ledger.py) that let every surface — CLI, MCP, context-card
generator — agree on risk classification and ontology graph walks
from one implementation.
homelab CLI: `service <name> explain|health|docs|log|actions|history`
(Service Console v0), `change preflight <service>`, `node <name>
relations`. Restart and client add/remove now append change-ledger
entries (ledger/*.jsonl, committed alongside the change they record).
mcp/server.py mirrors explain/preflight/get_relations/get_change_history
as MCP tools, card-first so agent orientation is one call instead of
several search_docs/get_page round-trips.
oikos/gen-topology.py now also emits a compact context card per host
and service (oikos/cards/*.md) — identity, blast radius, safe actions +
risk class, doc pointer, recent ledger history.
runbooks/*.md: service health check, config change + deploy, client
enrollment, incident investigation, and the five node lifecycle
transitions (provision/activate/migrate/deprecate/destroy), each with
machine-readable frontmatter (risk class, inputs, verification,
docs-update checklist). Wired into HERMES.md so agents load these
instead of rediscovering topology per-task.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the Oikos agent-OS kernel: oikos/policy.yaml (risk classes +
approval rules for every homelab/MCP command), oikos/ontology.yaml
(8-domain systems model, typed relationships, node lifecycle), and
OIKOS.md (OODA loop operating brief, linked from AGENTS.md).
Extends inventory.yaml with a stable service contract (doc_page,
config_repo, risk_notes) on all 17 services, and a structured
archaeology: section for the 13 destroyed LXCs (was scattered
comments + a narrative table). Fixes stale drift found in the
process: authentik's backend pointed at a retired LXC (124); core
has run on the VPS since 2026-05-31.
Adds oikos/gen-topology.py, generating infrastructure/topology.md
(Mermaid compute/ingress + storage views) from inventory.yaml.
build_host_files.py now carries state/storage/depends_on into
generated hosts/*.yaml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.
Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.
- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
register the new container.
