From 6b75f7302d87f97527eb6bfc7b80f180b2037707 Mon Sep 17 00:00:00 2001 From: dtoro Date: Tue, 7 Jul 2026 20:22:30 +0200 Subject: [PATCH] =?UTF-8?q?db=20as=20source=20of=20truth:=20wiki=E2=86=92s?= =?UTF-8?q?eeds,=20archive=20old=20artifacts,=20knowledge=20ingestion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Migrations 010 (content_hash) + 011 (search tsvector column) - new: internal/knowledge/seed.go — knowledge seed ingest engine - new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge - wire knowledge ingest into oikos seed pipeline - convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml - archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/ - delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py - remove empty knowledge/ directory tree --- ...0000-library-ssd-migration-to-ludo-mini.md | 0 ...06-03_150000-homelab-structure-revision.md | 0 ...-03_223218-dhcp-pool-exclude-static-ips.md | 0 ...2026-06-05_170000-prevent-dhcp-ip-drift.md | 0 ...-06_232200-authentik-frequent-login-fix.md | 0 ...4500-caddyfile-truncation-permanent-fix.md | 0 .../2026-07-05_strong-migration-assessment.md | 0 {knowledge => archive/knowledge}/GLOSSARY.md | 0 .../knowledge}/containers/101-jellyfin.md | 0 .../knowledge}/containers/102-nfs-export.md | 0 .../knowledge}/containers/103-paperless.md | 0 .../knowledge}/containers/104-gitea.md | 0 .../knowledge}/containers/105-apps.md | 0 .../knowledge}/containers/106-auth-outpost.md | 0 .../knowledge}/containers/107-dns.md | 0 .../knowledge}/containers/114-nextcloud.md | 0 .../containers/118-elementsynapse.md | 0 .../knowledge}/containers/119-sophia.md | 0 .../knowledge}/containers/120-mule-images.md | 0 .../knowledge}/containers/121-caddy.md | 0 .../knowledge}/containers/122-arriman.md | 0 .../knowledge}/containers/128-trmnl.md | 0 .../knowledge}/containers/129-house.md | 0 .../knowledge}/containers/130-grimmory.md | 0 .../knowledge}/containers/131-teddycloud.md | 0 .../knowledge}/containers/132-rclone.md | 0 .../knowledge}/containers/133-seanime.md | 0 .../knowledge}/containers/134-romm.md | 0 .../knowledge}/containers/index.md | 0 .../knowledge}/hosts/hubris.md | 0 .../wiki => archive/knowledge}/hosts/index.md | 0 .../knowledge}/hosts/strong.md | 0 {knowledge => archive/knowledge}/index.md | 0 .../knowledge}/infrastructure/auto-deploy.md | 0 .../knowledge}/infrastructure/backups.md | 0 .../knowledge}/infrastructure/dns.md | 0 .../infrastructure/homelab-context.md | 0 .../knowledge}/infrastructure/index.md | 0 .../knowledge}/infrastructure/ingress.md | 0 .../infrastructure/media-permissions.md | 0 .../knowledge}/infrastructure/mesh.md | 0 .../knowledge}/infrastructure/monitoring.md | 0 .../knowledge}/infrastructure/network.md | 0 .../knowledge}/infrastructure/ssh-access.md | 0 .../knowledge}/infrastructure/topology.md | 0 .../infrastructure/vps-hardening.md | 0 .../2026-04-21-hubris-crash-loop.md | 0 .../2026-05-31-authentik-vps-migration.md | 0 .../2026-06-01-mac-mini-onboarding.md | 0 ...26-06-03-moonlight-sunshine-wifi-jitter.md | 0 .../2026-06-06-authentik-session-lifetime.md | 0 .../2026-06-06-caddyfile-truncation.md | 0 .../knowledge}/investigations/index.md | 0 {knowledge => archive/knowledge}/log.md | 0 .../cert-sync-and-traefik-config.md | 0 .../knowledge}/sources/index.md | 0 .../knowledge}/vms/100-zimaos.md | 0 .../knowledge}/vms/108-haos.md | 0 .../wiki => archive/knowledge}/vms/index.md | 0 {ledger => archive/ledger}/2026-07.jsonl | 0 {mcp => archive/mcp}/build_host_files.py | 0 .../oikos-cards}/cards/host-apps.md | 0 .../oikos-cards}/cards/host-arriman.md | 0 .../oikos-cards}/cards/host-auth-outpost.md | 0 .../oikos-cards}/cards/host-caddy.md | 0 .../oikos-cards}/cards/host-dns.md | 0 .../oikos-cards}/cards/host-elementsynapse.md | 0 .../oikos-cards}/cards/host-gitea.md | 0 .../oikos-cards}/cards/host-grimmory.md | 0 .../oikos-cards}/cards/host-haos.md | 0 .../oikos-cards}/cards/host-house.md | 0 .../oikos-cards}/cards/host-hubris.md | 0 .../oikos-cards}/cards/host-jellyfin.md | 0 .../oikos-cards}/cards/host-mac-mini.md | 0 .../oikos-cards}/cards/host-mule-images.md | 0 .../oikos-cards}/cards/host-netbird-vps.md | 0 .../oikos-cards}/cards/host-nextcloud.md | 0 .../oikos-cards}/cards/host-nfs-export.md | 0 .../oikos-cards}/cards/host-paperless.md | 0 .../oikos-cards}/cards/host-rclone.md | 0 .../cards/host-republic-laptop.md | 0 .../oikos-cards}/cards/host-romm.md | 0 .../oikos-cards}/cards/host-seanime.md | 0 .../oikos-cards}/cards/host-sophia.md | 0 .../oikos-cards}/cards/host-strong.md | 0 .../oikos-cards}/cards/host-teddycloud.md | 0 .../oikos-cards}/cards/host-trmnl.md | 0 .../oikos-cards}/cards/host-zimaos.md | 0 .../oikos-cards}/cards/service-arr_stack.md | 0 .../oikos-cards}/cards/service-artifacto.md | 0 .../oikos-cards}/cards/service-authentik.md | 0 .../oikos-cards}/cards/service-caddy.md | 0 .../oikos-cards}/cards/service-dns.md | 0 .../oikos-cards}/cards/service-gitea.md | 0 .../oikos-cards}/cards/service-haos.md | 0 .../oikos-cards}/cards/service-homelab_mcp.md | 0 .../oikos-cards}/cards/service-jellyfin.md | 0 .../oikos-cards}/cards/service-matrix.md | 0 .../oikos-cards}/cards/service-nextcloud.md | 0 .../oikos-cards}/cards/service-paperless.md | 0 .../oikos-cards}/cards/service-photos.md | 0 .../oikos-cards}/cards/service-proxmox_ui.md | 0 .../cards/service-secrets_issuance.md | 0 .../oikos-cards}/cards/service-teddycloud.md | 0 .../oikos-cards}/cards/service-trmnl.md | 0 .../oikos-cards}/cards/service-zimaos.md | 0 cmd/oikos/main.go | 28 + internal/httpapi/knowledge.go | 142 + internal/httpapi/phase3.go | 10 - internal/knowledge/seed.go | 276 + .../containers/archive/123-claudio-bot.md | 90 - .../containers/archive/127-mule-photos-new.md | 313 - migrations/010_knowledge_hash.up.sql | 2 + migrations/011_knowledge_search.up.sql | 4 + oikos/__init__.py | 0 oikos/approve.py | 302 - oikos/decide.py | 140 - oikos/drift.py | 302 - oikos/ledger.py | 109 - oikos/policy.py | 72 - oikos/relations.py | 131 - oikos/scheduler.py | 230 - oikos/signal.py | 239 - scripts/convert-wiki.py | 397 + seeds/knowledge.yaml | 6708 +++++++++++++++++ 125 files changed, 7557 insertions(+), 1938 deletions(-) rename {.hermes/plans => archive/hermes-plans}/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-06-03_150000-homelab-structure-revision.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-06-03_223218-dhcp-pool-exclude-static-ips.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-06-05_170000-prevent-dhcp-ip-drift.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-06-06_232200-authentik-frequent-login-fix.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-06-06_234500-caddyfile-truncation-permanent-fix.md (100%) rename {.hermes/plans => archive/hermes-plans}/2026-07-05_strong-migration-assessment.md (100%) rename {knowledge => archive/knowledge}/GLOSSARY.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/101-jellyfin.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/102-nfs-export.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/103-paperless.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/104-gitea.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/105-apps.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/106-auth-outpost.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/107-dns.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/114-nextcloud.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/118-elementsynapse.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/119-sophia.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/120-mule-images.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/121-caddy.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/122-arriman.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/128-trmnl.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/129-house.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/130-grimmory.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/131-teddycloud.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/132-rclone.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/133-seanime.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/134-romm.md (100%) rename {knowledge/wiki => archive/knowledge}/containers/index.md (100%) rename {knowledge/wiki => archive/knowledge}/hosts/hubris.md (100%) rename {knowledge/wiki => archive/knowledge}/hosts/index.md (100%) rename {knowledge/wiki => archive/knowledge}/hosts/strong.md (100%) rename {knowledge => archive/knowledge}/index.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/auto-deploy.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/backups.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/dns.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/homelab-context.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/index.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/ingress.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/media-permissions.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/mesh.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/monitoring.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/network.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/ssh-access.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/topology.md (100%) rename {knowledge/wiki => archive/knowledge}/infrastructure/vps-hardening.md (100%) rename {knowledge/sources/investigations/archive => archive/knowledge/investigations}/2026-04-21-hubris-crash-loop.md (100%) rename {knowledge/sources/investigations/archive => archive/knowledge/investigations}/2026-05-31-authentik-vps-migration.md (100%) rename {knowledge/sources => archive/knowledge}/investigations/2026-06-01-mac-mini-onboarding.md (100%) rename {knowledge/sources => archive/knowledge}/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md (100%) rename {knowledge/sources => archive/knowledge}/investigations/2026-06-06-authentik-session-lifetime.md (100%) rename {knowledge/sources => archive/knowledge}/investigations/2026-06-06-caddyfile-truncation.md (100%) rename {knowledge/sources => archive/knowledge}/investigations/index.md (100%) rename {knowledge => archive/knowledge}/log.md (100%) rename {knowledge/sources => archive/knowledge}/references/cert-sync-and-traefik-config.md (100%) rename {knowledge => archive/knowledge}/sources/index.md (100%) rename {knowledge/wiki => archive/knowledge}/vms/100-zimaos.md (100%) rename {knowledge/wiki => archive/knowledge}/vms/108-haos.md (100%) rename {knowledge/wiki => archive/knowledge}/vms/index.md (100%) rename {ledger => archive/ledger}/2026-07.jsonl (100%) rename {mcp => archive/mcp}/build_host_files.py (100%) rename {oikos => archive/oikos-cards}/cards/host-apps.md (100%) rename {oikos => archive/oikos-cards}/cards/host-arriman.md (100%) rename {oikos => archive/oikos-cards}/cards/host-auth-outpost.md (100%) rename {oikos => archive/oikos-cards}/cards/host-caddy.md (100%) rename {oikos => archive/oikos-cards}/cards/host-dns.md (100%) rename {oikos => archive/oikos-cards}/cards/host-elementsynapse.md (100%) rename {oikos => archive/oikos-cards}/cards/host-gitea.md (100%) rename {oikos => archive/oikos-cards}/cards/host-grimmory.md (100%) rename {oikos => archive/oikos-cards}/cards/host-haos.md (100%) rename {oikos => archive/oikos-cards}/cards/host-house.md (100%) rename {oikos => archive/oikos-cards}/cards/host-hubris.md (100%) rename {oikos => archive/oikos-cards}/cards/host-jellyfin.md (100%) rename {oikos => archive/oikos-cards}/cards/host-mac-mini.md (100%) rename {oikos => archive/oikos-cards}/cards/host-mule-images.md (100%) rename {oikos => archive/oikos-cards}/cards/host-netbird-vps.md (100%) rename {oikos => archive/oikos-cards}/cards/host-nextcloud.md (100%) rename {oikos => archive/oikos-cards}/cards/host-nfs-export.md (100%) rename {oikos => archive/oikos-cards}/cards/host-paperless.md (100%) rename {oikos => archive/oikos-cards}/cards/host-rclone.md (100%) rename {oikos => archive/oikos-cards}/cards/host-republic-laptop.md (100%) rename {oikos => archive/oikos-cards}/cards/host-romm.md (100%) rename {oikos => archive/oikos-cards}/cards/host-seanime.md (100%) rename {oikos => archive/oikos-cards}/cards/host-sophia.md (100%) rename {oikos => archive/oikos-cards}/cards/host-strong.md (100%) rename {oikos => archive/oikos-cards}/cards/host-teddycloud.md (100%) rename {oikos => archive/oikos-cards}/cards/host-trmnl.md (100%) rename {oikos => archive/oikos-cards}/cards/host-zimaos.md (100%) rename {oikos => archive/oikos-cards}/cards/service-arr_stack.md (100%) rename {oikos => archive/oikos-cards}/cards/service-artifacto.md (100%) rename {oikos => archive/oikos-cards}/cards/service-authentik.md (100%) rename {oikos => archive/oikos-cards}/cards/service-caddy.md (100%) rename {oikos => archive/oikos-cards}/cards/service-dns.md (100%) rename {oikos => archive/oikos-cards}/cards/service-gitea.md (100%) rename {oikos => archive/oikos-cards}/cards/service-haos.md (100%) rename {oikos => archive/oikos-cards}/cards/service-homelab_mcp.md (100%) rename {oikos => archive/oikos-cards}/cards/service-jellyfin.md (100%) rename {oikos => archive/oikos-cards}/cards/service-matrix.md (100%) rename {oikos => archive/oikos-cards}/cards/service-nextcloud.md (100%) rename {oikos => archive/oikos-cards}/cards/service-paperless.md (100%) rename {oikos => archive/oikos-cards}/cards/service-photos.md (100%) rename {oikos => archive/oikos-cards}/cards/service-proxmox_ui.md (100%) rename {oikos => archive/oikos-cards}/cards/service-secrets_issuance.md (100%) rename {oikos => archive/oikos-cards}/cards/service-teddycloud.md (100%) rename {oikos => archive/oikos-cards}/cards/service-trmnl.md (100%) rename {oikos => archive/oikos-cards}/cards/service-zimaos.md (100%) create mode 100644 internal/httpapi/knowledge.go create mode 100644 internal/knowledge/seed.go delete mode 100644 knowledge/wiki/containers/archive/123-claudio-bot.md delete mode 100644 knowledge/wiki/containers/archive/127-mule-photos-new.md create mode 100644 migrations/010_knowledge_hash.up.sql create mode 100644 migrations/011_knowledge_search.up.sql delete mode 100644 oikos/__init__.py delete mode 100644 oikos/approve.py delete mode 100644 oikos/decide.py delete mode 100644 oikos/drift.py delete mode 100644 oikos/ledger.py delete mode 100644 oikos/policy.py delete mode 100644 oikos/relations.py delete mode 100644 oikos/scheduler.py delete mode 100644 oikos/signal.py create mode 100644 scripts/convert-wiki.py create mode 100644 seeds/knowledge.yaml diff --git a/.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md b/archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md similarity index 100% rename from .hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md rename to archive/hermes-plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md diff --git a/.hermes/plans/2026-06-03_150000-homelab-structure-revision.md b/archive/hermes-plans/2026-06-03_150000-homelab-structure-revision.md similarity index 100% rename from .hermes/plans/2026-06-03_150000-homelab-structure-revision.md rename to archive/hermes-plans/2026-06-03_150000-homelab-structure-revision.md diff --git a/.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md b/archive/hermes-plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md similarity index 100% rename from .hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md rename to archive/hermes-plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md diff --git a/.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md b/archive/hermes-plans/2026-06-05_170000-prevent-dhcp-ip-drift.md similarity index 100% rename from .hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md rename to archive/hermes-plans/2026-06-05_170000-prevent-dhcp-ip-drift.md diff --git a/.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md b/archive/hermes-plans/2026-06-06_232200-authentik-frequent-login-fix.md similarity index 100% rename from .hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md rename to archive/hermes-plans/2026-06-06_232200-authentik-frequent-login-fix.md diff --git a/.hermes/plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md b/archive/hermes-plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md similarity index 100% rename from .hermes/plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md rename to archive/hermes-plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md diff --git a/.hermes/plans/2026-07-05_strong-migration-assessment.md b/archive/hermes-plans/2026-07-05_strong-migration-assessment.md similarity index 100% rename from .hermes/plans/2026-07-05_strong-migration-assessment.md rename to archive/hermes-plans/2026-07-05_strong-migration-assessment.md diff --git a/knowledge/GLOSSARY.md b/archive/knowledge/GLOSSARY.md similarity index 100% rename from knowledge/GLOSSARY.md rename to archive/knowledge/GLOSSARY.md diff --git a/knowledge/wiki/containers/101-jellyfin.md b/archive/knowledge/containers/101-jellyfin.md similarity index 100% rename from knowledge/wiki/containers/101-jellyfin.md rename to archive/knowledge/containers/101-jellyfin.md diff --git a/knowledge/wiki/containers/102-nfs-export.md b/archive/knowledge/containers/102-nfs-export.md similarity index 100% rename from knowledge/wiki/containers/102-nfs-export.md rename to archive/knowledge/containers/102-nfs-export.md diff --git a/knowledge/wiki/containers/103-paperless.md b/archive/knowledge/containers/103-paperless.md similarity index 100% rename from knowledge/wiki/containers/103-paperless.md rename to archive/knowledge/containers/103-paperless.md diff --git a/knowledge/wiki/containers/104-gitea.md b/archive/knowledge/containers/104-gitea.md similarity index 100% rename from knowledge/wiki/containers/104-gitea.md rename to archive/knowledge/containers/104-gitea.md diff --git a/knowledge/wiki/containers/105-apps.md b/archive/knowledge/containers/105-apps.md similarity index 100% rename from knowledge/wiki/containers/105-apps.md rename to archive/knowledge/containers/105-apps.md diff --git a/knowledge/wiki/containers/106-auth-outpost.md b/archive/knowledge/containers/106-auth-outpost.md similarity index 100% rename from knowledge/wiki/containers/106-auth-outpost.md rename to archive/knowledge/containers/106-auth-outpost.md diff --git a/knowledge/wiki/containers/107-dns.md b/archive/knowledge/containers/107-dns.md similarity index 100% rename from knowledge/wiki/containers/107-dns.md rename to archive/knowledge/containers/107-dns.md diff --git a/knowledge/wiki/containers/114-nextcloud.md b/archive/knowledge/containers/114-nextcloud.md similarity index 100% rename from knowledge/wiki/containers/114-nextcloud.md rename to archive/knowledge/containers/114-nextcloud.md diff --git a/knowledge/wiki/containers/118-elementsynapse.md b/archive/knowledge/containers/118-elementsynapse.md similarity index 100% rename from knowledge/wiki/containers/118-elementsynapse.md rename to archive/knowledge/containers/118-elementsynapse.md diff --git a/knowledge/wiki/containers/119-sophia.md b/archive/knowledge/containers/119-sophia.md similarity index 100% rename from knowledge/wiki/containers/119-sophia.md rename to archive/knowledge/containers/119-sophia.md diff --git a/knowledge/wiki/containers/120-mule-images.md b/archive/knowledge/containers/120-mule-images.md similarity index 100% rename from knowledge/wiki/containers/120-mule-images.md rename to archive/knowledge/containers/120-mule-images.md diff --git a/knowledge/wiki/containers/121-caddy.md b/archive/knowledge/containers/121-caddy.md similarity index 100% rename from knowledge/wiki/containers/121-caddy.md rename to archive/knowledge/containers/121-caddy.md diff --git a/knowledge/wiki/containers/122-arriman.md b/archive/knowledge/containers/122-arriman.md similarity index 100% rename from knowledge/wiki/containers/122-arriman.md rename to archive/knowledge/containers/122-arriman.md diff --git a/knowledge/wiki/containers/128-trmnl.md b/archive/knowledge/containers/128-trmnl.md similarity index 100% rename from knowledge/wiki/containers/128-trmnl.md rename to archive/knowledge/containers/128-trmnl.md diff --git a/knowledge/wiki/containers/129-house.md b/archive/knowledge/containers/129-house.md similarity index 100% rename from knowledge/wiki/containers/129-house.md rename to archive/knowledge/containers/129-house.md diff --git a/knowledge/wiki/containers/130-grimmory.md b/archive/knowledge/containers/130-grimmory.md similarity index 100% rename from knowledge/wiki/containers/130-grimmory.md rename to archive/knowledge/containers/130-grimmory.md diff --git a/knowledge/wiki/containers/131-teddycloud.md b/archive/knowledge/containers/131-teddycloud.md similarity index 100% rename from knowledge/wiki/containers/131-teddycloud.md rename to archive/knowledge/containers/131-teddycloud.md diff --git a/knowledge/wiki/containers/132-rclone.md b/archive/knowledge/containers/132-rclone.md similarity index 100% rename from knowledge/wiki/containers/132-rclone.md rename to archive/knowledge/containers/132-rclone.md diff --git a/knowledge/wiki/containers/133-seanime.md b/archive/knowledge/containers/133-seanime.md similarity index 100% rename from knowledge/wiki/containers/133-seanime.md rename to archive/knowledge/containers/133-seanime.md diff --git a/knowledge/wiki/containers/134-romm.md b/archive/knowledge/containers/134-romm.md similarity index 100% rename from knowledge/wiki/containers/134-romm.md rename to archive/knowledge/containers/134-romm.md diff --git a/knowledge/wiki/containers/index.md b/archive/knowledge/containers/index.md similarity index 100% rename from knowledge/wiki/containers/index.md rename to archive/knowledge/containers/index.md diff --git a/knowledge/wiki/hosts/hubris.md b/archive/knowledge/hosts/hubris.md similarity index 100% rename from knowledge/wiki/hosts/hubris.md rename to archive/knowledge/hosts/hubris.md diff --git a/knowledge/wiki/hosts/index.md b/archive/knowledge/hosts/index.md similarity index 100% rename from knowledge/wiki/hosts/index.md rename to archive/knowledge/hosts/index.md diff --git a/knowledge/wiki/hosts/strong.md b/archive/knowledge/hosts/strong.md similarity index 100% rename from knowledge/wiki/hosts/strong.md rename to archive/knowledge/hosts/strong.md diff --git a/knowledge/index.md b/archive/knowledge/index.md similarity index 100% rename from knowledge/index.md rename to archive/knowledge/index.md diff --git a/knowledge/wiki/infrastructure/auto-deploy.md b/archive/knowledge/infrastructure/auto-deploy.md similarity index 100% rename from knowledge/wiki/infrastructure/auto-deploy.md rename to archive/knowledge/infrastructure/auto-deploy.md diff --git a/knowledge/wiki/infrastructure/backups.md b/archive/knowledge/infrastructure/backups.md similarity index 100% rename from knowledge/wiki/infrastructure/backups.md rename to archive/knowledge/infrastructure/backups.md diff --git a/knowledge/wiki/infrastructure/dns.md b/archive/knowledge/infrastructure/dns.md similarity index 100% rename from knowledge/wiki/infrastructure/dns.md rename to archive/knowledge/infrastructure/dns.md diff --git a/knowledge/wiki/infrastructure/homelab-context.md b/archive/knowledge/infrastructure/homelab-context.md similarity index 100% rename from knowledge/wiki/infrastructure/homelab-context.md rename to archive/knowledge/infrastructure/homelab-context.md diff --git a/knowledge/wiki/infrastructure/index.md b/archive/knowledge/infrastructure/index.md similarity index 100% rename from knowledge/wiki/infrastructure/index.md rename to archive/knowledge/infrastructure/index.md diff --git a/knowledge/wiki/infrastructure/ingress.md b/archive/knowledge/infrastructure/ingress.md similarity index 100% rename from knowledge/wiki/infrastructure/ingress.md rename to archive/knowledge/infrastructure/ingress.md diff --git a/knowledge/wiki/infrastructure/media-permissions.md b/archive/knowledge/infrastructure/media-permissions.md similarity index 100% rename from knowledge/wiki/infrastructure/media-permissions.md rename to archive/knowledge/infrastructure/media-permissions.md diff --git a/knowledge/wiki/infrastructure/mesh.md b/archive/knowledge/infrastructure/mesh.md similarity index 100% rename from knowledge/wiki/infrastructure/mesh.md rename to archive/knowledge/infrastructure/mesh.md diff --git a/knowledge/wiki/infrastructure/monitoring.md b/archive/knowledge/infrastructure/monitoring.md similarity index 100% rename from knowledge/wiki/infrastructure/monitoring.md rename to archive/knowledge/infrastructure/monitoring.md diff --git a/knowledge/wiki/infrastructure/network.md b/archive/knowledge/infrastructure/network.md similarity index 100% rename from knowledge/wiki/infrastructure/network.md rename to archive/knowledge/infrastructure/network.md diff --git a/knowledge/wiki/infrastructure/ssh-access.md b/archive/knowledge/infrastructure/ssh-access.md similarity index 100% rename from knowledge/wiki/infrastructure/ssh-access.md rename to archive/knowledge/infrastructure/ssh-access.md diff --git a/knowledge/wiki/infrastructure/topology.md b/archive/knowledge/infrastructure/topology.md similarity index 100% rename from knowledge/wiki/infrastructure/topology.md rename to archive/knowledge/infrastructure/topology.md diff --git a/knowledge/wiki/infrastructure/vps-hardening.md b/archive/knowledge/infrastructure/vps-hardening.md similarity index 100% rename from knowledge/wiki/infrastructure/vps-hardening.md rename to archive/knowledge/infrastructure/vps-hardening.md diff --git a/knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md b/archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md similarity index 100% rename from knowledge/sources/investigations/archive/2026-04-21-hubris-crash-loop.md rename to archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md diff --git a/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md b/archive/knowledge/investigations/2026-05-31-authentik-vps-migration.md similarity index 100% rename from knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md rename to archive/knowledge/investigations/2026-05-31-authentik-vps-migration.md diff --git a/knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md b/archive/knowledge/investigations/2026-06-01-mac-mini-onboarding.md similarity index 100% rename from knowledge/sources/investigations/2026-06-01-mac-mini-onboarding.md rename to archive/knowledge/investigations/2026-06-01-mac-mini-onboarding.md diff --git a/knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md b/archive/knowledge/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md similarity index 100% rename from knowledge/sources/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md rename to archive/knowledge/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md diff --git a/knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md b/archive/knowledge/investigations/2026-06-06-authentik-session-lifetime.md similarity index 100% rename from knowledge/sources/investigations/2026-06-06-authentik-session-lifetime.md rename to archive/knowledge/investigations/2026-06-06-authentik-session-lifetime.md diff --git a/knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md b/archive/knowledge/investigations/2026-06-06-caddyfile-truncation.md similarity index 100% rename from knowledge/sources/investigations/2026-06-06-caddyfile-truncation.md rename to archive/knowledge/investigations/2026-06-06-caddyfile-truncation.md diff --git a/knowledge/sources/investigations/index.md b/archive/knowledge/investigations/index.md similarity index 100% rename from knowledge/sources/investigations/index.md rename to archive/knowledge/investigations/index.md diff --git a/knowledge/log.md b/archive/knowledge/log.md similarity index 100% rename from knowledge/log.md rename to archive/knowledge/log.md diff --git a/knowledge/sources/references/cert-sync-and-traefik-config.md b/archive/knowledge/references/cert-sync-and-traefik-config.md similarity index 100% rename from knowledge/sources/references/cert-sync-and-traefik-config.md rename to archive/knowledge/references/cert-sync-and-traefik-config.md diff --git a/knowledge/sources/index.md b/archive/knowledge/sources/index.md similarity index 100% rename from knowledge/sources/index.md rename to archive/knowledge/sources/index.md diff --git a/knowledge/wiki/vms/100-zimaos.md b/archive/knowledge/vms/100-zimaos.md similarity index 100% rename from knowledge/wiki/vms/100-zimaos.md rename to archive/knowledge/vms/100-zimaos.md diff --git a/knowledge/wiki/vms/108-haos.md b/archive/knowledge/vms/108-haos.md similarity index 100% rename from knowledge/wiki/vms/108-haos.md rename to archive/knowledge/vms/108-haos.md diff --git a/knowledge/wiki/vms/index.md b/archive/knowledge/vms/index.md similarity index 100% rename from knowledge/wiki/vms/index.md rename to archive/knowledge/vms/index.md diff --git a/ledger/2026-07.jsonl b/archive/ledger/2026-07.jsonl similarity index 100% rename from ledger/2026-07.jsonl rename to archive/ledger/2026-07.jsonl diff --git a/mcp/build_host_files.py b/archive/mcp/build_host_files.py similarity index 100% rename from mcp/build_host_files.py rename to archive/mcp/build_host_files.py diff --git a/oikos/cards/host-apps.md b/archive/oikos-cards/cards/host-apps.md similarity index 100% rename from oikos/cards/host-apps.md rename to archive/oikos-cards/cards/host-apps.md diff --git a/oikos/cards/host-arriman.md b/archive/oikos-cards/cards/host-arriman.md similarity index 100% rename from oikos/cards/host-arriman.md rename to archive/oikos-cards/cards/host-arriman.md diff --git a/oikos/cards/host-auth-outpost.md b/archive/oikos-cards/cards/host-auth-outpost.md similarity index 100% rename from oikos/cards/host-auth-outpost.md rename to archive/oikos-cards/cards/host-auth-outpost.md diff --git a/oikos/cards/host-caddy.md b/archive/oikos-cards/cards/host-caddy.md similarity index 100% rename from oikos/cards/host-caddy.md rename to archive/oikos-cards/cards/host-caddy.md diff --git a/oikos/cards/host-dns.md b/archive/oikos-cards/cards/host-dns.md similarity index 100% rename from oikos/cards/host-dns.md rename to archive/oikos-cards/cards/host-dns.md diff --git a/oikos/cards/host-elementsynapse.md b/archive/oikos-cards/cards/host-elementsynapse.md similarity index 100% rename from oikos/cards/host-elementsynapse.md rename to archive/oikos-cards/cards/host-elementsynapse.md diff --git a/oikos/cards/host-gitea.md b/archive/oikos-cards/cards/host-gitea.md similarity index 100% rename from oikos/cards/host-gitea.md rename to archive/oikos-cards/cards/host-gitea.md diff --git a/oikos/cards/host-grimmory.md b/archive/oikos-cards/cards/host-grimmory.md similarity index 100% rename from oikos/cards/host-grimmory.md rename to archive/oikos-cards/cards/host-grimmory.md diff --git a/oikos/cards/host-haos.md b/archive/oikos-cards/cards/host-haos.md similarity index 100% rename from oikos/cards/host-haos.md rename to archive/oikos-cards/cards/host-haos.md diff --git a/oikos/cards/host-house.md b/archive/oikos-cards/cards/host-house.md similarity index 100% rename from oikos/cards/host-house.md rename to archive/oikos-cards/cards/host-house.md diff --git a/oikos/cards/host-hubris.md b/archive/oikos-cards/cards/host-hubris.md similarity index 100% rename from oikos/cards/host-hubris.md rename to archive/oikos-cards/cards/host-hubris.md diff --git a/oikos/cards/host-jellyfin.md b/archive/oikos-cards/cards/host-jellyfin.md similarity index 100% rename from oikos/cards/host-jellyfin.md rename to archive/oikos-cards/cards/host-jellyfin.md diff --git a/oikos/cards/host-mac-mini.md b/archive/oikos-cards/cards/host-mac-mini.md similarity index 100% rename from oikos/cards/host-mac-mini.md rename to archive/oikos-cards/cards/host-mac-mini.md diff --git a/oikos/cards/host-mule-images.md b/archive/oikos-cards/cards/host-mule-images.md similarity index 100% rename from oikos/cards/host-mule-images.md rename to archive/oikos-cards/cards/host-mule-images.md diff --git a/oikos/cards/host-netbird-vps.md b/archive/oikos-cards/cards/host-netbird-vps.md similarity index 100% rename from oikos/cards/host-netbird-vps.md rename to archive/oikos-cards/cards/host-netbird-vps.md diff --git a/oikos/cards/host-nextcloud.md b/archive/oikos-cards/cards/host-nextcloud.md similarity index 100% rename from oikos/cards/host-nextcloud.md rename to archive/oikos-cards/cards/host-nextcloud.md diff --git a/oikos/cards/host-nfs-export.md b/archive/oikos-cards/cards/host-nfs-export.md similarity index 100% rename from oikos/cards/host-nfs-export.md rename to archive/oikos-cards/cards/host-nfs-export.md diff --git a/oikos/cards/host-paperless.md b/archive/oikos-cards/cards/host-paperless.md similarity index 100% rename from oikos/cards/host-paperless.md rename to archive/oikos-cards/cards/host-paperless.md diff --git a/oikos/cards/host-rclone.md b/archive/oikos-cards/cards/host-rclone.md similarity index 100% rename from oikos/cards/host-rclone.md rename to archive/oikos-cards/cards/host-rclone.md diff --git a/oikos/cards/host-republic-laptop.md b/archive/oikos-cards/cards/host-republic-laptop.md similarity index 100% rename from oikos/cards/host-republic-laptop.md rename to archive/oikos-cards/cards/host-republic-laptop.md diff --git a/oikos/cards/host-romm.md b/archive/oikos-cards/cards/host-romm.md similarity index 100% rename from oikos/cards/host-romm.md rename to archive/oikos-cards/cards/host-romm.md diff --git a/oikos/cards/host-seanime.md b/archive/oikos-cards/cards/host-seanime.md similarity index 100% rename from oikos/cards/host-seanime.md rename to archive/oikos-cards/cards/host-seanime.md diff --git a/oikos/cards/host-sophia.md b/archive/oikos-cards/cards/host-sophia.md similarity index 100% rename from oikos/cards/host-sophia.md rename to archive/oikos-cards/cards/host-sophia.md diff --git a/oikos/cards/host-strong.md b/archive/oikos-cards/cards/host-strong.md similarity index 100% rename from oikos/cards/host-strong.md rename to archive/oikos-cards/cards/host-strong.md diff --git a/oikos/cards/host-teddycloud.md b/archive/oikos-cards/cards/host-teddycloud.md similarity index 100% rename from oikos/cards/host-teddycloud.md rename to archive/oikos-cards/cards/host-teddycloud.md diff --git a/oikos/cards/host-trmnl.md b/archive/oikos-cards/cards/host-trmnl.md similarity index 100% rename from oikos/cards/host-trmnl.md rename to archive/oikos-cards/cards/host-trmnl.md diff --git a/oikos/cards/host-zimaos.md b/archive/oikos-cards/cards/host-zimaos.md similarity index 100% rename from oikos/cards/host-zimaos.md rename to archive/oikos-cards/cards/host-zimaos.md diff --git a/oikos/cards/service-arr_stack.md b/archive/oikos-cards/cards/service-arr_stack.md similarity index 100% rename from oikos/cards/service-arr_stack.md rename to archive/oikos-cards/cards/service-arr_stack.md diff --git a/oikos/cards/service-artifacto.md b/archive/oikos-cards/cards/service-artifacto.md similarity index 100% rename from oikos/cards/service-artifacto.md rename to archive/oikos-cards/cards/service-artifacto.md diff --git a/oikos/cards/service-authentik.md b/archive/oikos-cards/cards/service-authentik.md similarity index 100% rename from oikos/cards/service-authentik.md rename to archive/oikos-cards/cards/service-authentik.md diff --git a/oikos/cards/service-caddy.md b/archive/oikos-cards/cards/service-caddy.md similarity index 100% rename from oikos/cards/service-caddy.md rename to archive/oikos-cards/cards/service-caddy.md diff --git a/oikos/cards/service-dns.md b/archive/oikos-cards/cards/service-dns.md similarity index 100% rename from oikos/cards/service-dns.md rename to archive/oikos-cards/cards/service-dns.md diff --git a/oikos/cards/service-gitea.md b/archive/oikos-cards/cards/service-gitea.md similarity index 100% rename from oikos/cards/service-gitea.md rename to archive/oikos-cards/cards/service-gitea.md diff --git a/oikos/cards/service-haos.md b/archive/oikos-cards/cards/service-haos.md similarity index 100% rename from oikos/cards/service-haos.md rename to archive/oikos-cards/cards/service-haos.md diff --git a/oikos/cards/service-homelab_mcp.md b/archive/oikos-cards/cards/service-homelab_mcp.md similarity index 100% rename from oikos/cards/service-homelab_mcp.md rename to archive/oikos-cards/cards/service-homelab_mcp.md diff --git a/oikos/cards/service-jellyfin.md b/archive/oikos-cards/cards/service-jellyfin.md similarity index 100% rename from oikos/cards/service-jellyfin.md rename to archive/oikos-cards/cards/service-jellyfin.md diff --git a/oikos/cards/service-matrix.md b/archive/oikos-cards/cards/service-matrix.md similarity index 100% rename from oikos/cards/service-matrix.md rename to archive/oikos-cards/cards/service-matrix.md diff --git a/oikos/cards/service-nextcloud.md b/archive/oikos-cards/cards/service-nextcloud.md similarity index 100% rename from oikos/cards/service-nextcloud.md rename to archive/oikos-cards/cards/service-nextcloud.md diff --git a/oikos/cards/service-paperless.md b/archive/oikos-cards/cards/service-paperless.md similarity index 100% rename from oikos/cards/service-paperless.md rename to archive/oikos-cards/cards/service-paperless.md diff --git a/oikos/cards/service-photos.md b/archive/oikos-cards/cards/service-photos.md similarity index 100% rename from oikos/cards/service-photos.md rename to archive/oikos-cards/cards/service-photos.md diff --git a/oikos/cards/service-proxmox_ui.md b/archive/oikos-cards/cards/service-proxmox_ui.md similarity index 100% rename from oikos/cards/service-proxmox_ui.md rename to archive/oikos-cards/cards/service-proxmox_ui.md diff --git a/oikos/cards/service-secrets_issuance.md b/archive/oikos-cards/cards/service-secrets_issuance.md similarity index 100% rename from oikos/cards/service-secrets_issuance.md rename to archive/oikos-cards/cards/service-secrets_issuance.md diff --git a/oikos/cards/service-teddycloud.md b/archive/oikos-cards/cards/service-teddycloud.md similarity index 100% rename from oikos/cards/service-teddycloud.md rename to archive/oikos-cards/cards/service-teddycloud.md diff --git a/oikos/cards/service-trmnl.md b/archive/oikos-cards/cards/service-trmnl.md similarity index 100% rename from oikos/cards/service-trmnl.md rename to archive/oikos-cards/cards/service-trmnl.md diff --git a/oikos/cards/service-zimaos.md b/archive/oikos-cards/cards/service-zimaos.md similarity index 100% rename from oikos/cards/service-zimaos.md rename to archive/oikos-cards/cards/service-zimaos.md diff --git a/cmd/oikos/main.go b/cmd/oikos/main.go index e3f2915..2f96e28 100644 --- a/cmd/oikos/main.go +++ b/cmd/oikos/main.go @@ -14,6 +14,7 @@ import ( "github.com/dtoro/oikos/internal/config" "github.com/dtoro/oikos/internal/db" "github.com/dtoro/oikos/internal/httpapi" + "github.com/dtoro/oikos/internal/knowledge" "github.com/dtoro/oikos/internal/notifier" "github.com/dtoro/oikos/internal/observability" "github.com/dtoro/oikos/internal/scheduler" @@ -125,6 +126,7 @@ Roles: notifier Run the notification service (Phase 3) all Run all roles in one process (dev mode) secret Secret management (Phase 5) + knowledge Convert wiki to knowledge seed (one-shot) version Print version info Environment: @@ -230,6 +232,32 @@ func runSeed(ctx context.Context, cfg config.Config) error { return err } + // Ingest knowledge seed (documents, investigations, runbooks) + knContent, err := os.ReadFile(seedsDir + "/knowledge.yaml") + if err != nil { + if os.IsNotExist(err) { + slog.Info("knowledge seed not found, skipping") + } else { + return fmt.Errorf("read knowledge seed: %w", err) + } + } else { + err = pool.SeedIngest(ctx, "knowledge.yaml", knContent, + func(ctx context.Context, tx pgx.Tx, data map[string]any) error { + r, err := knowledge.Ingest(ctx, tx, data) + if err != nil { + return err + } + slog.Info("knowledge ingested", + "documents", r.Documents, + "investigations", r.Investigations, + "runbooks", r.Runbooks) + return nil + }) + if err != nil { + return err + } + } + slog.Info("seed ingest complete") return nil } diff --git a/internal/httpapi/knowledge.go b/internal/httpapi/knowledge.go new file mode 100644 index 0000000..4a3a6cf --- /dev/null +++ b/internal/httpapi/knowledge.go @@ -0,0 +1,142 @@ +package httpapi + +import ( + "context" + "encoding/json" + + "github.com/dtoro/oikos/internal/httpapi/gen" + "github.com/google/uuid" +) + +func (s *Server) SearchKnowledge(ctx context.Context, request gen.SearchKnowledgeRequestObject) (gen.SearchKnowledgeResponseObject, error) { + q := request.Params.Q + limit := clampLimit(request.Params.Limit) + + rows, err := s.pool.Query(ctx, ` + SELECT e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags, + ts_rank(ke.search, plainto_tsquery('english', $1)) AS rank, + ts_headline('english', ke.content, plainto_tsquery('english', $1), + 'MaxWords=40, MinWords=15, ShortWord=3, MaxFragments=3, + FragmentDelimiter=" ... "') AS snippet + FROM knowledge_entities ke + JOIN entities e ON e.id = ke.entity_id + JOIN entity_types et ON et.name = e.type + WHERE ke.search @@ plainto_tsquery('english', $1) + ORDER BY rank DESC + LIMIT $2`, + q, limit) + if err != nil { + return nil, err + } + defer rows.Close() + + items := []gen.KnowledgeHit{} + + for rows.Next() { + var slug, eType, title, source, tagJSON string + var rank float32 + var snippet *string + + if err := rows.Scan(&slug, &eType, &title, &source, &tagJSON, &rank, &snippet); err != nil { + return nil, err + } + + var tags []string + json.Unmarshal([]byte(tagJSON), &tags) + + hitType := gen.Document + switch eType { + case "investigation": + hitType = gen.Investigation + case "runbook": + hitType = gen.Runbook + } + + id, _ := uuid.Parse("") + _ = id // not needed for response since we use slug + + items = append(items, gen.KnowledgeHit{ + Slug: slug, + Title: title, + Type: hitType, + Rank: &rank, + Snippet: snippet, + SourcePath: &source, + }) + } + if rows.Err() != nil { + return nil, rows.Err() + } + + if items == nil { + items = []gen.KnowledgeHit{} + } + + return gen.SearchKnowledge200JSONResponse{Items: items}, nil +} + +func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKnowledgeRequestObject) (gen.GetEntityKnowledgeResponseObject, error) { + entitySlug := request.EntityId + + rows, err := s.pool.Query(ctx, ` + SELECT e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags + FROM knowledge_entities ke + JOIN entities e ON e.id = ke.entity_id + JOIN entity_types et ON et.name = e.type + JOIN relationships r ON r.source_id = ke.entity_id + JOIN entities target ON target.id = r.target_id + WHERE target.slug = $1 + AND r.valid_to IS NULL + AND r.type IN ('documents', 'about') + UNION + SELECT e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags + FROM knowledge_entities ke + JOIN entities e ON e.id = ke.entity_id + JOIN entity_types et ON et.name = e.type + JOIN relationships r ON r.source_id = ke.entity_id + JOIN entity_types target_type ON target_type.name = r.target_id::text + JOIN entities ent ON ent.type = target_type.name AND ent.slug = $1 + WHERE r.valid_to IS NULL + AND r.type = 'procedure-for' + ORDER BY 1`, + entitySlug) + if err != nil { + return nil, err + } + defer rows.Close() + + items := []gen.KnowledgeHit{} + for rows.Next() { + var slug, eType, title, source, tagJSON string + var tags []string + + if err := rows.Scan(&slug, &eType, &title, &source, &tagJSON); err != nil { + return nil, err + } + json.Unmarshal([]byte(tagJSON), &tags) + + hitType := gen.Document + switch eType { + case "investigation": + hitType = gen.Investigation + case "runbook": + hitType = gen.Runbook + } + + items = append(items, gen.KnowledgeHit{ + Slug: slug, + Title: title, + Type: hitType, + SourcePath: &source, + }) + } + if rows.Err() != nil { + return nil, rows.Err() + } + + if items == nil { + items = []gen.KnowledgeHit{} + } + + return gen.GetEntityKnowledge200JSONResponse{Items: items}, nil +} \ No newline at end of file diff --git a/internal/httpapi/phase3.go b/internal/httpapi/phase3.go index 8703547..75ed5eb 100644 --- a/internal/httpapi/phase3.go +++ b/internal/httpapi/phase3.go @@ -1738,16 +1738,6 @@ func float32Ptr(f float32) *float32 { return &f } -// ─── Knowledge (stubs — tables don't exist yet) ──────────────────────── - -func (s *Server) SearchKnowledge(ctx context.Context, request gen.SearchKnowledgeRequestObject) (gen.SearchKnowledgeResponseObject, error) { - return nil, errNotImplemented -} - -func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKnowledgeRequestObject) (gen.GetEntityKnowledgeResponseObject, error) { - return nil, errNotImplemented -} - // ─── Agent Activity (stub) ───────────────────────────────────────────── func (s *Server) QueryAgentActivity(ctx context.Context, request gen.QueryAgentActivityRequestObject) (gen.QueryAgentActivityResponseObject, error) { diff --git a/internal/knowledge/seed.go b/internal/knowledge/seed.go new file mode 100644 index 0000000..c436c1c --- /dev/null +++ b/internal/knowledge/seed.go @@ -0,0 +1,276 @@ +package knowledge + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + + "github.com/google/uuid" + "github.com/jackc/pgx/v5" +) + +type SeedResult struct { + Documents int + Investigations int + Runbooks int +} + +func contentHash(s string) string { + h := sha256.Sum256([]byte(s)) + return hex.EncodeToString(h[:]) +} + +func Ingest(ctx context.Context, tx pgx.Tx, data map[string]any) (*SeedResult, error) { + r := &SeedResult{} + + docs, _ := data["documents"].([]any) + for _, raw := range docs { + d, _ := raw.(map[string]any) + if err := ingestDocument(ctx, tx, d); err != nil { + return nil, fmt.Errorf("document %v: %w", str(d, "slug"), err) + } + r.Documents++ + } + + invs, _ := data["investigations"].([]any) + for _, raw := range invs { + m, _ := raw.(map[string]any) + if err := ingestInvestigation(ctx, tx, m); err != nil { + return nil, fmt.Errorf("investigation %v: %w", str(m, "slug"), err) + } + r.Investigations++ + } + + rbs, _ := data["runbooks"].([]any) + for _, raw := range rbs { + m, _ := raw.(map[string]any) + if err := ingestRunbook(ctx, tx, m); err != nil { + return nil, fmt.Errorf("runbook %v: %w", str(m, "slug"), err) + } + r.Runbooks++ + } + + return r, nil +} + +func str(m map[string]any, key string) string { + s, _ := m[key].(string) + return s +} + +func strSlice(m map[string]any, key string) []string { + raw, _ := m[key].([]any) + var out []string + for _, v := range raw { + if s, ok := v.(string); ok { + out = append(out, s) + } + } + return out +} + +func mapVal(m map[string]any, key string) map[string]any { + v, _ := m[key].(map[string]any) + return v +} + +func ingestDocument(ctx context.Context, tx pgx.Tx, m map[string]any) error { + slug := str(m, "slug") + title := str(m, "title") + content := str(m, "content") + entitySlug := str(m, "entity_slug") + tags := strSlice(m, "tags") + atGlance, _ := m["at_glance"].(map[string]any) + clRaw, _ := m["changelog"].([]any) + + entityDocSlug := "document:" + slug + + if err := upsertKnowledgeEntity(ctx, tx, entityDocSlug, "document", title, content, slug, tags); err != nil { + return err + } + + attrs := map[string]any{} + if len(atGlance) > 0 { + attrs["at_glance"] = atGlance + } + if len(clRaw) > 0 { + attrs["changelog"] = clRaw + } + if len(attrs) > 0 { + attrsBytes, _ := json.Marshal(attrs) + _, err := tx.Exec(ctx, + `UPDATE entities SET attributes = attributes || $1, updated_at = now() + WHERE slug = $2`, string(attrsBytes), entityDocSlug) + if err != nil { + return fmt.Errorf("update document attrs: %w", err) + } + } + + if entitySlug != "" { + if err := createEdge(ctx, tx, entityDocSlug, entitySlug, "documents", nil); err != nil { + return fmt.Errorf("link document: %w", err) + } + } + + if len(atGlance) > 0 && entitySlug != "" { + backfillAttrs, _ := json.Marshal(atGlance) + _, err := tx.Exec(ctx, + `UPDATE entities SET attributes = $1 || attributes, updated_at = now() + WHERE slug = $2`, string(backfillAttrs), entitySlug) + if err != nil { + return fmt.Errorf("backfill entity attrs: %w", err) + } + } + + return nil +} + +func ingestInvestigation(ctx context.Context, tx pgx.Tx, m map[string]any) error { + slug := str(m, "slug") + title := str(m, "title") + content := str(m, "content") + date := str(m, "date") + status := str(m, "status") + duration := str(m, "duration") + aboutSlugs := strSlice(m, "about_slugs") + tags := strSlice(m, "tags") + + entitySlug := "investigation:" + slug + + if err := upsertKnowledgeEntity(ctx, tx, entitySlug, "investigation", title, content, slug, tags); err != nil { + return err + } + + attrs := map[string]any{} + if date != "" { + attrs["date"] = date + } + if status != "" { + attrs["status"] = status + } + if duration != "" { + attrs["duration"] = duration + } + if len(attrs) > 0 { + attrsBytes, _ := json.Marshal(attrs) + _, err := tx.Exec(ctx, + `UPDATE entities SET attributes = attributes || $1, updated_at = now() + WHERE slug = $2`, string(attrsBytes), entitySlug) + if err != nil { + return fmt.Errorf("update investigation attrs: %w", err) + } + } + + for _, aboutSlug := range aboutSlugs { + if err := createEdge(ctx, tx, entitySlug, aboutSlug, "about", nil); err != nil { + return fmt.Errorf("link investigation about %s: %w", aboutSlug, err) + } + } + + return nil +} + +func ingestRunbook(ctx context.Context, tx pgx.Tx, m map[string]any) error { + slug := str(m, "slug") + name := str(m, "name") + riskClass := str(m, "risk_class") + entityType := str(m, "entity_type") + content := str(m, "content") + tags := strSlice(m, "tags") + procedure, _ := m["procedure"].(map[string]any) + + entitySlug := "runbook:" + slug + + if err := upsertKnowledgeEntity(ctx, tx, entitySlug, "runbook", name, content, slug, tags); err != nil { + return err + } + + attrs := map[string]any{} + if riskClass != "" { + attrs["risk_class"] = riskClass + } + if len(procedure) > 0 { + attrs["procedure"] = procedure + } + if len(attrs) > 0 { + attrsBytes, _ := json.Marshal(attrs) + _, err := tx.Exec(ctx, + `UPDATE entities SET attributes = attributes || $1, updated_at = now() + WHERE slug = $2`, string(attrsBytes), entitySlug) + if err != nil { + return fmt.Errorf("update runbook attrs: %w", err) + } + } + + if entityType != "" { + if err := createEdge(ctx, tx, entitySlug, entityType, "procedure-for", nil); err != nil { + return fmt.Errorf("link runbook: %w", err) + } + } + + return nil +} + +func upsertKnowledgeEntity(ctx context.Context, tx pgx.Tx, slug, entityType, title, content, source string, tags []string) error { + id, err := getOrCreateEntity(ctx, tx, slug, entityType, title) + if err != nil { + return err + } + + hash := contentHash(content) + tagsJSON, _ := json.Marshal(tags) + + _, err = tx.Exec(ctx, + `INSERT INTO knowledge_entities (entity_id, title, content, source, tags, content_hash, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, now(), now()) + ON CONFLICT (entity_id) DO UPDATE SET + title = $2, content = $3, source = $4, tags = $5, + content_hash = $6, updated_at = now()`, + id, title, content, source, string(tagsJSON), hash) + return err +} + +func getOrCreateEntity(ctx context.Context, tx pgx.Tx, slug, entityType, name string) (uuid.UUID, error) { + var id uuid.UUID + err := tx.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id) + if err == nil { + return id, nil + } + if err != pgx.ErrNoRows { + return uuid.Nil, fmt.Errorf("lookup entity %s: %w", slug, err) + } + + id, err = uuid.NewV7() + if err != nil { + return uuid.Nil, fmt.Errorf("generate uuid: %w", err) + } + _, err = tx.Exec(ctx, + `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1, $2, $3, $4, NULL, '{}', 1, now(), now())`, + id, slug, entityType, name) + if err != nil { + return uuid.Nil, fmt.Errorf("create entity %s: %w", slug, err) + } + return id, nil +} + +func createEdge(ctx context.Context, tx pgx.Tx, sourceSlug, targetSlug, relType string, attrs map[string]any) error { + var sourceID, targetID uuid.UUID + if err := tx.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", sourceSlug).Scan(&sourceID); err != nil { + return fmt.Errorf("source %s: %w", sourceSlug, err) + } + if err := tx.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", targetSlug).Scan(&targetID); err != nil { + return fmt.Errorf("target %s: %w", targetSlug, err) + } + + attrsBytes, _ := json.Marshal(attrs) + _, err := tx.Exec(ctx, + `INSERT INTO relationships (source_id, target_id, type, attributes, valid_from, valid_to) + VALUES ($1, $2, $3, $4, now(), NULL) + ON CONFLICT (source_id, target_id, type) WHERE valid_to IS NULL + DO UPDATE SET attributes = EXCLUDED.attributes`, + sourceID, targetID, relType, string(attrsBytes)) + return err +} \ No newline at end of file diff --git a/knowledge/wiki/containers/archive/123-claudio-bot.md b/knowledge/wiki/containers/archive/123-claudio-bot.md deleted file mode 100644 index 384bff3..0000000 --- a/knowledge/wiki/containers/archive/123-claudio-bot.md +++ /dev/null @@ -1,90 +0,0 @@ -# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04) - -> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini. -> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob. -> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea. -> See [deprecation plan](../../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) for full details. - -Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications. - -## At a glance -- **Hostname:** `claudio-bot` -- **IP:** `192.168.8.230` -- **Privilege:** **unprivileged** -- **Resources:** 1 core / 512 MiB RAM / 8 GiB rootfs -- **Mounts:** none from `/mnt/library` -- **Public hostname:** none - -## Stack - -Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](../118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). - -Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed). - -## Configuration - -Secrets at `/etc/claudio-bot/` (mode 600): -- `config.yaml` -- `matrix.token` -- `anthropic.key` -- `lmstudio.key` -- `ipc.token` - -Deploy git creds at `/etc/claudio-deploy/git-credentials`. - -### LLM backend (pluggable) - -`llm.backend` in `config.yaml`: -- `anthropic` — Claude API via `anthropic.key` -- `lmstudio` — OpenAI-compat HTTP via `lmstudio.key` (bearer) - -Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168.8.174:1234` (since 2026-04-25). Switch back with `backend:` + `systemctl restart claudio-bot`. Original config saved at `/etc/claudio-bot/config.yaml.bak`. The LM Studio agent in `bot_core/lmstudio.py` translates Anthropic → OpenAI tool definitions. - -## IPC - -`http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by: -- [claudio-monitor on hubris](../../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) -- The (currently disabled) [restic backup wrapper](../../infrastructure/backups.md) (token in `/etc/restic/bot.token`) - -> Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together. - -## Plugins - -Module under `plugins/.py` exposing a `Plugin` class; add `` to `plugins:` in `config.yaml`. Plugins can register slash commands, Claude tools, and `on_notify` / `on_proposal_response` hooks. - -Active plugins: -- `system` — `ping`, `help`, `status`, `list_plugins` -- `backup` — ingests `/notify` from the backup wrapper (currently silent — backups disabled) -- `monitor` — `/monitor status`, `/monitor history [N]`, `/monitor clear `. Tools: `get_alerts`, `get_monitor_history` for NL queries via Claude. - -## Auto-deploy - -Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf. - -`app.ini` `ALLOWED_HOST_LIST` on [gitea](../104-gitea.md) includes `192.168.8.230`. - -## Related -- [elementsynapse (118)](../118-elementsynapse.md) -- [Monitoring (claudio-monitor)](../../infrastructure/monitoring.md) -- [Backups (disabled)](../../infrastructure/backups.md) -- [Auto-deploy](../../infrastructure/auto-deploy.md) - -## Changelog - -### 2026-06-04 — LXC destroyed; replaced by Hermes Agent -LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd -units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on -Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job. -`@claudio:avispero` Matrix account decommissioned. - -### 2026-04-28 — wiki entry created -Initial documentation. - -### 2026-04-25 — LLM backend switched to LM Studio -`backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field. - -### 2026-04-21 — `monitor` plugin added -Receives events from [claudio-monitor](../../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. - -### 2026-04-20 — claudio-bot deployed -LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server. diff --git a/knowledge/wiki/containers/archive/127-mule-photos-new.md b/knowledge/wiki/containers/archive/127-mule-photos-new.md deleted file mode 100644 index 0966251..0000000 --- a/knowledge/wiki/containers/archive/127-mule-photos-new.md +++ /dev/null @@ -1,313 +0,0 @@ -# 127 — `mule-photos-new` - -Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch -at `photos-new.hubris.network`. Production [LXC 120](../120-mule-images.md) keeps -running on the legacy stack at `photos.hubris.network` until M5 cutover. - -## At a glance -- **Hostname:** `mule-photos-new` -- **IP:** `192.168.8.181` -- **Privilege:** unpriv -- **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap -- **Features:** `nesting=1,fuse=1,keyctl=1` -- **Mounts:** *(none — see scratch copy below)* -- **Public hostname:** [`photos-new.hubris.network`](../../infrastructure/dns.md) → [caddy (121)](../121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`) - -## Stack (`/opt/mule-image`) - -`/opt/mule-image` is the working tree of `dtoro/mule-image` on branch -`new`. Compose lives at `docker-compose.photoprism.yml`; LXC-127-only -overrides at `docker-compose.photoprism.override.yml` (untracked — see -[Why an override exists](#why-an-override-exists)). Invoked with -`--env-file .env.photoprism`. - -| Service | Container | Port | Notes | -| ----------- | ------------- | ----------------- | -------------------------------------------------- | -| mariadb | `pp-mariadb` | `127.0.0.1:3306` | MariaDB 11; named volume `pp_mariadb_data`; init SQL provisions `mule_sidecar.marks` table | -| photoprism | `pp-app` | `:2342` | `docker.io/photoprism/photoprism:latest`, runs as uid 33 (PP_UID/GID env) | -| sidecar | `pp-sidecar` | `:8000` (overridden) | Go + Gin service for rename / folder mutations / heap convert / dup detect | -| vite (host) | systemd unit | `:5173` | SvelteKit dev server (`mule-vite.service`), `npm run dev` in `/opt/mule-image/web` | - -PhotoPrism is enterprise-tier (`tier: 1`) per the build tag (`-Plus`). TF -vision pipeline and EXIF backwrite are **disabled** in M0 — `PP_READONLY=true` -keeps the originals view read-only as the M0 safety net. - -## Library — writable rsync scratch copy (NOT the real admin Photos) - -Unlike LXC 120 (which mounts `/mnt/library` directly), this LXC has **no -bind-mount of the production library**. Instead a one-shot rsync of the -admin's Photos lives on the LXC's own rootfs: - -- **Host source (read-only reference):** `/mnt/library/homecloud/admin/files/Photos` (~4.9 GB) -- **LXC scratch (writable):** `/srv/photos-scratch` (owner `www-data:media`, mode 0775) - -This means **sidecar rename / folder mutation operations land in the -scratch copy, not the real admin library**. The scratch is not -auto-synced — it's a snapshot from `2026-05-17`. To refresh from -production: - -```bash -# on hubris (LXC 127 must be stopped to mount its rootfs) -pct stop 127 -pct mount 127 -rsync -aHAX --info=stats2 --chown=100033:110000 --no-perms \ - --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r \ - /mnt/library/homecloud/admin/files/Photos/ \ - /var/lib/lxc/127/rootfs/srv/photos-scratch/ -pct unmount 127 -pct start 127 -``` - -`--chown=100033:110000` accounts for the unprivileged-LXC ID shift -(host 100033 = LXC `www-data`, host 110000 = LXC `media`). Don't try a -bind-mount of `/mnt/library/...` — the admin Photos tree is 0750 and -unprivileged LXCs can't see through. - -## Auth — Authentik OIDC - -PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](../106-auth-outpost.md). - -- **Provider/Application slug:** `mule-photos-new` -- **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/` -- **Redirect URI:** `https://photos-new.hubris.network/api/v1/oidc/redirect` -- **Scopes:** `openid profile email` -- **Initiated by clicking the OIDC button** at `/library/login` → `GET /api/v1/oidc/login` → 302 to Authentik authorize. -- `OIDC_REGISTER=true` + `OIDC_ROLE=admin` so the first SSO login auto-creates a PhotoPrism admin account. - -Local PhotoPrism admin (username `admin`, password in -`/root/mule-photos-new-secrets.txt` on hubris) stays available as a -fallback. - -## Why an override exists - -`docker-compose.photoprism.override.yml` is **only on LXC 127** (not in -the git repo). After the 2026-05-17/18 fixes landed upstream -(commits `cce1d87` for OIDC env names and `3d8e050` for worker caps), -the override has shrunk to just one stanza — the cross-host sidecar -bind: - -```yaml -services: - sidecar: - ports: !override - - "0.0.0.0:8000:8000" -``` - -Upstream binds the sidecar to `127.0.0.1:8000` because the M4 design -colocates Caddy with the sidecar. On this test LXC Caddy lives on a -different host (LXC 121), so the port has to be reachable from the -LAN. Delete this file at M4 cutover. - -## Auto-deploy - -Mirrors the LXC 120 pattern. - -- **Webhook listener:** `mule-deploy-webhook.service` → `python3 /opt/mule-deploy/webhook.py` on `0.0.0.0:9797`. -- **Branch filter:** `refs/heads/new` (LXC 120 still owns `main`). -- **HMAC secret:** `/etc/mule-deploy/secret` (mode 0600). -- **Deploy script:** `/opt/mule-deploy/deploy.sh` — `git fetch && git reset --hard origin/new`, fix `pp/{storage,import}` ownership to `33:10000` (PP container uid), `docker compose ... up -d --build --force-recreate` with both compose files, `systemctl restart mule-vite`. -- **Gitea webhook id 9** on `dtoro/mule-image` pointed at `http://192.168.8.181:9797/deploy`. - -Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected. - -**Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST` -in `/etc/gitea/app.ini` on [LXC 104](../104-gitea.md). LXC 127's -`192.168.8.181` was missing on first bring-up; every push delivered -status 0 with the message `webhook can only call allowed HTTP servers`. -Adding the IP and `systemctl restart gitea` is enough — same list is -also the gating mechanism for the LXC 120 webhook. Verify any future -test instance is added before relying on auto-deploy. - -## Bootstrap secrets - -Saved on hubris at `/root/mule-photos-new-secrets.txt` (mode 0600): - -- `PP_ADMIN_PASSWORD` — initial PhotoPrism `admin` login -- `PP_DB_PASSWORD` — MariaDB `photoprism` user -- `PP_DB_ROOT_PASSWORD` — MariaDB root -- `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` — generated by `ak shell` against Authentik - -`SIDECAR_DB_PASSWORD` is still the literal placeholder -`replace-at-m4-bringup` because `mariadb/init/01-sidecar.sql` hardcodes -it; rotate before this stack ever goes public. - -## Health checks - -```bash -# from hubris -pct exec 127 -- curl -sf http://127.0.0.1:2342/api/v1/status # PP -pct exec 127 -- curl -sf http://127.0.0.1:8000/api/sidecar/healthz # sidecar -pct exec 127 -- curl -sf http://127.0.0.1:5173/ # Vite - -# through Caddy -curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \ - https://photos-new.hubris.network/api/v1/oidc/login -i | head -2 # 302 → auth.hubris.network -``` - -> **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack -> validated here was promoted into production on [LXC 120](../120-mule-images.md) -> via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This -> page is retained for archaeology; everything below is historic. See the -> 2026-05-22 entry in [120-mule-images.md](../120-mule-images.md#changelog) for -> the cutover detail. - -## Changelog - -### 2026-05-22 — Destroyed - -`pct destroy 127` after curl verification of the new 120 stack passed -end-to-end. dnsmasq `photos-new.hubris.network` line removed; gitea -webhook id 9 + `192.168.8.181` ALLOWED_HOST_LIST entry removed; caddy -`photos-new.hubris.network` site block dropped from `dtoro/caddy-conf`. -Authentik `mule-photos-new` app + provider deleted. - -### 2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app - -User reported `failed to get state: securecookie: the value is not valid` after -authenticating at Authentik. Root cause: PhotoPrism rotates the `Session:` HMAC -key in `pp/storage/config/hub.yml` on **every container start** (it's the hub -auto-refresh; no env flag disables it, and with `Status: ""` it regenerates -each boot). The key signs OIDC state cookies, so every restart invalidated -every in-flight login. - -`/opt/mule-deploy/deploy.sh` was doing `docker compose up -d --build ---force-recreate` after a `pull --ignore-buildable`. Both moves recreate -pp-app: `--force-recreate` unconditionally, and the pull bumps the -`photoprism:latest` digest which makes plain `up -d` recreate too. Every -auto-deploy on a code push therefore broke every OIDC login. - -Fixed by: - -1. Dropping the `pull` step from the auto-deploy. Image refreshes for pp-app - / mariadb are now an admin operation (run `docker compose pull` manually - when you want a new PhotoPrism build). -2. Force-recreating only the **sidecar** (its image rebuilds on every push - anyway). `up -d photoprism mariadb` reconciles in place — only restarts - if their compose declaration actually changes. - -Verified by running `deploy.sh` twice and watching `hub.yml`'s `Session:` -field and pp-app's container PID; both stayed stable across the deploy. -The sidecar's PID changed as expected. - -The fix lives in `/opt/mule-deploy/deploy.sh` on LXC 127. `/opt/mule-deploy` -is not a git checkout — keep this Changelog entry as the source of truth. - -### 2026-05-18 — OIDC bridge + indexer + folder fixes - -The `new` branch's PhotoPrism stack required several iterations to be -actually usable. Fixes pushed upstream so they apply to anyone running -the M0 compose; the LXC override file shrank to just the cross-host -sidecar port (see [Why an override exists](#why-an-override-exists)). - -**OIDC end-to-end** (upstream commits `4abe6d7`, `9a3ad3e`, `cce1d87`, -plus Caddy `/library/* → /` bounce on LXC 121). - -- The SvelteKit `/login` had a `// OIDC SSO ships in M4` placeholder - but no button. Added a "Sign in with {provider}" button conditional - on `/api/v1/config.ext.oidc.enabled`; click sends the browser to - `/api/v1/oidc/login`. -- The compose file passed OIDC values through `PHOTOPRISM_OIDC_ISSUER_URL` - / `_CLIENT_ID` / `_CLIENT_SECRET` / `_PROVIDER_NAME` — names PhotoPrism - silently ignores. The actual env-var names are `PHOTOPRISM_OIDC_URI` - / `_CLIENT` / `_SECRET` / `_PROVIDER` (see `photoprism show config`). - Renamed upstream; user-facing keys in `.env.photoprism` (OIDC_ISSUER_URL, - OIDC_CLIENT_ID, …) are unchanged. -- PhotoPrism's OIDC callback does **not** set `auth_token` / `auth_session` - cookies. It returns an HTML page that writes the session into - `localStorage` under `pp::session.{id,token,user,provider}` - and then runs `window.location.href = "/library/login"`. Caddy on this - test instance bounces `/library/*` back to `/`, and the SvelteKit root - layout (`bootstrapSessionFromPhotoPrism()`) reads those localStorage - entries on mount, fetches `/api/v1/session/` with the cookied - token, and adopts the session into the SPA store. - -**Indexer caps** (upstream commit `3d8e050`). A fresh index of ~1.2k -photos pushed the LXC load average above 50 with the default -`PHOTOPRISM_INDEX_WORKERS` (NumCPU/2 = 3 here, each forking TF + -ffmpeg + libvips). Compose now reads `PP_WORKERS` / `PP_INDEX_WORKERS` -from `.env.photoprism`, defaulting to 2. Both set explicitly on this -LXC to keep sibling containers happy. - -**Library went RW.** Flipped `PP_READONLY=false` and `PP_ORIGINALS_MODE=rw` -in `.env.photoprism` so the indexer can actually run — `READONLY=true` -disables it entirely. Safe because `/srv/photos-scratch` is the -rsync scratch copy on the LXC rootfs, not the real admin Photos tree. - -**Admin role.** OIDC creates users with `OIDC_ROLE` ONLY on first -registration. The `dtoro` user was created in an earlier flow before -the env-var-name fix, so it landed as `guest` and saw no photos in -the UI. Promoted manually: - -```sql -UPDATE auth_users SET user_role='admin', super_admin=1, can_invite=1 - WHERE user_name='dtoro'; -DELETE FROM auth_sessions WHERE user_name='dtoro'; -``` - -Stale sessions are dropped so a fresh OIDC login mints an admin token. - -**Video pre-transcode pass** (no upstream change — operational fix on -this LXC). Only 11/45 `.mov` originals had a `.avc` sidecar; the rest -forced 12–21 s inline libx264 transcodes on first playback, serialised -one ffmpeg at a time. Measured cold vs warm: - -| Path | TTFB | -| --------------------------------- | ------- | -| Thumbnail `fit_1280` (warm) | ~2 ms | -| Video playback with `.avc` sidecar | ~2 ms | -| Video playback without sidecar | 12–21 s | -| Thumbnail GET *during* a transcode | ~2 ms (no sibling slowdown — 6-core LXC, ffmpeg ~6%/core, nvme util 0.01%) | - -Mitigation: - -```bash -pct exec 127 -- docker exec -d pp-app /opt/photoprism/bin/photoprism convert -``` - -Walks the library, builds every missing `.avc` next to its original -(`/photoprism/storage/sidecar///.mov.avc`), -idempotent on re-run, two ffmpeg processes in parallel. Took ~8 min -to bring coverage to 45/45. Previously-cold videos verified to serve -at ~2 ms TTFB after the pass. Future imports get AVC sidecars -automatically as part of indexing; a re-run is only needed if videos -ever land outside the indexer's path. - -**Folder tree** (upstream commits `8083328`, `505fef5`, `cfd85a1`). -PhotoPrism's `path:` operator is exact-match by default but supports -a `*` wildcard. Without it, every internal tree node (year folders, -since photos always nest under YYYY/MM) returned zero hits — both in -the timeline and in the sidecar's folder-count fan-out. Fixed both -to emit `path:"*"`. Also relaxed the root-folder client-side -filter (was clipping to `Path === ''`, which is always empty) so `/` -shows the whole library. Root badge in the sidebar now reads -`config.count.all` directly instead of subtracting Σ(folderCounts) — -the subtraction double-counted after the recursive switch. - -### 2026-05-17 — Bring-up - -LXC 127 created from `debian-13-standard_13.1-2`, joined to vmbr0 with -static IP `192.168.8.181`. Docker engine + Node 20 installed. - -`dtoro/mule-image` cloned at branch `new`, compose stack -(`docker-compose.photoprism.yml`) brought up: MariaDB 11 + PhotoPrism -`:latest` (`-Plus` build) + Go sidecar (built locally). Vite dev server -running as `mule-deploy` via `mule-vite.service` on port 5173. - -Authentik OIDC application `mule-photos-new` provisioned via `ak shell` -(`OAuth2Provider` + `Application` + STRICT `RedirectURI`). PhotoPrism's -OIDC button delegates to Authentik; `OIDC_REGISTER=true` / -`OIDC_ROLE=admin` so the first SSO login becomes admin. - -Admin's Photos library rsynced (~4.9 GB, 1206 files) into -`/srv/photos-scratch` on the LXC rootfs (no bind-mount). Sidecar -mutations land in the scratch copy, not the real library. - -Caddy site `photos-new.hubris.network` added in `dtoro/caddy-conf`; -dnsmasq entry on LXC 124 → `192.168.8.175`. dnsmasq required a `restart` -(not `reload`) for the new `address=` line to take effect. - -`docker-compose.photoprism.override.yml` (LXC-only, untracked) pins two -upstream issues: sidecar bound to `127.0.0.1` (cross-host Caddy can't -reach), and OIDC env-var name mismatch -(`PHOTOPRISM_OIDC_ISSUER_URL` vs `PHOTOPRISM_OIDC_URI` and friends). -Should land upstream on `new` next iteration. diff --git a/migrations/010_knowledge_hash.up.sql b/migrations/010_knowledge_hash.up.sql new file mode 100644 index 0000000..3bae15c --- /dev/null +++ b/migrations/010_knowledge_hash.up.sql @@ -0,0 +1,2 @@ +-- Migration 010: Add content_hash to knowledge_entities for drift detection. +ALTER TABLE knowledge_entities ADD COLUMN IF NOT EXISTS content_hash TEXT; \ No newline at end of file diff --git a/migrations/011_knowledge_search.up.sql b/migrations/011_knowledge_search.up.sql new file mode 100644 index 0000000..b75f154 --- /dev/null +++ b/migrations/011_knowledge_search.up.sql @@ -0,0 +1,4 @@ +ALTER TABLE knowledge_entities ADD COLUMN IF NOT EXISTS search tsvector + GENERATED ALWAYS AS (to_tsvector('english', COALESCE(title, '') || ' ' || COALESCE(content, ''))) STORED; + +CREATE INDEX IF NOT EXISTS idx_knowledge_search ON knowledge_entities USING GIN(search); \ No newline at end of file diff --git a/oikos/__init__.py b/oikos/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/oikos/approve.py b/oikos/approve.py deleted file mode 100644 index 89449b5..0000000 --- a/oikos/approve.py +++ /dev/null @@ -1,302 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/approve.py — the approval engine (escalate route of the OODA loop). - -Repo-side half of the Week-3 approval flow. This module owns the request/ -grant lifecycle and the HMAC signing; it does NOT talk to Matrix directly. -There is no dedicated Matrix bot in this homelab — alerts already go out -as the operator's own Hermes agent posting to @dtoro:avispero (see -knowledge/wiki/infrastructure/monitoring.md's homelab-health-watchdog). The integration -contract is: - - 1. An agent or the Week-3 scheduler calls `request()` (or the CLI - `request` subcommand) to open an approval. This prints the Matrix- - ready message text (evidence, options, reply format). - 2. Hermes posts that text to Matrix using its existing send capability - (the same one homelab-health-watchdog already uses) and later reads - the operator's reply/reaction. - 3. Hermes calls back `reply()` (or `oikos/approve.py reply - approve|deny`) with the operator's decision. This module verifies - `requires_phrase` for destructive actions, then issues a short-TTL - HMAC-signed grant. - 4. Mutating `homelab` commands call `check_grant()` before executing a - config_mutation/destructive action. - -Storage: approvals/.jsonl, same append-only-JSONL-with-latest- -state-wins convention as oikos/signal.py. - -Grant signing key: secrets/oikos-approval-hmac.yaml (SOPS, recipients: -apps + hubris — see .sops.yaml). Decrypted on demand; never cached to -disk in plaintext. -""" - -from __future__ import annotations - -import hashlib -import hmac -import json -import os -import subprocess -import sys -from datetime import datetime, timedelta, timezone -from functools import lru_cache -from pathlib import Path - -import yaml - -REPO = Path(__file__).resolve().parent.parent -APPROVALS_DIR = REPO / "approvals" -HMAC_SECRET = REPO / "secrets" / "oikos-approval-hmac.yaml" -AGE_KEY = Path(os.environ.get("SOPS_AGE_KEY_FILE", "/etc/age/key.txt")) - -VALID_STATES = ("pending", "approved", "denied", "expired", "executed") -REQUEST_TTL_HOURS = 24 -GRANT_TTL_MINUTES = 15 - - -def _month_path(dt: datetime | None = None) -> Path: - dt = dt or datetime.now(timezone.utc) - return APPROVALS_DIR / f"{dt.strftime('%Y-%m')}.jsonl" - - -def _all_entries() -> list[dict]: - if not APPROVALS_DIR.exists(): - return [] - out = [] - for path in sorted(APPROVALS_DIR.glob("*.jsonl")): - for line in path.read_text().splitlines(): - if not line.strip(): - continue - try: - out.append(json.loads(line)) - except json.JSONDecodeError: - continue - return out - - -def _next_id(dt: datetime | None = None) -> str: - dt = dt or datetime.now(timezone.utc) - prefix = f"appr-{dt.strftime('%Y-%m-%d')}-" - existing = [e["id"] for e in _all_entries() if e.get("id", "").startswith(prefix)] - n = 1 - while f"{prefix}{n:04d}" in existing: - n += 1 - return f"{prefix}{n:04d}" - - -def _append(entry: dict) -> dict: - APPROVALS_DIR.mkdir(exist_ok=True) - entry = {k: v for k, v in entry.items() if v is not None} - with _month_path().open("a") as f: - f.write(json.dumps(entry, sort_keys=False) + "\n") - return entry - - -def current(request_id: str) -> dict | None: - matches = [e for e in _all_entries() if e.get("id") == request_id] - if not matches: - return None - matches.sort(key=lambda e: e.get("ts", "")) - return matches[-1] - - -def list_approvals(state: str | None = None) -> list[dict]: - latest: dict[str, dict] = {} - for e in sorted(_all_entries(), key=lambda e: e.get("ts", "")): - rid = e.get("id") - if rid: - latest[rid] = e - now = datetime.now(timezone.utc).isoformat(timespec="seconds") - out = [] - for e in latest.values(): - if e.get("state") == "pending" and e.get("expires_at") and e["expires_at"] < now: - e = {**e, "state": "expired"} - if state and e.get("state") != state: - continue - out.append(e) - out.sort(key=lambda e: e.get("ts", ""), reverse=True) - return out - - -def _matrix_message(entry: dict) -> str: - lines = [ - f"[Oikos approval {entry['id']}] {entry['entity']} — {entry['action']}", - f"risk: {entry['risk']}", - f"evidence: {entry['evidence']}", - ] - if entry.get("verification"): - lines.append(f"verification: {entry['verification']}") - if entry.get("requires_phrase"): - lines.append(f'reply: "approve {entry["id"]} {entry["confirmation_phrase"]}" or "deny {entry["id"]}"') - else: - lines.append(f'reply: ✅ to approve, ❌ to deny (or "approve {entry["id"]}" / "deny {entry["id"]}")') - return "\n".join(lines) - - -def request(entity: str, action: str, risk: str, evidence: str, *, - verification: str | None = None, signal_id: str | None = None, - requires_phrase: bool = False, requested_by: str | None = None, - ttl_hours: int = REQUEST_TTL_HOURS) -> dict: - """Open a new approval request. `requires_phrase=True` (use for - `destructive`-class actions per oikos/policy.yaml) generates a - confirmation phrase the operator must echo back — a reaction alone - can't authorize it.""" - rid = _next_id() - now = datetime.now(timezone.utc) - phrase = f"{action} {entity}" if requires_phrase else None - entry = { - "id": rid, - "ts": now.isoformat(timespec="seconds"), - "entity": entity, - "action": action, - "risk": risk, - "evidence": evidence, - "verification": verification, - "signal_id": signal_id, - "requires_phrase": requires_phrase, - "confirmation_phrase": phrase, - "requested_by": requested_by or os.environ.get("HOMELAB_AGENT_ID"), - "state": "pending", - "expires_at": (now + timedelta(hours=ttl_hours)).isoformat(timespec="seconds"), - } - recorded = _append(entry) - recorded["matrix_message"] = _matrix_message(recorded) - return recorded - - -@lru_cache(maxsize=1) -def _hmac_key() -> str: - if not HMAC_SECRET.exists(): - raise RuntimeError(f"no secret at {HMAC_SECRET} — has it been provisioned?") - env = {**os.environ} - if AGE_KEY.exists(): - env["SOPS_AGE_KEY_FILE"] = str(AGE_KEY) - proc = subprocess.run(["sops", "-d", str(HMAC_SECRET)], - capture_output=True, text=True, env=env) - if proc.returncode != 0: - raise RuntimeError( - f"sops decrypt of oikos-approval-hmac failed (is this host a recipient?): " - f"{proc.stderr.strip()}") - return yaml.safe_load(proc.stdout)["hmac_key"] - - -def _sign(request_id: str, entity: str, action: str, expires_at: str) -> str: - key = _hmac_key().encode() - msg = f"{request_id}:{entity}:{action}:{expires_at}".encode() - return hmac.new(key, msg, hashlib.sha256).hexdigest() - - -def reply(request_id: str, decision: str, *, phrase: str | None = None, - decided_by: str | None = None) -> dict: - """Record the operator's decision. On approval, issues a short-TTL - HMAC-signed grant token. Raises ValueError if a required confirmation - phrase is missing or wrong, or if the request already expired.""" - if decision not in ("approve", "deny"): - raise ValueError("decision must be 'approve' or 'deny'") - entry = current(request_id) - if entry is None: - raise ValueError(f"unknown approval id: {request_id}") - if entry.get("state") != "pending": - raise ValueError(f"{request_id} is not pending (state={entry.get('state')})") - now_s = datetime.now(timezone.utc).isoformat(timespec="seconds") - if entry.get("expires_at") and entry["expires_at"] < now_s: - _append({**entry, "ts": now_s, "state": "expired"}) - raise ValueError(f"{request_id} expired at {entry['expires_at']}") - - if decision == "deny": - return _append({**entry, "ts": now_s, "state": "denied", "decided_by": decided_by}) - - if entry.get("requires_phrase"): - if phrase != entry.get("confirmation_phrase"): - raise ValueError( - "confirmation phrase missing or incorrect — a reaction alone " - "cannot approve a destructive action") - - grant_expires = (datetime.now(timezone.utc) + timedelta(minutes=GRANT_TTL_MINUTES)) \ - .isoformat(timespec="seconds") - grant_token = _sign(request_id, entry["entity"], entry["action"], grant_expires) - return _append({**entry, "ts": now_s, "state": "approved", "decided_by": decided_by, - "grant_token": grant_token, "grant_expires": grant_expires}) - - -def check_grant(request_id: str, entity: str, action: str) -> tuple[bool, str]: - """Verify a request carries a valid, unexpired, matching grant, AND - consume it — a grant is exact-bound (this exact request id + entity + - action) and single-use: this call both checks and marks it "executed" - in the same step, so a second call for the same request_id fails with - "not approved" even if the grant's TTL hasn't expired yet. Callers - (homelab CLI mutating commands) must call this immediately before - executing, exactly once.""" - entry = current(request_id) - if entry is None: - return False, "unknown approval id" - if entry.get("state") != "approved": - return False, f"not approved (state={entry.get('state')})" - if entry.get("entity") != entity or entry.get("action") != action: - return False, "grant does not match entity/action" - now_s = datetime.now(timezone.utc).isoformat(timespec="seconds") - if not entry.get("grant_expires") or entry["grant_expires"] < now_s: - return False, "grant expired" - expected = _sign(request_id, entity, action, entry["grant_expires"]) - if not hmac.compare_digest(expected, entry.get("grant_token", "")): - return False, "grant signature invalid" - _append({**entry, "ts": now_s, "state": "executed"}) - return True, "ok" - - -def main() -> int: - import argparse - p = argparse.ArgumentParser(description="oikos approval engine") - sub = p.add_subparsers(dest="cmd", required=True) - - r = sub.add_parser("request") - r.add_argument("entity") - r.add_argument("action") - r.add_argument("risk") - r.add_argument("evidence") - r.add_argument("--verification") - r.add_argument("--signal-id") - r.add_argument("--requires-phrase", action="store_true") - r.add_argument("--ttl-hours", type=int, default=REQUEST_TTL_HOURS) - - ls = sub.add_parser("list") - ls.add_argument("--state", choices=VALID_STATES) - - rp = sub.add_parser("reply") - rp.add_argument("id") - rp.add_argument("decision", choices=["approve", "deny"]) - rp.add_argument("--phrase") - rp.add_argument("--decided-by") - - ck = sub.add_parser("check") - ck.add_argument("id") - ck.add_argument("entity") - ck.add_argument("action") - - args = p.parse_args() - if args.cmd == "request": - entry = request(args.entity, args.action, args.risk, args.evidence, - verification=args.verification, signal_id=args.signal_id, - requires_phrase=args.requires_phrase, ttl_hours=args.ttl_hours) - print(json.dumps({k: v for k, v in entry.items() if k != "matrix_message"}, indent=2)) - print() - print("--- post this to Matrix ---") - print(entry["matrix_message"]) - elif args.cmd == "list": - for e in list_approvals(state=args.state): - print(json.dumps(e)) - elif args.cmd == "reply": - try: - entry = reply(args.id, args.decision, phrase=args.phrase, decided_by=args.decided_by) - except (ValueError, RuntimeError) as e: - print(f"error: {e}", file=sys.stderr) - return 1 - print(json.dumps(entry, indent=2)) - elif args.cmd == "check": - ok, reason = check_grant(args.id, args.entity, args.action) - print(f"{'GRANTED' if ok else 'DENIED'}: {reason}") - return 0 if ok else 1 - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/decide.py b/oikos/decide.py deleted file mode 100644 index 22ac271..0000000 --- a/oikos/decide.py +++ /dev/null @@ -1,140 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/decide.py — the Decide stage of the OODA loop (decision classifier). - -Scores a proposed action against three inputs and routes it: - - 1. risk class — oikos/policy.yaml (read_only / reversible_low / - config_mutation / destructive) - 2. blast radius — oikos/relations.py transitive impact graph - 3. confidence — prior ledger history of this exact action on this - exact entity, falling back to "is this a - well-known mechanical action" when there's no - history yet - -Routes: **auto-act** (execute — risk is within unattended policy, blast -radius is contained, confidence isn't low) or **escalate** (request -operator approval via oikos/approve.py — anything else). The classifier -can only make an action MORE cautious than policy says, never less: if -policy already requires approval, escalate always wins regardless of -confidence or radius. - -Note on the third OODA route, "queue": that's the Signal engine's -info-severity routing (oikos/signal.py SEVERITY_ROUTE — informational -findings that need no action land in the console/weekly report, not -here). This module only classifies things that might actually need to -run, so it only ever returns auto-act or escalate. -""" - -from __future__ import annotations - -import json -import sys -from pathlib import Path - -REPO = Path(__file__).resolve().parent.parent -sys.path.insert(0, str(REPO)) -from oikos import ledger as oikos_ledger # noqa: E402 -from oikos import policy as oikos_policy # noqa: E402 -from oikos import relations as oikos_relations # noqa: E402 - -# Actions we consider "well-known mechanical" absent any ledger history — -# mirrors oikos/policy.py's safe_actions_for_service() baseline. -_ROUTINE_ACTIONS = {"restart", "service-restart", "health-check", "view-logs", - "view-docs", "sync", "cache-clear"} - -# A blast radius at or below this size counts as "contained" for auto-act -# purposes. Anything wider always escalates regardless of risk/confidence. -_CONTAINED_RADIUS = 1 - - -def _confidence(action: str, entity_id: str) -> tuple[str, str]: - hist = oikos_ledger.history(entity_id, limit=50) - successes = [h for h in hist if h.get("action") == action and h.get("result") == "ok"] - failures = [h for h in hist - if h.get("action") == action and str(h.get("result", "")).startswith("failed")] - if failures and not successes: - return "low", f"{len(failures)} prior failed attempt(s) of '{action}' on this entity" - if successes: - return "high", f"{len(successes)} prior successful run(s) of '{action}' on this entity" - if action in _ROUTINE_ACTIONS: - return "medium", "no history yet, but this is a well-known mechanical action" - return "low", "no ledger history and not a recognized routine action" - - -def classify(action: str, entity: str, *, service_name: str | None = None, - record: bool = False) -> dict: - """Classify one proposed action against one entity. `entity` may be a - bare name (resolved via oikos/relations.py) or an already-namespaced - id ("service:x" / "host:x"). If bare and ambiguous (matches both a - host and a service), the first resolved id is used — pass a - namespaced id explicitly to disambiguate. - """ - entity_ids = oikos_relations.resolve(entity) - entity_id = entity_ids[0] - - action = oikos_policy.canonical_action(action) - # Per-service policy overrides are keyed by service name (e.g. "caddy"). - # Most single-purpose nodes are named identically to the service they - # run, so infer it from the entity's bare name when not given explicitly. - if service_name is None: - service_name = entity_id.split(":", 1)[1] if ":" in entity_id else entity_id - - risk = (oikos_policy.classify_action(action, service_name) - or oikos_policy.classify_command(action)) - if risk is None: - risk = "config_mutation" # unknown action: default to the cautious class - - approval = oikos_policy.approval_for(risk) - radius = oikos_relations.blast_radius(entity_id) - contained = len(radius) <= _CONTAINED_RADIUS - confidence, why = _confidence(action, entity_id) - - if approval != "none": - route = "escalate" - reasoning = f"risk class '{risk}' requires approval ({approval}) per oikos/policy.yaml" - elif not contained: - route = "escalate" - reasoning = f"blast radius not contained ({len(radius)} entities: {', '.join(radius)})" - elif confidence == "low": - route = "escalate" - reasoning = f"low confidence — {why}" - else: - route = "auto-act" - reasoning = (f"risk '{risk}' is unattended-safe, blast radius contained " - f"({radius or 'none'}), confidence {confidence} — {why}") - - result = { - "entity": entity_id, - "action": action, - "risk": risk, - "approval": approval, - "blast_radius": radius, - "contained": contained, - "confidence": confidence, - "confidence_reason": why, - "route": route, - "reasoning": reasoning, - } - if record: - oikos_ledger.append(entity_id, f"decide:{action}", risk, - result=route, notes=reasoning) - return result - - -def main() -> int: - import argparse - p = argparse.ArgumentParser(description="oikos decision classifier") - p.add_argument("action") - p.add_argument("entity") - p.add_argument("--service-name", help="disambiguate policy overrides for a service action") - p.add_argument("--record", action="store_true", - help="append this classification to the change ledger") - args = p.parse_args() - result = classify(args.action, args.entity, service_name=args.service_name, - record=args.record) - print(json.dumps(result, indent=2)) - return 0 if result["route"] == "auto-act" else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/drift.py b/oikos/drift.py deleted file mode 100644 index 8e598f9..0000000 --- a/oikos/drift.py +++ /dev/null @@ -1,302 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/drift.py — drift detectors (Week 3 reliability layer). - -Each detector returns a list of finding dicts: - {kind, severity, entity, evidence, likely_cause, recommended_action, - verification} -matching the Signal schema in oikos/signal.py. `run_all()` runs every -detector and raises a Signal for each finding, skipping ones that already -have an open Signal for the same (entity, kind) — see -oikos/signal.py open_signal_for(). SSH-based detectors degrade to a -"probe-unreachable" info finding instead of a false drift signal when the -target can't be reached (no ssh, no network) — drift means "state -disagrees," not "couldn't check." - -Two detectors are fully local (no SSH, always safe to run from anywhere -with the repo checked out): SOPS-recipient-vs-inventory and lifecycle -consistency. Two need live access to hubris (pct list, caddy backend -config) and gracefully no-op when unreachable. - -Not yet implemented — needs data this repo doesn't structure yet: - - DNS records vs inventory services (no Technitium API wiring here) - - Generic tracked-config-repo cleanliness for every service (needs the - `mutation_path`/local-checkout-path field per service; only caddy's - path is documented today, so that's the one config-cleanliness check - implemented below) -""" - -from __future__ import annotations - -import re -import subprocess -import sys -from pathlib import Path - -REPO = Path(__file__).resolve().parent.parent -sys.path.insert(0, str(REPO)) -from oikos import relations as oikos_relations # noqa: E402 -from oikos import signal as oikos_signal # noqa: E402 - -import yaml # noqa: E402 - -SOPS_FILE = REPO / ".sops.yaml" -INVENTORY = REPO / "inventory.yaml" - -_AGE_RE = re.compile(r"age1[a-z0-9]+") - - -def _load_inventory() -> dict: - return yaml.safe_load(INVENTORY.read_text()) - - -def _sops_rules() -> list[dict]: - """Parse .sops.yaml's creation_rules: [{path_regex, recipients: [...]}].""" - doc = yaml.safe_load(SOPS_FILE.read_text()) - return [ - {"path_regex": r.get("path_regex", ""), - "recipients": _AGE_RE.findall(r.get("age", "") or "")} - for r in doc.get("creation_rules", []) - ] - - -def check_sops_recipients(inv: dict | None = None) -> list[dict]: - """Two directions of drift: - 1. A recipient pubkey in .sops.yaml that matches no host's age_pubkey - in inventory — likely a removed/revoked host whose secrets weren't - fully cleaned up (or a stale rule). - 2. An enrolled host (non-empty age_pubkey) missing from the - secrets/hello.yaml rule — every enrolled client should be able to - decrypt the bootstrap canary; absence usually means enrollment - stalled partway. - """ - inv = inv or _load_inventory() - hosts = inv.get("hosts", {}) - known_pubkeys = {e["age_pubkey"]: name for name, e in hosts.items() - if e.get("age_pubkey")} - rules = _sops_rules() - findings = [] - - all_recipients = {pk for r in rules for pk in r["recipients"]} - for pk in all_recipients: - if pk not in known_pubkeys: - findings.append({ - "kind": "sops-orphan-recipient", "severity": "warning", - "entity": "repo:Homelab-Docs", - "evidence": f"age recipient {pk[:20]}... appears in .sops.yaml but " - f"matches no host's age_pubkey in inventory.yaml", - "likely_cause": "host removed without full secret revocation, or a stale rule", - "recommended_action": {"runbook": "lifecycle-destroy-node", "risk": "destructive"}, - "verification": "grep .sops.yaml; check inventory.yaml + archaeology", - }) - - hello_rule = next((r for r in rules if "hello" in r["path_regex"]), None) - if hello_rule: - for pubkey, name in known_pubkeys.items(): - if pubkey not in hello_rule["recipients"]: - findings.append({ - "kind": "sops-missing-recipient", "severity": "warning", - "entity": f"host:{name}", - "evidence": f"{name}'s age_pubkey is not a recipient of secrets/hello.yaml", - "likely_cause": "client-add --finalize-pubkey ran incompletely", - "recommended_action": {"runbook": "client-enrollment", "risk": "config_mutation"}, - "verification": f"homelab client add {name} --finalize-pubkey ", - }) - return findings - - -def check_lifecycle_consistency(inv: dict | None = None) -> list[dict]: - """state: vs reality, using the ontology graph (oikos/relations.py) — - no SSH needed, this only reasons over inventory.yaml itself.""" - inv = inv or _load_inventory() - hosts = inv.get("hosts", {}) - archaeology = inv.get("archaeology", {}) - findings = [] - - for name, e in hosts.items(): - state = e.get("state", "active") - if state == "deprecated": - rel = oikos_relations.relations(f"host:{name}", inv) - if rel["affected_by"]: - findings.append({ - "kind": "lifecycle-inconsistent", "severity": "warning", - "entity": f"host:{name}", - "evidence": f"deprecated but still has inbound edges: {', '.join(rel['affected_by'])}", - "likely_cause": "deprecation started before dependents were migrated off", - "recommended_action": {"runbook": "lifecycle-deprecate-node", "risk": "config_mutation"}, - "verification": f"homelab node {name} relations", - }) - elif state == "destroyed": - findings.append({ - "kind": "lifecycle-inconsistent", "severity": "critical", - "entity": f"host:{name}", - "evidence": "state is 'destroyed' but the host still has a live hosts.: entry " - "(destroyed nodes belong in the archaeology: section, not hosts:)", - "likely_cause": "lifecycle-destroy-node runbook step 6 (move to archaeology) was skipped", - "recommended_action": {"runbook": "lifecycle-destroy-node", "risk": "destructive"}, - "verification": "grep -A3 '^ " + name + ":' inventory.yaml", - }) - - # A pve_id collision between an active host and an archaeology entry on - # the same Proxmox node usually means the ID was reused without the old - # entry's data being fully accounted for. - active_by_id = {(e.get("host"), e.get("pve_id")): name - for name, e in hosts.items() if e.get("pve_id")} - for aname, ae in archaeology.items(): - key = (None, ae.get("pve_id")) # archaeology doesn't record which node - for (host, pve_id), name in active_by_id.items(): - if pve_id == ae.get("pve_id") and name != aname: - findings.append({ - "kind": "lifecycle-pve-id-reuse", "severity": "info", - "entity": f"host:{name}", - "evidence": f"pve_id {pve_id} is both active ({name}) and in archaeology ({aname}, " - f"destroyed {ae.get('destroyed', '?')})", - "likely_cause": "normal ID reuse after destroy — informational only", - "recommended_action": None, - "verification": None, - }) - return findings - - -def _ssh_hubris(cmd: list[str]) -> subprocess.CompletedProcess | None: - """Best-effort ssh to hubris. Returns None (not a CompletedProcess with - nonzero rc) if ssh itself can't run/connect at all, so callers can - distinguish "unreachable" from "reachable but the command failed.""" - try: - return subprocess.run(["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5", - "root@192.168.8.77", " ".join(cmd)], - capture_output=True, text=True, timeout=15) - except (subprocess.TimeoutExpired, OSError): - return None - - -def check_pct_list(inv: dict | None = None) -> list[dict]: - """Inventory LXCs/VMs on hubris vs live `pct list` + `qm list`.""" - inv = inv or _load_inventory() - hosts = inv.get("hosts", {}) - proc = _ssh_hubris(["pct", "list"]) - if proc is None or proc.returncode != 0: - return [{ - "kind": "probe-unreachable", "severity": "info", "entity": "host:hubris", - "evidence": "could not reach hubris to run `pct list`", - "likely_cause": "no network path from this client to hubris, or ssh key not authorized", - "recommended_action": None, "verification": "homelab ssh hubris -- pct list", - }] - live_ids = set() - for line in proc.stdout.splitlines()[1:]: - parts = line.split() - if parts and parts[0].isdigit(): - live_ids.add(int(parts[0])) - - inventory_ids = {e["pve_id"] for name, e in hosts.items() - if e.get("kind") == "lxc" and e.get("host") == "hubris" and e.get("pve_id")} - - findings = [] - for missing in sorted(inventory_ids - live_ids): - name = next(n for n, e in hosts.items() if e.get("pve_id") == missing) - findings.append({ - "kind": "inventory-vs-live", "severity": "critical", "entity": f"host:{name}", - "evidence": f"pve_id {missing} ({name}) is in inventory.yaml but absent from " - f"`pct list` on hubris", - "likely_cause": "destroyed outside the lifecycle-destroy-node runbook, or migrated " - "without updating inventory", - "recommended_action": {"runbook": "lifecycle-destroy-node", "risk": "destructive"}, - "verification": "homelab ssh hubris -- pct list", - }) - for extra in sorted(live_ids - inventory_ids): - findings.append({ - "kind": "inventory-vs-live", "severity": "warning", "entity": "host:hubris", - "evidence": f"pve_id {extra} exists on hubris (`pct list`) but has no inventory.yaml entry", - "likely_cause": "created outside the provision-node runbook", - "recommended_action": {"runbook": "lifecycle-provision-node", "risk": "config_mutation"}, - "verification": "homelab ssh hubris -- pct config " + str(extra), - }) - return findings - - -def check_caddy_backends(inv: dict | None = None) -> list[dict]: - """Caddy's /etc/caddy (a git checkout of dtoro/caddy-conf, per - knowledge/wiki/containers/121-caddy.md) vs inventory service backend IPs. Best-effort - grep for reverse_proxy targets; skips services whose Caddyfile snippet - doesn't use a bare IP (e.g. references a Caddy snippet/import).""" - inv = inv or _load_inventory() - hosts = inv.get("hosts", {}) - services = inv.get("services", {}) - caddy_ip = hosts.get("caddy", {}).get("lan_ip") - if not caddy_ip: - return [] - try: - proc = subprocess.run( - ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=5", f"root@{caddy_ip}", - "grep -rhoE 'reverse_proxy[^{]*' /etc/caddy/ 2>/dev/null"], - capture_output=True, text=True, timeout=15, - ) - except (subprocess.TimeoutExpired, OSError): - proc = None - if proc is None or proc.returncode != 0: - return [{ - "kind": "probe-unreachable", "severity": "info", "entity": "host:caddy", - "evidence": "could not reach caddy to inspect /etc/caddy", - "likely_cause": "no network path from this client to caddy, or ssh key not authorized", - "recommended_action": None, "verification": "homelab ssh caddy -- grep -r reverse_proxy /etc/caddy", - }] - - live_ips = set(re.findall(r"\d+\.\d+\.\d+\.\d+", proc.stdout)) - findings = [] - for svc, e in services.items(): - if not isinstance(e, dict): - continue - backend = e.get("backend") - if not backend or not (e.get("url") or e.get("public_host")): - continue - backend_ip = hosts.get(backend, {}).get("lan_ip") - if backend_ip and live_ips and backend_ip not in live_ips: - findings.append({ - "kind": "caddy-backend-mismatch", "severity": "warning", "entity": f"service:{svc}", - "evidence": f"inventory backend IP {backend_ip} ({backend}) not found in any " - f"reverse_proxy directive on caddy", - "likely_cause": "backend migrated (lan_ip changed) without updating the Caddyfile, " - "or the service uses a snippet/import this grep can't resolve", - "recommended_action": {"runbook": "config-change-deploy", "risk": "config_mutation"}, - "verification": f"homelab service {svc} health", - }) - return findings - - -DETECTORS = [check_sops_recipients, check_lifecycle_consistency, - check_pct_list, check_caddy_backends] - - -def run_all(inv: dict | None = None, *, raise_signals: bool = False) -> list[dict]: - inv = inv or _load_inventory() - findings: list[dict] = [] - for detector in DETECTORS: - findings.extend(detector(inv)) - if raise_signals: - for f in findings: - if oikos_signal.open_signal_for(f["entity"], f["kind"]) is not None: - continue # already open, don't duplicate - oikos_signal.raise_signal( - f["kind"], f["severity"], f["entity"], f["evidence"], - likely_cause=f.get("likely_cause"), - recommended_action=f.get("recommended_action"), - verification=f.get("verification"), - ) - return findings - - -def main() -> int: - import argparse - import json - p = argparse.ArgumentParser(description="oikos drift detectors") - p.add_argument("--raise-signals", action="store_true", - help="raise a Signal for each new finding (default: print only)") - args = p.parse_args() - findings = run_all(raise_signals=args.raise_signals) - for f in findings: - print(json.dumps(f)) - print(f"\n{len(findings)} finding(s)", file=sys.stderr) - return 1 if any(f["severity"] == "critical" for f in findings) else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/ledger.py b/oikos/ledger.py deleted file mode 100644 index a867dd2..0000000 --- a/oikos/ledger.py +++ /dev/null @@ -1,109 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/ledger.py — append-only change ledger. - -Every mutation an agent or operator performs (reversible_low and above, -per oikos/policy.yaml) gets one JSON line in ledger/.jsonl: -timestamp, acting agent identity, entity, action, risk class, approval -reference, verification result. Append-only, committed like any tracked -file — never edited or reordered in place. - -CLI: - python3 oikos/ledger.py append [--verification ...] [--result ...] - python3 oikos/ledger.py history [--limit N] -""" - -from __future__ import annotations - -import json -import os -import socket -import sys -from datetime import datetime, timezone -from pathlib import Path - -REPO = Path(__file__).resolve().parent.parent -LEDGER_DIR = REPO / "ledger" - - -def _agent_identity() -> str: - """Best-effort identity for the acting agent. HOMELAB_AGENT_ID lets - approval-engine callers stamp the real requester; falls back to the - local hostname.""" - return os.environ.get("HOMELAB_AGENT_ID") or socket.gethostname().split(".")[0] - - -def append(entity: str, action: str, risk: str, *, verification: str | None = None, - result: str | None = None, approval_ref: str | None = None, - agent: str | None = None, notes: str | None = None) -> dict: - """Append one change entry. Returns the recorded dict (None fields dropped).""" - entry = { - "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"), - "agent": agent or _agent_identity(), - "entity": entity, - "action": action, - "risk": risk, - "approval_ref": approval_ref, - "verification": verification, - "result": result, - "notes": notes, - } - entry = {k: v for k, v in entry.items() if v is not None} - LEDGER_DIR.mkdir(exist_ok=True) - month = datetime.now(timezone.utc).strftime("%Y-%m") - path = LEDGER_DIR / f"{month}.jsonl" - with path.open("a") as f: - f.write(json.dumps(entry, sort_keys=False) + "\n") - return entry - - -def history(entity: str, limit: int = 20) -> list[dict]: - """Most recent `limit` ledger entries for `entity`, newest first.""" - entries: list[dict] = [] - if not LEDGER_DIR.exists(): - return entries - for path in sorted(LEDGER_DIR.glob("*.jsonl")): - for line in path.read_text().splitlines(): - if not line.strip(): - continue - try: - e = json.loads(line) - except json.JSONDecodeError: - continue - if e.get("entity") == entity: - entries.append(e) - entries.sort(key=lambda e: e.get("ts", ""), reverse=True) - return entries[:limit] - - -def main() -> int: - import argparse - p = argparse.ArgumentParser(description="oikos change ledger") - sub = p.add_subparsers(dest="cmd", required=True) - - sp = sub.add_parser("append") - sp.add_argument("entity") - sp.add_argument("action") - sp.add_argument("risk") - sp.add_argument("--verification") - sp.add_argument("--result") - sp.add_argument("--approval-ref") - sp.add_argument("--notes") - - sh = sub.add_parser("history") - sh.add_argument("entity") - sh.add_argument("--limit", type=int, default=20) - - args = p.parse_args() - if args.cmd == "append": - entry = append(args.entity, args.action, args.risk, - verification=args.verification, result=args.result, - approval_ref=args.approval_ref, notes=args.notes) - print(json.dumps(entry, indent=2)) - elif args.cmd == "history": - for e in history(args.entity, args.limit): - print(json.dumps(e)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/policy.py b/oikos/policy.py deleted file mode 100644 index 97d345d..0000000 --- a/oikos/policy.py +++ /dev/null @@ -1,72 +0,0 @@ -"""oikos/policy.py — load oikos/policy.yaml and classify actions. - -Shared by bin/homelab, mcp/server.py, and oikos/gen-topology.py so every -surface agrees on risk classes. See OIKOS.md for the operating model. -""" - -from __future__ import annotations - -from pathlib import Path - -import yaml - -REPO = Path(__file__).resolve().parent.parent -POLICY_FILE = REPO / "oikos" / "policy.yaml" - - -def load() -> dict: - return yaml.safe_load(POLICY_FILE.read_text()) - - -def classify_command(cmd: str) -> str | None: - """Risk class for a `homelab ` subcommand.""" - return load().get("commands", {}).get(cmd) - - -# Synonyms for the canonical action keys in oikos/policy.yaml `actions:`. -# `homelab restart ` is the real CLI verb; "restart" is what an -# agent proposing the action is most likely to say. Keep this list in sync -# with anything oikos/decide.py or the CLI classifies by name. -ACTION_ALIASES = { - "restart": "service-restart", -} - - -def canonical_action(action: str) -> str: - return ACTION_ALIASES.get(action, action) - - -def classify_action(action: str, service: str | None = None) -> str | None: - """Risk class for a generic action, honoring per-service overrides.""" - action = canonical_action(action) - pol = load() - if service: - override = pol.get("service_overrides", {}).get(service, {}).get(action) - if override: - return override - return pol.get("actions", {}).get(action) - - -def approval_for(risk: str) -> str: - return load().get("risk_classes", {}).get(risk, {}).get("approval", "unknown") - - -def safe_actions_for_service(name: str, svc_entry: dict) -> list[dict]: - """Actions an agent can propose for this service, each tagged with its - risk class and whether operator approval is required. Derived from what - the service entry actually declares — no action is offered that the - service doesn't support. - """ - out = [ - {"action": "health-check", "risk": "read_only", "approval": "none"}, - {"action": "view-logs", "risk": "read_only", "approval": "none"}, - {"action": "view-docs", "risk": "read_only", "approval": "none"}, - ] - if svc_entry.get("backend"): - risk = classify_action("service-restart", name) - out.append({"action": "restart", "risk": risk, "approval": approval_for(risk)}) - if svc_entry.get("config_repo"): - risk = classify_action("tracked-config-edit", name) - out.append({"action": "edit-config-and-deploy", "risk": risk, - "approval": approval_for(risk)}) - return out diff --git a/oikos/relations.py b/oikos/relations.py deleted file mode 100644 index df1ee3e..0000000 --- a/oikos/relations.py +++ /dev/null @@ -1,131 +0,0 @@ -"""oikos/relations.py — walk the ontology graph derived from inventory.yaml. - -Wires up the subset of oikos/ontology.yaml relationships that are already -structured data today: hosts, mounts, provides, configured-by, depends-on. -Everything else in the ontology (physical, external, identity domains) is -documented but thin — not yet backed by inventory fields, so it doesn't -appear in the graph until those fields are populated. - -Entities are namespaced ("host:name", "service:name", "repo:name", -"mount:path") because host and service names collide in this inventory -(e.g. "jellyfin" is both a service and its own LXC). - -Impact polarity: build_impacts() returns edges in the direction -"if SOURCE fails/disappears, TARGET is affected" — this is not the same -direction as how the fact is stored in inventory (e.g. a mount is stored -as guest -> pool, but if the POOL fails the GUEST is impacted, so the -impact edge runs pool -> guest). -""" - -from __future__ import annotations - -from pathlib import Path - -import yaml - -REPO = Path(__file__).resolve().parent.parent -INVENTORY = REPO / "inventory.yaml" - - -def _hid(name: str) -> str: - return f"host:{name}" - - -def _sid(name: str) -> str: - return f"service:{name}" - - -def _rid(name: str) -> str: - return f"repo:{name}" - - -def _mid(name: str) -> str: - return f"mount:{name}" - - -def load_inventory() -> dict: - return yaml.safe_load(INVENTORY.read_text()) - - -def resolve(entity: str, inv: dict | None = None) -> list[str]: - """Map a bare name (as typed on the CLI) to every namespaced id it - could refer to. A bare name may match a host AND a service.""" - inv = inv or load_inventory() - if ":" in entity: - return [entity] - ids = [] - if entity in inv.get("hosts", {}): - ids.append(_hid(entity)) - if entity in inv.get("services", {}): - ids.append(_sid(entity)) - if not ids: - ids.append(entity) - return ids - - -def build_impacts(inv: dict | None = None) -> dict[str, set[str]]: - inv = inv or load_inventory() - hosts = inv.get("hosts", {}) - services = inv.get("services", {}) - impacts: dict[str, set[str]] = {} - - def add(source: str, target: str) -> None: - impacts.setdefault(source, set()).add(target) - - for name, e in hosts.items(): - hid = _hid(name) - parent = e.get("host") - if parent: - add(_hid(parent), hid) # host failing -> guest impacted - for mount in e.get("mounts", []): - add(_mid(mount), hid) # pool failing -> mounter impacted - for dep in e.get("depends_on", []) or []: - add(_hid(dep), hid) # dependency failing -> dependent impacted - - for svc, e in services.items(): - if not isinstance(e, dict): - continue - sid = _sid(svc) - backend = e.get("backend") - if backend and backend in hosts: - add(_hid(backend), sid) # backend failing -> service impacted - config_repo = e.get("config_repo") - if config_repo: - add(_rid(config_repo), _hid(backend)) # bad config -> backend impacted - - return impacts - - -def blast_radius(entity_id: str, inv: dict | None = None) -> list[str]: - """Transitive closure: every entity affected if `entity_id` fails.""" - impacts = build_impacts(inv) - seen: set[str] = set() - stack = [entity_id] - while stack: - cur = stack.pop() - for nxt in impacts.get(cur, ()): - if nxt not in seen: - seen.add(nxt) - stack.append(nxt) - return sorted(seen) - - -def relations(entity_id: str, inv: dict | None = None) -> dict: - """One entity's direct edges plus its full transitive blast radius.""" - impacts = build_impacts(inv) - impacts_on = sorted(impacts.get(entity_id, ())) - affected_by = sorted(src for src, targets in impacts.items() if entity_id in targets) - return { - "entity": entity_id, - "impacts": impacts_on, - "affected_by": affected_by, - "blast_radius": blast_radius(entity_id, inv), - } - - -def relations_for_name(name: str, inv: dict | None = None) -> list[dict]: - """CLI/MCP entry point: resolve a bare name and report relations for - every matching entity id (usually one; two if host and service share - a name).""" - inv = inv or load_inventory() - return [relations(eid, inv) for eid in resolve(name, inv)] diff --git a/oikos/scheduler.py b/oikos/scheduler.py deleted file mode 100644 index b1e05ed..0000000 --- a/oikos/scheduler.py +++ /dev/null @@ -1,230 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/scheduler.py — the Observe stage: periodic probes -> Signals + state cache. - -Intended to run under a systemd timer (see scripts/sync/linux/ for the -existing timer pattern this should mirror — deploy target: LXC 105 -alongside the MCP server) every 5-15 minutes. Each run: - - 1. Probes every service's HTTP health and writes oikos/state.json — the - snapshot `homelab service health` serves by default (cache- - first reads); pass --live on that command to force a fresh probe - instead of trusting the cache. - 2. Probes disk usage on hubris + strong (best-effort SSH; degrades to a - "probe-unreachable" info Signal rather than a false disk-threshold - alarm when unreachable). - 3. Runs the Week-3 drift detectors (oikos/drift.py). - 4. Raises Signals for anything over threshold or drifted — skipping - duplicates via oikos_signal.open_signal_for() — and auto-resolves - any open Signal whose condition has since cleared. - -Temperature probing is NOT implemented here: LXCs don't expose host -sensors, and hubris/strong's actual sensor path (lm-sensors vs vendor -tool) hasn't been confirmed on either box yet — a real check needs that -groundwork first rather than a guessed command. Backup-freshness is -likewise deferred: `backs-up-to` isn't populated in inventory yet (see -oikos/ontology.yaml — it's a documented thin field), so there's nothing -structured to check freshness against. - -CLI: - python3 oikos/scheduler.py run # probe, raise signals, write state.json - python3 oikos/scheduler.py run --dry-run # probe and print only, no side effects -""" - -from __future__ import annotations - -import json -import subprocess -import sys -from datetime import datetime, timezone -from pathlib import Path - -import yaml - -REPO = Path(__file__).resolve().parent.parent -sys.path.insert(0, str(REPO)) -from oikos import drift as oikos_drift # noqa: E402 -from oikos import signal as oikos_signal # noqa: E402 - -INVENTORY = REPO / "inventory.yaml" -STATE_FILE = REPO / "oikos" / "state.json" - -DISK_WARN_PCT = 85 -DISK_CRIT_PCT = 95 -HEALTH_TIMEOUT_S = 5 - - -def _load_inventory() -> dict: - return yaml.safe_load(INVENTORY.read_text()) - - -# Services whose public `url`/`endpoint` isn't a plain-GET health check — -# mirrors the special-casing bin/homelab's `doctor` command already does. -# A generic per-service `health:` inventory field (checked URL/command -# distinct from the public url) would remove the need for this; until -# that's backfilled, these are the two known exceptions. -_HEALTH_OVERRIDES = { - # FastMCP's streamable-http endpoint expects a JSON-RPC POST handshake, - # not a bare GET — a plain curl gets 4xx even when the server is fully - # healthy (confirmed against the live server: consistently 400, never - # 2xx/3xx, for a GET). Any HTTP response at all (vs. no response/ - # connection refused) proves the process is up and answering, which is - # what "service-down" alerting actually cares about here. - "homelab_mcp": {"extra_curl_args": ["-H", "Accept: text/event-stream"], - "any_response_ok": True}, - "secrets_issuance": { - "url_transform": lambda url: url.rstrip("/").rsplit("/", 1)[0] + "/health", - }, - # Token-gated at the app level (401 without a token is correct, not - # down) — see knowledge/wiki/containers/128-trmnl.md, which documents a dedicated - # /health endpoint returning 200 unauthenticated. - "trmnl": {"url_transform": lambda url: url.rstrip("/") + "/health"}, -} - - -def probe_service_health(name: str, entry: dict) -> dict: - url = entry.get("url") or entry.get("endpoint") - if not url: - return {"service": name, "checked": False} - override = _HEALTH_OVERRIDES.get(name, {}) - check_url = override.get("url_transform", lambda u: u)(url) - curl_cmd = ["curl", "-sS", "-o", "/dev/null", "-w", "%{http_code}", - "--max-time", str(HEALTH_TIMEOUT_S), *override.get("extra_curl_args", []), check_url] - try: - proc = subprocess.run(curl_cmd, capture_output=True, text=True) - code = proc.stdout.strip() - except OSError: - code = "" - ok = bool(code) if override.get("any_response_ok") else code.startswith(("2", "3")) - return {"service": name, "checked": True, "url": url, "checked_url": check_url, - "http_code": code or None, "ok": ok} - - -def _ssh(host_ip: str, remote_cmd: str, timeout: int = 10) -> str | None: - try: - proc = subprocess.run( - ["ssh", "-o", "BatchMode=yes", "-o", f"ConnectTimeout={min(timeout, 5)}", - f"root@{host_ip}", remote_cmd], - capture_output=True, text=True, timeout=timeout, - ) - except (subprocess.TimeoutExpired, OSError): - return None - return proc.stdout.strip() if proc.returncode == 0 else None - - -def probe_disk(name: str, entry: dict) -> dict | None: - """`df -P /` on a proxmox-host's own root fs. LXC-level disk usage is - already covered by MCP's get_lxc_state; this probe is host-level.""" - if entry.get("kind") != "proxmox-host" or not entry.get("lan_ip"): - return None - out = _ssh(entry["lan_ip"], "df -P / | tail -1") - if out is None: - return {"host": name, "checked": False} - parts = out.split() - if len(parts) < 5 or not parts[4].rstrip("%").isdigit(): - return {"host": name, "checked": False} - return {"host": name, "checked": True, "used_pct": int(parts[4].rstrip("%"))} - - -def run(dry_run: bool = False) -> dict: - inv = _load_inventory() - now = datetime.now(timezone.utc).isoformat(timespec="seconds") - - services_state = {} - for name, entry in inv.get("services", {}).items(): - if not isinstance(entry, dict): - continue - result = probe_service_health(name, entry) - services_state[name] = result - if not result.get("checked"): - continue - sig_kind = "service-down" - if result["ok"]: - open_sig = oikos_signal.open_signal_for(f"service:{name}", sig_kind) - if open_sig and not dry_run: - oikos_signal.resolve(open_sig["id"], note="health probe recovered") - else: - if not dry_run and oikos_signal.open_signal_for(f"service:{name}", sig_kind) is None: - oikos_signal.raise_signal( - sig_kind, "critical", f"service:{name}", - f"{result['url']} -> {result.get('http_code') or 'no response'}", - likely_cause="backend down, crashed, or ingress misconfigured", - recommended_action={"runbook": "service-health-check", "risk": "reversible_low"}, - verification=f"homelab service {name} health --live", - ) - - hosts_state = {} - for name, entry in inv.get("hosts", {}).items(): - disk = probe_disk(name, entry) - if disk is None: - continue - hosts_state[name] = disk - if not disk.get("checked"): - continue - pct = disk["used_pct"] - sig_kind = "disk-threshold" - if pct >= DISK_CRIT_PCT: - severity = "critical" - elif pct >= DISK_WARN_PCT: - severity = "warning" - else: - severity = None - open_sig = oikos_signal.open_signal_for(f"host:{name}", sig_kind) - if severity is None: - if open_sig and not dry_run: - oikos_signal.resolve(open_sig["id"], note=f"disk usage back to {pct}%") - elif open_sig is None and not dry_run: - oikos_signal.raise_signal( - sig_kind, severity, f"host:{name}", f"root filesystem {pct}% used", - likely_cause="data growth or a runaway log", - recommended_action={"runbook": "config-change-deploy", "risk": "config_mutation"}, - verification=f"homelab ssh {name} -- df -h /", - ) - - drift_findings = oikos_drift.run_all(inv, raise_signals=not dry_run) - - snapshot = { - "generated_at": now, - "services": services_state, - "hosts": hosts_state, - "drift_findings": len(drift_findings), - } - if not dry_run: - STATE_FILE.write_text(json.dumps(snapshot, indent=2) + "\n") - return snapshot - - -def read_state() -> dict | None: - """Load the last scheduler snapshot, or None if it's never run.""" - if not STATE_FILE.exists(): - return None - try: - return json.loads(STATE_FILE.read_text()) - except json.JSONDecodeError: - return None - - -def cached_service_health(name: str) -> dict | None: - state = read_state() - if state is None: - return None - entry = state.get("services", {}).get(name) - if entry is None: - return None - return {**entry, "as_of": state.get("generated_at")} - - -def main() -> int: - import argparse - p = argparse.ArgumentParser(description="oikos scheduler (Observe stage)") - sub = p.add_subparsers(dest="cmd", required=True) - r = sub.add_parser("run") - r.add_argument("--dry-run", action="store_true") - args = p.parse_args() - if args.cmd == "run": - snapshot = run(dry_run=args.dry_run) - print(json.dumps(snapshot, indent=2)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/oikos/signal.py b/oikos/signal.py deleted file mode 100644 index afd0595..0000000 --- a/oikos/signal.py +++ /dev/null @@ -1,239 +0,0 @@ -#!/usr/bin/env python3 -"""oikos/signal.py — the Signal engine (attention layer). - -A Signal is anything the lab notices that needs attention and possibly -action: pending updates, high temperature, low disk, a service down, a -cert expiring, a stale backup, inventory drift, a node stuck mid- -lifecycle-transition. See OIKOS.md / oikos/ontology.yaml "Signals — the -attention layer". - -Storage: signals/.jsonl, same append-only-JSONL convention as -oikos/ledger.py, but a signal's *state* changes over its lifecycle -(raised -> acknowledged -> acting -> resolved | muted). Each state change -is a NEW line with the same id — never edit a line in place, so git -history stays a legible append-only audit trail. `current()` / -`list_signals()` reduce the log to each id's latest state. - -CLI: - python3 oikos/signal.py raise [--likely-cause] [--action-runbook] [--action-risk] [--verification] - python3 oikos/signal.py list [--state raised] [--entity strong] [--severity warning] - python3 oikos/signal.py ack - python3 oikos/signal.py resolve [--note ...] - python3 oikos/signal.py mute [--ttl-hours 24] -""" - -from __future__ import annotations - -import json -import sys -from datetime import datetime, timedelta, timezone -from pathlib import Path - -REPO = Path(__file__).resolve().parent.parent -SIGNALS_DIR = REPO / "signals" - -VALID_SEVERITIES = ("info", "warning", "critical") -VALID_STATES = ("raised", "acknowledged", "acting", "resolved", "muted") -# Routing per oikos/ontology.yaml "Signals" section. -SEVERITY_ROUTE = { - "info": "console+report", - "warning": "matrix-digest", - "critical": "matrix-immediate", -} - - -def _month_path(dt: datetime | None = None) -> Path: - dt = dt or datetime.now(timezone.utc) - return SIGNALS_DIR / f"{dt.strftime('%Y-%m')}.jsonl" - - -def _all_entries() -> list[dict]: - if not SIGNALS_DIR.exists(): - return [] - out = [] - for path in sorted(SIGNALS_DIR.glob("*.jsonl")): - for line in path.read_text().splitlines(): - if not line.strip(): - continue - try: - out.append(json.loads(line)) - except json.JSONDecodeError: - continue - return out - - -def _next_id(dt: datetime | None = None) -> str: - dt = dt or datetime.now(timezone.utc) - prefix = f"sig-{dt.strftime('%Y-%m-%d')}-" - existing = [e["id"] for e in _all_entries() if e.get("id", "").startswith(prefix)] - n = 1 - while f"{prefix}{n:04d}" in existing: - n += 1 - return f"{prefix}{n:04d}" - - -def _append(entry: dict) -> dict: - SIGNALS_DIR.mkdir(exist_ok=True) - entry = {k: v for k, v in entry.items() if v is not None} - with _month_path().open("a") as f: - f.write(json.dumps(entry, sort_keys=False) + "\n") - return entry - - -def raise_signal(kind: str, severity: str, entity: str, evidence: str, *, - likely_cause: str | None = None, - recommended_action: dict | None = None, - verification: str | None = None, - signal_id: str | None = None) -> dict: - """Raise a new Signal, or (if `signal_id` names an existing one and it's - not currently open) re-raise it. Returns the recorded entry.""" - if severity not in VALID_SEVERITIES: - raise ValueError(f"severity must be one of {VALID_SEVERITIES}") - sid = signal_id or _next_id() - entry = { - "id": sid, - "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"), - "kind": kind, - "severity": severity, - "entity": entity, - "evidence": evidence, - "likely_cause": likely_cause, - "recommended_action": recommended_action, - "verification": verification, - "state": "raised", - "route": SEVERITY_ROUTE.get(severity, "console+report"), - } - return _append(entry) - - -def _transition(signal_id: str, state: str, *, note: str | None = None, - mute_until: str | None = None) -> dict: - if state not in VALID_STATES: - raise ValueError(f"state must be one of {VALID_STATES}") - prior = current(signal_id) - if prior is None: - raise ValueError(f"unknown signal id: {signal_id}") - entry = {**prior, "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"), - "state": state, "note": note, "mute_until": mute_until} - return _append(entry) - - -def acknowledge(signal_id: str, note: str | None = None) -> dict: - return _transition(signal_id, "acknowledged", note=note) - - -def start_acting(signal_id: str, note: str | None = None) -> dict: - return _transition(signal_id, "acting", note=note) - - -def resolve(signal_id: str, note: str | None = None) -> dict: - return _transition(signal_id, "resolved", note=note) - - -def mute(signal_id: str, ttl_hours: int = 24, note: str | None = None) -> dict: - until = (datetime.now(timezone.utc) + timedelta(hours=ttl_hours)).isoformat(timespec="seconds") - return _transition(signal_id, "muted", note=note, mute_until=until) - - -def current(signal_id: str) -> dict | None: - """Latest recorded state for one signal id, or None if unknown.""" - matches = [e for e in _all_entries() if e.get("id") == signal_id] - if not matches: - return None - matches.sort(key=lambda e: e.get("ts", "")) - return matches[-1] - - -def list_signals(state: str | None = None, entity: str | None = None, - severity: str | None = None, kind: str | None = None) -> list[dict]: - """Every signal's latest state, optionally filtered. Auto-expires mute - (a signal muted past its mute_until is treated as raised again).""" - latest: dict[str, dict] = {} - for e in sorted(_all_entries(), key=lambda e: e.get("ts", "")): - sid = e.get("id") - if sid: - latest[sid] = e - - now = datetime.now(timezone.utc).isoformat(timespec="seconds") - out = [] - for e in latest.values(): - if e.get("state") == "muted" and e.get("mute_until") and e["mute_until"] < now: - e = {**e, "state": "raised", "note": "mute expired"} - if state and e.get("state") != state: - continue - if entity and e.get("entity") != entity: - continue - if severity and e.get("severity") != severity: - continue - if kind and e.get("kind") != kind: - continue - out.append(e) - out.sort(key=lambda e: e.get("ts", ""), reverse=True) - return out - - -def open_signal_for(entity: str, kind: str) -> dict | None: - """The currently-open (raised/acknowledged/acting) signal for this - entity+kind, if any. Used by probes to avoid raising duplicates and to - auto-resolve when a condition clears.""" - for e in list_signals(entity=entity, kind=kind): - if e.get("state") in ("raised", "acknowledged", "acting"): - return e - return None - - -def main() -> int: - import argparse - p = argparse.ArgumentParser(description="oikos signal engine") - sub = p.add_subparsers(dest="cmd", required=True) - - r = sub.add_parser("raise") - r.add_argument("kind") - r.add_argument("severity", choices=VALID_SEVERITIES) - r.add_argument("entity") - r.add_argument("evidence") - r.add_argument("--likely-cause") - r.add_argument("--action-runbook") - r.add_argument("--action-risk") - r.add_argument("--verification") - - ls = sub.add_parser("list") - ls.add_argument("--state", choices=VALID_STATES) - ls.add_argument("--entity") - ls.add_argument("--severity", choices=VALID_SEVERITIES) - ls.add_argument("--kind") - - for name in ("ack", "resolve"): - sp = sub.add_parser(name) - sp.add_argument("id") - sp.add_argument("--note") - - mt = sub.add_parser("mute") - mt.add_argument("id") - mt.add_argument("--ttl-hours", type=int, default=24) - mt.add_argument("--note") - - args = p.parse_args() - if args.cmd == "raise": - action = None - if args.action_runbook or args.action_risk: - action = {"runbook": args.action_runbook, "risk": args.action_risk} - entry = raise_signal(args.kind, args.severity, args.entity, args.evidence, - likely_cause=args.likely_cause, recommended_action=action, - verification=args.verification) - print(json.dumps(entry, indent=2)) - elif args.cmd == "list": - for e in list_signals(state=args.state, entity=args.entity, - severity=args.severity, kind=args.kind): - print(json.dumps(e)) - elif args.cmd == "ack": - print(json.dumps(acknowledge(args.id, args.note), indent=2)) - elif args.cmd == "resolve": - print(json.dumps(resolve(args.id, args.note), indent=2)) - elif args.cmd == "mute": - print(json.dumps(mute(args.id, args.ttl_hours, args.note), indent=2)) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/convert-wiki.py b/scripts/convert-wiki.py new file mode 100644 index 0000000..06cf561 --- /dev/null +++ b/scripts/convert-wiki.py @@ -0,0 +1,397 @@ +#!/usr/bin/env python3 +"""One-shot: convert knowledge/wiki/ to seeds/knowledge.yaml.""" + +import os, re, yaml +from pathlib import Path +from hashlib import sha256 + +REPO = Path("/Users/dtoro/Projects/oikos") +WIKI = REPO / "knowledge/wiki" +SOURCES = REPO / "knowledge/sources" +GLOSSARY = REPO / "knowledge/GLOSSARY.md" + +# Maps wiki path components to entity slugs +# Format: (path_pattern, entity_slug) +PATH_TO_ENTITY = { + # Containers + "containers/101-jellyfin": "lxc:jellyfin", + "containers/102-nfs-export": "lxc:nfs-export", + "containers/103-paperless": "lxc:paperless", + "containers/104-gitea": "lxc:gitea", + "containers/105-apps": "lxc:apps", + "containers/106-auth-outpost": "lxc:auth-outpost", + "containers/107-dns": "lxc:dns", + "containers/114-nextcloud": "lxc:nextcloud", + "containers/118-elementsynapse": "lxc:elementsynapse", + "containers/119-sophia": "lxc:sophia", + "containers/120-mule-images": "lxc:mule-images", + "containers/121-caddy": "lxc:caddy", + "containers/122-arriman": "lxc:arriman", + "containers/128-trmnl": "lxc:trmnl", + "containers/129-house": "lxc:house", + "containers/130-grimmory": "lxc:grimmory", + "containers/131-teddycloud": "lxc:teddycloud", + "containers/132-rclone": "lxc:rclone", + "containers/133-seanime": "lxc:seanime", + "containers/134-romm": "lxc:romm", + # Hosts + "hosts/hubris": "proxmox-host:hubris", + "hosts/strong": "proxmox-host:strong", + # VMs + "vms/100-zimaos": "vm:zimaos", + "vms/108-haos": "vm:haos", + # Infrastructure → services + "infrastructure/auto-deploy": None, + "infrastructure/backups": None, + "infrastructure/dns": "service:dns", + "infrastructure/homelab-context": "service:homelab_mcp", + "infrastructure/ingress": "service:caddy", + "infrastructure/media-permissions": "service:jellyfin", + "infrastructure/mesh": None, + "infrastructure/monitoring": None, + "infrastructure/network": None, + "infrastructure/ssh-access": None, + "infrastructure/topology": None, + "infrastructure/vps-hardening": "standalone-server:netbird-vps", +} + +def parse_page(path): + """Parse a wiki page into structured sections.""" + if not path.exists(): + return None + text = path.read_text() + lines = text.split('\n') + + # Title is first H1 + title = "" + for line in lines: + if line.startswith('# ') and not line.startswith('## '): + title = line[2:].strip() + break + + # Find sections by H2 headings + sections = {} + current_heading = "_preamble" + current_content = [] + + for line in lines: + if line.startswith('## ') and not line.startswith('### '): + if current_content: + sections[current_heading] = '\n'.join(current_content).strip() + current_heading = line[3:].strip().lower() + current_content = [] + else: + current_content.append(line) + if current_content: + sections[current_heading] = '\n'.join(current_content).strip() + + # Parse at-a-glance + at_glance = {} + ag_text = sections.get('at a glance', '') + for line in ag_text.split('\n'): + line = line.strip() + # Strip leading bullet + line = re.sub(r'^[-*]\s+', '', line) + # Match **Key:** value or **Key Word:** value + m = re.match(r'\*\*([^*]+?):?\*\*\s+(.+)', line) + if not m: + m = re.match(r'([A-Z][a-z]+(?:\s+[A-Z][a-z]+)*):\s+(.+)', line) + if m: + key = m.group(1).lower().strip().replace(' ', '_').replace('/', '_') + val = m.group(2).strip() + # Strip trailing parenthetical notes + val = re.sub(r'\s*\([^)]*\)$', '', val) + # Strip markdown formatting from value + val = re.sub(r'\*\*([^*]+)\*\*', r'\1', val) + val = re.sub(r'`([^`]+)`', r'\1', val) + # Simplify link text + val = re.sub(r'\[([^\]]+)\]\([^)]+\)', r'\1', val) + val = re.sub(r'↔', '', val).strip() + # Normalize keys + key_map = { + 'cores': 'cores', 'core': 'cores', + 'ram': 'ram', 'memory': 'ram', + 'mounts': 'mounts', 'mount': 'mounts', + 'host': 'host', 'ip': 'ip', + 'public_host': 'public_host', 'public_hostname': 'public_host', + 'lan_ip': 'lan_ip', + 'os': 'os', 'kind': 'kind', + 'runtime': 'runtime', 'role': 'role', + 'pve_id': 'pve_id', 'privilege': 'privileged', + 'resources': 'resources', 'gpu': 'gpu', + 'swap': 'swap', 'rootfs': 'rootfs', + 'version': 'version', 'hardware': 'hardware', + } + key = key_map.get(key, key) + at_glance[key] = val + + # Parse changelog + changelog = [] + cl_text = sections.get('changelog', '') + current_entry = None + for line in cl_text.split('\n'): + m = re.match(r'###\s+(\d{4}-\d{2}-\d{2})\s+[—–-]\s+(.+)', line) + if m: + if current_entry: + changelog.append(current_entry) + current_entry = {'date': m.group(1), 'title': m.group(2).strip(), 'body': ''} + elif current_entry is not None: + stripped = line.strip() + if stripped and not stripped.startswith('#'): + if current_entry['body']: + current_entry['body'] += ' ' + current_entry['body'] += stripped + if current_entry: + changelog.append(current_entry) + + # Tags from path + parts = path.relative_to(REPO).parts + tags = [] + if 'containers' in parts: + tags.append('container') + elif 'hosts' in parts: + tags.append('host') + elif 'vms' in parts: + tags.append('vm') + elif 'infrastructure' in parts: + tags.append('infrastructure') + + # Determine slug from relative path + rel = str(path.relative_to(REPO)) + if rel.startswith('knowledge/wiki/'): + slug_rel = rel[len('knowledge/wiki/'):] + elif rel.startswith('knowledge/sources/investigations/'): + slug_rel = rel[len('knowledge/sources/'):] + else: + slug_rel = rel + + slug = slug_rel.replace('.md', '') + + # Entity mapping + entity_slug = PATH_TO_ENTITY.get(slug, None) + + return { + 'slug': slug, + 'title': title, + 'content': text, + 'entity_slug': entity_slug, + 'tags': tags, + 'at_glance': at_glance, + 'changelog': changelog, + 'is_investigation': 'investigations' in rel, + } + + +def parse_investigation(path): + """Parse an investigation page.""" + if not path.exists(): + return None + text = path.read_text() + lines = text.split('\n') + + title = "" + for line in lines: + if line.startswith('# '): + title = line[2:].strip() + break + + # Extract date from title or filename + date = "" + status = "resolved" + duration = "" + + for line in lines[:30]: + m = re.search(r'(\d{4}-\d{2}-\d{2})', line) + if m: + date = m.group(1) + break + + for line in lines: + if '**Status:**' in line: + status = line.split('**Status:**')[-1].strip().lower() + if '**Duration:**' in line: + duration = line.split('**Duration:**')[-1].strip() + + # Extract entity references for about_slugs + about_slugs = [] + entity_patterns = [ + (r'\bcaddy\b', 'service:caddy'), + (r'\bauthentik\b', 'service:authentik'), + (r'\bdns\b', 'service:dns'), + (r'\bgitea\b', 'service:gitea'), + (r'\bjellyfin\b', 'service:jellyfin'), + (r'\bmatrix\b', 'service:matrix'), + (r'\bpaperless\b', 'service:paperless'), + (r'\bnextcloud\b', 'service:nextcloud'), + (r'\bartifacto\b', 'service:artifacto'), + (r'\barriman\b', 'lxc:arriman'), + (r'\btrmnl\b', 'service:trmnl'), + (r'\bmac-mini\b', 'workstation:mac-mini'), + (r'\bhubris\b', 'proxmox-host:hubris'), + (r'\bstrong\b', 'proxmox-host:strong'), + ] + for pattern, slug in entity_patterns: + if re.search(pattern, text, re.IGNORECASE): + about_slugs.append(slug) + + rel = str(path.relative_to(SOURCES)) + slug = rel.replace('.md', '') + + return { + 'slug': slug, + 'title': title, + 'date': date, + 'status': status, + 'duration': duration, + 'content': text, + 'about_slugs': about_slugs, + 'tags': ['investigation'], + } + + +def main(): + documents = [] + investigations = [] + runbooks = [] + + # Container pages + containers_dir = WIKI / "containers" + for f in sorted(containers_dir.glob("*.md")): + if 'index' in f.name or 'archive' in str(f): + continue + result = parse_page(f) + if result and result['title']: + documents.append(result) + print(f" document: {result['slug']} → {result['entity_slug']}") + + # Host pages + hosts_dir = WIKI / "hosts" + for f in sorted(hosts_dir.glob("*.md")): + if 'index' in f.name: + continue + result = parse_page(f) + if result and result['title']: + documents.append(result) + print(f" document: {result['slug']} → {result['entity_slug']}") + + # VM pages + vms_dir = WIKI / "vms" + for f in sorted(vms_dir.glob("*.md")): + if 'index' in f.name: + continue + result = parse_page(f) + if result and result['title']: + documents.append(result) + print(f" document: {result['slug']} → {result['entity_slug']}") + + # Infrastructure pages + infra_dir = WIKI / "infrastructure" + for f in sorted(infra_dir.glob("*.md")): + if 'index' in f.name: + continue + result = parse_page(f) + if result and result['title']: + documents.append(result) + print(f" document: {result['slug']} → {result['entity_slug']}") + + # Investigation pages + inv_dir = SOURCES / "investigations" + for f in sorted(inv_dir.glob("*.md")): + if 'index' in f.name or 'archive' in str(f): + continue + result = parse_investigation(f) + if result and result['title']: + investigations.append(result) + print(f" investigation: {result['slug']} → {result['about_slugs']}") + + # Archive investigations too + inv_archive = inv_dir / "archive" + if inv_archive.exists(): + for f in sorted(inv_archive.glob("*.md")): + result = parse_investigation(f) + if result and result['title']: + investigations.append(result) + print(f" investigation: {result['slug']} → {result['about_slugs']}") + + # Runbooks from .agents/skills/ + skills_dir = REPO / ".agents" / "skills" + for skill_dir in sorted(skills_dir.iterdir()): + if not skill_dir.is_dir(): + continue + skill_file = skill_dir / "SKILL.md" + if not skill_file.exists(): + continue + text = skill_file.read_text() + lines = text.split('\n') + title = "" + for line in lines: + if line.startswith('# '): + title = line[2:].strip() + break + + # Extract risk_class and entity_type from frontmatter + risk_class = "read_only" + entity_type = "service" + for line in lines[:30]: + m = re.match(r'\*\*risk_class:\*\*\s*(\w+)', line, re.IGNORECASE) + if m: + risk_class = m.group(1) + m = re.match(r'\*\*applies_to:\*\*\s*(\w[\w-]*)', line, re.IGNORECASE) + if m: + entity_type = m.group(1) + + name = skill_dir.name + runbooks.append({ + 'slug': name, + 'name': title or name, + 'risk_class': risk_class, + 'entity_type': entity_type, + 'procedure': {}, # SKILL.md is narrative, not structured yet + 'content': text, + 'tags': ['skill', 'runbook'], + }) + print(f" runbook: {name}") + + # Build seed YAML + seed = { + 'version': 1, + 'documents': [{ + 'slug': d['slug'], + 'title': d['title'], + 'content': d['content'], + 'entity_slug': d['entity_slug'], + 'tags': d['tags'], + 'at_glance': d['at_glance'], + 'changelog': d['changelog'], + } for d in documents], + 'investigations': [{ + 'slug': i['slug'], + 'title': i['title'], + 'date': i['date'], + 'status': i['status'], + 'duration': i['duration'], + 'content': i['content'], + 'about_slugs': i['about_slugs'], + 'tags': i['tags'], + } for i in investigations], + 'runbooks': [{ + 'slug': r['slug'], + 'name': r['name'], + 'risk_class': r['risk_class'], + 'entity_type': r['entity_type'], + 'procedure': r['procedure'], + 'content': r['content'], + 'tags': r['tags'], + } for r in runbooks], + } + + out_path = REPO / "seeds" / "knowledge.yaml" + out_path.write_text(yaml.dump(seed, allow_unicode=True, width=120, sort_keys=False)) + print(f"\nWrote {out_path}") + print(f" {len(documents)} documents") + print(f" {len(investigations)} investigations") + print(f" {len(runbooks)} runbooks") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/seeds/knowledge.yaml b/seeds/knowledge.yaml new file mode 100644 index 0000000..3aec741 --- /dev/null +++ b/seeds/knowledge.yaml @@ -0,0 +1,6708 @@ +version: 1 +documents: +- slug: containers/101-jellyfin + title: 101 — `jellyfin` + content: "# 101 — `jellyfin`\n\nMedia server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to\ + \ LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.\n\n## At a glance\n-\ + \ **Host:** **strong** (migrated from hubris 2026-07-05)\n- **IP:** `192.168.8.246`\n- **Privilege:** privileged (recreated\ + \ on strong as priv)\n- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs\n- **GPU:** `/dev/dri/renderD128`\ + \ + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config\n- **Mounts:** `/mnt/media_local`\ + \ ↔ `/mnt/library`\n- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)\ + \ → `:8096`\n- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)\n- **FFmpeg:** jellyfin-ffmpeg7 7.1.4\n\ + \n## Service / port map\n\n| Service | Listen | Notes |\n| -------- | ------ | ----- |\n| jellyfin | `:8096` | HTTP (caddy\ + \ terminates TLS) |\n\n## Hardware acceleration (VAAPI)\n\nGPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993`\ + \ and\n`dev1: /dev/dri/card0,gid=44` in\n`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID\ + \ 993) and\n`video` groups inside the container.\n\n| GPU | Model | Role |\n|-----|-------|------|\n| Radeon 680M | iGPU\ + \ (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |\n| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1\ + \ encoding |\n\nEncoding settings (`/etc/jellyfin/encoding.xml`):\n- `HardwareAccelerationType`: `vaapi`\n- `VaapiDevice`:\ + \ `/dev/dri/renderD128`\n- `EnableHardwareEncoding`: `true`\n- `AllowHevcEncoding`: `true`\n- `AllowAv1Encoding`: `true`\n\ + - `EnableTonemapping`: `true`\n- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1\n- `EnableThrottling`: `true`\n-\ + \ `EnableSegmentDeletion`: `true`\n\nTrickplay (`/etc/jellyfin/system.xml`):\n- `EnableHwAcceleration`: `true`\n- `EnableHwEncoding`:\ + \ `true`\n\n## Authentik SSO (OIDC)\n\nJellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)\n\ + v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin\nhandles auth directly via OIDC redirect\ + \ flow.\n\n### Architecture\n\n```\nUser → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096\n \ + \ ↓\n Login\ + \ page with \"Sign in with Authentik\" button\n ↓ (click)\n\ + \ /sso/OID/start/Authentik\n \ + \ ↓ (302 redirect)\n auth.hubris.network OIDC\n\ + \ ↓ (login)\n \ + \ /sso/OID/redirect/Authentik?code=...&state=...\n \ + \ ↓\n Jellyfin SSO plugin validates token → logged in\n```\n\n\ + ### Components\n\n1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`\n - Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`\n\ + \ - Provider name: `Authentik`\n - OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`\n - `SchemeOverride`:\ + \ `https` (required — without it, plugin generates\n `http://` redirect URIs that Authentik rejects)\n - `EnableAuthorization`:\ + \ `false` (prevents plugin from overwriting admin\n permissions on each SSO login — see\n [jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))\n\ + \ - `OidScopes`: `[\"email\"]` (openid+profile added by default by the plugin;\n must be non-null or `OidChallenge()`\ + \ throws `ArgumentNullException`)\n\n2. **Authentik OIDC provider** — `Provider for Jellyfin` (PK 6)\n - Client ID:\ + \ `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`\n - Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`\n\ + \ - Application slug: `jellyfin`\n\n3. **SSO button injection** — `/usr/share/jellyfin/web/sso-inject.js`\n - Injected\ + \ via `