4 Commits

Author SHA1 Message Date
b87735a111 chore: graph view, dns-zone gap, fleet deploy/cleanup tooling
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.

dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.

Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.

Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
2026-07-29 13:37:27 +02:00
1540f74342 feat(audit): read-only knowledge-graph drift report + skill
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.

Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
2026-07-29 13:37:16 +02:00
c7729b2ef6 fix(scheduler): stop monitoring deprecated/destroyed targets
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.

coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
2026-07-29 13:37:08 +02:00
b8b4aa2aee feat(remote): route LXC/VM checks through the Proxmox host, not direct SSH
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.

Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.

cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
2026-07-29 13:37:00 +02:00
26 changed files with 1675 additions and 148 deletions

View File

@@ -0,0 +1,83 @@
---
name: knowledge-graph-audit
risk_class: read_only
inputs: []
verification: "audit_knowledge_graph returns a report with summary.total_findings"
docs_update_checklist: []
---
# Knowledge-graph audit
Goal: validate that the knowledge graph (entities, relationships, checks) and
the monitoring built on it reflect live reality — without mutating anything.
Read-only. Run this before trusting health, blast-radius, or coverage answers,
and whenever something feels off (a healthy host reports `down`, a retired
service still alarms, the graph looks thin).
## 1. Run the drift report
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
ranked list of findings, each with `{category, severity, count, entities,
evidence, suggested_runbook}`, plus a `summary` with totals by category.
The DB-side categories:
- **orphan_checks** — check entities with truncated/random slugs left by the
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
(the scheduler already skips these, but the rows should be retired).
- **down_checks** — enabled probes reporting `down`. Remediation:
`service-health-check` (then check whether the failure is real or a
probe-config/routing problem — see step 3).
- **unknown_checks** — probes that ran but reported `unknown` (usually a
misconfigured or not-yet-deployed probe script).
- **unmonitored** — active entities whose type declares monitoring but have no
enabled `check_def`.
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
## 2. Triage
`severity: critical` (down_checks) first. For each finding, read `evidence` and
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
before acting — the report is a pointer, not a verdict.
## 3. Common probe-failure causes
A `down_checks` finding that is NOT a real outage is usually one of:
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
exec` automatically when both are present.
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
## 4. What this audit does NOT cover (follow-ups)
Live-infrastructure discovery is out of scope for the DB report and must be done
manually until that machinery lands:
- **Ghost vs missing entities** — cross-check `pct list` / `qm list` (on
`host:hubris`, `host:strong`) and `docker ps` against `list_entities`. A guest
with no entity, or an entity with no guest, is drift.
- **Misplaced parent** — compare each guest's actual Proxmox host against its
`hosts` edge (migrations leave these stale).
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
- **Unmodeled certs** — Caddy-managed TLS certs with no `certificate` entity.
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
runtime-created entities not in version control.
## 5. Acting on findings
This skill is read-only — make no changes here. Route each confirmed finding to
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
to confirm the finding cleared.
Docs-update checklist: none — the audit reads state; it changes nothing. If a
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
in that remediation session.

View File

@@ -1 +1 @@
0.13.0
0.14.0

View File

@@ -2,12 +2,27 @@
# cpu_check.sh — CPU usage % and thermal temperature.
set -euo pipefail
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
if [ -z "$USAGE" ]; then
os=$(uname -s)
if [ "$os" = "Darwin" ]; then
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
# Usage is 100 minus the idle figure that precedes the literal `idle`.
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
/^CPU usage/ {
for (i = 1; i <= NF; i++) {
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
}
}' || true)
else
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
if [ -z "$USAGE" ]; then
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
fi
fi
[ -z "$USAGE" ] && USAGE=0
TEMP=""
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)

125
internal/audit/audit.go Normal file
View File

@@ -0,0 +1,125 @@
// Package audit produces read-only drift reports over the knowledge graph and
// monitoring state. It is the shared engine behind the
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
//
// It surfaces the structural gaps an operator otherwise discovers only by
// accident: orphan check entities, checks targeting retired entities, probes
// stuck down/unknown, unmonitored declared types, and live edges pointing at
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
// follow-up that needs host-hop execution; these categories are pure DB
// queries, so the report is cheap, safe to run unattended, and testable.
package audit
import (
"context"
"github.com/dtoro/oikos/internal/db"
)
// Finding is one drift item the operator should look at.
type Finding struct {
Category string `json:"category"`
Severity string `json:"severity"` // info | warning | critical
Count int `json:"count"`
Entities []string `json:"entities"`
Evidence string `json:"evidence"`
SuggestedRunbook string `json:"suggested_runbook"`
}
// Summary tallies findings by category.
type Summary struct {
TotalFindings int `json:"total_findings"`
ByCategory map[string]int `json:"by_category"`
}
// Report runs every drift check and returns the findings plus a summary.
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
specs := []struct {
finding Finding
query string
}{
{
Finding{Category: "orphan_checks", Severity: "warning",
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
`SELECT e.slug FROM entities e
WHERE e.type = 'check'
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
},
{
Finding{Category: "dead_checks", Severity: "warning",
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
`SELECT e.slug FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
},
{
Finding{Category: "down_checks", Severity: "critical",
Evidence: "enabled checks reporting health=down",
SuggestedRunbook: "service-health-check"},
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
WHERE cd.enabled AND cd.last_health = 'down'`,
},
{
Finding{Category: "unknown_checks", Severity: "warning",
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
SuggestedRunbook: "knowledge-graph-audit"},
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
WHERE cd.enabled AND cd.last_health = 'unknown'`,
},
{
Finding{Category: "unmonitored", Severity: "warning",
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
SuggestedRunbook: "knowledge-graph-audit"},
`SELECT DISTINCT e.slug FROM signals sg
JOIN entities e ON e.id = sg.target_entity_id
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
},
{
Finding{Category: "dangling_edges", Severity: "warning",
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
SuggestedRunbook: "lifecycle-destroy-node"},
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
JOIN entities src ON src.id = r.source_id
JOIN entities tgt ON tgt.id = r.target_id
WHERE r.valid_to IS NULL
AND src.state NOT IN ('destroyed','deprecated')
AND tgt.state IN ('destroyed','deprecated')`,
},
}
findings := make([]Finding, 0, len(specs))
summary := Summary{ByCategory: map[string]int{}}
for _, sp := range specs {
f := runFinding(ctx, pool, sp.finding, sp.query)
findings = append(findings, f)
summary.TotalFindings += f.Count
summary.ByCategory[f.Category] = f.Count
}
return findings, summary
}
const entityCap = 50
// runFinding runs a single-column slug query and folds the rows into a Finding.
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
rows, err := pool.Query(ctx, query)
if err != nil {
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
return f
}
defer rows.Close()
for rows.Next() {
var slug string
if err := rows.Scan(&slug); err != nil {
continue
}
f.Count++
if len(f.Entities) < entityCap {
f.Entities = append(f.Entities, slug)
}
}
return f
}

View File

@@ -0,0 +1,69 @@
package audit
import (
"context"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// Integration tests against a real Postgres, guarded by
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
func newAuditPool(t *testing.T) *db.Pool {
t.Helper()
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
if base == "" {
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
}
return createTestDB(t, base)
}
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
pool := newAuditPool(t)
ctx := context.Background()
// An orphan check entity (truncated random slug, the legacy bug shape).
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
// An active entity + a check_def on it stuck down.
target := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
checkE := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
// A deprecated entity still carrying an enabled check (dead_checks).
dep := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
depCheck := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
findings, summary := Report(ctx, pool)
byCat := map[string]int{}
for _, f := range findings {
byCat[f.Category] = f.Count
}
if byCat["orphan_checks"] < 1 {
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
}
if byCat["down_checks"] < 1 {
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
}
if byCat["dead_checks"] < 1 {
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
}
if summary.TotalFindings < 3 {
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
}
}

View File

@@ -0,0 +1,67 @@
package audit
import (
"context"
"fmt"
"math/rand"
"os"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/jackc/pgx/v5"
)
// createTestDB provisions a throwaway migrated database, same convention as
// internal/scheduler/coverage_test.go.
func createTestDB(t *testing.T, baseURL string) *db.Pool {
t.Helper()
ctx := context.Background()
admin, err := pgx.Connect(ctx, baseURL)
if err != nil {
t.Fatalf("connect admin: %v", err)
}
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
admin.Close(ctx)
t.Fatalf("create test db: %v", err)
}
admin.Close(ctx)
at := strings.LastIndex(baseURL, "/")
testURL := baseURL[:at+1] + dbName
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
testURL += baseURL[at+q:]
}
pool, err := db.New(ctx, testURL)
if err != nil {
t.Fatalf("connect test db: %v", err)
}
if err := pool.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
t.Cleanup(func() {
pool.Close()
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
admin.Close(ctx)
}
})
return pool
}
func getenvOrDefault(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
t.Helper()
if _, err := pool.Exec(ctx, q, args...); err != nil {
t.Fatalf("exec %s: %v", q, err)
}
}

View File

@@ -380,6 +380,12 @@ func resolveHost(attrs map[string]any) string {
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
return ip
}
// public_ipv4 before mesh: the scheduler container has no mesh interface,
// so a standalone-server reachable only by mesh IP (netbird-vps) is
// unprobeable even though a public IPv4 is available.
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
return ip
}
if mesh, ok := attrs["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if ip, ok := nb["ip"].(string); ok && ip != "" {
@@ -409,6 +415,13 @@ func resolveSSHUser(attrs map[string]any) string {
return u
}
}
// Workstations carry their login as a top-level `user` attribute
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
// no explicit ssh.user was set, so a host that genuinely wants root still
// gets root.
if u, ok := attrs["user"].(string); ok && u != "" {
return u
}
return "root"
}

View File

@@ -63,7 +63,9 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
e.slug AS entity_slug
FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
LEFT JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled = true
AND (tgt.id IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
AND (cd.last_run_at IS NULL
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));

View File

@@ -696,7 +696,9 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
e.slug AS entity_slug
FROM check_defs cd
JOIN entities e ON e.id = cd.entity_id
LEFT JOIN entities tgt ON tgt.id = cd.target_id
WHERE cd.enabled = true
AND (tgt.id IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
AND (cd.last_run_at IS NULL
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
`

20
internal/httpapi/audit.go Normal file
View File

@@ -0,0 +1,20 @@
package httpapi
import (
"net/http"
"github.com/dtoro/oikos/internal/audit"
)
// serveAuditDrift returns a read-only DB-side drift report: orphan check
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
// declared types, and dangling edges. Companion to the knowledge-graph-audit
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
findings, summary := audit.Report(req.Context(), s.pool)
writeJSON(w, map[string]any{
"findings": findings,
"summary": summary,
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
})
}

View File

@@ -24,7 +24,12 @@ import (
const (
defaultLimit = 50
maxLimit = 200
graphNodeCap = 500
// graphNodeCap bounds the whole-graph view. The cognition transactional
// types (execution, task) are audit records, not topology, and previously
// crowded out every host/lxc/service; the default whole-graph view below
// excludes them so the cap is spent on the actual fleet graph. Operators
// still reach executions/tasks via list_entities.
graphNodeCap = 2000
)
// actorInfo returns the caller's (type, label) from the request context,
@@ -307,14 +312,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
// alphabetically. Without this the cap fills with exec:* rows and
// drops every host/lxc/service/vm — and every edge those entities
// connect — because edges require both endpoints in the node set.
// Exclude the cognition transactional types (execution/task): they
// are audit records rather than topology, and at ~380 rows they
// consumed most of the old 500-node cap.
nodes, err = s.queryEntities(ctx, `
SELECT `+entityCols+`
FROM entities e
LEFT JOIN entity_status st ON st.entity_id = e.id
WHERE e.id IN (
WHERE e.type NOT IN ('execution','task')
AND e.id IN (
SELECT e2.id FROM entities e2
LEFT JOIN relationships r ON r.valid_to IS NULL
AND (r.source_id = e2.id OR r.target_id = e2.id)
WHERE e2.type NOT IN ('execution','task')
GROUP BY e2.id
ORDER BY count(r.type) DESC, e2.slug
LIMIT $1

View File

@@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
// Drift audit: read-only DB-side report of orphan checks, checks on
// retired targets, stuck down/unknown probes, unmonitored declared types,
// and dangling edges. Companion to the knowledge-graph-audit skill.
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
// unlike ListExecutions which sorts by target for pagination) and the
// per-session "what did this session do" digest.

View File

@@ -5,7 +5,6 @@ package mcp
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"fmt"
"html"
@@ -25,6 +24,7 @@ import (
"github.com/dtoro/oikos/internal/execlog"
"github.com/dtoro/oikos/internal/observability"
"github.com/dtoro/oikos/internal/policy"
"github.com/dtoro/oikos/internal/remote"
"github.com/google/jsonschema-go/jsonschema"
"github.com/google/uuid"
"github.com/modelcontextprotocol/go-sdk/mcp"
@@ -461,31 +461,12 @@ func sshExecStream(ctx context.Context, host, user, command string, sink execlog
}
}
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
var attrs string
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
if err != nil {
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
}
var m map[string]interface{}
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
return "", "", fmt.Errorf("parse attributes: %w", err)
}
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
return ip, sshUser, nil
}
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
for _, proto := range []string{"netbird", "tailscale"} {
if p, ok := mesh[proto].(map[string]interface{}); ok {
if ip, ok := p["ip"].(string); ok && ip != "" {
return ip, sshUser, nil
}
}
}
}
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
// wrapper over the shared resolver (internal/remote), kept so slug-based
// callers keep working; the shared resolver also prefers public_ipv4 over
// mesh and honors a per-entity ssh.user.
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
}
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
@@ -564,109 +545,28 @@ func isPrivateHost(host string) bool {
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
// `qm guest exec <pve_id> -- ...` for a VM.
//
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
// "strong", not "host:strong") — see pct_create's entity registration. The
// pre-existing pct_exec handler queried resolveHost with that bare value
// directly, which can never match a "host:*" slug and always fails; this
// prefixes it correctly.
//
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
// which broke on nested shell quoting and forced manual escaping workarounds
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
// `host` attribute is optional: if absent, fall back to looking up the
// `hosts` relationship on the VM entity, then to hubris (the documented
// default Proxmox host) — same fallback chain as LXCs.
// Delegates to the shared resolver (internal/remote), the single path used by
// both the MCP `run` tool and the scheduler's checks. The historical notes
// (host attr without prefix, vm host-resolution chain, nested-quoting
// handling via base64) all still hold — they now live in remote.guestWrap.
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
if strings.HasPrefix(targetSlug, "host:") {
host, user, err = resolveHost(ctx, pool, targetSlug)
return host, user, func(cmd string) string { return cmd }, err
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
if err != nil {
return "", "", nil, err
}
if strings.HasPrefix(targetSlug, "lxc:") {
var pveID, hostAttr string
// COALESCE the host column: many older LXC entities (seeded from
// inventory, not provisioned by pct_create) have pve_id but no host
// attribute at all. Scanning a SQL NULL into a plain string errors
// the whole row, wrongly reporting "missing pve_id" even when it was
// present — COALESCE avoids the NULL, "" is handled below.
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
}
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
host, user, err = resolveHost(ctx, pool, hostSlug)
id := pveID
return host, user, func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
}, err
}
if strings.HasPrefix(targetSlug, "vm:") {
// VMs: same host-resolution chain as LXCs (attributes.host →
// `hosts` relationship → hubris default), but reached via
// `qm guest exec` instead of `pct exec`. Requires the QEMU
// guest agent running inside the VM (the standard Proxmox
// setup; ZimaOS/HAOS in this fleet already have it).
var pveID, hostAttr string
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
}
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
host, user, err = resolveHost(ctx, pool, hostSlug)
id := pveID
return host, user, func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
// default; pipe through `jq -r .out` if available, else cat.
// The base64 round-trip mirrors the LXC path so nested quoting
// (the original VM-target pain point — session 55927f0a) is
// handled identically to LXC dispatch.
return fmt.Sprintf(
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
id, b64, id, b64)
}, err
}
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
return et.Host, et.User, et.Wrap, nil
}
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
// LXC/VM target. Resolution order:
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
// "host:" prefix in inventory.yaml and pct_create).
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
// looked up in the relationships table — the canonical graph source.
// 3. "hubris" as a documented default Proxmox host fallback.
//
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
// This slug-based wrapper looks up the entity id so slug callers keep working;
// the shared resolver takes an id directly.
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
hostSlug := strings.TrimSpace(hostAttr)
if hostSlug == "" {
// Fall back to the `hosts` relationship — the graph edge from
// the Proxmox host to this LXC/VM. This is the canonical source
// for "who owns this VM" in inventory.yaml; the `host` attribute
// is a denormalized shortcut that not every entity has.
var relHostSlug string
// hosts relationship: source=host, target=lxc/vm. Look up the
// source slug given the target.
if err := pool.QueryRow(ctx, `
SELECT e.slug FROM relationships r
JOIN entities e ON e.id = r.source_id
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
AND r.type = 'hosts' AND r.valid_to IS NULL
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
hostSlug = relHostSlug
var id uuid.UUID
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
id = uuid.Nil
}
}
if hostSlug == "" {
hostSlug = "hubris" // documented default Proxmox host when unset
}
if !strings.HasPrefix(hostSlug, "host:") {
hostSlug = "host:" + hostSlug
}
return hostSlug
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
}
// classifyAndGate is the shared classify→execute-or-queue path for every

View File

@@ -6,6 +6,7 @@ import (
"fmt"
"strings"
"github.com/dtoro/oikos/internal/audit"
"github.com/dtoro/oikos/internal/db"
"github.com/dtoro/oikos/internal/policy"
"github.com/google/uuid"
@@ -782,6 +783,14 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
`), "fleet_snapshot"), nil
}},
{tool: &mcp.Tool{Name: "audit_knowledge_graph", Description: "Read-only drift report over the knowledge graph and monitoring: orphan check entities, checks targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored declared entity types, and live edges pointing at destroyed targets. Returns ranked findings with a suggested remediation runbook each. Use this to validate the graph is complete and consistent before trusting health/blast-radius answers. Does NOT mutate anything.",
InputSchema: objSchema(),
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
findings, summary := audit.Report(ctx, pool)
b, _ := json.Marshal(map[string]any{"findings": findings, "summary": summary})
return textResult(string(b)), nil
}},
{tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key",
InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}),
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {

257
internal/remote/remote.go Normal file
View File

@@ -0,0 +1,257 @@
// Package remote resolves how to execute a command on a target entity and
// turns a plain shell command into whatever must be sent over the SSH
// connection that reaches it.
//
// The canonical access model: a host or workstation is reached by direct SSH
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
// credential per host (the host's root key), no per-guest keys, sshd, or
// lan_ip required for execution. Network probes (http/ping) still hit a
// guest's lan_ip directly; only command execution host-hops.
//
// This is the single resolver shared by the scheduler's check execution and
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
// directly (broken for headless/keyless/mesh-only guests), while MCP
// host-hopped (working). Keeping one path keeps them in lockstep.
package remote
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"strings"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// DefaultUser is the SSH user when an entity declares no ssh.user. The
// Proxmox hosts and their guests are all administered as root.
const DefaultUser = "root"
// ExecTarget is a resolved execution endpoint: the SSH address and user to
// connect to, plus Wrap, which rewrites a plain command for transport.
type ExecTarget struct {
Host string
User string
// Wrap turns a plain shell command into the form that must be sent over
// the SSH connection to this target: the identity function for a host,
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
// nested quoting identical across both guest kinds.
Wrap func(cmd string) string
}
// IsGuest reports whether an entity type is reached via pct/qm exec through a
// Proxmox host rather than by direct SSH. docker-container is reached via its
// host's docker socket, not pct, so it is not a guest here.
func IsGuest(entityType string) bool {
return entityType == "lxc" || entityType == "vm"
}
// ResolveHost resolves a `host:<slug>` to its reachable network address and
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
// container has no mesh interface — a standalone-server with only a mesh IP
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
//
// fallbackUser is used when the entity declares no ssh.user; callers pass
// their configured default (the scheduler uses "root", the MCP run tool uses
// its configured OIKOS_SSH_USER).
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
var raw string
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
}
var m map[string]any
if err = json.Unmarshal([]byte(raw), &m); err != nil {
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
}
if v, ok := m["lan_ip"].(string); ok && v != "" {
addr = v
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
addr = v
} else if mesh, ok := m["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if v, ok := nb["ip"].(string); ok && v != "" {
addr = v
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
addr = v
}
}
}
if addr == "" {
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
}
user = fallbackUser
if ssh, ok := m["ssh"].(map[string]any); ok {
if u, ok := ssh["user"].(string); ok && u != "" {
user = u
}
}
return addr, user, nil
}
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
// stored without the "host:" prefix), the `hosts` relationship on the guest
// (the canonical graph edge), then "hubris" as the documented default.
//
// entityID is the guest's entity id; the relationship lookup uses it
// directly rather than a slug subquery.
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
hostSlug := strings.TrimSpace(hostAttr)
if hostSlug == "" {
var relHostSlug string
if err := pool.QueryRow(ctx, `
SELECT e.slug FROM relationships r
JOIN entities e ON e.id = r.source_id
WHERE r.target_id = $1
AND r.type = 'hosts' AND r.valid_to IS NULL
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
hostSlug = relHostSlug
}
}
if hostSlug == "" {
hostSlug = "hubris" // documented default Proxmox host when unset
}
if !strings.HasPrefix(hostSlug, "host:") {
hostSlug = "host:" + hostSlug
}
return hostSlug
}
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
switch {
case strings.HasPrefix(targetSlug, "host:"):
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
var (
id uuid.UUID
pveID string
typ string
hostAttr string
)
if err := pool.QueryRow(ctx,
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
}
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
}
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
}
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
// target's id and type — the data the scheduler has at check-execution time
// (check_defs carry target_id + target_type, not a slug). Guests route via
// pct/qm exec; everything else (hosts, workstations, services resolved to
// their hosting machine) is reached by direct SSH to the entity's own address.
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
if IsGuest(targetType) {
var (
pveID string
hostAttr string
)
if err := pool.QueryRow(ctx,
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
targetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", targetID)
}
hostSlug := ResolveProxmoxHostSlug(ctx, pool, targetID, hostAttr)
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(targetType, pveID)}, nil
}
// Host-like target: reach it directly at its own address. Services and
// other non-host entities that reach here should already have had their
// host address baked into check config at seed time; this path covers
// host/workstation targets whose address is resolved live.
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
if err != nil {
return ExecTarget{}, err
}
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
}
// resolveHostByID is ResolveHost keyed by entity id.
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
var raw string
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
return "", "", fmt.Errorf("entity %s not found", id)
}
var m map[string]any
if err = json.Unmarshal([]byte(raw), &m); err != nil {
return "", "", fmt.Errorf("parse attributes: %w", err)
}
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
if v, ok := m[key].(string); ok && v != "" {
addr = v
break
}
}
if addr == "" {
if mesh, ok := m["mesh"].(map[string]any); ok {
if nb, ok := mesh["netbird"].(map[string]any); ok {
if v, ok := nb["ip"].(string); ok && v != "" {
addr = v
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
addr = v
}
}
}
}
if addr == "" {
return "", "", fmt.Errorf("no IP found for entity %s", id)
}
user = fallbackUser
if ssh, ok := m["ssh"].(map[string]any); ok {
if u, ok := ssh["user"].(string); ok && u != "" {
user = u
}
}
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
// Workstations carry their login as a top-level `user` attribute
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
// explicit ssh.user was set, so a host that genuinely wants root still
// gets root.
user = u
}
return addr, user, nil
}
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
func guestWrap(entityType, pveID string) func(cmd string) string {
if entityType == "vm" {
return func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
// falling back to the raw form. Mirrors the LXC base64 round-trip.
return fmt.Sprintf(
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
pveID, b64, pveID, b64)
}
}
return func(cmd string) string {
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
}
}

View File

@@ -0,0 +1,172 @@
package remote
import (
"context"
"encoding/base64"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/google/uuid"
)
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
// same convention as internal/scheduler/coverage_test.go.
func TestIsGuest(t *testing.T) {
cases := map[string]bool{
"lxc": true, "vm": true,
"proxmox-host": false, "workstation": false,
"service": false, "docker-container": false,
}
for typ, want := range cases {
if got := IsGuest(typ); got != want {
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
}
}
}
func TestGuestWrapLXC(t *testing.T) {
w := guestWrap("lxc", "132")
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
t.Fatalf("lxc wrap must use pct exec: %q", out)
}
if strings.Contains(out, "qm guest exec") {
t.Fatalf("lxc wrap must not use qm: %q", out)
}
// The base64 payload must round-trip to the original command.
i := strings.Index(out, "echo ")
j := strings.LastIndex(out, " | base64 -d | bash")
if i < 0 || j < 0 || j <= i {
t.Fatalf("cannot locate base64 payload in %q", out)
}
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
if err != nil {
t.Fatalf("decode payload: %v", err)
}
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
t.Fatalf("round-trip mismatch: %q", string(dec))
}
}
func TestGuestWrapVM(t *testing.T) {
w := guestWrap("vm", "108")
out := w("uname -a")
if !strings.Contains(out, "qm guest exec 108") {
t.Fatalf("vm wrap must use qm guest exec: %q", out)
}
}
func TestResolveExecTargetUnsupported(t *testing.T) {
// No DB needed: an unsupported slug prefix errors before any query.
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
t.Fatal("expected error for unsupported target prefix")
}
}
// --- integration tests (require a real Postgres) ---
func newRemotePool(t *testing.T) *db.Pool {
t.Helper()
base := testDatabaseURL(t)
return createTestDB(t, base)
}
func testDatabaseURL(t *testing.T) string {
t.Helper()
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
if u == "" {
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
}
return u
}
func TestResolveHostPrefersLAN(t *testing.T) {
pool := newRemotePool(t)
ctx := context.Background()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
uuid.New())
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
if err != nil {
t.Fatalf("ResolveHost: %v", err)
}
if addr != "10.0.0.1" {
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
}
if user != "root" {
t.Errorf("user = %q, want root", user)
}
}
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
// public_ipv4 — the scheduler container has no mesh interface.
pool := newRemotePool(t)
ctx := context.Background()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
uuid.New())
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
if err != nil {
t.Fatalf("ResolveHost: %v", err)
}
if addr != "82.165.190.79" {
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
}
}
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
// HOST's address (the host-hop target), wrapped as `pct exec`.
pool := newRemotePool(t)
ctx := context.Background()
hostID := uuid.New()
guestID := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
if err != nil {
t.Fatalf("ResolveExecTargetForCheck: %v", err)
}
if et.Host != "192.168.8.77" {
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
}
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
if !strings.Contains(out, "pct exec 132") {
t.Errorf("guest wrap must use pct exec 132, got %q", out)
}
}
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
// A host-like target resolves to its own address with identity wrap.
pool := newRemotePool(t)
ctx := context.Background()
hid := uuid.New()
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
if err != nil {
t.Fatalf("ResolveExecTargetForCheck: %v", err)
}
if et.Host != "192.168.178.182" {
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
}
// Workstation's top-level `user` must be honored (the mac-mini fix).
if et.User != "dtoro" {
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
}
if cmd := et.Wrap("uptime"); cmd != "uptime" {
t.Errorf("host wrap must be identity, got %q", cmd)
}
}

View File

@@ -0,0 +1,68 @@
package remote
import (
"context"
"fmt"
"math/rand"
"os"
"strings"
"testing"
"github.com/dtoro/oikos/internal/db"
"github.com/jackc/pgx/v5"
)
// createTestDB provisions a throwaway migrated database off baseURL, the same
// convention as internal/scheduler/coverage_test.go. The base URL must point
// at a Postgres superuser-capable connection.
func createTestDB(t *testing.T, baseURL string) *db.Pool {
t.Helper()
ctx := context.Background()
admin, err := pgx.Connect(ctx, baseURL)
if err != nil {
t.Fatalf("connect admin: %v", err)
}
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
admin.Close(ctx)
t.Fatalf("create test db: %v", err)
}
admin.Close(ctx)
at := strings.LastIndex(baseURL, "/")
testURL := baseURL[:at+1] + dbName
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
testURL += baseURL[at+q:]
}
pool, err := db.New(ctx, testURL)
if err != nil {
t.Fatalf("connect test db: %v", err)
}
if err := pool.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
t.Cleanup(func() {
pool.Close()
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
admin.Close(ctx)
}
})
return pool
}
func getenvOrDefault(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
t.Helper()
if _, err := pool.Exec(ctx, q, args...); err != nil {
t.Fatalf("exec %s: %v", q, err)
}
}

View File

@@ -85,7 +85,16 @@ func coverageSweep(ctx context.Context, pool *db.Pool) {
case !mon.Declared:
undeclared++
case mon.None():
// Explicitly unmonitorable. Nothing to say.
// Explicitly unmonitorable. Nothing to raise — but a type that
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
// open `unmonitored` signals from before the change. They are no
// longer a gap, so close them; otherwise they linger forever,
// because resolveCoverageSignal only runs from the hasCheck path
// and a None() entity never gains a check.
if resolveCoverageSignal(ctx, pool, e.id) {
resolved++
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
}
case e.hasCheck:
if resolveCoverageSignal(ctx, pool, e.id) {
resolved++

View File

@@ -23,6 +23,7 @@ import (
"github.com/dtoro/oikos/internal/db"
"github.com/dtoro/oikos/internal/db/sqlcgen"
"github.com/dtoro/oikos/internal/observability"
"github.com/dtoro/oikos/internal/remote"
"github.com/google/uuid"
"golang.org/x/sync/errgroup"
"golang.org/x/sys/unix"
@@ -111,7 +112,7 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
q := sqlcgen.New(pool)
start := time.Now()
result := executeCheck(ctx, cd)
result := executeCheck(ctx, pool, cd)
// Stamp the run before processing the result: due-ness must advance even
// when a check fails, or a permanently failing check would be re-run on
@@ -277,8 +278,10 @@ type checkResult struct {
err error
}
// executeCheck dispatches to the appropriate checker by kind.
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
// executeCheck dispatches to the appropriate checker by kind. pool is needed
// by the ssh-script path, which resolves the target's execution endpoint
// (guests route through their Proxmox host; see internal/remote).
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
switch cd.Kind {
case "http":
return checkHTTP(ctx, cd)
@@ -291,7 +294,7 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
case "ping":
return checkPing(ctx, cd)
case "ssh-script":
return checkSSHScript(ctx, cd)
return checkSSHScript(ctx, pool, cd)
case "backup-freshness":
return checkBackupFreshness(ctx, cd)
default:
@@ -706,8 +709,16 @@ func parsePingLatency(output []byte) float64 {
return val
}
// checkSSHScript executes an allowlisted script on a remote host via SSH.
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
// checkSSHScript executes an allowlisted script on a remote target via SSH.
//
// Routing follows the canonical access model (internal/remote): an LXC or VM
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
// key of its own. Hosts and workstations are reached by direct SSH, resolved
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
// a re-seed. Services and other entities fall back to the host address baked
// into check config at seed time (their hosting container's address).
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
cfg := struct {
Host string `json:"host"`
Port int `json:"port"`
@@ -725,12 +736,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
if cfg.Host == "" || cfg.Script == "" {
return checkResult{health: "healthy"}
}
if cfg.Port == 0 {
cfg.Port = 22
}
if cfg.User == "" {
cfg.User = sshUser
}
if !allowlistedScript(cfg.Script) {
return checkResult{
@@ -753,13 +758,45 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
// the remote command. The script name itself is allowlisted above.
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
}
port := strconv.Itoa(cfg.Port)
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
// Resolve the execution endpoint. Guests host-hop; host/workstation types
// resolve their address + user live; everything else uses the baked config.
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
wrap := func(cmd string) string { return cmd }
targetType := ""
if cd.TargetType != nil {
targetType = *cd.TargetType
}
if cd.TargetID != nil {
switch {
case remote.IsGuest(targetType):
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
if err != nil {
return checkResult{
health: "down", signalKind: "ssh-script",
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
}
}
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
case isMachine(targetType):
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
if err == nil {
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
} else {
// Log the resolution failure so an opaque ssh "down" doesn't
// hide that the real cause was host/user resolution (e.g. a
// missing attribute), then fall back to the baked config below.
slog.Warn("scheduler: machine target resolution failed, using baked config",
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
}
}
}
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
if err != nil {
return checkResult{
health: "down", signalKind: "ssh-script",
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
err: err,
}
}
@@ -797,6 +834,32 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
}
}
// isMachine reports whether a target type is a physical/virtual machine that
// should be reached by direct SSH at its own resolved address (rather than the
// baked hosting-container address a service uses). These are the machine
// subtypes in the ontology.
func isMachine(entityType string) bool {
switch entityType {
case "proxmox-host", "standalone-server", "workstation", "appliance":
return true
}
return false
}
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
func oru(v, def int) int {
if v != 0 {
return v
}
return def
}
func orStr(v, def string) string {
if v != "" {
return v
}
return def
}
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
func allowlistedScript(name string) bool {

View File

@@ -0,0 +1,410 @@
# Plan: Make health reflect reality + complete the knowledge graph
Status: ready for implementation · Created 2026-07-29
## Context
`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
Investigation showed the problem is systemic, not local: **49 enabled checks report
`down`**, almost all `ssh-script`, because the resource/updates probes assume
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target**
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
topology view).
The DB is the source of truth; live state was verified via the REST API
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
## Decisions (confirmed with operator)
1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
`qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
use direct SSH with the correct per-target user.
2. **Canonical host-hop access**`pct exec`/`qm guest exec` through the proxmox host is
the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
`health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
becomes monitorable with zero manual steps (Track E).
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
permanent false alarms from retired things.
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
re-export seeds, seed skills, raise graph cap.
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
it against the DB graph, producing a ranked drift report; the operator acts on findings
via existing lifecycle runbooks. No auto-fix. (Track F.)
## Findings (evidence)
### A. Health-check reality gaps (49 checks `down`)
**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
`-i /etc/oikos/ssh_key`, not a default-key test):**
The MCP `run` tool already reaches every guest correctly via
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
(`attributes.host``hosts` edge → hubris default), SSH there, run
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
| Class | Targets (verified) | Root cause |
|---|---|---|
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
(it's a homelab client with root key + scripts). So the host hop is the reliable path.
**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
script deployment, not topology.
Health aggregation itself is correct: `WorstHealthForTarget`
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
ssh-script drags an otherwise-healthy entity to `down`.
### B. Dead/stale data
- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
(e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
routes to it and alarms permanently.
### C. Knowledge-graph gaps
- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
checker all exist, but **0** certificate entities. Cert expiry is invisible.
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
checker exists → every zone is an `unmonitored` signal.
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
---
## Work breakdown
### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
hop already has working root SSH. Hosts/workstations keep direct SSH.
**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
(`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
the scheduler SSHes guests directly (broken), MCP host-hops (works).
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
keep direct SSH (they ARE the host).
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
approval. Verify one LXC end-to-end (rclone) before fanning out.
**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
scripts must live in the guest.
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
`checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
- Backfill once now (all guests + mac-mini). See Track E for the automated version.
**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
`attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
**A4. macOS-aware check scripts.**
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
(`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
(`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
with `security_updates=0` or read `softwareupdate --list`).
- Each must still emit `{"health":..,"metrics":{..}}` JSON
(`internal/scheduler/scheduler.go:767`).
**A5. Reachability for external/mesh-only hosts.**
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
key or proxy via hubris. Confirm before assuming direct SSH works.
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
Open Questions.)
- `lxc:rclone` no longer a special case — handled by A1's pct routing.
**A6. ICMP-blocked VMs.**
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
(`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
Lower priority — confirm haos blocks ICMP before building.
### Track B — Lifecycle monitoring gate
**B1. Skip monitoring for deprecated/destroyed targets.**
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
`state` ∈ {`deprecated`,`destroyed`}.
- Implement by joining target state in `ListEnabledCheckDefs`
(`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
whose target is retired — **or** in a `housekeeping` sweep
(`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
(no write needed at runtime).
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
### Track C — Dead-data cleanup
**C1. Delete 24 orphan check_defs + check entities.**
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
`slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
`enabled=false` first (already verified).
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
the rows first; this is `config_mutation` → operator approval.
**C2. Retire the secrets route.**
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
`deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
(or keep as archaeology). Decide with operator.
**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
- DB shows these with live `hosts` edges on strong, never cleaned up:
`lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
`lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
container still exists, destroy via `pct destroy` first (destructive → approval).
- They currently generate checks and pollute the graph/health view.
### Track D — Knowledge graph
**D1. Model TLS certificates.**
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
cert) + `uses-certificate` edges from each `ingress-route`.
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
checker's discovery, or seed from Caddyfile and backfill `expires` live.
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
against real entities instead of nothing.
**D2. dns-zone monitoring gap.**
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]``monitoring: none`
with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
`unmonitored` noise. Re-seed.
**D3. Re-export seeds to fix drift.**
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
**D4. Seed skills from disk.**
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
ingest. `get_skills()` then returns data.
**D5. Raise graph node cap.**
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
### Track E — Auto-provision monitoring when a new entity is created
Goal: the operator's request — "make sure this is handled automatically in the future
when the agent creates new entities." Today `ensureDefaultChecks`
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
admits the gap. A new entity should become monitorable with zero manual steps.
**E1. Hook script-deploy into entity creation / provisioning.**
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
after writing check_defs, it also ensures the target can answer:
- **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
(reuse the host resolution from A1). Idempotent (skip if present + unchanged).
- **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
over SSH; locally on mac-mini).
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
automated. mac-mini still needs its `dtoro` key (A3) once.
**E2. Tie into the lifecycle `provisioning → active` gate.**
- The ontology already requires `health-check-answering` for `provisioning → active`
(`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
- Make that gate actually run one check against the new entity and require a non-`down`
verdict before the transition is allowed. This closes the loop: an entity isn't "active"
(and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
**E3. Re-run on re-seed / attribute change.**
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
(`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
and re-deploy scripts to the new guest.
**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
script-pushed + check_defs + a passing `health-check-answering` gate before going active.
### Track F — Read-only knowledge-graph audit skill
Goal: the operator's request — a skill that auto-discovers live infra and validates the
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
finding to the relevant lifecycle runbook.
**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
`get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
entities, evidence, suggested_runbook}`.
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
what's deployed at `/opt/oikos/checks/` per target.
- Report categories (the gaps this investigation found):
1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
rclone class — though rclone's parent is correct; this catches real migrations).
4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
the guest (the strong-guest/rclone class).
6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
via `oikos export` diff.
8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
**F2. Author the skill.**
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
`risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
`config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
tracks for cert/seed/graph-cap work). No mutating steps.
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
(bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
empty-skills-table gap, Track D4).
**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
---
## Validation
After each track, verify via API (read-only, no approval):
- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
route via `pct exec 132` on hubris.
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
after scripts pushed inside + routed via strong's `pct exec`.
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
`health-check-answering` before reaching `active` (E1/E2).
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
- `GET /api/v1/entities?type=skill&limit=50` → >0.
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
`grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
go through the shared `pct exec`/`qm guest exec` resolver.
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
the 9 categories; running it against current state reproduces this plan's findings
(orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
is read-only — confirm it performs no DB writes (audit-log shows only reads).
Unit/integration tests to add/update:
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
`pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
`public_ipv4` preferred for standalone-server (`defaults_test.go`).
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
(new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
- E1: new-entity creation triggers script push (mock pct/SSH in test).
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
category counts (ghost, missing, orphan, undeployed, drift).
## Risks
- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
guests. This is already true (pct exec requires the host up) and is a net improvement
(one credential vs many), but a host outage now fails all its guest checks together —
which is the *correct* blast radius (guests are unreachable when their host is down).
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
`resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
Idempotent + content-checked; never clobber a same-named operator script without diffing.
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
proxying via hubris; confirm before assuming direct SSH works (A5).
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
reachable probe, or grant an operator override.
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
read_only; ensure discovery commands are in the read-only allowlist (no state change).
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
## Open questions (none blocking; confirm during implementation)
- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
`monitoring: none`? (A5)
- secrets ingress: keep as archaeology or destroy the route? (C2)
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
stacks only? (F1)
## Suggested order
A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
→ D1 → D4 → D3 → D5. Validate after each track.

View File

@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
#
# These are leftovers from the old shortSlug() collision bug: check entities
# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
# live target and are disabled. They pollute the entity table and the checks
# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
#
# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
# to actually delete. Review the listed slugs first — they must all match the
# legacy random-slug pattern and be disabled.
#
# Usage:
# cleanup-orphan-checks.sh # dry-run: list what would be deleted
# cleanup-orphan-checks.sh --apply # delete check_defs rows, then entities
#
# Connects via the OIKOS_TEST... no — via the running postgres container by
# default, or OIKOS_PSQL if set.
set -euo pipefail
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
# pointing at a real target. A random-slug check that IS enabled and has a live
# target is a working check with a bad slug — keep it (deleting would drop
# monitoring), and flag it for a slug fix instead.
ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
AND NOT EXISTS (SELECT 1 FROM check_defs cd
WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
echo "== details (slug, state, enabled) =="
$PSQL_CMD -F ' | ' -Ac "
SELECT e.slug, COALESCE(e.state,'(null)'),
COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
FROM entities e
WHERE $ORPHAN_PRED
ORDER BY e.slug;" | head -60
if [ "${1:-}" != "--apply" ]; then
echo
echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
echo " check_defs whose check entity is an orphan, then those entities."
exit 0
fi
echo
echo "== applying (config_mutation) =="
# Delete check_defs first (FK), then the orphan check entities.
$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
BEGIN;
DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
DELETE FROM entities e WHERE $ORPHAN_PRED;
COMMIT;
SQL
echo "== remaining orphans (should be 0) =="
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"

View File

@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
#
# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
# entities in the graph long after the containers are gone or repurposed.
# They generate checks and pollute health/graph views. This reports them and
# their DB state + which Proxmox host each is parented on, so the operator can
# confirm the container is really gone and retire the entity via the
# lifecycle-destroy-node runbook (a destructive, approval-gated action).
#
# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
# lifecycle-deprecate-node.
set -euo pipefail
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
echo "== stray test/scratch LXC entities =="
$PSQL_CMD -F ' | ' -Ac "
SELECT e.slug, COALESCE(e.state,'active') AS state,
e.attributes->>'pve_id' AS pve_id,
COALESCE(h.slug,'(no host)') AS host
FROM entities e
LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
LEFT JOIN entities h ON h.id = r.source_id
WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
ORDER BY e.slug;"
echo
echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
echo "deprecated. If a container still exists, pct destroy it first."

View File

@@ -6372,6 +6372,23 @@ investigations:
tags:
- investigation
runbooks:
- slug: knowledge-graph-audit
name: Knowledge-graph audit
risk_class: read_only
entity_type: entity
procedure: {}
content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
\ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
\ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
\ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
\ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
\ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
\ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
\ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
\ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
\ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
\ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
\ follow-up until that machinery lands.\n"
- slug: client-enrollment
name: Client enrollment
risk_class: read_only

View File

@@ -379,8 +379,11 @@ entity_types:
layer: infrastructure
lifecycle: infrastructure
description: DNS zone (e.g. split-horizon hubris.network).
monitoring: [dns] # NOTE: no `dns` checker exists yet — this is a
# real gap and coverageSweep will report it
monitoring: none # no `dns` checker exists yet; declaring [dns]
# made every zone an unresolvable `unmonitored`
# signal. Flip back to [dns] when a checker lands.
# Requires ontology re-ingest to take effect;
# coverageSweep then auto-clears the stale signals.
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
dns-record:
parent: entity

94
tools/deploy-checks.sh Executable file
View File

@@ -0,0 +1,94 @@
#!/usr/bin/env bash
# deploy-checks.sh — push the check scripts into every monitored target.
#
# Background: an ssh-script check runs the script INSIDE the target, so the
# script must exist at /opt/oikos/checks/ on the target itself — not just on
# the proxmox host. The scheduler routes LXC/VM checks through the host via
# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
# script still has to be present inside the guest. This script deploys them.
#
# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
# and/or pass --host to install on a host/workstation over SSH.
#
# Usage:
# deploy-checks.sh # on a proxmox host: push to every LXC/VM here
# deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
# deploy-checks.sh --checks /path # override the source checks dir
#
# Idempotent: skips a script whose deployed copy is byte-identical.
set -euo pipefail
CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
DEST=/opt/oikos/checks
die() { echo "deploy-checks: $*" >&2; exit 1; }
deploy_to_guest() {
local id="$1" vm="$2" # vm=0 for LXC, 1 for VM
local kind=pct; [ "$vm" = "1" ] && kind=qm
echo "[deploy-checks] $kind $id"
# Ensure the destination dir exists inside the guest.
if [ "$kind" = "pct" ]; then
pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo " skip (pct exec failed)"; return; }
else
# qm guest exec returns JSON; best-effort for VMs (guest agent required).
qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo " skip (qm guest exec failed)"; return; }
fi
local pushed=0 skipped=0
for script in "$CHECKS_DIR"/*.sh; do
local name; name=$(basename "$script")
[ "$name" = "deploy-checks.sh" ] && continue
[ "$name" = "install.sh" ] && continue
if [ "$kind" = "pct" ]; then
pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
else
# qm has no push; copy via the guest agent file write if available.
qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
fi
done
echo " pushed=$pushed skipped=$skipped"
}
deploy_to_host() {
local target="$1" # user@ip or slug resolved by caller
echo "[deploy-checks] host $target"
ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
|| echo " WARNING: install on $target failed"
}
if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
# Host/workstation install mode.
if [ "${1:-}" = "--host" ]; then
[ $# -ge 2 ] || die "--host needs a target (user@ip)"
deploy_to_host "$2"
exit 0
fi
# Proxmox-host mode: push to every local LXC and VM.
if command -v pct >/dev/null 2>&1; then
# LXC containers: ID and status. Skip stopped ones.
while IFS= read -r line; do
[ -z "$line" ] && continue
id=$(awk '{print $1}' <<<"$line")
status=$(awk '{print $2}' <<<"$line")
[ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
deploy_to_guest "$id" 0
done < <(pct list 2>/dev/null | tail -n +2)
else
echo "deploy-checks: 'pct' not found — not a Proxmox host."
echo " On a host/workstation, use: deploy-checks.sh --host user@ip"
exit 0
fi
if command -v qm >/dev/null 2>&1; then
while IFS= read -r line; do
[ -z "$line" ] && continue
id=$(awk '{print $1}' <<<"$line")
status=$(awk '{print $2}' <<<"$line")
[ "$status" = "running" ] || continue
deploy_to_guest "$id" 1
done < <(qm list 2>/dev/null | tail -n +2)
fi
echo "[deploy-checks] done"

View File

@@ -1,13 +1,35 @@
#!/usr/bin/env bash
# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
#
# On a plain host/workstation this installs the scripts locally (the pulling
# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
# guest, because an ssh-script check runs the script INSIDE the target — a
# script present only on the host does nothing for a guest reached via
# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
set -euo pipefail
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
if [ -f "$CHECK_SETUP" ]; then
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
else
echo "[setup-checks] no checks/install.sh found, skipping"
fi
# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
# failing push to one guest must not abort the whole hook. Warn (not skip
# silently) if deploy-checks.sh itself is absent — without it guests never get
# scripts and the pct-exec routing reports every guest check down.
if command -v pct >/dev/null 2>&1; then
if [ ! -f "$DEPLOY" ]; then
echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
elif [ ! -x "$DEPLOY" ]; then
echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
else
"$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
fi
fi