Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- disk_usage_check.sh: sed 's/-/0/' for filesystems without inodes
- checkdefaults.Ensure: includes target_id in check_defs INSERT so
signals get proper target slug instead of null
Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
- InsertMetricSample now includes ts=now() (TimescaleDB hypertable requires it)
- ssh-script: pass host and port separately (ssh uses -p flag, not host:port)
- ssh-script: use OIKOS_SSH_USER from config/env, default root
- Add -o LogLevel=ERROR to suppress SSH warnings polluting JSON output
- Use Output() (stdout only) instead of CombinedOutput()
- Set OIKOS_SSH_USER=root in scheduler docker-compose service
- Switch Dockerfile from distroless/static to alpine:3.21
- Install openssh-client-default in runtime image
- Mount SSH key in scheduler service (docker-compose)
- Add NET_RAW capability for ping checks
- Wire OIKOS_SSH_KEY_PATH and OIKOS_SSH_USER env vars in scheduler
- sshExec uses configured key path with StrictHostKeyChecking=no
Add three new pages completing the control-room web UI:
- Agent activity: polls /agent-activity every 5s, filterable by type/agent
- Knowledge search: FTS over /knowledge/search with snippet + entity links
- Audit trail: browseable audit log with actor/action/entity filters
Enhanced live events page with correlation-id clustering (Groups toggle).
Added fetchAgentActivity/searchKnowledge/fetchAudit to the API client.
11 nav items now cover all planned control-room views.
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nomos (from oikonomos, steward of the oikos) avoids the name collision
with Nous Research's Hermes Agent. N0 enumerates the full rename scope:
cmd/, hermes/ dir, env vars, config fields, compose service, Caddy vhost,
identity-preserving DB slug migration + seed update, persona docs.
History, the Matrix bot user, and legacy bin/hermes stay untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Swap anthropic-sdk-go for openai-go against the OpenRouter API; default
model deepseek/deepseek-v4-flash with Exacto routing and ZDR provider
pinning. Record why Nous Hermes Agent (and hosted MCP connectors) were
rejected for the resident role.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hermes becomes an LLM-backed agent loop (hand-rolled tool loop over the
existing mcpClient, not the public MCP connector), with Postgres-backed
sessions, SSE chat streaming, and Authentik-gated /agent routing. The
control-room plan is amended to make the chat the main entry point.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documents a full-project review (confirmed bugs, security gaps, user- and
agent-perspective gaps) and a realtime control-room web UI plan, per prior
codebase exploration on this branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Consolidate Oikos on mac-mini (2026-07-06)
- Client lifecycle in Go (2026-07-07)
- Comprehensive audit & next steps (2026-07-07)
- DB as source of truth (2026-07-07)
- MCP tool completion (2026-07-07)
Paths fixed in index.md to reflect planes/done/ locations.
Active plans remaining: Prometheus LXC (Planned), implementation audit (Active).
Removed via Gitea API using Infisical-stored token:
- Hook 10: 192.168.8.205:9811 (homelab-mcp-deploy)
- Hook 11: 192.168.8.205:9821 (secrets-issuance-deploy)
- Hook 14: 192.168.8.205:9831 (oikos-console-deploy) — found during cleanup
All three pointed to apps/105 (192.168.8.205). No remaining webhooks
fire to the old Python stack. Zero hooks remain on dtoro/oikos.
Cutover checklist updated. Only item remaining: archive apps/105 LXC.
scripts/cleanup-apps105-webhooks.sh: lists webhooks on dtoro/oikos,
filters for apps/105 deployment endpoints, prompts for confirmation,
deletes matching hooks via Gitea API.
Usage: GITEA_TOKEN='...' ./scripts/cleanup-apps105-webhooks.sh
Also updated cutover-checklist.md with the script reference and LXC
archive command.
Plan #1 at 98% (code complete). Three fixes applied to remaining cutover items:
1. watchdog.sh — dual-path health checking (LAN 192.168.8.175 + mesh/Caddy
proxy). Only pages when BOTH paths fail. Partial failure logged but not
paged (distinguishes stack problem from mesh/Caddy issue).
2. deploy.sh — pre-deploy pg_dump before each deploy saves to
/opt/oikos/backups/pre-deploy-<sha>.sql. Rollback script now has a
guaranteed recovery point.
3. docs/operations/rollback.md — runbook documenting automated rollback,
manual recovery, decision tree, backup schedule, and rehearsal log.
Two operational items remain (require operator on Proxmox/Gitea):
- Remove Gitea webhooks ids 10, 11 from dtoro/Homelab-Docs
- Archive apps/105 LXC (pct stop 105 + archive)
All active config (seeds, compose, scripts) is already clean of apps/105 refs.
Infisical bootstrap code is complete (bootstrap-infisical.sh + Go backend).
Added 12-point verification table with file:line evidence for every item.
Noted two minor deviations: inline SQL (not sqlc) and synthetic Infisical
IDs (real Infisical pending consolidation plan). Precondition checks
enumerated with hard/soft classification.
Plan #4 done. Audit inventory verified against codebase:
- 9 superseded oikos/*.py files deleted (only gen-topology.py remains)
- bin/homelab deleted, bin/oikos deleted, oikos/cards/ deleted
- .hermes/plans/ already archived to archive/hermes-plans/ (all 7 files)
- TRMNL plan already in Done table
- seanime + romm documented in seeds/knowledge.yaml (DB-native, no wiki needed)
- Traefik references valid (VPS still runs traefik for public termination)
- ADR-0011 exists (client lifecycle); consolidation plan is Go rewrite record
- Prometheus plan updated: Python refs replaced with Go scheduler, check_defs,
MCP request_execution; LXC 131 identified as teddycloud
Remaining items (cutover, Infisical, watchdog, rollback, apps/105) belong to
consolidation plan (#1). 4 of 6 plans now Done.
Plan #5 done. Wiki already archived to archive/knowledge/. seeds/knowledge.yaml
has 24 docs + 6 investigations + 3 runbooks.
- MCP search_knowledge: upgraded from ILIKE to PostgreSQL ts_rank/ts_headline
- MCP get_entity_knowledge: new tool, walks relationship edges to return
all docs/investigations/runbooks linked to an entity
- HTTP endpoints (SearchKnowledge, GetEntityKnowledge) already used full FTS
- Plan index + audit cross-reference updated
Thin client model (rev 2): no git clone, no sync timer.
Changes:
- Fetches only CLIENTS.md, AGENTS.md, OIKOS.md, tools/ from raw Gitea URL
- Enrolls via POST /api/v1/clients/enroll (replaces dead Python
secrets-issuance service)
- Receives age keypair + Infisical identity from Oikos API
- Context poller replaces 5-minute git pull (launchd/systemd timer hits
GET /api/v1/clients/{slug}/context?since=)
- Removed --no-secrets, --no-mesh flags (degraded modes TBD)
- Removed dead bin/homelab symlink
- Removed Gitea credential configuration (no git clone = no git auth)
- Kept --with-mcp and --with-hermes flags for optional tooling
- auto-setup scripts run from fetched tools/ directory
Phase 1 implementation from the client-lifecycle plan.
- Migration 012: provisioning_steps table, context_version, context_files,
enrolled_at column, slug+type index for machine entities
- API endpoints (openapi.yaml + generated code):
POST /clients/enroll — age key issuance, Infisical identity, state transition
GET /clients/{slug}/context — agent file delta polling (replaces git pull)
GET /clients/{slug}/secrets — scoped secret listing
POST /entities/provision — compute entity creation with constraint validation
GET /entities/{slug}/provision/status — step-by-step provisioning progress
- Handlers in impl.go: enrollment with state validation and age key generation,
provisioning with execution tracking and relationship creation,
context endpoint with since-based delta queries
- Server struct extended with secretsBackend interface for key storage
- All tests pass, build clean
CLIENTS.md is the entry point for a machine joining the homelab.
AGENTS.md is the AI agent persona layer on top. Both fetched, but
CLIENTS.md comes first.
Problem: Repo had no developer guide, no client onboarding doc, no agent
dev instructions. Stale files (675KB SQL dump, one-off convert script,
legacy MCP builder) cluttered the tree. Client enrollment was a documented
intention with no Go implementation.
Changes:
- New docs: CONTRIBUTING.md (dev setup), CLIENTS.md (client onboarding),
.agents/dev/CONTRIBUTING.md (agent codebase map)
- New plan: plans/2026-07-07-client-lifecycle-in-go.md — full client
lifecycle (planned→provisioning→active→deprecated→destroyed) in Go,
replacing archived Python secrets-issuance, adding client API endpoints
and 6 missing MCP tools
- Cleanup: deleted archive/convert-wiki.py (one-off), archive/mcp/
build_host_files.py (legacy), backups/pre-deploy-7f7d039.sql (local)
- Fixes: plans/index.md duplicate row removed, README.md repo layout
updated for current state, AGENTS.md header points to new guides
Risk: low. Docs only + stale file deletion. No code changes. New plan is
proposal, not implementation.
Verification: git diff reviewed, all changes are prose/docs/plans.
