Files

dtoro b5c1247093 docs: streamline & consolidate the tree (phase 6)

Problem: after the wiki-hq reorg, agent-instruction and human-doc domains
were still scattered across the repo root, with three now-redundant stub
files cluttering it. The organizing principle wasn't visible in the layout.

Change — enforce three clear buckets:
- .agents/  = how agents operate: OIKOS.md, HERMES.md (moved from root),
  shared/ conventions, domains/ schemas, skills/, and operations/ (operator
  cheatsheet + enrollment + hermes-agent, moved from root).
- knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now
  including investigations/ (incident records are evidence/sources).
- root = substrate + two entry points (AGENTS.md, README.md), plus plans/
  as its own design-intent domain.

Moves:
- investigations/ -> knowledge/sources/investigations/ (incl. archive/, index).
- operations/ -> .agents/operations/.
- HERMES.md -> .agents/HERMES.md.
- Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md
  (its 7 remaining linkers repointed to .agents/OIKOS.md).

Consumers updated:
- inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards.
- tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md.
- bin/homelab help string -> .agents/operations/hermes-agent.md.
- knowledge/operations schemas, llm-wiki, page-templates, incident-investigation
  skill, AGENTS.md/README nav -> new investigations/operations paths.
- All markdown links rewritten via the path-resolving mapper.

Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/,
plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml.

Verification: docs-lint at baseline (2 intentional cross-repo refs, no new
breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all
doc_page targets resolve; Hermes provisioning scripts point at the new path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-06 18:12:14 +02:00

3.4 KiB

Raw Blame History

Investigation: Caddyfile truncation — all LAN services down (2026-06-06)

Date: 2026-06-06 Status: resolved Duration: ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)

Symptom

All *.hubris.network URLs except photos.hubris.network and auth.hubris.network returned tlsv1 alert internal error or TCP timeouts from LAN/mesh clients. dig @192.168.8.2 and dig @100.122.255.254 both resolved to 192.168.8.175 correctly — DNS was fine. The issue was at the Caddy level.

Root cause

The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the dtoro/caddy-conf git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: photos.hubris.network, prism.hubris.network, and photos2.hubris.network.

Timeline

Time (UTC+2)	Event
Jun 04 23:43	Last successful git-push deploy — full Caddyfile (260 lines)
Jun 06 ~12:00	Caddyfile manually edited locally, truncating to 3 sites
Jun 06 12:39	Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten"
Jun 06 14:13	Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow)
Jun 06 22:34	Investigation began
Jun 06 22:43	Caddyfile restored from `origin/master`, `systemctl reload caddy`

Evidence

git diff HEAD -- Caddyfile on LXC 121: +3 / -159 lines
Git reflog: HEAD at 32575ce (fix: sab port 8081→8082), working tree diverged
Backup file Caddyfile.bak.1780263919: 225 lines, full original config
git stash list shows one auto-stash entry
origin/master at 1b977aa: 260 lines, all site blocks present

Secondary root cause found during investigation

elementsynapse (LXC 118) had iface eth0 inet dhcp internally despite pct set 118 --net0 ... ip=192.168.8.239/24. On DHCP lease renewal, dhclient grabbed .244 from Technitium's pool. Caddy's reverse_proxy 192.168.8.239:8008 was hitting a dead IP.

This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.

Fix applied

Caddyfile → git checkout --force origin/master -- Caddyfile + systemctl reload caddy
elementsynapse → replaced iface eth0 inet dhcp with static, killed dhclient, verified connectivity

Permanent safeguards (all deployed)

Safeguard	Location	What it does
Site-count guard	`/etc/caddy/scripts/deploy.sh`	Refuses reload if <20 `hubris.network` site blocks
Dirty-tree auto-stash	`/etc/caddy/scripts/deploy.sh`	Stashes local edits before git pull
Auto-backup	`/etc/caddy/scripts/deploy.sh`	Saves Caddyfile.bak. before any change, keeps 5
Caddy backend health	`/etc/cron.d/caddy-backend-health` on hubris	Runs `check-caddy-backends.sh` every 10 min
DNS sync	`/etc/cron.d/dns-sync` on LXC 107	Runs `dns-sync.py` every 10 min (was missing since 2026-06-04)

DHCP drift investigation (previous incident) — not filed as its own investigation; see the DNS sync fix
Caddy (121)
elementsynapse (118)
dns-sync script
check-caddy-backends script

3.4 KiB Raw Blame History