dtoro
b5c1247093
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
61 lines
3.4 KiB
Markdown
61 lines
3.4 KiB
Markdown
# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
|
|
|
|
**Date:** 2026-06-06
|
|
**Status:** resolved
|
|
**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
|
|
|
|
## Symptom
|
|
|
|
All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
|
|
|
|
## Root cause
|
|
|
|
The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
|
|
|
|
### Timeline
|
|
|
|
| Time (UTC+2) | Event |
|
|
|---|---|
|
|
| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
|
|
| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
|
|
| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
|
|
| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
|
|
| Jun 06 22:34 | Investigation began |
|
|
| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
|
|
|
|
### Evidence
|
|
|
|
- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
|
|
- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
|
|
- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
|
|
- `git stash list` shows one auto-stash entry
|
|
- `origin/master` at `1b977aa`: 260 lines, all site blocks present
|
|
|
|
### Secondary root cause found during investigation
|
|
|
|
**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
|
|
|
|
This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
|
|
|
|
## Fix applied
|
|
|
|
1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
|
|
2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
|
|
|
|
## Permanent safeguards (all deployed)
|
|
|
|
| Safeguard | Location | What it does |
|
|
|---|---|---|
|
|
| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
|
|
| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
|
|
| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
|
|
| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
|
|
| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
|
|
|
|
## Related
|
|
|
|
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
|
|
- [Caddy (121)](../../wiki/containers/121-caddy.md)
|
|
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
|
|
- [dns-sync script](../../../scripts/dns-sync.py)
|
|
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh) |