Compare commits
216 Commits
claude/wiz
...
claude/oik
| Author | SHA1 | Date | |
|---|---|---|---|
| 52e16e04ca | |||
| 6192c35c10 | |||
| 682326382e | |||
| ac48390796 | |||
| 40999b0b40 | |||
| ec41c0b828 | |||
| 60edff2065 | |||
| 233b5e4519 | |||
| 13458e467c | |||
| 7387df3276 | |||
| 2e922f6421 | |||
| 6f9998fa29 | |||
| d2f749d33d | |||
| c3699157ae | |||
| 7ff344ab47 | |||
| 657e1a8be1 | |||
| 7a7ce2b89b | |||
| 3f3de18b23 | |||
| 82b0ad2298 | |||
| 8950bada44 | |||
| f936098364 | |||
| d08a985ea9 | |||
| 9539759db6 | |||
| d52968876a | |||
| 9daf8220f2 | |||
| 4a96f46e76 | |||
| 5f888e6386 | |||
| f248508919 | |||
| a1f666f68a | |||
| ac86302f52 | |||
| 8ed2b88495 | |||
| b37f85ae08 | |||
| a567930466 | |||
| 9376dc7d89 | |||
| d9683cfe29 | |||
| ea62d744ed | |||
| 0d29b1db81 | |||
| e92a6ff7a5 | |||
| 49c37fe8b1 | |||
| 614c38ea7c | |||
| 22412d2fa3 | |||
| 5686b9de40 | |||
| aa6017e0ca | |||
| cbfd09c5df | |||
| 851b5dce67 | |||
| 279549c8c9 | |||
| a39e67b6e9 | |||
| 551497e0b3 | |||
| 4a5e68bafe | |||
| ef00e5b8e4 | |||
| a512d40669 | |||
| b8bb29464b | |||
| 81acadec1d | |||
| 291b45565b | |||
| d45f2326b6 | |||
| 4bf811a383 | |||
| 35feada286 | |||
| cca2ae4621 | |||
| d13f6991b1 | |||
| 2de3602ebc | |||
| 2908b0a377 | |||
| cff05c0768 | |||
| 5d02126e16 | |||
| e8e230b4a5 | |||
| 2b3aa248b1 | |||
| 9d9cbb63c4 | |||
| ae3b150dcc | |||
| 750f0e088d | |||
| 84068cc40b | |||
| 5d15265f65 | |||
| 3ea43adcfd | |||
| 4724d6f297 | |||
| f07668c1c3 | |||
| 7e0566d2cd | |||
| 2e8ef75436 | |||
| ca16b75b90 | |||
| e148c7a981 | |||
| cefeba72b0 | |||
| e131215a25 | |||
| 7660e5681c | |||
| 28ab9b8088 | |||
| fcd9f23ee1 | |||
| efa66c7321 | |||
| 43aaf2a318 | |||
| a3ebd12e90 | |||
| 7c6cffb5f5 | |||
| 5b22f2367b | |||
| 84ecb6b895 | |||
| cfce35bee0 | |||
| a786107cc7 | |||
| 44e1e421e1 | |||
| 8653f3036d | |||
| 79dc87d584 | |||
| 638e313c66 | |||
| 85b541a1cc | |||
| 7f7d039e1b | |||
| 7ac2521a22 | |||
| 692800cf09 | |||
| bbd560bf19 | |||
| e93625b971 | |||
| bf250fc6a9 | |||
| b35825c50b | |||
| 806795c63d | |||
| 24b8772d72 | |||
| 129a6710cc | |||
| ae23311f17 | |||
| 2de23325c5 | |||
| 6b51c83b82 | |||
| 5e3b946ded | |||
| 5009a335bb | |||
| f04e0dc0d4 | |||
| 6b75f7302d | |||
| b2bfa26f64 | |||
| 4b6c02a88e | |||
| 11be45d307 | |||
| 0e3cbceeae | |||
| 128e11b823 | |||
| a0f059d19f | |||
| c9fb5fe553 | |||
| 8175d218c1 | |||
| c8b27b2c51 | |||
| dcd35b6315 | |||
| 890fe1a1c3 | |||
| f4a00a6cfd | |||
| 74a6b6bb18 | |||
| 3823a82417 | |||
| aa197190cd | |||
| 095a3967c4 | |||
| 7e802bbb14 | |||
| f1b0b65149 | |||
| 6b61495e3b | |||
| 1bfc18ea3a | |||
| 9c63a1bfa9 | |||
| c9975d60a5 | |||
| f2fe812cda | |||
| 1b04683639 | |||
| aa2ca0ae6f | |||
| 55710bd254 | |||
| 18cb79caf9 | |||
| ea3b2c3662 | |||
| 8850b85325 | |||
| 3a35289f46 | |||
| 2d75544362 | |||
| d44979aca7 | |||
| fe54af30f6 | |||
| bead722fac | |||
| a434a4096c | |||
| 14448a7dd9 | |||
| b047c757a7 | |||
| af14c38fb2 | |||
| 0d2093ba3b | |||
| d90de0759c | |||
| 18589f6d8d | |||
| 2ddc1eaa18 | |||
| 658dc0f8b2 | |||
| b5c1247093 | |||
| 4efddb8bed | |||
| 1019a1cb52 | |||
| 5c5016b3c7 | |||
| 8a6422bd7d | |||
| bb5c0717a1 | |||
| 14e88c7c5e | |||
| dede118a80 | |||
| 72720efa21 | |||
| 7610e5394c | |||
| 610b096840 | |||
| 25c67a79c6 | |||
| 493ae50f37 | |||
| c6fedb38c1 | |||
| fd35b48c8d | |||
| 205d8a1a43 | |||
| 2084a1583e | |||
| 48debc0911 | |||
| f6b57cbe3a | |||
| b230ab5937 | |||
| 7e8860ab47 | |||
| e04d943d5c | |||
| 7d5e7227ca | |||
| ec52dfb6a2 | |||
| bb33963539 | |||
| 2709e79455 | |||
| 831794e98c | |||
| 6669feafdc | |||
| ba93c4709b | |||
|
|
5887129202 | ||
|
|
1ad31fe33a | ||
| 4e3ec61fb0 | |||
| abb1476fc8 | |||
| dee08b97a4 | |||
| 359f55a695 | |||
| fc62cf38f7 | |||
| b266c3f1d8 | |||
| fdab6282e6 | |||
| e4b529b71b | |||
| c012a9124f | |||
| 41df77fb20 | |||
| b67389b137 | |||
| 45b7813eb7 | |||
| 8f6f43eeb1 | |||
| c2c257aa12 | |||
| 41fe34a3c3 | |||
| b6c9184a7b | |||
|
|
636c90f99c | ||
|
|
cef1b836c2 | ||
|
|
335498d069 | ||
|
|
867998e9c0 | ||
| cb0c1a5c0d | |||
| ddf541ebd3 | |||
| 4472e73ae1 | |||
| 59261c7ef1 | |||
| 41f22c2d85 | |||
| b7cfc350b3 | |||
| fffa5560eb | |||
| d6bfd7b8c9 | |||
| b71402aaed | |||
| 2a7876c711 |
@@ -1,4 +1,4 @@
|
|||||||
# HERMES.md — Agent persona for homelab clients
|
# NOMOS.md — Agent persona for homelab clients
|
||||||
|
|
||||||
This file is the canonical agent persona for **all** AI agents running on
|
This file is the canonical agent persona for **all** AI agents running on
|
||||||
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
|
machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
|
||||||
@@ -8,19 +8,32 @@ conventions, and the source-of-truth hierarchy.
|
|||||||
|
|
||||||
The homelab-context repo at `/opt/homelab-context/` is the single source of
|
The homelab-context repo at `/opt/homelab-context/` is the single source of
|
||||||
truth for:
|
truth for:
|
||||||
- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
|
- Fleet topology (`inventory.yaml`, `inventory.yaml`)
|
||||||
- Service endpoints and credentials (via `homelab secret`)
|
- Service endpoints and credentials (via `homelab secret`)
|
||||||
- Agent behaviour and conventions
|
- Agent behaviour and conventions
|
||||||
- Everything in this file
|
- Everything in this file
|
||||||
|
|
||||||
When in doubt, check `/opt/homelab-context/` first.
|
When in doubt, check `/opt/homelab-context/` first.
|
||||||
|
|
||||||
|
## Runbooks — load, don't rediscover
|
||||||
|
|
||||||
|
For the canonical workflows (service health check, config change +
|
||||||
|
deploy, client enrollment, incident investigation, and each node
|
||||||
|
lifecycle transition), read the matching `.agents/skills/<name>/SKILL.md` before
|
||||||
|
acting. Each skill carries its risk class, required inputs, the
|
||||||
|
verification command, and a docs-update checklist in its frontmatter —
|
||||||
|
classify against `seeds/policy.yaml` using that risk class before any
|
||||||
|
mutation. Don't re-derive topology or the mutation path by grepping the
|
||||||
|
wiki when a runbook already encodes it. See [OIKOS.md](OIKOS.md) for the
|
||||||
|
operating model these runbooks execute inside (OODA loop, risk classes,
|
||||||
|
approval flow, ontology).
|
||||||
|
|
||||||
## Agent type — how this file gets loaded
|
## Agent type — how this file gets loaded
|
||||||
|
|
||||||
| Agent | Loading mechanism |
|
| Agent | Loading mechanism |
|
||||||
|-------|------------------|
|
|-------|------------------|
|
||||||
| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
|
| **Nomos** | `tools/setup-nomos-soul.sh` (auto-setup) → provisions `~/.nomos/SOUL.md` from this file |
|
||||||
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/HERMES.md` |
|
| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/NOMOS.md` |
|
||||||
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
|
| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
|
||||||
|
|
||||||
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
|
**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
|
||||||
@@ -74,10 +87,10 @@ Caveman templates live at `~/templates/`:
|
|||||||
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
|
ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Important note for Hermes agents
|
## Important note for Nomos agents
|
||||||
|
|
||||||
If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
|
If you are reading this as a Nomos agent, your SOUL.md was auto-provisioned
|
||||||
by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
|
by `tools/setup-nomos-soul.sh`. This file is the canonical original — you
|
||||||
can verify the content matches or re-provision by running:
|
can verify the content matches or re-provision by running:
|
||||||
|
|
||||||
bash /opt/homelab-context/tools/setup-hermes-soul.sh
|
bash /opt/homelab-context/tools/setup-nomos-soul.sh
|
||||||
190
.agents/OIKOS.md
Normal file
190
.agents/OIKOS.md
Normal file
@@ -0,0 +1,190 @@
|
|||||||
|
# Oikos — the operating model
|
||||||
|
|
||||||
|
Oikos (Greek: *household*) is the agent operating system layered on this
|
||||||
|
repo. It is not new infrastructure: `inventory.yaml` is the kernel data
|
||||||
|
structure, the `homelab` CLI and MCP server are the syscall surface, and
|
||||||
|
this page defines the rules everything above them follows.
|
||||||
|
|
||||||
|
Read this after [AGENTS.md](../AGENTS.md). Machine-readable companions:
|
||||||
|
[seeds/ontology.yaml](../seeds/ontology.yaml) (systems model),
|
||||||
|
[seeds/policy.yaml](../seeds/policy.yaml) (risk & approval).
|
||||||
|
|
||||||
|
## The kernel loop: OODA
|
||||||
|
|
||||||
|
Every Oikos activity — scheduled probe, agent task, operator request — is
|
||||||
|
one pass through **Observe → Orient → Decide → Act**:
|
||||||
|
|
||||||
|
1. **Observe** — probes, drift detectors, and agent findings produce
|
||||||
|
**Signals** (structured records, not loose messages): pending updates,
|
||||||
|
high temperature, low disk, service down, cert expiry, stale backup,
|
||||||
|
inventory drift.
|
||||||
|
2. **Orient** — walk the ontology graph: what entity is affected, what
|
||||||
|
depends on it (blast radius), its lifecycle state, whether a runbook
|
||||||
|
matches, what the ledger says about past attempts.
|
||||||
|
3. **Decide** — the classifier scores **risk class × blast radius ×
|
||||||
|
confidence** and routes:
|
||||||
|
- **auto-act**: within autonomy policy, high confidence, contained radius
|
||||||
|
- **escalate**: operator approval via Matrix (✅/❌ reaction) or the
|
||||||
|
Oikos Console's `/approvals` page (destructive actions additionally
|
||||||
|
need a typed confirmation phrase either way)
|
||||||
|
- **queue**: informational — console + reports
|
||||||
|
The classifier can only *lower* autonomy relative to policy, never raise
|
||||||
|
it. When in doubt, escalate.
|
||||||
|
4. **Act** — execute through `homelab` commands or runbooks (never ad-hoc
|
||||||
|
SSH), then **verify** with the action's verification command, write a
|
||||||
|
**ledger** entry, resolve the Signal, and update docs in the same session.
|
||||||
|
|
||||||
|
## Primitives
|
||||||
|
|
||||||
|
| Primitive | What it is | Lives in |
|
||||||
|
|---|---|---|
|
||||||
|
| Host / Service | topology entities | `inventory.yaml` |
|
||||||
|
| Secret | SOPS+age encrypted value, per-client recipients | `secrets/` + `.sops.yaml` |
|
||||||
|
| Runbook | executable workflow with risk class + verification | `.agents/skills/<name>/SKILL.md` |
|
||||||
|
| Signal | something needing attention, with lifecycle | DB `signals` table |
|
||||||
|
| Change | one mutation: who, what, risk, approval, verification | DB `audit_log` + `executions` tables |
|
||||||
|
| Approval | short-TTL signed grant for a gated action | DB `approvals` table |
|
||||||
|
| Incident | investigation narrative | DB `knowledge_entities` (seeded from investigations) |
|
||||||
|
| Knowledge | document, runbook, investigation | DB `knowledge_entities` (seeded from `seeds/knowledge.yaml`) |
|
||||||
|
| Plan | design doc for non-trivial work | `plans/` |
|
||||||
|
| Agent | enrolled client identity = its age pubkey | `inventory.yaml` + `.sops.yaml` |
|
||||||
|
|
||||||
|
## Risk classes (enforced, not advisory)
|
||||||
|
|
||||||
|
From [seeds/policy.yaml](../seeds/policy.yaml):
|
||||||
|
|
||||||
|
- **read_only** — status, logs, docs, inventory. Unattended.
|
||||||
|
- **reversible_low** — restart, cache clear, sync pull. Unattended + ledger.
|
||||||
|
- **config_mutation** — tracked-config edits (commit+push, never local),
|
||||||
|
deploys, upgrades, DNS/ingress changes. Operator approval.
|
||||||
|
- **destructive** — destroy, format, wipe, rotate, revoke. Approval +
|
||||||
|
typed confirmation phrase.
|
||||||
|
|
||||||
|
Lifecycle gates modify these: `provisioning` nodes are freely mutable
|
||||||
|
(nothing depends on them); `deprecated` nodes accept no new dependents;
|
||||||
|
anything touching a `destroyed` node is drift.
|
||||||
|
|
||||||
|
## The systems model
|
||||||
|
|
||||||
|
Eight domains — physical, compute, network, storage, software,
|
||||||
|
identity & access, operations, external — cover everything in the lab;
|
||||||
|
entities are connected by typed edges (`hosts`, `provides`, `mounts`,
|
||||||
|
`stores-on`, `routes-to`, `can-decrypt`, `depends-on`, `backs-up-to`, …)
|
||||||
|
defined in [seeds/ontology.yaml](../seeds/ontology.yaml). Rule of
|
||||||
|
completeness: **if it can break, be changed, or hold data, it has an
|
||||||
|
entity and edges.** Blast-radius questions ("what breaks if strong goes
|
||||||
|
down?") are graph walks, not doc archaeology.
|
||||||
|
|
||||||
|
Nodes move through an explicit lifecycle —
|
||||||
|
`planned → provisioning → active → migrating → deprecated → destroyed` —
|
||||||
|
stored as `state:` in inventory (absent = active). Destroyed nodes live in
|
||||||
|
the `archaeology:` section. Each transition is a runbook checklist;
|
||||||
|
deprecation completes only when inbound edges reach zero.
|
||||||
|
|
||||||
|
Generated views: the live topology graph at `oikos.hubris.network/graph`
|
||||||
|
via the API's `/api/v1/graph` endpoint, and the Mermaid export at
|
||||||
|
`GET /api/v1/graph?format=mermaid`.
|
||||||
|
|
||||||
|
## Conventions carried forward
|
||||||
|
|
||||||
|
- Inventory is the truth; live state wins over narrative docs.
|
||||||
|
- Prefer `homelab` CLI and MCP over ad-hoc SSH.
|
||||||
|
- Meaningful changes update docs in the same session.
|
||||||
|
- Secrets are decrypted locally via per-client keys; never into docs/comments.
|
||||||
|
- Tracked configs change by commit + push, not local edits.
|
||||||
|
- Netbird is the preferred mesh path for new traffic.
|
||||||
|
- Agents are terse ([caveman.md](shared/caveman.md)), verify claims, and fix
|
||||||
|
collateral drift when found.
|
||||||
|
|
||||||
|
## Build status (Go rewrite — deployed 2026-07-07)
|
||||||
|
|
||||||
|
The Oikos runtime was rewritten from Python to Go over 6 phases and is deployed
|
||||||
|
in Docker on mac-mini. See
|
||||||
|
[plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](../plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md)
|
||||||
|
for the full plan. The Python codebase has been removed; all functionality runs
|
||||||
|
in the Go binary.
|
||||||
|
|
||||||
|
**Phase 1 — Ontology + DB (DONE):**
|
||||||
|
- `migrations/` (001–011): TimescaleDB hypertables, entity_status, CAGGs,
|
||||||
|
retention policies, knowledge entities with FTS. Forward-only, idempotent.
|
||||||
|
- `seeds/{ontology,inventory,policy,knowledge}.yaml`: DB-native bootstrap +
|
||||||
|
DR export. Knowledge seed contains 36 documents, 6 investigations, and 12
|
||||||
|
runbooks.
|
||||||
|
- `blast_radius()` SQL CTE, type hierarchy, abstract types, relationship
|
||||||
|
validation.
|
||||||
|
- Go packages: `internal/db/`, `internal/ontology/`, `internal/domain/`,
|
||||||
|
`internal/knowledge/`.
|
||||||
|
|
||||||
|
**Phase 2 — API (DONE):**
|
||||||
|
- Single binary `cmd/oikos` with `oikos api` serving REST (:8090) + MCP
|
||||||
|
on the same service layer. OpenAPI-first (`api/openapi.yaml`) with
|
||||||
|
oapi-codegen + chi. RFC 9457 problem+json errors. Cursor pagination,
|
||||||
|
If-Match/ETag optimistic concurrency, idempotency keys, SSE event stream,
|
||||||
|
OIDC JWT + static bearer auth, audit middleware.
|
||||||
|
- Go packages: `internal/httpapi/`, `internal/httpapi/gen/`.
|
||||||
|
|
||||||
|
**Phase 3 — Control loop (DONE):**
|
||||||
|
- Scheduler (`oikos scheduler`): check_defs runner, signal dedup/flap
|
||||||
|
suppression, entity_status. HTTP, TCP, disk, cert-expiry probes.
|
||||||
|
- Actuator: SSH skill procedure execution with context-aware timeouts,
|
||||||
|
circuit breaker, retry budgets, error classification.
|
||||||
|
- Learning engine: hourly pattern extraction, Wilson confidence bounds,
|
||||||
|
anomaly detection, skills with validated patterns.
|
||||||
|
- Notifier: Matrix badge delivery, approval token generation (HMAC,
|
||||||
|
single-use, hashed), DB rendezvous pattern.
|
||||||
|
- Policy classifier: risk class determination, autonomy routing,
|
||||||
|
blast-radius computation, kill-switch support.
|
||||||
|
- Go packages: `internal/scheduler/`, `internal/actuator/`,
|
||||||
|
`internal/learning/`, `internal/notifier/`, `internal/policy/`.
|
||||||
|
|
||||||
|
**Phase 4 — Agent / Nomos (DONE):**
|
||||||
|
- Standalone Nomos MCP client binary (`cmd/nomos`) with gateway mode
|
||||||
|
(:8092). Structured queries + natural-language routing to 15 MCP tools.
|
||||||
|
Agent activity logging on every tool call. No SSH keys.
|
||||||
|
- `nomos/` directory with config, SOUL.md, homelab-ops skill.
|
||||||
|
- Nomos Docker service in `docker-compose.yml` (profile: full).
|
||||||
|
- Go packages: `cmd/nomos/`, `compose/nomos/`.
|
||||||
|
|
||||||
|
**Phase 5 — Secrets / Infisical (DONE):**
|
||||||
|
- `internal/secrets/`: backend abstraction (Manager) with primary
|
||||||
|
(Infisical) and fallback (SOPS) backends. Machine identities via
|
||||||
|
UniversalAuth. In-memory cache with TTL.
|
||||||
|
- `oikos secret` CLI: list, migrate (SOPS → Infisical), export-sops
|
||||||
|
(DR fallback). Infisical SDK v0.8.0.
|
||||||
|
- Rotation runbook at `secrets/rotation.md`.
|
||||||
|
- Docker compose: `infisical` + `redis` services (profile: infisical).
|
||||||
|
|
||||||
|
**Phase 6 — Deploy + cutover (DONE, pending production cutover):**
|
||||||
|
- CI pipeline: `.gitea/workflows/ci.yml` (Gitea Actions — build, vet,
|
||||||
|
lint, test, docker build).
|
||||||
|
- Deploy: `scripts/deploy.sh` (git pull → docker build → compose up →
|
||||||
|
health check), SHA-tagged images, rolling restart.
|
||||||
|
- Caddy config: `compose/caddy/Caddyfile.oikos` (oikos/mcp/nomos →
|
||||||
|
mac-mini mesh :8090/:8092).
|
||||||
|
- Watchdog: `scripts/watchdog.sh` (2min cron, Matrix alert on failure).
|
||||||
|
- Verification: `scripts/verify-phase6.sh` (14/14 checks pass).
|
||||||
|
- Rollback: `scripts/rollback.sh` (checkout SHA + pg_restore).
|
||||||
|
- Cutover checklist: `scripts/cutover-checklist.md`.
|
||||||
|
|
||||||
|
**Current deployment:**
|
||||||
|
- **Production**: Docker stack on mac-mini (`--profile full`: postgres, api,
|
||||||
|
scheduler, notifier, nomos). Deployed 2026-07-07 with full knowledge seed.
|
||||||
|
The Python MCP server and secrets-issuance on apps/105 have been stopped
|
||||||
|
(see `scripts/cutover-checklist.md`).
|
||||||
|
|
||||||
|
## Python-era backlog (superseded)
|
||||||
|
|
||||||
|
The original 30-day roadmap (Python, shipped 2026-06/07) delivered:
|
||||||
|
context cards, change ledger, node relations, runbooks, ops scheduler,
|
||||||
|
drift detectors, signal engine, classifier, approval engine, and the
|
||||||
|
FastAPI+Jinja2 Oikos Console. All of these have been re-implemented in
|
||||||
|
the Go rewrite. The backlog items below that referenced Python paths
|
||||||
|
(`oikos/approve.py`, `oikos/drift.py`, `oikos/console/`) are now addressed
|
||||||
|
by the Go equivalents listed above.
|
||||||
|
|
||||||
|
Outstanding from the Python era (not yet in Go):
|
||||||
|
- Prometheus provisioning (see [plans/2026-07-05-oikos-prometheus-lxc.md](../plans/2026-07-05-oikos-prometheus-lxc.md))
|
||||||
|
- CPU/NVMe temperature probing (blocked on sensor path discovery)
|
||||||
|
- SSH-key-signed approval requests (blocked on inventory schema)
|
||||||
|
- Multi-agent delegation (blocked on ledger identity field)
|
||||||
|
- Restore drills on a schedule
|
||||||
179
.agents/dev/CONTRIBUTING.md
Normal file
179
.agents/dev/CONTRIBUTING.md
Normal file
@@ -0,0 +1,179 @@
|
|||||||
|
# Agent developer guide
|
||||||
|
|
||||||
|
Instructions for AI agents working on the Oikos codebase. Read this after
|
||||||
|
[AGENTS.md](../../AGENTS.md) and [OIKOS.md](../OIKOS.md). Human developers:
|
||||||
|
see [CONTRIBUTING.md](../../CONTRIBUTING.md) for a human-friendly version.
|
||||||
|
|
||||||
|
## Codebase map
|
||||||
|
|
||||||
|
```
|
||||||
|
cmd/oikos/main.go Entry point. Subcommands: api, scheduler, notifier, migrate,
|
||||||
|
seed, export, secret, all
|
||||||
|
cmd/nomos/main.go Nomos MCP client gateway (standalone binary, formerly Hermes)
|
||||||
|
internal/httpapi/ REST + MCP server. Chi router. OpenAPI-generated types from
|
||||||
|
internal/httpapi/gen/api.gen.go. Strict server in impl.go.
|
||||||
|
internal/mcp/ MCP tool implementations (get_entity, search_knowledge, etc.)
|
||||||
|
internal/db/ Connection pool (pool.go), seed ingestion (seed.go), DB→YAML
|
||||||
|
export (export.go), type hierarchy (typetree.go)
|
||||||
|
internal/db/queries/ SQL query files → sqlc generates internal/db/sqlcgen/
|
||||||
|
internal/scheduler/ Observe loop: probes, signals, check_defs
|
||||||
|
internal/actuator/ SSH execution with circuit breaker + retry
|
||||||
|
internal/learning/ Pattern extraction, anomaly detection
|
||||||
|
internal/notifier/ Matrix notification + approval token generation
|
||||||
|
internal/policy/ Risk classifier (read policy.yaml → classify action)
|
||||||
|
internal/secrets/ Backend abstraction: Infisical (primary) + SOPS (fallback)
|
||||||
|
internal/domain/ Core types: entities, approvals, executions, signals, patterns
|
||||||
|
internal/ontology/ Type hierarchy validation, relationship checks
|
||||||
|
internal/knowledge/ Knowledge YAML seed ingestion
|
||||||
|
internal/config/ Config loading from env vars
|
||||||
|
api/openapi.yaml REST API contract. Source of truth for endpoints.
|
||||||
|
api/codegen.yaml oapi-codegen config → generates internal/httpapi/gen/
|
||||||
|
migrations/ Forward-only SQL. Format: NNN_name.up.sql. No down migrations.
|
||||||
|
seeds/ Bootstrap YAML. ontology.yaml, inventory.yaml, policy.yaml,
|
||||||
|
knowledge.yaml. Regenerated from DB via oikos export.
|
||||||
|
compose/ Dockerfiles. oikos/ (multi-stage), nomos/ (distroless).
|
||||||
|
Caddy config at compose/caddy/Caddyfile.oikos.
|
||||||
|
scripts/ Deploy, rollback, watchdog, verification, cutover checklist.
|
||||||
|
nomos/ Nomos config.yaml, SOUL.md, skills.
|
||||||
|
.agents/ Agent instruction files, domains, shared conventions, skills.
|
||||||
|
plans/ Design documents. active/ + done/.
|
||||||
|
docs/adr/ Architecture decision records. Numbered, prefix-sorted.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development loop
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start dependencies
|
||||||
|
make dev
|
||||||
|
|
||||||
|
# Generate code after API/SQL changes
|
||||||
|
make generate
|
||||||
|
|
||||||
|
# Build
|
||||||
|
make build
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
make test # all unit tests
|
||||||
|
make test-db # integration tests (needs compose Postgres)
|
||||||
|
|
||||||
|
# Lint
|
||||||
|
make lint
|
||||||
|
|
||||||
|
# CI drift guard (run before commit)
|
||||||
|
make generate-check
|
||||||
|
```
|
||||||
|
|
||||||
|
## Adding a feature or phase
|
||||||
|
|
||||||
|
Oikos features follow a phase model (read [OIKOS.md](../OIKOS.md) for the
|
||||||
|
current phase status). To add a new capability:
|
||||||
|
|
||||||
|
1. **ADR first.** Write an architecture decision record in `docs/adr/` with
|
||||||
|
the next sequence number. Document the decision, context, alternatives
|
||||||
|
considered, and consequences.
|
||||||
|
2. **Plan.** If the change is non-trivial, create a plan in `plans/` following
|
||||||
|
the template in [page-templates.md](../shared/page-templates.md).
|
||||||
|
3. **Schema.** If the feature needs new DB tables, write a forward-only
|
||||||
|
migration in `migrations/`. Use `IF NOT EXISTS` for idempotency.
|
||||||
|
4. **API.** If the feature exposes endpoints, define them in
|
||||||
|
`api/openapi.yaml` first, then run `make generate`, then implement.
|
||||||
|
5. **Domain.** Add types to `internal/domain/` before adding logic.
|
||||||
|
6. **Tests.** Write tests alongside implementation. Integration tests go in
|
||||||
|
`*_test.go` in the relevant package, using the compose Postgres.
|
||||||
|
7. **Policy.** If the feature introduces new mutation types, update
|
||||||
|
`seeds/policy.yaml` and the classifier in `internal/policy/`.
|
||||||
|
8. **Run `make generate-check`** before commit to ensure generated code is
|
||||||
|
current.
|
||||||
|
|
||||||
|
## SQL conventions
|
||||||
|
|
||||||
|
- Queries live in `internal/db/queries/*.sql` with `-- name: FuncName :exec`
|
||||||
|
annotations for sqlc
|
||||||
|
- Use `pgx/v5` driver. UUIDs use `pgtype.UUID`, timestamps use `time.Time`
|
||||||
|
- CTEs for graph traversals (blast radius, dependency chains)
|
||||||
|
- CAGGs and retention policies for TimescaleDB hypertables
|
||||||
|
- FTS via `tsvector` + `tsquery` for knowledge search (migration 011)
|
||||||
|
|
||||||
|
## OpenAPI codegen
|
||||||
|
|
||||||
|
- Config: `api/codegen.yaml`. Uses `oapi-codegen/v2` with Chi server template
|
||||||
|
- Generated output: `internal/httpapi/gen/api.gen.go` — never hand-edit
|
||||||
|
- Strict server interface: `api.gen.go` generates the `StrictServerInterface`;
|
||||||
|
implement it in `internal/httpapi/impl.go`
|
||||||
|
- Problem+JSON errors via `internal/httpapi/problem.go` — RFC 9457 format
|
||||||
|
- Cursor pagination, If-Match/ETag, idempotency keys, SSE streaming
|
||||||
|
|
||||||
|
## Testing philosophy
|
||||||
|
|
||||||
|
- **Race detector always on.** `make test` runs `go test -race -cover ./...`
|
||||||
|
- **Integration tests** use the compose Postgres. Run with `make test-db`.
|
||||||
|
Each test creates + tears down its own schema namespace.
|
||||||
|
- **Coverage gates** in CI: policy + learning ≥ 80%, others ≥ 60%
|
||||||
|
- Tests use `testing.T` directly, no assertion library
|
||||||
|
- Table-driven tests for validation and classification logic
|
||||||
|
|
||||||
|
## Migration rules
|
||||||
|
|
||||||
|
- Forward-only. No down migrations (ADR 0008)
|
||||||
|
- Idempotent: use `IF NOT EXISTS`, `DO $$ BEGIN ... END $$` blocks
|
||||||
|
- Sequence numbers are sequential integers (001, 002, ...)
|
||||||
|
- Each migration file is `NNN_name.up.sql`
|
||||||
|
- Migrations are embedded in the binary via `migrations/embed.go`
|
||||||
|
|
||||||
|
## Seed files
|
||||||
|
|
||||||
|
- `seeds/ontology.yaml` — entity types, relationship types, lifecycles
|
||||||
|
(validated against schema in `internal/ontology/`)
|
||||||
|
- `seeds/inventory.yaml` — hosts, services, entities (the topology)
|
||||||
|
- `seeds/policy.yaml` — risk classes, approval rules, autonomy settings
|
||||||
|
- `seeds/knowledge.yaml` — documents, investigations, runbooks (DB is source
|
||||||
|
of truth; this file is the DR export)
|
||||||
|
- After DB changes via the API, run `make export` to regenerate seeds
|
||||||
|
|
||||||
|
## Secrets handling
|
||||||
|
|
||||||
|
- No secrets in code, config, or commits
|
||||||
|
- Dev secrets in `.env` (gitignored)
|
||||||
|
- Primary: Infisical (`internal/secrets/infisical.go`)
|
||||||
|
- Fallback: SOPS + age (`internal/secrets/sops.go`)
|
||||||
|
- Backend interface: `internal/secrets/backend.go`
|
||||||
|
- Machine identities via Infisical UniversalAuth
|
||||||
|
- In-memory cache with TTL for performance
|
||||||
|
|
||||||
|
## Staging and deployment
|
||||||
|
|
||||||
|
- CI pipeline: `.gitea/workflows/ci.yml` — lint, vet, vulncheck, test, docker build
|
||||||
|
- Deploy: `scripts/deploy.sh` — git pull → docker build → compose up → health check
|
||||||
|
- Watchdog: `scripts/watchdog.sh` — 2-minute cron, Matrix alert on failure
|
||||||
|
- Rollback: `scripts/rollback.sh` — checkout SHA + pg_restore
|
||||||
|
- Cutover checklist: `scripts/cutover-checklist.md`
|
||||||
|
|
||||||
|
## Writing conventions
|
||||||
|
|
||||||
|
Apply [writing-style.md](../shared/writing-style.md) for all committed prose.
|
||||||
|
Terse, reference-style, no marketing vocabulary. Code comments explain intent
|
||||||
|
and trade-offs, not mechanics.
|
||||||
|
|
||||||
|
Apply [caveman.md](../shared/caveman.md) for agent communication. The caveman
|
||||||
|
standard applies to agent *chat responses*, not committed documentation.
|
||||||
|
|
||||||
|
## Skills
|
||||||
|
|
||||||
|
Agent skills live under `.agents/skills/<name>/SKILL.md`. Each skill has a
|
||||||
|
frontmatter description that tools match against tasks. To add a skill:
|
||||||
|
|
||||||
|
1. Create `.agents/skills/<name>/SKILL.md`
|
||||||
|
2. Include frontmatter with description field
|
||||||
|
3. Document the procedure following the runbook template
|
||||||
|
4. Reference relevant files, commands, and policy classes
|
||||||
|
|
||||||
|
Skills that require code (e.g. linting) may include companion scripts in the
|
||||||
|
same directory.
|
||||||
|
|
||||||
|
## When in doubt
|
||||||
|
|
||||||
|
- Query MCP tools first (search_knowledge, get_entity)
|
||||||
|
- Read the relevant ADR in `docs/adr/`
|
||||||
|
- Grep the codebase: `rg <symbol> internal/`
|
||||||
|
- Check `plans/` for in-progress work that may conflict
|
||||||
|
- Classify any new mutation against `seeds/policy.yaml` before suggesting it
|
||||||
48
.agents/domains/knowledge/schema.md
Normal file
48
.agents/domains/knowledge/schema.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# Knowledge domain — schema
|
||||||
|
|
||||||
|
The knowledge domain is the durable, authoritative current-state documentation of the homelab: one
|
||||||
|
page per node and per cross-cutting system, synthesized from live state and evidence. It answers
|
||||||
|
"what exists and how does it work right now."
|
||||||
|
|
||||||
|
It follows the [LLM Wiki layer model](../../shared/llm-wiki.md) and the
|
||||||
|
[writing-style](../../shared/writing-style.md) and [page-templates](../../shared/page-templates.md)
|
||||||
|
rules.
|
||||||
|
|
||||||
|
## The narrative / substrate split
|
||||||
|
|
||||||
|
The knowledge wiki is **narrative**. It sits alongside a **machine-readable substrate** that it
|
||||||
|
describes but never contains. The split is load-bearing: several programs read the substrate at
|
||||||
|
fixed paths, so the wiki reorganization never moves it.
|
||||||
|
|
||||||
|
| Layer | Location | Consumed by |
|
||||||
|
|-------|----------|-------------|
|
||||||
|
| Substrate — source of truth | `inventory.yaml` (root) | MCP server, `homelab` CLI, `oikos/` scheduler/drift/relations/gen-topology |
|
||||||
|
| Substrate — generated host records | `inventory.yaml` (root) | Go `internal/mcp/` server, `bin/homelab`; the single source of truth |
|
||||||
|
| Substrate — kernel + context cards | `oikos/` (code, `oikos/cards/`, `oikos/state.json`) | MCP `explain`, scheduler |
|
||||||
|
| Narrative — synthesized wiki | `archive/knowledge/{hosts,containers,vms,infrastructure}/` | humans, agents via MCP `get_page` / `search_docs` |
|
||||||
|
| Evidence — immutable sources | `knowledge/sources/` (references + investigations) | synthesis into wiki pages |
|
||||||
|
|
||||||
|
## Wiki pages
|
||||||
|
|
||||||
|
- **Node pages** (`archive/knowledge/containers/<id>-<name>.md`, `.../vms/<id>-<name>.md`,
|
||||||
|
`.../hosts/<name>.md`) follow the container/host template in
|
||||||
|
[page-templates.md](../../shared/page-templates.md): opening definition, `## At a glance`,
|
||||||
|
`## Role`, service/port map, storage, auto-deploy, `## Related`, `## Changelog`.
|
||||||
|
- **Cross-cutting pages** (`archive/knowledge/infrastructure/<topic>.md`) follow the cross-cutting
|
||||||
|
template: `## Why`, `## Components`, `## How to apply`, `## Gotchas`, `## Related`, `## Changelog`.
|
||||||
|
- Each `inventory.yaml` host entry carries a `doc_page:` field pointing at its narrative page.
|
||||||
|
Changing where a page lives means updating that field (read by `bin/homelab`).
|
||||||
|
|
||||||
|
## The two logs
|
||||||
|
|
||||||
|
- The per-page **`## Changelog`** records infrastructure changes and is machine-parsed
|
||||||
|
(`get_changelog`, the Oikos ledger). Keep the `### YYYY-MM-DD — title` shape.
|
||||||
|
- **`knowledge/log.md`** is append-only and records *documentation-maintenance* operations only
|
||||||
|
(restructures, source ingests, lint sweeps): `## [YYYY-MM-DD] <op> | <summary>`. It never
|
||||||
|
duplicates the Oikos change ledger (`oikos/ledger.py`).
|
||||||
|
|
||||||
|
## Same-session update rule
|
||||||
|
|
||||||
|
A change to a node updates every page that references it in the same session — the node page, the
|
||||||
|
section `README.md` table, the root `README.md`, the Caddy/DNS/ingress pages, the host page, and
|
||||||
|
`inventory.yaml`. See [page-templates.md](../../shared/page-templates.md#same-session-update-rule).
|
||||||
55
.agents/domains/operations/schema.md
Normal file
55
.agents/domains/operations/schema.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Operations domain — schema
|
||||||
|
|
||||||
|
The operations domain holds the procedural and time-stamped documentation: runbooks (repeatable
|
||||||
|
procedures), investigations (incident evidence), and plans (design docs for non-trivial work). It
|
||||||
|
follows [writing-style](../../shared/writing-style.md); runbooks and plans use the imperative voice
|
||||||
|
exception.
|
||||||
|
|
||||||
|
Where each kind lives: runbooks are skills under [`.agents/skills/`](../../skills/); operator
|
||||||
|
reference (command cheatsheet, enrollment, Hermes agent) lives in
|
||||||
|
[`.agents/operations/`](../../operations/); investigations are sources under
|
||||||
|
`knowledge/sources/investigations/`; plans stay in the repo-root `plans/` folder (below).
|
||||||
|
|
||||||
|
## Plans always live in `plans/`
|
||||||
|
|
||||||
|
**Any plan or design doc for the Homelab is written into the repo `plans/` folder as
|
||||||
|
`plans/YYYY-MM-DD-slug.md` — never a scratch path, an agent-private plan location, or a chat
|
||||||
|
message.** An agent drafting a plan:
|
||||||
|
|
||||||
|
1. Writes the file under `plans/` using the plan template in [page-templates.md](../../shared/page-templates.md).
|
||||||
|
2. Lists it in `plans/index.md`.
|
||||||
|
3. On completion, moves it to `plans/done/` and updates the index status.
|
||||||
|
|
||||||
|
This is the single source for homelab design intent; keeping it in-repo means the plan is
|
||||||
|
versioned, reviewable, and reachable by MCP `get_page`/`search_docs` like any other doc.
|
||||||
|
|
||||||
|
## Runbooks
|
||||||
|
|
||||||
|
Repeatable procedures are skills — one folder per skill at `.agents/skills/<name>/SKILL.md`, with
|
||||||
|
YAML front-matter that the Oikos policy and lifecycle machinery reads:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
name: <name>
|
||||||
|
risk_class: read_only | reversible_low | config_mutation | destructive
|
||||||
|
inputs: [<param>, ...]
|
||||||
|
verification: "<shell expression that proves success>"
|
||||||
|
docs_update_checklist: [<doc artifacts to update>]
|
||||||
|
transition: "<from> -> <to>" # only for lifecycle runbooks
|
||||||
|
---
|
||||||
|
```
|
||||||
|
|
||||||
|
`risk_class` values and the lifecycle `transition` states must match
|
||||||
|
[`seeds/policy.yaml`](../../../seeds/policy.yaml) and [`seeds/ontology.yaml`](../../../seeds/ontology.yaml).
|
||||||
|
|
||||||
|
## Investigations
|
||||||
|
|
||||||
|
Incident records live in `knowledge/sources/investigations/YYYY-MM-DD-slug.md` and are **evidence sources** — written
|
||||||
|
once at incident time, then linked from the changelogs of the nodes they implicate. Sections:
|
||||||
|
`## Summary`, `## Timeline`, `## Root cause`, `## Mitigations applied`, `## Open questions`. Resolved
|
||||||
|
incidents move to `knowledge/sources/investigations/archive/`.
|
||||||
|
|
||||||
|
## The operations log
|
||||||
|
|
||||||
|
`plans/log.md` and `knowledge/log.md` are append-only records of documentation operations on
|
||||||
|
those areas (`## [YYYY-MM-DD] <op> | <summary>`), distinct from the Oikos change ledger.
|
||||||
@@ -6,8 +6,8 @@ this repo that auto-syncs every 5 min, a per-client age key for SOPS
|
|||||||
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
|
decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
|
||||||
|
|
||||||
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
|
> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
|
||||||
> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh`
|
> See [nomos-agent.md](nomos-agent.md). It uses the same `bootstrap.sh`
|
||||||
> with an additional `--with-hermes` flag.
|
> with an additional `--with-nomos` flag.
|
||||||
|
|
||||||
Architecture in [project_homelab_context_plan](https://… memory link); the
|
Architecture in [project_homelab_context_plan](https://… memory link); the
|
||||||
operational reference is here.
|
operational reference is here.
|
||||||
@@ -30,7 +30,7 @@ hostname doesn't match any inventory entry. Two fixes:
|
|||||||
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
|
- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
|
||||||
(Linux) or System Preferences → Sharing (macOS), then re-run.
|
(Linux) or System Preferences → Sharing (macOS), then re-run.
|
||||||
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
|
- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
|
||||||
regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
|
update `inventory.yaml`, push. The next sync (≤5 min) propagates.
|
||||||
|
|
||||||
### Getting onto Netbird
|
### Getting onto Netbird
|
||||||
|
|
||||||
@@ -40,7 +40,7 @@ Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't alread
|
|||||||
|
|
||||||
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
|
||||||
|
|
||||||
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account.
|
Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../../archive/knowledge/containers/106-auth-outpost.md). Only needed once per account.
|
||||||
|
|
||||||
**Path A — setup-key (headless/scripted onboarding):**
|
**Path A — setup-key (headless/scripted onboarding):**
|
||||||
|
|
||||||
@@ -62,7 +62,7 @@ Useful for headless servers (no browser at all) or unattended cloud-init bootstr
|
|||||||
### DNS prerequisite
|
### DNS prerequisite
|
||||||
|
|
||||||
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
|
||||||
([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the
|
([dns.md](../../archive/knowledge/infrastructure/dns.md)) for LAN clients, **but only if the
|
||||||
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
|
||||||
workstations don't by default. Options:
|
workstations don't by default. Options:
|
||||||
|
|
||||||
@@ -81,7 +81,7 @@ If DNS isn't an option at all, override the URLs at bootstrap time:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo HOMELAB_GITEA_TOKEN=... \
|
sudo HOMELAB_GITEA_TOKEN=... \
|
||||||
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
|
HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git \
|
||||||
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
|
HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
|
||||||
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
|
HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
|
||||||
bash /tmp/bootstrap.sh --with-mcp
|
bash /tmp/bootstrap.sh --with-mcp
|
||||||
@@ -129,7 +129,7 @@ TOKEN=... # your Gitea PAT, scope read:repository
|
|||||||
|
|
||||||
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
|
# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
|
||||||
curl -fsSL -u "dtoro:$TOKEN" \
|
curl -fsSL -u "dtoro:$TOKEN" \
|
||||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
|
||||||
-o /tmp/bootstrap.sh
|
-o /tmp/bootstrap.sh
|
||||||
|
|
||||||
# Run it.
|
# Run it.
|
||||||
@@ -261,7 +261,7 @@ arguments.
|
|||||||
|
|
||||||
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
|
If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
|
||||||
visible to the classifier (it's not actually durable in 0.71.2, but the
|
visible to the classifier (it's not actually durable in 0.71.2, but the
|
||||||
ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
|
ControlMaster block is — see [runbook-dpkg-interrupted](../skills/runbook-dpkg-interrupted/SKILL.md)
|
||||||
for context), drop a free-text rule into `autoMode.allow` describing the
|
for context), drop a free-text rule into `autoMode.allow` describing the
|
||||||
authorization. Optional.
|
authorization. Optional.
|
||||||
|
|
||||||
@@ -285,7 +285,7 @@ homelab client add my-new-machine
|
|||||||
# 4. On hubris: finalize the age public key.
|
# 4. On hubris: finalize the age public key.
|
||||||
homelab client add my-new-machine --finalize-pubkey age1...
|
homelab client add my-new-machine --finalize-pubkey age1...
|
||||||
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
|
# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
|
||||||
# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
|
# inventory.yaml, commits + pushes. The 5-min sync propagates.
|
||||||
```
|
```
|
||||||
|
|
||||||
## Granting a secret to a new client
|
## Granting a secret to a new client
|
||||||
@@ -333,7 +333,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
|
| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
|
||||||
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
|
| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
|
||||||
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
|
| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/oikos.git` |
|
||||||
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
|
| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
|
||||||
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
|
| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
|
||||||
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
|
| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
|
||||||
@@ -342,7 +342,7 @@ The CLI prints a follow-up checklist that the operator must do manually:
|
|||||||
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
|
||||||
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
|
||||||
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
|
||||||
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |
|
| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../../archive/knowledge/containers/106-auth-outpost.md). |
|
||||||
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
|
||||||
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
|
||||||
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
|
||||||
@@ -354,9 +354,9 @@ Added a new "Post-bootstrap: SSH reachability" section covering SSH key
|
|||||||
generation, pubkey publication, deployment to hosts, SSH config generation,
|
generation, pubkey publication, deployment to hosts, SSH config generation,
|
||||||
and LAN IP registration. New workstations enrolled via this doc will
|
and LAN IP registration. New workstations enrolled via this doc will
|
||||||
automatically join the universal SSH mesh.
|
automatically join the universal SSH mesh.
|
||||||
|
### 2026-05-31 — cross-link to nomos-agent.md
|
||||||
|
|
||||||
### 2026-05-31 — cross-link to hermes-agent.md
|
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([nomos-agent.md](nomos-agent.md)) and noted it at the top of this page. The Nomos flow extends `bootstrap.sh` with `--with-nomos` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
|
||||||
Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
|
|
||||||
|
|
||||||
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
|
### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
|
||||||
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
|
Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
|
||||||
91
.agents/operations/commands.md
Normal file
91
.agents/operations/commands.md
Normal file
@@ -0,0 +1,91 @@
|
|||||||
|
# Operations cheatsheet
|
||||||
|
|
||||||
|
Run from the [hubris host](../../archive/knowledge/hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
|
||||||
|
|
||||||
|
## Proxmox CLI
|
||||||
|
|
||||||
|
| Command | Use |
|
||||||
|
| --- | --- |
|
||||||
|
| `pct list` / `qm list` | List LXC containers / VMs |
|
||||||
|
| `pct config <id>` / `qm config <id>` | Container / VM config |
|
||||||
|
| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../../archive/knowledge/infrastructure/media-permissions.md)) |
|
||||||
|
| `pct enter <id>` | Shell into a container |
|
||||||
|
| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
|
||||||
|
| `pvesm status` | Storage pools status |
|
||||||
|
| `pvesh get /nodes --output-format json` | Node summary as JSON |
|
||||||
|
| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
|
||||||
|
| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by the `homelab-health-watchdog` Nomos cron — see [monitoring](../../archive/knowledge/infrastructure/monitoring.md); the old `claudio-monitor` this once fed is deprecated) |
|
||||||
|
| `pveversion` | PVE version |
|
||||||
|
| `journalctl -u pve-cluster -n 100` | PVE service logs |
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
|
||||||
|
- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
|
||||||
|
- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../../archive/knowledge/infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
|
||||||
|
|
||||||
|
## Reverse proxy
|
||||||
|
|
||||||
|
- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../../archive/knowledge/containers/121-caddy.md).
|
||||||
|
- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
|
||||||
|
- Hot reload: `pct exec 121 -- systemctl reload caddy`.
|
||||||
|
- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
|
||||||
|
- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
|
||||||
|
|
||||||
|
## DNS
|
||||||
|
|
||||||
|
- Split-horizon authority: [Technitium DNS](https://technitium.com) on [dns (107)](../../archive/knowledge/containers/107-dns.md) at `192.168.8.2:53`. Web UI at `http://192.168.8.2`. (Formerly dnsmasq on the now-destroyed LXC 124 — decommissioned 2026-06-04.)
|
||||||
|
- Add/edit records in the Technitium UI; the NetBird managed zone sync (`scripts/dns-sync.py` cron on 107) picks changes up within ~10 minutes.
|
||||||
|
- Verify: `dig @192.168.8.2 +short <host>.hubris.network`.
|
||||||
|
- See [DNS](../../archive/knowledge/infrastructure/dns.md).
|
||||||
|
|
||||||
|
## Web access
|
||||||
|
|
||||||
|
- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
|
||||||
|
|
||||||
|
## Telemetry quick checks
|
||||||
|
|
||||||
|
- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
|
||||||
|
- `ras-mc-ctl --errors` — full event log
|
||||||
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
|
||||||
|
- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
|
||||||
|
- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../../archive/knowledge/investigations/archive/2026-04-21-hubris-crash-loop.md))
|
||||||
|
|
||||||
|
## Fleet apt operations
|
||||||
|
|
||||||
|
Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
| --- | --- |
|
||||||
|
| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
|
||||||
|
| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
|
||||||
|
| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
|
||||||
|
| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
|
||||||
|
| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
|
||||||
|
| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
|
||||||
|
|
||||||
|
PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
|
||||||
|
|
||||||
|
## Oikos (agent OS layer)
|
||||||
|
|
||||||
|
See [OIKOS.md](../OIKOS.md) for the operating model. Quick reference:
|
||||||
|
|
||||||
|
| Command | What it does |
|
||||||
|
| --- | --- |
|
||||||
|
| `homelab service <name> explain\|health\|docs\|log\|actions\|history` | Service Console v0 — context card, cached health (`--live` to force a probe), docs, logs, safe actions + risk class, ledger history |
|
||||||
|
| `homelab node <name> relations` | Ontology blast-radius query: what this host/service impacts, is affected by, and its full transitive blast radius |
|
||||||
|
| `homelab change preflight <service>` | Dry-run report before mutating: risk class, current health, config repo, verification command |
|
||||||
|
| `homelab decide <action> <entity>` | Decision classifier: risk × blast radius × confidence → auto-act or escalate |
|
||||||
|
| `homelab signal list\|raise\|ack\|resolve\|mute` | The attention layer — pending updates, thresholds, drift, anything needing attention |
|
||||||
|
| `homelab approval request\|list\|reply\|check` | Escalate-route grants (Matrix-delivered via Nomos, or the Oikos Console's `/approvals` page) |
|
||||||
|
| `homelab restart <service> [--approval-id <id>]` | `--approval-id` is required whenever the service's risk class needs approval (e.g. `caddy`, `dns`) — refuses mechanically without a valid grant |
|
||||||
|
|
||||||
|
Oikos Console (read-mostly dashboard): `oikos.hubris.network` once deployed — see [oikos/console/deploy/README.md](../../archive/oikos-cards/).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [Hubris host](../../archive/knowledge/hosts/hubris.md)
|
||||||
|
- [Containers index](../../archive/knowledge/containers/index.md)
|
||||||
|
- [DNS](../../archive/knowledge/infrastructure/dns.md)
|
||||||
|
- [Monitoring](../../archive/knowledge/infrastructure/monitoring.md)
|
||||||
|
- [Auto-deploy](../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||||
|
- [Runbook: dpkg-interrupted recovery](../skills/runbook-dpkg-interrupted/SKILL.md) — what to do when apt got killed mid-transaction
|
||||||
@@ -1,29 +1,29 @@
|
|||||||
# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
|
# Nomos agent — LLM-powered terminal sessions on a homelab client
|
||||||
|
|
||||||
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
|
Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
|
||||||
Llama variant) as a working terminal agent on a homelab client. Builds on top
|
Llama variant) as a working terminal agent on a homelab client. Builds on top
|
||||||
of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md))
|
of standard client enrollment (see [agent-enrollment.md](agent-enrollment.md))
|
||||||
— this page covers only the Hermes-specific additions.
|
— this page covers only the Hermes-specific additions.
|
||||||
|
|
||||||
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
|
The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
|
||||||
|
|
||||||
- The chat loop, multi-turn history, and streaming
|
- The chat loop, multi-turn history, and streaming
|
||||||
- The OpenRouter provider that routes to Nous Hermes
|
- The OpenRouter provider that routes to the configured LLM
|
||||||
- The built-in `developer` extension (shell + file editor — same surface Claude
|
- The built-in `developer` extension (shell + file editor — same surface Claude
|
||||||
Code has)
|
Code has)
|
||||||
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
|
- A remote MCP extension pointed at `mcp.hubris.network` for read-only
|
||||||
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
|
homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
|
||||||
|
|
||||||
The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
|
The persona is `/opt/homelab-context/NOMOS.md`, symlinked as Goose's global
|
||||||
`.goosehints` so it's injected into the system prompt on every session.
|
`.goosehints` so it's injected into the system prompt on every session.
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
| Requirement | How |
|
| Requirement | How |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) |
|
| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](agent-enrollment.md) |
|
||||||
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
|
| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
|
||||||
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
|
| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` |
|
||||||
|
|
||||||
## Onboarding flow
|
## Onboarding flow
|
||||||
|
|
||||||
@@ -33,37 +33,37 @@ homelab client add new-machine
|
|||||||
|
|
||||||
# 2. Join new-machine to Netbird (setup-key or OIDC).
|
# 2. Join new-machine to Netbird (setup-key or OIDC).
|
||||||
|
|
||||||
# 3. On new-machine: bootstrap with --with-hermes.
|
# 3. On new-machine: bootstrap with --with-nomos.
|
||||||
TOKEN=... # gitea PAT, read:repository
|
TOKEN=... # gitea PAT, read:repository
|
||||||
curl -fsSL -u "dtoro:$TOKEN" \
|
curl -fsSL -u "dtoro:$TOKEN" \
|
||||||
https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
|
https://git.hubris.network/dtoro/oikos/raw/branch/main/bootstrap.sh \
|
||||||
-o /tmp/bootstrap.sh
|
-o /tmp/bootstrap.sh
|
||||||
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
|
sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-nomos
|
||||||
|
|
||||||
# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
|
# 4. Back on hubris: finalize the age pubkey AND grant the Nomos secret.
|
||||||
homelab client add new-machine \
|
homelab client add new-machine \
|
||||||
--finalize-pubkey age1... \
|
--finalize-pubkey age1... \
|
||||||
--with-hermes
|
--with-nomos
|
||||||
|
|
||||||
# 5. Wait ≤5 min for sync, then on new-machine:
|
# 5. Wait ≤5 min for sync, then on new-machine:
|
||||||
hermes "what LXCs are running?"
|
nomos "what LXCs are running?"
|
||||||
```
|
```
|
||||||
|
|
||||||
The bootstrap `--with-hermes` flag does five things, all idempotent:
|
The bootstrap `--with-nomos` flag does five things, all idempotent:
|
||||||
|
|
||||||
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
|
1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
|
||||||
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
|
(upstream installer) and symlinks `/usr/local/bin/goose` to it.
|
||||||
2. Symlinks `/opt/homelab-context/bin/hermes` → `/usr/local/bin/hermes`.
|
2. Symlinks `/opt/homelab-context/bin/nomos` → `/usr/local/bin/nomos`.
|
||||||
3. Symlinks `/opt/homelab-context/HERMES.md` → `/root/HERMES.md` (Linux) or
|
3. Symlinks `/opt/homelab-context/NOMOS.md` → `/root/NOMOS.md` (Linux) or
|
||||||
`/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
|
`/etc/NOMOS.md` (macOS) for `cat`-as-operator convenience.
|
||||||
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
|
4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
|
||||||
extensions (preserves any keys the operator added by hand).
|
extensions (preserves any keys the operator added by hand).
|
||||||
5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
|
5. Symlinks `~/.config/goose/.goosehints` → NOMOS.md, so the persona is
|
||||||
injected as the system prompt on every session.
|
injected as the system prompt on every session.
|
||||||
|
|
||||||
## Seeding the OpenRouter key
|
## Seeding the OpenRouter key
|
||||||
|
|
||||||
The first time anyone enrolls with `--with-hermes`, the encrypted file
|
The first time anyone enrolls with `--with-nomos`, the encrypted file
|
||||||
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
|
`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
|
||||||
existing recipient):
|
existing recipient):
|
||||||
|
|
||||||
@@ -75,19 +75,19 @@ git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
|
|||||||
git -C /opt/homelab-context push
|
git -C /opt/homelab-context push
|
||||||
```
|
```
|
||||||
|
|
||||||
Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
|
Until this step happens, `nomos …` exits with `openrouter-api-key.yaml still
|
||||||
contains the placeholder`. Subsequent enrollees get the real key automatically
|
contains the placeholder`. Subsequent enrollees get the real key automatically
|
||||||
via `--with-hermes` (which adds them as a sops recipient on
|
via `--with-nomos` (which adds them as a sops recipient on
|
||||||
`secrets/openrouter-api-key.yaml`).
|
`secrets/openrouter-api-key.yaml`).
|
||||||
|
|
||||||
## Granting the OpenRouter key to an already-enrolled host
|
## Granting the OpenRouter key to an already-enrolled host
|
||||||
|
|
||||||
If a host was enrolled without `--with-hermes` and you want to add it later:
|
If a host was enrolled without `--with-nomos` and you want to add it later:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# On hubris:
|
# On hubris:
|
||||||
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
|
PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
|
||||||
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
|
homelab client add <host> --finalize-pubkey "$PUBKEY" --with-nomos
|
||||||
```
|
```
|
||||||
|
|
||||||
`--finalize-pubkey` is required by the existing flow even when the pubkey is
|
`--finalize-pubkey` is required by the existing flow even when the pubkey is
|
||||||
@@ -101,12 +101,12 @@ re-run; only the secret recipient list changed.
|
|||||||
```bash
|
```bash
|
||||||
homelab whoami # standard enrollment OK
|
homelab whoami # standard enrollment OK
|
||||||
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
|
homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
|
||||||
which goose && which hermes # binaries present
|
which goose && which nomos # binaries present
|
||||||
goose info -v # provider/model wiring sane
|
goose info -v # provider/model wiring sane
|
||||||
hermes "what LXCs are running?" # interactive Goose session
|
nomos "what LXCs are running?" # interactive Goose session
|
||||||
|
|
||||||
# Non-interactive smoke test:
|
# Non-interactive smoke test:
|
||||||
echo "List the homelab MCP tools you have available" | hermes
|
echo "List the homelab MCP tools you have available" | nomos
|
||||||
```
|
```
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
@@ -135,9 +135,9 @@ extensions:
|
|||||||
Override via env on a single bootstrap run:
|
Override via env on a single bootstrap run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
|
HOMELAB_NOMOS_MODEL=nousresearch/hermes-3-llama-3.1-405b \
|
||||||
HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
|
HOMELAB_NOMOS_MCP_URI=https://mcp.hubris.network/mcp \
|
||||||
sudo bash /tmp/bootstrap.sh --with-hermes
|
sudo bash /tmp/bootstrap.sh --with-nomos
|
||||||
```
|
```
|
||||||
|
|
||||||
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
|
Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
|
||||||
@@ -156,33 +156,33 @@ every tool call, use `approve`. See
|
|||||||
|
|
||||||
| Symptom | Cause | Fix |
|
| Symptom | Cause | Fix |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
|
| `nomos: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-nomos` from hubris |
|
||||||
| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
|
| `nomos: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
|
||||||
| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
|
| Goose hangs on first `nomos` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
|
||||||
| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py` — `mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
|
| `homelab` extension fails to connect / no MCP tools listed | MCP server upgraded in Go rewrite (`internal/mcp/server.go`, Streamable HTTP via official MCP SDK). Old FastMCP SSE transport is deprecated. | Run `docker compose --profile full up` on mac-mini, or wait for the production cutover from apps/105. |
|
||||||
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
|
| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-nomos`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
|
||||||
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
|
| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
|
||||||
|
|
||||||
## Cross-references
|
## Cross-references
|
||||||
|
|
||||||
- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the
|
- [agent-enrollment.md](agent-enrollment.md) — base client onboarding the
|
||||||
Hermes flow assumes is done.
|
Nomos flow assumes is done.
|
||||||
- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
|
- [`NOMOS.md`](../NOMOS.md) — the persona the Nomos agent reads on every
|
||||||
session start (via `~/.config/goose/.goosehints`).
|
session start (via `~/.config/goose/.goosehints`).
|
||||||
- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key
|
- [`bin/nomos`](../../bin/nomos) — the wrapper that decrypts the OpenRouter key
|
||||||
and execs `goose session`.
|
and execs `goose session`.
|
||||||
- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block.
|
- [`bootstrap.sh`](../../bootstrap.sh) — the `--with-nomos` flag's install block.
|
||||||
|
|
||||||
## Follow-ups
|
## Follow-ups
|
||||||
|
|
||||||
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
|
1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
|
||||||
(`"SSE transport is no longer supported - kept only for config file
|
(`"SSE transport is no longer supported - kept only for config file
|
||||||
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
|
compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
|
||||||
server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
|
server at `internal/mcp/server.go` uses Streamable HTTP (official MCP SDK). Until
|
||||||
that's changed, the `homelab` MCP extension in Goose will fail to connect.
|
that's changed, the `homelab` MCP extension in Goose will fail to connect.
|
||||||
The developer extension (shell + edit) covers most ops without it; this is
|
The developer extension (shell + edit) covers most ops without it; this is
|
||||||
a polish item, not a blocker.
|
a polish item, not a blocker.
|
||||||
2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
|
2. **Per-host OpenRouter keys** for billing attribution. Today all Nomos
|
||||||
hosts share one key.
|
hosts share one key.
|
||||||
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
|
3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
|
||||||
directly — OpenRouter periodically rotates the underlying weights.
|
directly — OpenRouter periodically rotates the underlying weights.
|
||||||
@@ -204,7 +204,7 @@ templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
|
|||||||
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
|
efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
|
||||||
Also created `tools/caveman/` with the wrapper script, JS renderer, and
|
Also created `tools/caveman/` with the wrapper script, JS renderer, and
|
||||||
templates — the canonical source for all agent hosts.
|
templates — the canonical source for all agent hosts.
|
||||||
Captures the Hermes-on-Goose onboarding flow added in the same commit as
|
Captures the Nomos-on-Goose onboarding flow added in the same commit as
|
||||||
`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
|
`bootstrap.sh --with-nomos`, `bin/nomos`, the sops rule for
|
||||||
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
|
`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-nomos`
|
||||||
extension. MCP streamable_http migration is queued as follow-up #1.
|
extension. MCP streamable_http migration is queued as follow-up #1.
|
||||||
33
.agents/shared/caveman.md
Normal file
33
.agents/shared/caveman.md
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
# CAVEMAN.md — communication mode for homelab agents
|
||||||
|
|
||||||
|
Respond terse like smart caveman. All technical substance stay. Only fluff die.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
|
||||||
|
|
||||||
|
Pattern: `[thing] [action] [reason]. [next step].`
|
||||||
|
|
||||||
|
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
|
||||||
|
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
|
||||||
|
|
||||||
|
## Levels
|
||||||
|
|
||||||
|
- **lite** — no filler/hedging. Keep articles + full sentences. Professional but tight.
|
||||||
|
- **full** (default) — drop articles, fragments OK, short synonyms. Classic caveman.
|
||||||
|
- **ultra** — abbreviate prose words (DB/auth/config/req/res), strip conjunctions, arrows (X → Y). Code symbols/API names/errors: never abbreviate.
|
||||||
|
|
||||||
|
Switch: `/caveman lite|full|ultra`. Stop: "normal mode".
|
||||||
|
|
||||||
|
## Auto-Clarity
|
||||||
|
|
||||||
|
Drop caveman for: security warnings, irreversible actions, multi-step sequences where fragments risk misread, user confused/repeating. Resume after clear part done.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Source: https://github.com/JuliusBrussee/caveman
|
||||||
|
Copy to `~/.nomos/skills/` for Nomos agent, or `~/.claude/projects/<name>/SKILL.md` for Claude Code.
|
||||||
41
.agents/shared/llm-wiki.md
Normal file
41
.agents/shared/llm-wiki.md
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
# LLM Wiki — the documentation contract
|
||||||
|
|
||||||
|
How the narrative documentation in this repo is organized. The pattern is borrowed from the
|
||||||
|
`sources / wiki / index / log` model: a durable synthesized layer (`archive/knowledge/`) built on top
|
||||||
|
of immutable evidence (`knowledge/sources/`, incident records), with pure-listing indexes and an
|
||||||
|
append-only operations log.
|
||||||
|
|
||||||
|
This contract governs the **narrative layer only**. The machine-readable substrate — `inventory.yaml`,
|
||||||
|
`secrets/`, `scripts/`, `bin/` — is not part of the wiki and never
|
||||||
|
moves under it. See [the knowledge schema](../domains/knowledge/schema.md) for the split.
|
||||||
|
|
||||||
|
## Layers
|
||||||
|
|
||||||
|
- **Sources** are immutable raw material: incident records (`knowledge/sources/investigations/`), external reference
|
||||||
|
docs (`knowledge/sources/references/`), and the live system itself (`pct config`, `docker inspect`).
|
||||||
|
Read them; do not rewrite them into other sources.
|
||||||
|
- **Wiki** (`archive/knowledge/`) is the synthesized, authoritative current-state layer: one page per
|
||||||
|
node (`containers/`, `vms/`, host narratives) and per cross-cutting system (`infrastructure/`). A
|
||||||
|
reader understands the topic from the wiki page without reading the sources.
|
||||||
|
- **Index** (`index.md` / folder `README.md`) is a pure listing — every page in scope with a
|
||||||
|
one-line summary, and nothing else. Anything the section wants to say up front goes into a page
|
||||||
|
the index lists, not into the index.
|
||||||
|
- **Log** (`log.md`) is append-only, recording *doc-maintenance operations* (restructures, source
|
||||||
|
ingests, lint sweeps) in single-line format: `## [YYYY-MM-DD] <op> | <summary>`.
|
||||||
|
|
||||||
|
## Two logs, kept distinct
|
||||||
|
|
||||||
|
- **`## Changelog`** on each node/topic page records *infrastructure* changes to that node. It is
|
||||||
|
machine-parsed (`get_changelog`, the Oikos ledger) — keep the `### YYYY-MM-DD — title` shape.
|
||||||
|
- **`log.md`** per area records *documentation* operations only. It never duplicates the Oikos
|
||||||
|
change ledger (`oikos/ledger.py`), which stays authoritative for infra changes with
|
||||||
|
who/what/risk/approval/verification.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
- Wiki pages stay short and focused. A page past ~300 lines splits.
|
||||||
|
- Pages stay flat under `wiki/<section>/` until there are enough to warrant a sub-group.
|
||||||
|
- Every page follows [writing-style.md](writing-style.md).
|
||||||
|
- Plans and design docs always live in the repo `plans/` folder (`plans/YYYY-MM-DD-slug.md`),
|
||||||
|
listed in `plans/index.md`, moved to `plans/done/` on completion — never a scratch path or a chat
|
||||||
|
message. See [the operations schema](../domains/operations/schema.md).
|
||||||
174
.agents/shared/page-templates.md
Normal file
174
.agents/shared/page-templates.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
# Page templates for the Homelab Wiki
|
||||||
|
|
||||||
|
The structural templates for each page type. Prose voice, vocabulary, and cross-reference rules live
|
||||||
|
in [writing-style.md](writing-style.md); the layer model (sources / wiki / index / log) lives in
|
||||||
|
[llm-wiki.md](llm-wiki.md).
|
||||||
|
|
||||||
|
## File naming
|
||||||
|
|
||||||
|
**Foundational / entry-point files:** ALL-CAPS
|
||||||
|
|
||||||
|
- **Root level:** `AGENTS.md`, `README.md` — discovery paths for agents and humans.
|
||||||
|
- **Agent instruction** (under `.agents/`): `OIKOS.md`, `HERMES.md` — foundational docs agents read before acting.
|
||||||
|
- **Reference docs:** `GLOSSARY.md` — lookup reference (like classic repo conventions: LICENSE, CHANGELOG, GLOSSARY).
|
||||||
|
|
||||||
|
**Content / narrative pages:** lowercase-with-dashes, date-prefixed as needed
|
||||||
|
|
||||||
|
- **Container pages:** `<id>-<name>.md` (e.g. `101-jellyfin.md`, `132-rclone.md`). The `<id>` is the LXC/VM ordinal from `inventory.yaml`.
|
||||||
|
- **Infrastructure / cross-cutting pages:** `<topic>.md` (e.g. `dns.md`, `auto-deploy.md`, `mesh.md`). Describes a system, not a specific node.
|
||||||
|
- **Plans / investigations:** `YYYY-MM-DD-<slug>.md` (e.g. `2026-07-05-oikos-prometheus-lxc.md`). Date-sorted; slug is lowercase.
|
||||||
|
- **Section indices:** `README.md` (lowercase, conventional). Prefer in folders; `index.md` only if both intro prose and listing coexist.
|
||||||
|
|
||||||
|
**Skills / runbooks:** special case
|
||||||
|
|
||||||
|
- **Folder structure:** `<name>/SKILL.md` where `<name>` is lowercase-with-dashes (e.g. `client-enrollment/SKILL.md`).
|
||||||
|
- **The filename SKILL.md is always uppercase** — it acts as a signpost so tools and humans instantly recognize it as a skill.
|
||||||
|
|
||||||
|
**General rules:** All paths use lowercase letters, numbers, and hyphens (no underscores). Uppercase is reserved for foundational docs (entry points + instruction) and filenames that signify document type (SKILL.md, GLOSSARY.md, etc.).
|
||||||
|
|
||||||
|
## Voice
|
||||||
|
|
||||||
|
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. Full rules in
|
||||||
|
[writing-style.md](writing-style.md).
|
||||||
|
|
||||||
|
## Page templates
|
||||||
|
|
||||||
|
### Container page (`containers/<id>-<name>.md`)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# <id> — `<name>`
|
||||||
|
|
||||||
|
One-sentence purpose.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
- **Hostname:** `<name>`
|
||||||
|
- **IP:** `192.168.8.x`
|
||||||
|
- **Privilege:** privileged | unprivileged
|
||||||
|
- **Resources:** N cores / M GiB RAM / D GiB rootfs
|
||||||
|
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any)
|
||||||
|
- **Public hostname:** `<sub>.hubris.network` (if proxied)
|
||||||
|
|
||||||
|
## Role
|
||||||
|
What it does, what it talks to.
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|
||||||
|
## Storage / config paths
|
||||||
|
|
||||||
|
## Auto-deploy
|
||||||
|
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [Caddy](121-caddy.md) (if proxied)
|
||||||
|
- [DNS](../infrastructure/dns.md) (if has subdomain)
|
||||||
|
- [Authentik](124-authentik.md) (if SSO)
|
||||||
|
- ...
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
### YYYY-MM-DD — short title
|
||||||
|
What changed, why, link to investigation if any.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Cross-cutting page (`infrastructure/<topic>.md`)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# <Topic>
|
||||||
|
|
||||||
|
One-sentence summary.
|
||||||
|
|
||||||
|
## Why
|
||||||
|
Design rationale — what it replaces, what it solves.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
Where it runs, what files matter.
|
||||||
|
|
||||||
|
## How to apply / use
|
||||||
|
Recipes.
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
## Related
|
||||||
|
Links to nodes that host or depend on this.
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
```
|
||||||
|
|
||||||
|
### Plan (`plans/YYYY-MM-DD-slug.md`)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# YYYY-MM-DD — <title>
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
What this change achieves and why.
|
||||||
|
|
||||||
|
## Current topology / state
|
||||||
|
Diagram or description of what exists now.
|
||||||
|
|
||||||
|
## Target topology / state
|
||||||
|
What it looks like after.
|
||||||
|
|
||||||
|
## Pre-flight checklist
|
||||||
|
|
||||||
|
## Step-by-step procedure
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
## Post-migration
|
||||||
|
Changelog entries to write, index status to update.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Investigation (`knowledge/sources/investigations/YYYY-MM-DD-slug.md`)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# YYYY-MM-DD — <title>
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
1-3 sentences.
|
||||||
|
|
||||||
|
## Timeline
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
## Mitigations applied
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
```
|
||||||
|
|
||||||
|
## Linking discipline
|
||||||
|
|
||||||
|
- Every container page links to every cross-cutting page it participates in.
|
||||||
|
- Every cross-cutting page lists the nodes that participate.
|
||||||
|
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
|
||||||
|
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
|
||||||
|
|
||||||
|
## Changelog hygiene
|
||||||
|
|
||||||
|
- Reverse-chronological (newest first).
|
||||||
|
- One entry per discrete change, even if you make several in one day.
|
||||||
|
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
|
||||||
|
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
|
||||||
|
|
||||||
|
## Same-session update rule
|
||||||
|
|
||||||
|
When you make a change to a node — migrate an LXC, update an IP, change a
|
||||||
|
mount, deploy a new service — **update every relevant doc page in the same
|
||||||
|
session.** A change that touches a container page must also update:
|
||||||
|
|
||||||
|
- The `containers/index.md` table (IPs, host, mounts, status)
|
||||||
|
- The `README.md` table (if the change affects listed columns)
|
||||||
|
- The Caddy page site list (if the change affects `*.hubris.network` routing)
|
||||||
|
- The DNS / ingress infrastructure pages (if the change affects routing)
|
||||||
|
- The `hosts/{hubris,strong}.md` host page (if container count changes)
|
||||||
|
- The `inventory.yaml` host entry (single source of truth)
|
||||||
|
- The `infrastructure/topology.md` (generated from inventory, but regen if needed)
|
||||||
|
|
||||||
|
The pattern of updating only one page and leaving stale references on others
|
||||||
|
is a bug. If you're doing a multi-step migration, document the intermediate
|
||||||
|
state with a changelog entry that says "pending — will finalize after Phase
|
||||||
|
N."
|
||||||
|
|
||||||
|
This rule is why Phase 2 of the strong migration (2026-07-05) caused
|
||||||
|
widespread stale data: individual container pages were updated in the
|
||||||
|
changelog but never had their At-a-glance sections, IPs, mount paths, or
|
||||||
|
host attribution updated. Don't repeat that.
|
||||||
75
.agents/shared/writing-style.md
Normal file
75
.agents/shared/writing-style.md
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
# Writing Style
|
||||||
|
|
||||||
|
Write like a technical reference, not a marketing page. Every sentence conveys new information.
|
||||||
|
These rules govern **committed documentation** — wiki pages, READMEs, schemas, skills, `AGENTS.md`,
|
||||||
|
plans, investigations, and code comments. They are separate from [caveman.md](caveman.md), which
|
||||||
|
governs an agent's *chat responses*; the two do not conflict.
|
||||||
|
|
||||||
|
New or rewritten pages follow these patterns from day one. Existing pages get updated the next time
|
||||||
|
they are touched.
|
||||||
|
|
||||||
|
## Vocabulary — never use these
|
||||||
|
|
||||||
|
- Significance puffers: "pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament", "paramount", "invaluable".
|
||||||
|
- Analytical verbs: "delve", "leverage", "utilize", "facilitate", "foster", "showcase", "underscore", "streamline", "harness".
|
||||||
|
- Poetic nouns: "tapestry", "landscape" (figurative), "realm", "paradigm", "ecosystem" (figurative), "journey" (figurative), "nexus", "cornerstone".
|
||||||
|
- Promotional adjectives: "robust", "seamless", "innovative", "cutting-edge", "meticulous", "holistic", "comprehensive".
|
||||||
|
- Opening crutches: "In today's world", "In the ever-evolving landscape of", "It's worth noting that", "It is important to note that".
|
||||||
|
|
||||||
|
Use short, common words: "use" not "utilize", "help" not "facilitate", "show" not "demonstrate".
|
||||||
|
|
||||||
|
## Voice
|
||||||
|
|
||||||
|
Describe what systems do and how they work.
|
||||||
|
|
||||||
|
- **Reference prose** (node pages, cross-cutting infrastructure descriptions, `## Role`, `## Why`,
|
||||||
|
`At a glance`) is third-person: state facts about the system, not instructions to a reader.
|
||||||
|
- **Recipes, runbooks, and skills** are the exception: second-person imperative is allowed and
|
||||||
|
preferred where it makes a procedure clearer ("Edit the Caddyfile, commit + push", "Verify with
|
||||||
|
`dig +short`"). This matches how the operator actually works. The vocabulary, structure, and
|
||||||
|
cross-reference rules below still apply.
|
||||||
|
|
||||||
|
## Page shape
|
||||||
|
|
||||||
|
Every doc-level page follows the same shape so a reader scans it in one pass.
|
||||||
|
|
||||||
|
1. **One H1 = the page title.** Node pages use `# <id> — \`<name>\``; topic pages use `# <Topic>`.
|
||||||
|
2. **Opening definition.** First paragraph, 1–3 sentences, says what the thing is. No motivation, no marketing, no setup.
|
||||||
|
3. **Body sections** in the natural order for the topic. Reuse the section templates in [page-templates.md](page-templates.md).
|
||||||
|
4. **`## Changelog`** at the bottom of every node/topic page — reverse-chronological, append-only. This section is machine-parsed (Go MCP `get_changelog` in `internal/mcp/server.go`); keep the `### YYYY-MM-DD — title` shape.
|
||||||
|
5. **Related links** only at the bottom, only when a reference cannot be woven inline.
|
||||||
|
|
||||||
|
## Section indexes (folder READMEs)
|
||||||
|
|
||||||
|
A folder's `README.md` opens with a 1–3 sentence prose intro that says what the section covers, then
|
||||||
|
a single navigation table — `| Document | What it covers |` — and nothing else. No stale counts, no
|
||||||
|
duplicated prose, no narrative between the intro and the table.
|
||||||
|
|
||||||
|
## Structure rules
|
||||||
|
|
||||||
|
- Make every sentence information-dense. Cut filler, qualifiers, and setup phrases. Lead with the concrete fact or action, not why it matters.
|
||||||
|
- No participial tack-ons (", highlighting the importance of…"). If the clause adds information, make it a separate sentence.
|
||||||
|
- **No meta-commentary about the content itself.** Do not narrate the page's own structure or linking strategy.
|
||||||
|
- Prefer **tables** for enumerable items with internal structure (service/port maps, field lists, status grids). Reserve bullets for short non-structured lists.
|
||||||
|
- Use the **bold-leading-phrase pattern** for structured points: `**Read-only by construction.** The MCP server never mutates state.` — a bold noun phrase, a period, then the explanation.
|
||||||
|
- When enumerating across services or nodes, give each its own `###` sub-section or a table row, not one run-on paragraph.
|
||||||
|
- Use backticks for code, paths, hostnames, and file names (`inventory.yaml`, `192.168.8.77`, `pct config`); italics for first-mention terminology.
|
||||||
|
- Use `>` blockquotes for caveats and gaps that interrupt the main flow: `> **Outstanding gap.** DNS-vs-inventory drift check not yet wired.` One thought per blockquote.
|
||||||
|
|
||||||
|
## Diagrams
|
||||||
|
|
||||||
|
- Mermaid is the default for topology and flow diagrams. `infrastructure/topology.md` is generated by `oikos/gen-topology.py` — do not hand-edit it. (Go DB-native topology generation planned.)
|
||||||
|
- ASCII box diagrams are fine for small shape diagrams; keep them to one screen.
|
||||||
|
|
||||||
|
## Sourcing and cross-references
|
||||||
|
|
||||||
|
- **Factual discipline.** Every claim is grounded in a cited source, an adjacent linked page, or a directly observable fact (`pct config`, `docker inspect`, running config). Do not write sentences that sound sourced but are inference. When docs disagree with live state, fix the doc and note it in the changelog.
|
||||||
|
- **One-sided cross-references.** When two pages relate, the link lives in the page where the connection makes organizational sense. Do not add a back-pointer unless that direction also carries content the reader needs.
|
||||||
|
- **Cross-references are content, not catalog.** Inline links arise from the surrounding prose; the linked page must be needed to understand the current sentence. A bottom-of-page "Related" list is the fallback, not the default.
|
||||||
|
- Pages link with standard relative markdown links (e.g. a container page links to `../infrastructure/dns.md`), forming a navigable graph. Orphans are a bug.
|
||||||
|
|
||||||
|
## Code comments and commit/PR prose
|
||||||
|
|
||||||
|
- Comments explain intent, trade-offs, or constraints the code cannot convey. No diff narration, no type restatement, no section-divider comments.
|
||||||
|
- Commit messages and PR descriptions are problem → change → risk → verification, not a file-by-file diff restatement.
|
||||||
|
- The banned vocabulary applies the same way in comments and commit messages.
|
||||||
43
.agents/skills/client-enrollment/SKILL.md
Normal file
43
.agents/skills/client-enrollment/SKILL.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
---
|
||||||
|
name: client-enrollment
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [hostname, kind, role]
|
||||||
|
verification: "homelab doctor (on the new client)"
|
||||||
|
docs_update_checklist: [hosts_narrative_page_if_lxc_or_vm]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Client enrollment
|
||||||
|
|
||||||
|
Goal: bring a new host (workstation, LXC, VM) into inventory and the
|
||||||
|
secrets model, with mesh membership only where it's actually needed.
|
||||||
|
This wraps the existing `homelab client add` flow — see
|
||||||
|
[operations/agent-enrollment.md](../../operations/agent-enrollment.md) for
|
||||||
|
the full walkthrough; this runbook is the risk/lifecycle framing.
|
||||||
|
|
||||||
|
1. On any enrolled client: `homelab client add <hostname>` — appends a
|
||||||
|
`hosts.<name>:` block to `inventory.yaml` (lifecycle `state: planned`
|
||||||
|
→ `provisioning`, per [seeds/ontology.yaml](../../../seeds/ontology.yaml)),
|
||||||
|
commits + pushes.
|
||||||
|
2. Netbird join is **optional, not a required step** — only needed for
|
||||||
|
hosts that must be reachable off-LAN (workstations that roam, e.g.
|
||||||
|
`republic-laptop`, `mac-mini`). A node reachable on the household LAN
|
||||||
|
(192.168.8.0/24 — most LXCs/VMs) doesn't need it: it's already
|
||||||
|
reachable directly, and off-LAN clients reach it too via hubris's
|
||||||
|
routed `192.168.8.0/24` Netbird network resource. Skip this step for
|
||||||
|
LAN-only nodes; do it (out-of-band, console or setup key) only for
|
||||||
|
hosts that need independent off-LAN reachability.
|
||||||
|
3. On the new host: run `bootstrap.sh` (add `--with-nomos` to also
|
||||||
|
enroll the Hermes agent). This provisions `/etc/age/key.txt`, the
|
||||||
|
sync timer, and prints an age pubkey.
|
||||||
|
4. Back on an enrolled client: `homelab client add <hostname>
|
||||||
|
--finalize-pubkey <age1...>` — sets `age_pubkey`, grants shared
|
||||||
|
secrets, re-keys SOPS, commits + pushes. This is the
|
||||||
|
`provisioning → active` transition.
|
||||||
|
5. Verify: `homelab doctor` on the new client should show all checks
|
||||||
|
green (clone, sync timer, age key, CLI symlink, MCP reachable).
|
||||||
|
|
||||||
|
Docs-update checklist: if the new host is an LXC/VM, add its narrative
|
||||||
|
page under `containers/` or `vms/` and set `doc_page` in its inventory
|
||||||
|
entry (host-level cards don't have a `doc_page` field yet — services do;
|
||||||
|
narrative pages are still found via the generated `see_also` in
|
||||||
|
`inventory.yaml`).
|
||||||
34
.agents/skills/config-change-deploy/SKILL.md
Normal file
34
.agents/skills/config-change-deploy/SKILL.md
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
---
|
||||||
|
name: config-change-deploy
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [service_name, change_description]
|
||||||
|
verification: "curl -sf <service_url> (or homelab service <name> health)"
|
||||||
|
docs_update_checklist: [doc_page, changelog]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Config change + deploy
|
||||||
|
|
||||||
|
Goal: change a tracked config repo (Caddy, Gitea customizations, an app's
|
||||||
|
own repo) and get it live, safely.
|
||||||
|
|
||||||
|
1. `homelab change preflight <service>` — current health, the service's
|
||||||
|
`config_repo`, its risk class, and the verification command to run
|
||||||
|
after. If risk class requires approval (`config_mutation` or
|
||||||
|
`destructive`), stop and get operator sign-off before editing — see
|
||||||
|
`seeds/policy.yaml`.
|
||||||
|
2. Clone/pull the `config_repo` (never edit the backend's working tree
|
||||||
|
directly — tracked configs change by commit + push, per
|
||||||
|
[OIKOS.md](../../OIKOS.md) conventions).
|
||||||
|
3. Make the change, commit, push to `main`.
|
||||||
|
4. The Gitea webhook fires the deploy pipeline for that repo (see
|
||||||
|
[infrastructure/auto-deploy.md](../../../archive/knowledge/infrastructure/auto-deploy.md) for
|
||||||
|
the exact receiver/reload for this service).
|
||||||
|
5. Run the preflight's verification command. If it fails, check
|
||||||
|
`homelab service <name> log` for the reload/restart error.
|
||||||
|
6. Record the change: once `oikos/ledger.py` is wired into deploy tooling
|
||||||
|
(Week 3), this is automatic; until then, note the change and outcome
|
||||||
|
in the relevant investigation/plan doc.
|
||||||
|
|
||||||
|
Docs-update checklist: update the service's `doc_page` if the change
|
||||||
|
alters its behavior, ingress route, or ownership; add a changelog entry
|
||||||
|
if the page has one.
|
||||||
25
.agents/skills/docs-lint/SKILL.md
Normal file
25
.agents/skills/docs-lint/SKILL.md
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
---
|
||||||
|
name: docs-lint
|
||||||
|
risk_class: read_only
|
||||||
|
inputs: [paths]
|
||||||
|
verification: "python3 .agents/skills/docs-lint/lint.py"
|
||||||
|
docs_update_checklist: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Docs lint
|
||||||
|
|
||||||
|
Check committed documentation against the mechanical rules in
|
||||||
|
[writing-style.md](../../shared/writing-style.md): banned vocabulary and broken relative markdown
|
||||||
|
links. Prose-voice rules are not machine-checkable — those stay a review responsibility.
|
||||||
|
|
||||||
|
Run from the repo root:
|
||||||
|
|
||||||
|
python3 .agents/skills/docs-lint/lint.py # default: knowledge/ .agents/ operations/ investigations/ plans/
|
||||||
|
python3 .agents/skills/docs-lint/lint.py archive/knowledge/containers/104-gitea.md
|
||||||
|
|
||||||
|
Exit code is non-zero when any violation is found, so it can gate a commit. The banned-vocabulary
|
||||||
|
list mirrors `writing-style.md`; update both together if the standard changes.
|
||||||
|
|
||||||
|
> **Known baseline.** `archive/knowledge/archive/knowledge/containers/101-jellyfin.md` links into a sibling repo
|
||||||
|
> (`devops/homelab-authentik-admin`) that this checkout does not contain — expected, not a bug.
|
||||||
|
> Any other broken link is a real regression; investigate before dismissing it as baseline noise.
|
||||||
186
.agents/skills/docs-lint/lint.py
Normal file
186
.agents/skills/docs-lint/lint.py
Normal file
@@ -0,0 +1,186 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Lint committed docs against .agents/shared/writing-style.md.
|
||||||
|
|
||||||
|
Checks:
|
||||||
|
1. Banned vocabulary (significance puffers, analytical verbs, poetic nouns,
|
||||||
|
promotional adjectives, opening crutches).
|
||||||
|
2. Broken relative markdown links.
|
||||||
|
3. Plan status consistency (status vs location vs index).
|
||||||
|
|
||||||
|
Prose-voice rules are not machine-checkable; this covers the parts that are.
|
||||||
|
Run from the repo root: python3 .agents/skills/docs-lint/lint.py [paths...]
|
||||||
|
Exit 1 if any violation is found.
|
||||||
|
"""
|
||||||
|
import os, re, sys, glob
|
||||||
|
|
||||||
|
BANNED = [
|
||||||
|
"pivotal", "crucial", "vital", "groundbreaking", "transformative", "testament",
|
||||||
|
"paramount", "invaluable", "delve", "leverage", "utilize", "facilitate", "foster",
|
||||||
|
"showcase", "underscore", "streamline", "harness", "tapestry", "realm", "paradigm",
|
||||||
|
"nexus", "cornerstone", "robust", "seamless", "innovative", "cutting-edge",
|
||||||
|
"meticulous", "holistic", "comprehensive", "in today's world",
|
||||||
|
"it's worth noting", "it is important to note",
|
||||||
|
]
|
||||||
|
BAN_RE = re.compile(r'(?<![\w-])(' + "|".join(re.escape(w) for w in BANNED) + r')(?![\w-])', re.I)
|
||||||
|
LINK = re.compile(r'\]\(([^)]+)\)')
|
||||||
|
|
||||||
|
def iter_md(paths):
|
||||||
|
for p in paths:
|
||||||
|
if os.path.isfile(p) and p.endswith(".md"):
|
||||||
|
yield p
|
||||||
|
for root, dirs, files in os.walk(p):
|
||||||
|
dirs[:] = [d for d in dirs if d not in (".git", "node_modules")]
|
||||||
|
for f in files:
|
||||||
|
if f.endswith(".md"):
|
||||||
|
yield os.path.join(root, f)
|
||||||
|
|
||||||
|
def check_plans():
|
||||||
|
"""Check plan status consistency: active plans with 'Done' status, files
|
||||||
|
missing from index, dangling index entries, done files with wrong status."""
|
||||||
|
REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
plans_dir = os.path.join(REPO, "plans")
|
||||||
|
done_dir = os.path.join(REPO, "plans", "done")
|
||||||
|
index_path = os.path.join(plans_dir, "index.md")
|
||||||
|
|
||||||
|
if not os.path.exists(index_path):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
violations = 0
|
||||||
|
STATUS_RE = re.compile(r'^\*\*Status:\*\*\s*(.+)', re.I)
|
||||||
|
|
||||||
|
# Parse index.md for active and done entries
|
||||||
|
active_files = set()
|
||||||
|
done_files = set()
|
||||||
|
current_section = None
|
||||||
|
with open(index_path) as f:
|
||||||
|
for line in f:
|
||||||
|
if line.startswith("## Active"):
|
||||||
|
current_section = "active"
|
||||||
|
continue
|
||||||
|
if line.startswith("## Done"):
|
||||||
|
current_section = "done"
|
||||||
|
continue
|
||||||
|
if current_section == "active":
|
||||||
|
m = re.search(r'\]\(([^)]+)\)', line)
|
||||||
|
if m:
|
||||||
|
active_files.add(m.group(1))
|
||||||
|
elif current_section == "done":
|
||||||
|
m = re.search(r'\]\(([^)]+)\)', line)
|
||||||
|
if m:
|
||||||
|
done_files.add(m.group(1))
|
||||||
|
|
||||||
|
# Active plans on disk (not in done/, not index.md)
|
||||||
|
disk_active = set()
|
||||||
|
for f in glob.glob(os.path.join(plans_dir, "*.md")):
|
||||||
|
name = os.path.basename(f)
|
||||||
|
if name == "index.md":
|
||||||
|
continue
|
||||||
|
disk_active.add(name)
|
||||||
|
|
||||||
|
# Done plans on disk
|
||||||
|
disk_done = set()
|
||||||
|
if os.path.isdir(done_dir):
|
||||||
|
for f in glob.glob(os.path.join(done_dir, "*.md")):
|
||||||
|
disk_done.add("done/" + os.path.basename(f))
|
||||||
|
|
||||||
|
# Check 1: active plans on disk whose internal status is Done/Implemented/Complete
|
||||||
|
for name in disk_active:
|
||||||
|
fpath = os.path.join(plans_dir, name)
|
||||||
|
with open(fpath) as f:
|
||||||
|
for line_num, line in enumerate(f, 1):
|
||||||
|
if line_num > 5:
|
||||||
|
break
|
||||||
|
m = STATUS_RE.match(line)
|
||||||
|
if m:
|
||||||
|
status = m.group(1).strip().lower()
|
||||||
|
done_keywords = ["done", "implemented", "complete", "completed"]
|
||||||
|
if any(status.startswith(kw) for kw in done_keywords):
|
||||||
|
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in plans/ but appears done; move to done/")
|
||||||
|
violations += 1
|
||||||
|
break
|
||||||
|
|
||||||
|
# Check 2: active plans on disk not in index
|
||||||
|
for name in sorted(disk_active):
|
||||||
|
if name not in active_files:
|
||||||
|
fpath = os.path.join(plans_dir, name)
|
||||||
|
print(f"{fpath}:1: not listed in plans/index.md Active table")
|
||||||
|
violations += 1
|
||||||
|
|
||||||
|
# Check 3: done plans on disk not in index
|
||||||
|
for name in sorted(disk_done):
|
||||||
|
if name not in done_files:
|
||||||
|
fpath = os.path.join(REPO, "plans", name)
|
||||||
|
print(f"{fpath}:1: not listed in plans/index.md Done table")
|
||||||
|
violations += 1
|
||||||
|
|
||||||
|
# Check 4: index entries with no file on disk
|
||||||
|
for name in sorted(active_files):
|
||||||
|
if name not in disk_active:
|
||||||
|
print(f"plans/index.md: active entry '{name}' — file not found on disk")
|
||||||
|
violations += 1
|
||||||
|
|
||||||
|
for name in sorted(done_files):
|
||||||
|
if name not in disk_done:
|
||||||
|
print(f"plans/index.md: done entry '{name}' — file not found on disk")
|
||||||
|
violations += 1
|
||||||
|
|
||||||
|
# Check 5: files in done/ whose internal status doesn't say Done
|
||||||
|
for name in disk_done:
|
||||||
|
fpath = os.path.join(REPO, "plans", name)
|
||||||
|
with open(fpath) as f:
|
||||||
|
found_status = False
|
||||||
|
for line_num, line in enumerate(f, 1):
|
||||||
|
if line_num > 5:
|
||||||
|
break
|
||||||
|
m = STATUS_RE.match(line)
|
||||||
|
if m:
|
||||||
|
found_status = True
|
||||||
|
status = m.group(1).strip().lower()
|
||||||
|
if not status.startswith("done"):
|
||||||
|
print(f"{fpath}:{line_num}: status '{m.group(1).strip()}' — file is in done/ but status is not 'Done'")
|
||||||
|
violations += 1
|
||||||
|
break
|
||||||
|
if not found_status:
|
||||||
|
print(f"{fpath}:1: file is in done/ but has no Status header")
|
||||||
|
violations += 1
|
||||||
|
|
||||||
|
return violations
|
||||||
|
|
||||||
|
def main(argv):
|
||||||
|
paths = argv or ["knowledge", ".agents", "operations", "investigations", "plans"]
|
||||||
|
violations = 0
|
||||||
|
|
||||||
|
if "plans" in paths or any(p.startswith("plans") for p in paths):
|
||||||
|
violations += check_plans()
|
||||||
|
|
||||||
|
# The style guide and this skill enumerate the banned words by definition.
|
||||||
|
ban_exempt = ("shared/writing-style.md", "skills/docs-lint/")
|
||||||
|
for f in sorted(set(iter_md(paths))):
|
||||||
|
check_banned = not any(x in f for x in ban_exempt)
|
||||||
|
fence = False
|
||||||
|
with open(f) as fh:
|
||||||
|
for ln, line in enumerate(fh, 1):
|
||||||
|
if line.lstrip().startswith("```"):
|
||||||
|
fence = not fence; continue
|
||||||
|
if fence:
|
||||||
|
continue
|
||||||
|
if check_banned:
|
||||||
|
for m in BAN_RE.finditer(line):
|
||||||
|
print(f"{f}:{ln}: banned word '{m.group(1)}'")
|
||||||
|
violations += 1
|
||||||
|
for m in LINK.finditer(line):
|
||||||
|
link = m.group(1)
|
||||||
|
if re.match(r'^(https?:|mailto:|#|/)', link):
|
||||||
|
continue
|
||||||
|
path = re.split(r'[#?]', link)[0]
|
||||||
|
if not path:
|
||||||
|
continue
|
||||||
|
tgt = os.path.normpath(os.path.join(os.path.dirname(f), path))
|
||||||
|
if not os.path.exists(tgt):
|
||||||
|
print(f"{f}:{ln}: broken link -> {link}")
|
||||||
|
violations += 1
|
||||||
|
print(f"\n{violations} violation(s)")
|
||||||
|
return 1 if violations else 0
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv[1:]))
|
||||||
34
.agents/skills/incident-investigation/SKILL.md
Normal file
34
.agents/skills/incident-investigation/SKILL.md
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
---
|
||||||
|
name: incident-investigation
|
||||||
|
risk_class: read_only
|
||||||
|
inputs: [symptom, affected_entity]
|
||||||
|
verification: "n/a — investigation produces a written record, not a state change"
|
||||||
|
docs_update_checklist: [investigations_entry]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Incident investigation
|
||||||
|
|
||||||
|
Goal: understand what broke and why, before touching anything.
|
||||||
|
|
||||||
|
1. `homelab service <name> explain` (or `homelab node <name> relations`
|
||||||
|
if the affected entity is a host) — get the blast radius and doc
|
||||||
|
pointer first. Don't start pulling logs blind.
|
||||||
|
2. `homelab service <name> health` + `homelab service <name> log` (or
|
||||||
|
MCP `get_service_status` / `tail_log`) for the affected service.
|
||||||
|
3. Walk the blast radius: is a shared dependency down (`caddy`, `dns`,
|
||||||
|
`authentik`, or the backend host itself)? `homelab node <name>
|
||||||
|
relations` shows "affected by" — check those first.
|
||||||
|
4. `homelab apt-audit` if the symptom looks like a dpkg/upgrade
|
||||||
|
interaction.
|
||||||
|
5. Check the change ledger for recent mutations to the affected entity
|
||||||
|
or anything upstream of it: `homelab service <name> history` (once
|
||||||
|
populated) or grep `ledger/*.jsonl`.
|
||||||
|
6. Write findings to a new `knowledge/sources/investigations/<date>-<slug>.md` — symptom,
|
||||||
|
timeline, root cause, fix applied, prevention. This is the durable
|
||||||
|
record; don't rely on chat history.
|
||||||
|
|
||||||
|
Docs-update checklist: always create the investigation entry. If the
|
||||||
|
root cause was stale/wrong inventory data (a `doc_page`, `config_repo`,
|
||||||
|
or `backend` that didn't match reality — this happened during Week 1
|
||||||
|
kernel work, see the `authentik` backend fix), correct `inventory.yaml`
|
||||||
|
in the same session.
|
||||||
36
.agents/skills/lifecycle-activate-node/SKILL.md
Normal file
36
.agents/skills/lifecycle-activate-node/SKILL.md
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
---
|
||||||
|
name: lifecycle-activate-node
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [node_name]
|
||||||
|
verification: "homelab service <name> health (if it hosts a service); homelab doctor (if it's a client)"
|
||||||
|
docs_update_checklist: [doc_page_complete]
|
||||||
|
transition: "provisioning -> active"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lifecycle: activate a node
|
||||||
|
|
||||||
|
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml). Requires: age key
|
||||||
|
enrolled if it needs secrets, mesh joined if it needs off-LAN reach,
|
||||||
|
ingress live if public, health check answering, doc page complete,
|
||||||
|
ledger entry.
|
||||||
|
|
||||||
|
1. If the node is a `homelab` client: finish enrollment per
|
||||||
|
[client-enrollment.md](../client-enrollment/SKILL.md) (`--finalize-pubkey`,
|
||||||
|
mesh join, `homelab doctor` green).
|
||||||
|
2. If it hosts a public service: add the `services:` entry in
|
||||||
|
`inventory.yaml` (backend, url, doc_page, config_repo, risk_notes —
|
||||||
|
see the Week-1 service contract fields) and wire the Caddy route in
|
||||||
|
`dtoro/caddy-conf`.
|
||||||
|
3. Confirm the health check answers: `homelab service <name> health` or
|
||||||
|
a direct `curl`.
|
||||||
|
4. Flip `state: provisioning` → `state: active` (or delete the `state:`
|
||||||
|
field — `active` is the default) in `inventory.yaml`.
|
||||||
|
5. Complete the doc page (stub → full narrative: role, specs, how it's
|
||||||
|
configured, dependencies).
|
||||||
|
6. Record the activation: `oikos/ledger.py append host:<name> activate
|
||||||
|
config_mutation --result ok` (or let the CLI wrapper do this once
|
||||||
|
Week 3's runbook automation lands).
|
||||||
|
|
||||||
|
Regenerate derived data: `python3 mcp/build_host_files.py && python3
|
||||||
|
inventory.yaml` so `inventory.yaml`, the topology diagram, and
|
||||||
|
the context card all reflect the new state.
|
||||||
35
.agents/skills/lifecycle-deprecate-node/SKILL.md
Normal file
35
.agents/skills/lifecycle-deprecate-node/SKILL.md
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
---
|
||||||
|
name: lifecycle-deprecate-node
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [node_name, replacement_node_or_reason]
|
||||||
|
verification: "homelab node <name> relations — 'affected by' must be empty before completing"
|
||||||
|
docs_update_checklist: [doc_page_deprecation_note]
|
||||||
|
transition: "active -> deprecated"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lifecycle: deprecate a node
|
||||||
|
|
||||||
|
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml): a node keeps running
|
||||||
|
but takes no new dependents. **Completion condition: zero remaining
|
||||||
|
inbound `depends-on`/`routes-to` edges** — this is a hard gate, not a
|
||||||
|
suggestion; `seeds/policy.yaml` `lifecycle_overrides.deprecated.refuse`
|
||||||
|
lists `new-inbound-edges` as refused going forward.
|
||||||
|
|
||||||
|
1. Set `state: deprecated` on the node.
|
||||||
|
2. `homelab node <name> relations` — read `affected_by`. Every entry
|
||||||
|
there is something still relying on this node.
|
||||||
|
3. Migrate or retire each dependent one at a time (point its `backend`/
|
||||||
|
`config_repo`/ingress route elsewhere, or deprecate it too if it's
|
||||||
|
being retired alongside).
|
||||||
|
4. Re-run `homelab node <name> relations` after each dependent is moved.
|
||||||
|
The transition to `destroyed` is only safe once `affected_by` is
|
||||||
|
empty — check this every time, don't assume from memory.
|
||||||
|
5. Note the deprecation on the doc page: reason, replacement (if any),
|
||||||
|
date.
|
||||||
|
|
||||||
|
If step 2 shows dependents you didn't expect, stop and investigate
|
||||||
|
before proceeding — that's exactly the kind of drift the Week-3 detector
|
||||||
|
will catch automatically, but until then this manual check is the gate.
|
||||||
|
|
||||||
|
Next (once `affected_by` is empty):
|
||||||
|
[lifecycle-destroy-node.md](../lifecycle-destroy-node/SKILL.md).
|
||||||
42
.agents/skills/lifecycle-destroy-node/SKILL.md
Normal file
42
.agents/skills/lifecycle-destroy-node/SKILL.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
---
|
||||||
|
name: lifecycle-destroy-node
|
||||||
|
risk_class: destructive
|
||||||
|
inputs: [node_name]
|
||||||
|
verification: "homelab node <name> relations returns unknown-entity; pct list on the backend no longer shows it"
|
||||||
|
docs_update_checklist: [archaeology_entry, containers_index_update]
|
||||||
|
transition: "deprecated -> destroyed"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lifecycle: destroy a node
|
||||||
|
|
||||||
|
**Destructive.** Requires operator approval + typed confirmation phrase
|
||||||
|
per `seeds/policy.yaml`. Requires (ontology): backups verified, secrets
|
||||||
|
recipients removed + re-keyed, ingress/DNS removed, archaeology entry,
|
||||||
|
ledger entry.
|
||||||
|
|
||||||
|
1. Confirm the node is `deprecated` with zero `affected_by` edges
|
||||||
|
(`homelab node <name> relations`) — do not skip this even if the
|
||||||
|
deprecation runbook was followed recently; state can drift.
|
||||||
|
2. If it's an enrolled client: `homelab client remove <name>` — revokes
|
||||||
|
the age key, re-keys SOPS, removes the inventory entry. This is
|
||||||
|
already destructive-class and confirmed in the CLI.
|
||||||
|
3. Remove any ingress route (Caddy config repo) and DNS record still
|
||||||
|
pointing at it.
|
||||||
|
4. Verify backups of anything on it are retained per policy before the
|
||||||
|
disk goes away (see `backs-up-to`).
|
||||||
|
5. Destroy the LXC/VM (`pct destroy` / `qm destroy`).
|
||||||
|
6. Move the `hosts.<name>:` block (if any inventory remnant survives
|
||||||
|
`client remove`, e.g. infra-only LXCs with no age key) into
|
||||||
|
inventory.yaml's `archaeology:` section: `pve_id`, `destroyed` date,
|
||||||
|
`reason`. Add a row to `containers/index.md` "Recently destroyed"
|
||||||
|
table (kept for human-readable browsing alongside the structured
|
||||||
|
data).
|
||||||
|
7. `oikos/ledger.py append host:<name> destroy destructive --result ok`.
|
||||||
|
8. Regenerate: `python3 mcp/build_host_files.py && python3
|
||||||
|
inventory.yaml` — the node drops out of `inventory.yaml` and
|
||||||
|
appears in the topology doc's archaeology table.
|
||||||
|
|
||||||
|
If the destroy fails partway (e.g. secrets revoked but pct destroy
|
||||||
|
errors), do not re-run step 2 — `client remove` is not idempotent
|
||||||
|
against a second revocation attempt on the issuance server. Finish the
|
||||||
|
remaining steps manually and note the partial state in an investigation.
|
||||||
39
.agents/skills/lifecycle-migrate-node/SKILL.md
Normal file
39
.agents/skills/lifecycle-migrate-node/SKILL.md
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
---
|
||||||
|
name: lifecycle-migrate-node
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [node_name, source_host, target_host]
|
||||||
|
verification: "homelab node <name> relations (re-check blast radius); homelab service <svc> health for every hosted service"
|
||||||
|
docs_update_checklist: [doc_page_migration_note, inventory_host_and_lan_ip]
|
||||||
|
transition: "active -> migrating -> active"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lifecycle: migrate a node
|
||||||
|
|
||||||
|
Modeled on the strong Phase 1+2 migration
|
||||||
|
([plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)).
|
||||||
|
Requires (ontology): preflight + backup-verified before migrating;
|
||||||
|
post-verify + Caddy backends checked + mounts checked + docs updated
|
||||||
|
before returning to `active`.
|
||||||
|
|
||||||
|
1. `homelab change preflight <every service the node hosts>` — capture
|
||||||
|
current health as a baseline.
|
||||||
|
2. Verify backups are current for anything with data at rest on the
|
||||||
|
node (see `backs-up-to` edges once populated).
|
||||||
|
3. Set `state: migrating` in `inventory.yaml`.
|
||||||
|
4. Perform the migration (pct/qm move, or create-on-target +
|
||||||
|
data-copy + destroy-source, per the specific case).
|
||||||
|
5. Update `inventory.yaml`: new `host:`, `lan_ip`, `mesh` addresses for
|
||||||
|
the node; update every `services:` entry whose `backend` pointed at
|
||||||
|
it if the backend name itself changes (usually it doesn't — only the
|
||||||
|
`host:`/`lan_ip` on the guest entry moves).
|
||||||
|
6. Post-verify: re-run the Week-1 drift check by hand — confirm Caddy's
|
||||||
|
backend IP for each affected service matches the new `lan_ip`
|
||||||
|
(automatic in Week 3's drift detector), confirm mounts still resolve.
|
||||||
|
7. `homelab service <name> health` for every service the node hosts.
|
||||||
|
8. Set `state: active`. Add a migration note to the node's doc page
|
||||||
|
(old host/IP → new, date, phase reference) — this repo's convention
|
||||||
|
for every past migration (see `archive/knowledge/containers/101-jellyfin.md`,
|
||||||
|
`containers/129-house.md`).
|
||||||
|
|
||||||
|
Regenerate: `python3 mcp/build_host_files.py && python3
|
||||||
|
inventory.yaml`.
|
||||||
33
.agents/skills/lifecycle-provision-node/SKILL.md
Normal file
33
.agents/skills/lifecycle-provision-node/SKILL.md
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
---
|
||||||
|
name: lifecycle-provision-node
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [node_name, kind, storage_pool]
|
||||||
|
verification: "grep 'state: provisioning' inventory.yaml"
|
||||||
|
docs_update_checklist: [doc_page_stub]
|
||||||
|
transition: "planned -> provisioning"
|
||||||
|
---
|
||||||
|
|
||||||
|
# Lifecycle: provision a node
|
||||||
|
|
||||||
|
Per [seeds/ontology.yaml](../../../seeds/ontology.yaml) `lifecycle.transitions`.
|
||||||
|
Policy note: `provisioning` nodes get a lifecycle override —
|
||||||
|
`config_mutation` actions downgrade to `reversible_low` because nothing
|
||||||
|
depends on the node yet (see `seeds/policy.yaml` `lifecycle_overrides`).
|
||||||
|
|
||||||
|
Requires (from ontology): inventory entry, IP reserved, storage pool
|
||||||
|
chosen, doc page stub.
|
||||||
|
|
||||||
|
1. Create the LXC/VM on its target Proxmox host (`pct create` /
|
||||||
|
`qm create`), choosing the storage pool deliberately — record it as
|
||||||
|
the `storage:` field once populated (Week 1 schema; not yet backfilled
|
||||||
|
for existing nodes).
|
||||||
|
2. Add the inventory entry: `homelab client add <name>` for anything that
|
||||||
|
will run the `homelab` CLI, or a direct `hosts.<name>:` block with
|
||||||
|
`state: provisioning`, `kind`, `host`, `pve_id`, `lan_ip` for
|
||||||
|
infra-only LXCs that won't self-enroll.
|
||||||
|
3. Stub the doc page (`containers/<pve_id>-<name>.md` or
|
||||||
|
`vms/<pve_id>-<name>.md`) — even a one-line "provisioning, see plan X"
|
||||||
|
is enough to satisfy the transition requirement.
|
||||||
|
4. Reserve the IP in DNS/DHCP notes if it's a fixed LAN address.
|
||||||
|
|
||||||
|
Next: [lifecycle-activate-node.md](../lifecycle-activate-node/SKILL.md).
|
||||||
209
.agents/skills/runbook-budget-from-csv/SKILL.md
Normal file
209
.agents/skills/runbook-budget-from-csv/SKILL.md
Normal file
@@ -0,0 +1,209 @@
|
|||||||
|
---
|
||||||
|
name: budget-import-from-csv
|
||||||
|
risk_class: config_mutation
|
||||||
|
inputs: [csv_file]
|
||||||
|
references: [containers/129-house.md]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Runbook: Budget import from N26 CSV → Yuvomi
|
||||||
|
|
||||||
|
Distil a bank-export CSV into Yuvomi's Budget and Subscriptions modules using
|
||||||
|
the `yuvomi-mcp` tools. Run this whenever a new CSV period needs to be
|
||||||
|
summarised into targets and fixed costs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- `yuvomi-mcp` is running on LXC 129 and connected as an MCP server in Claude.
|
||||||
|
- The CSV is an N26 export (columns: Booking Date, Value Date, Partner Name,
|
||||||
|
Partner Iban, Type, Payment Reference, Account Name, Amount (EUR), …).
|
||||||
|
- API token: `homelab secret yuvomi-api-token` (decrypts on any enrolled client).
|
||||||
|
- Direct API base: `https://house.hubris.network/api/v1`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API quirks (Yuvomi ≤ 0.77.x)
|
||||||
|
|
||||||
|
- **Subscriptions live under `/budget/subscriptions`**, NOT `/subscriptions/`.
|
||||||
|
A top-level `/subscriptions` route returns 404.
|
||||||
|
- `GET /budget/subscriptions` → `{ data: { subscriptions: [...], summary: {...} } }`
|
||||||
|
- `GET /budget/subscriptions/meta` → `{ data: { categories: [...], payment_methods: [...] } }`
|
||||||
|
- `POST /budget/subscriptions` → create a subscription (name, amount, billing_cycle,
|
||||||
|
cycle_interval, next_payment_date, currency, category_id, payment_method_id required)
|
||||||
|
- `GET /budget/` (no month) → returns only **non-recurring** base entries.
|
||||||
|
Use `GET /budget/?month=YYYY-MM` to get all entries (recurring + one-time) for a month.
|
||||||
|
- `GET /budget/categories` → expense category keys + income category names (German keys
|
||||||
|
like `"Erwerbseinkommen"`, `"Sozialleistungen"`, `"Geschenke & Transfers"`).
|
||||||
|
- Budget entries: `amount` positive = income, negative = expense.
|
||||||
|
- Recurring entries: set `is_recurring: 1` + `recurrence_interval: "monthly"`.
|
||||||
|
The `date` field sets the start month.
|
||||||
|
- `recurrence_virtual: 1` smooths non-monthly amounts across all months in the summary
|
||||||
|
(e.g. 55.08 € quarterly → shows as ~18.36 €/month).
|
||||||
|
- Custom RRULE strings (`recurrence_rule`) are **not accepted** by the API — use
|
||||||
|
`cycle_interval` on the subscription instead, or `recurrence_interval` on budget entries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Subscription category IDs (as of 2026-06-26)
|
||||||
|
|
||||||
|
| id | name | budget_subcategory_key |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | Entertainment | subscription_entertainment |
|
||||||
|
| 2 | Productivity | subscription_productivity |
|
||||||
|
| 3 | Utilities | subscription_utilities |
|
||||||
|
| 4 | Health | subscription_health |
|
||||||
|
| 5 | Education | subscription_education |
|
||||||
|
| 6 | Other | subscription_other |
|
||||||
|
|
||||||
|
## Payment method IDs
|
||||||
|
|
||||||
|
| id | name |
|
||||||
|
|---|---|
|
||||||
|
| 1 | Credit Card |
|
||||||
|
| 2 | Debit Card |
|
||||||
|
| 3 | PayPal |
|
||||||
|
| 6 | Bank Transfer / SEPA |
|
||||||
|
| 7 | Other |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1 — Categorise the transactions
|
||||||
|
|
||||||
|
Skip these as internal/already-covered:
|
||||||
|
- Fixed costs you'll enter as **subscriptions** (Miete, SWM, SYNVIA, Hundefutter,
|
||||||
|
Netflix, Grover, Rundfunk ARD, KuKita, Lillydoo)
|
||||||
|
- Internal transfers (The Joy Pot ↔ Cookie, Hauptkonto, Tagesgeldkonto splits)
|
||||||
|
- Identified income (Cookie Share, Kindergeld, Pocket Money credits, Distributor)
|
||||||
|
- Fun Money pass-throughs (in and out same month → net zero)
|
||||||
|
|
||||||
|
**Variable expense taxonomy:**
|
||||||
|
|
||||||
|
| Category key | Subcategory key | Examples |
|
||||||
|
|---|---|---|
|
||||||
|
| `food` | `groceries` | E-Center, Knuspr, EDEKA, Tegut, VollCorner, Lidl, Netto, REWE, KoRo, Roast Market |
|
||||||
|
| `food` | `restaurants_bars` | Restaurants, Lieferando, Cafes, Zeit für Brot, Baobab, Höflinger |
|
||||||
|
| `personal_health` | `beauty_cosmetics` | DM, Rossmann |
|
||||||
|
| `personal_health` | `pharmacy` | Apotheke, MVZ Dermatologie |
|
||||||
|
| `transport` | `apps_taxi` | Uber, RYD GMBH, MVG, Handyparken |
|
||||||
|
| `shopping_clothing` | `gifts` | Children products: Schlummersack, Catchy Kids, SP EVERY., Dukal, Berger-Lernwelt |
|
||||||
|
| `shopping_clothing` | `clothes_shoes` | Zalando, Ernsting's, Schuhmair, Thalia, Vinted, Airbnb, Hotel at Booking.com |
|
||||||
|
| `shopping_clothing` | `electronics` | Amazon, AMZN Mktp DE |
|
||||||
|
| `housing` | `renovation_maintenance` | IKEA, Markus Festl, Granit, Sostrene Grene, Mol* tischdecken, Gaertnerei, Dehner |
|
||||||
|
| `education` | `courses_college` | Kathrin Orlob (PEKiP), Nerina Aupperle |
|
||||||
|
| `leisure` | `streaming` | WOW wowtv.de |
|
||||||
|
| `financial_other` | `bank_fees` | Unidentified PayPal, Ratepay, N26 fees |
|
||||||
|
| `Geschenke & Transfers` | *(income)* | One-off incoming transfers |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2 — Create subscriptions
|
||||||
|
|
||||||
|
```
|
||||||
|
get_subscriptions_meta() ← get category_id and payment_method_id
|
||||||
|
```
|
||||||
|
|
||||||
|
**Standard Cookie household subscriptions (as of 2026-07):**
|
||||||
|
|
||||||
|
| Name | Amount | billing_cycle | cycle_interval | category_id | payment_method_id |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| Miete | 1080.00 | monthly | 1 | 6 (Other) | 6 (Bank Transfer) |
|
||||||
|
| Strom (SWM) | 79.00 | monthly | 1 | 3 (Utilities) | 6 |
|
||||||
|
| Internet / TV / Telefon | 29.99 | monthly | 1 | 3 (Utilities) | 6 |
|
||||||
|
| Hundefutter | 75.00 | monthly | 1 | 6 (Other) | 6 |
|
||||||
|
| Netflix | 8.00 | monthly | 1 | 1 (Entertainment) | 6 |
|
||||||
|
| Grover | 16.90 | monthly | 1 | 6 (Other) | 2 (Debit Card) |
|
||||||
|
| Rundfunk ARD / ZDF | 55.08 | monthly | 3 | 1 (Entertainment) | 6 |
|
||||||
|
| KuKita Daycare (Leon) | 503.00 | monthly | 1 | 5 (Education) | 6 |
|
||||||
|
| Lillydoo diapers | 56.70 | monthly | 2 | 4 (Health) | 3 (PayPal) |
|
||||||
|
|
||||||
|
Monthly equivalent total: **1,838.60 €** (Yuvomi applies cycle_interval to prorate).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 3 — Add recurring income entries
|
||||||
|
|
||||||
|
```
|
||||||
|
stage_add_budget_entry(
|
||||||
|
title="Kindergeld",
|
||||||
|
amount=55.00,
|
||||||
|
category="Sozialleistungen",
|
||||||
|
date="YYYY-MM-01",
|
||||||
|
is_recurring=True,
|
||||||
|
recurrence_interval="monthly",
|
||||||
|
)
|
||||||
|
commit_pending(pending_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Standard recurring income:**
|
||||||
|
|
||||||
|
| Title | Amount | category |
|
||||||
|
|---|---|---|
|
||||||
|
| Kindergeld | +55.00 | Sozialleistungen |
|
||||||
|
| Cookie Share | +2650.00 | Erwerbseinkommen *(see recommended amount below)* |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 4 — Post variable transactions
|
||||||
|
|
||||||
|
For each non-skipped CSV row, call `stage_add_budget_entry` with the mapped
|
||||||
|
category/subcategory and the actual transaction amount and date. Use the Partner
|
||||||
|
Name + Payment Reference as the title (truncate to 100 chars).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 5 — Verify
|
||||||
|
|
||||||
|
```
|
||||||
|
get_budget_summary("YYYY-MM")
|
||||||
|
list_subscriptions()
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected for a full month with KuKita:
|
||||||
|
- Fixed expenses ≥ 1,838 € (subscriptions)
|
||||||
|
- Variable expenses ≥ 500 € (groceries alone)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cookie Share: how much to transfer monthly
|
||||||
|
|
||||||
|
Calculated from Jan–Jun 2026 data (Cookie account, one-offs stripped):
|
||||||
|
|
||||||
|
| | €/month |
|
||||||
|
|---|---|
|
||||||
|
| **Fixed costs (subscriptions)** | **1,839** |
|
||||||
|
| Miete | 1,080 |
|
||||||
|
| KuKita *(permanent from Jul 2026)* | 503 |
|
||||||
|
| Strom + SYNVIA + Rundfunk + Netflix + Grover + Hundefutter + Lillydoo | 256 |
|
||||||
|
| **Variable (6-month averages)** | **1,032** |
|
||||||
|
| Groceries | 595 |
|
||||||
|
| Children products | 142 |
|
||||||
|
| Dining & cafes | 100 |
|
||||||
|
| Transport | 66 |
|
||||||
|
| Drugstore | 52 |
|
||||||
|
| Clothing, Amazon, Pharmacy | 77 |
|
||||||
|
| **Total monthly spend** | **≈ 2,871** |
|
||||||
|
| Minus Kindergeld (fixed income) | −55 |
|
||||||
|
| Minus Pocket Money (conservative ~600 €) | −600 |
|
||||||
|
| **→ Recommended Cookie Share** | **≈ 2,650 €** |
|
||||||
|
| With 200 € buffer | **≈ 2,850 €** |
|
||||||
|
|
||||||
|
**Current Cookie Share (Jun 2026): 1,995 € — shortfall ~655 €.**
|
||||||
|
|
||||||
|
The gap was covered by irregular Pocket Money top-ups (avg 962 €/mo over 6 months, but
|
||||||
|
highly variable: 121 €–3,000 €). KuKita starting in June is the biggest step-up; raising
|
||||||
|
Cookie Share to **2,650 €** makes the budget self-sufficient without relying on top-ups.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-29 — Corrections from first real import
|
||||||
|
- Subscriptions endpoint is `/budget/subscriptions`, NOT `/subscriptions/` (404).
|
||||||
|
- `recurrence_rule` RRULE strings are rejected by the API; use `cycle_interval` instead.
|
||||||
|
- `GET /budget/` (no filter) returns only non-recurring entries; use `?month=` for full view.
|
||||||
|
- Added Cookie Share recommendation (2,650 €/month) based on 6-month expense analysis.
|
||||||
|
- Added full category taxonomy table.
|
||||||
|
|
||||||
|
### 2026-06-29 — Initial runbook
|
||||||
|
Created from Jan–Jun 2026 N26 Cookie account analysis.
|
||||||
@@ -1,3 +1,9 @@
|
|||||||
|
---
|
||||||
|
name: recover-dpkg-interrupted
|
||||||
|
risk_class: reversible_low
|
||||||
|
verification: "dpkg --audit (should be clean); apt-get check"
|
||||||
|
---
|
||||||
|
|
||||||
# Runbook — recover from dpkg-interrupted state
|
# Runbook — recover from dpkg-interrupted state
|
||||||
|
|
||||||
You're here because an apt run got killed mid-transaction and the target now
|
You're here because an apt run got killed mid-transaction and the target now
|
||||||
@@ -96,9 +102,9 @@ Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Operations cheatsheet](commands.md)
|
- [Operations cheatsheet](../../operations/commands.md)
|
||||||
- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
|
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
|
||||||
- [Hubris host page](../hosts/hubris.md)
|
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
30
.agents/skills/service-health-check/SKILL.md
Normal file
30
.agents/skills/service-health-check/SKILL.md
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
---
|
||||||
|
name: service-health-check
|
||||||
|
risk_class: read_only
|
||||||
|
inputs: [service_name]
|
||||||
|
verification: "homelab service <name> health"
|
||||||
|
docs_update_checklist: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Service health check
|
||||||
|
|
||||||
|
Goal: determine whether a service is actually healthy, without ad-hoc SSH.
|
||||||
|
|
||||||
|
1. `homelab service <name> explain` — read the context card: backend,
|
||||||
|
blast radius, doc pointer, risk notes.
|
||||||
|
2. `homelab service <name> health` — live health probe (HTTP code against
|
||||||
|
the service's `url`/`endpoint`). Once the Week-3 scheduler ships, this
|
||||||
|
reads a cached snapshot by default; pass `--live` to force a fresh probe.
|
||||||
|
3. If unhealthy, `homelab service <name> log` (or MCP `tail_log`) for the
|
||||||
|
last 200 lines.
|
||||||
|
4. Cross-check blast radius: `homelab node <name> relations` — is this
|
||||||
|
entity's own backend host healthy? A downstream failure (e.g. `strong`
|
||||||
|
down) will show up here before the service's own logs explain anything.
|
||||||
|
5. If the fix is a restart: classify first (`seeds/policy.yaml` —
|
||||||
|
`service-restart` is `reversible_low` unless the service has a
|
||||||
|
`service_overrides` entry, e.g. `caddy`/`dns` are `config_mutation`).
|
||||||
|
Unattended agents may act on `reversible_low` without approval.
|
||||||
|
|
||||||
|
Docs-update checklist: none for a pure health check. If the investigation
|
||||||
|
reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` in
|
||||||
|
the same session.
|
||||||
82
.agents/skills/session-review/SKILL.md
Normal file
82
.agents/skills/session-review/SKILL.md
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
---
|
||||||
|
name: session-review
|
||||||
|
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
|
||||||
|
risk_class: reversible_low
|
||||||
|
inputs: [session_id]
|
||||||
|
---
|
||||||
|
# Session review
|
||||||
|
|
||||||
|
Analyze Nomos chat sessions from the live database, diff objectives
|
||||||
|
against outcomes, and propose fixes.
|
||||||
|
|
||||||
|
## 1. Retrieve session data
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List recent sessions
|
||||||
|
curl -s http://localhost:8092/sessions | jq '.sessions[:5]'
|
||||||
|
|
||||||
|
# Fetch one session with messages
|
||||||
|
curl -s http://localhost:8092/sessions/{session_id} | jq .
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Classify the session
|
||||||
|
|
||||||
|
For each session determine:
|
||||||
|
|
||||||
|
| Dimension | Check |
|
||||||
|
|-----------|-------|
|
||||||
|
| Objective | What was the user trying to accomplish? |
|
||||||
|
| Outcome | Was it achieved? (read final assistant text) |
|
||||||
|
| Tool calls | Count, unique tools, redundancy (e.g., N+1 fan-out) |
|
||||||
|
| Blockers | Missing action? Missing tool? Model refusal? Empty response? |
|
||||||
|
| User frustration | Did the user need to clarify/correct/repeat? |
|
||||||
|
| Message sizes | Content blob sizes — truncation needed? |
|
||||||
|
|
||||||
|
## 3. Key failure signatures
|
||||||
|
|
||||||
|
| Signature | Root cause | Fix |
|
||||||
|
|-----------|-----------|-----|
|
||||||
|
| Agent: "I can't run X — only supports Y" | Missing action in `request_execution` | Add action in `internal/mcp/server.go` |
|
||||||
|
| Agent: "No local knowledge on that" + no web tool | Missing `http_get` / web fetch MCP tool | Add MCP tool |
|
||||||
|
| Empty assistant bubble (text="", no tools) | Model returned blank completion | Retry + error surfacing |
|
||||||
|
| Non-English boilerplate refusal | Flash-tier model degradation | Response quality guard |
|
||||||
|
| >30 tool calls per turn, same tool repeated | N+1 fan-out instead of bulk tool | Enrich bulk tools + tighten SOUL.md |
|
||||||
|
| Message >50KB in DB | Raw tool results persisted verbatim | Truncation in `store.go` |
|
||||||
|
|
||||||
|
## 4. Extract patterns across sessions
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# All sessions summary
|
||||||
|
curl -s http://localhost:8092/sessions | jq -r '.sessions[] | "\(.id[:8]) \(.title[:80]) \(.created_at[:16])"'
|
||||||
|
|
||||||
|
# Message count + tool count per session
|
||||||
|
for id in $(curl -s http://localhost:8092/sessions | jq -r '.sessions[].id'); do
|
||||||
|
msgs=$(curl -s "http://localhost:8092/sessions/$id" | jq '.messages | length')
|
||||||
|
tools=$(curl -s "http://localhost:8092/sessions/$id" | jq '[.messages[].content.tool_calls | length] | add')
|
||||||
|
echo "$id $msgs msgs $tools tools"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Output format
|
||||||
|
|
||||||
|
```
|
||||||
|
Session: {id[:8]} — "{title[:60]}"
|
||||||
|
Messages: {N} ({user}/{assistant})
|
||||||
|
Tool calls: {total} across {turns} turns
|
||||||
|
Top tools: {name:count, name:count, ...}
|
||||||
|
Objective: {one-line summary}
|
||||||
|
Outcome: ✅ / ❌ / ⚠️
|
||||||
|
Blockers: {list or "none"}
|
||||||
|
Fixes needed: {concrete actions}
|
||||||
|
Severity: blocker | friction | cosmetic
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related files
|
||||||
|
|
||||||
|
- `cmd/nomos/agent.go` — agent loop, tool building, response guards
|
||||||
|
- `cmd/nomos/store.go` — session + message persistence
|
||||||
|
- `internal/mcp/server.go` — all tool implementations including `request_execution`
|
||||||
|
- `web/src/lib/components/ToolCallGroup.svelte` — tool result display
|
||||||
|
- `nomos/SOUL.md` — agent persona and tool selection rules
|
||||||
|
- `plans/2026-07-09-chat-sessions-improvements.md` — prior session findings
|
||||||
|
- `plans/2026-07-09-session-execution-and-ux-fixes.md` — latest plan
|
||||||
11
.claude/launch.json
Normal file
11
.claude/launch.json
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"version": "0.0.1",
|
||||||
|
"configurations": [
|
||||||
|
{
|
||||||
|
"name": "web",
|
||||||
|
"runtimeExecutable": "npm",
|
||||||
|
"runtimeArgs": ["--prefix", "web", "run", "dev"],
|
||||||
|
"port": 5173
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
72
.gitea/workflows/ci.yml
Normal file
72
.gitea/workflows/ci.yml
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# Oikos CI (Gitea Actions). Gates the deploy webhook on a green run (plan M1).
|
||||||
|
# Mirrors `make lint`, `make test`, and the generated-code drift guard.
|
||||||
|
name: ci
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build-test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: timescale/timescaledb:2.17.2-pg16
|
||||||
|
env:
|
||||||
|
POSTGRES_DB: oikos
|
||||||
|
POSTGRES_USER: oikos
|
||||||
|
POSTGRES_PASSWORD: oikos_dev
|
||||||
|
ports:
|
||||||
|
- 5432:5432
|
||||||
|
options: >-
|
||||||
|
--health-cmd "pg_isready -U oikos"
|
||||||
|
--health-interval 5s
|
||||||
|
--health-timeout 5s
|
||||||
|
--health-retries 10
|
||||||
|
env:
|
||||||
|
OIKOS_TEST_DATABASE_URL: postgres://oikos:oikos_dev@postgres:5432/oikos?sslmode=disable
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- uses: actions/setup-go@v5
|
||||||
|
with:
|
||||||
|
go-version: "1.26"
|
||||||
|
cache: true
|
||||||
|
|
||||||
|
- name: go vet
|
||||||
|
run: go vet ./...
|
||||||
|
|
||||||
|
- name: golangci-lint
|
||||||
|
uses: golangci/golangci-lint-action@v6
|
||||||
|
with:
|
||||||
|
version: latest
|
||||||
|
args: --timeout 5m
|
||||||
|
continue-on-error: true # advisory until the lint baseline is clean
|
||||||
|
|
||||||
|
- name: govulncheck
|
||||||
|
run: |
|
||||||
|
go install golang.org/x/vuln/cmd/govulncheck@latest
|
||||||
|
govulncheck ./... || true # advisory
|
||||||
|
|
||||||
|
- name: generated code is up to date
|
||||||
|
run: make generate-check
|
||||||
|
|
||||||
|
- name: build
|
||||||
|
run: go build ./...
|
||||||
|
|
||||||
|
- name: test (race + coverage)
|
||||||
|
run: go test -race -covermode=atomic -coverprofile=coverage.out -timeout 300s ./...
|
||||||
|
|
||||||
|
- name: coverage gates (policy + learning ≥ 80%, others ≥ 60%)
|
||||||
|
run: |
|
||||||
|
go tool cover -func=coverage.out | tail -1
|
||||||
|
# Note: policy/ and learning/ packages land in Phase 3; enforce
|
||||||
|
# their 80% gate then. For now, report total coverage.
|
||||||
|
|
||||||
|
docker-build:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: docker build (verify image builds; no push)
|
||||||
|
run: docker build -f compose/oikos/Dockerfile -t oikos:ci .
|
||||||
27
.gitignore
vendored
27
.gitignore
vendored
@@ -1 +1,26 @@
|
|||||||
.DS_Store
|
.DS_Store
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
|
||||||
|
# Regenerated every scheduler run; ephemeral health-probe cache.
|
||||||
|
oikos/state.json
|
||||||
|
|
||||||
|
# Compiled binaries (Go rewrite — bin/oikos, bin/nomos)
|
||||||
|
bin/oikos
|
||||||
|
bin/nomos
|
||||||
|
oikos/oikos
|
||||||
|
|
||||||
|
# Legacy Python oikos (superseded by cmd/oikos Go binary — Phase 1-6 rewrite).
|
||||||
|
# oikos/ kernel files are still imported by bin/homelab for operational CLI
|
||||||
|
# commands (ssh, pct, logs, restart, status, open, secret, client, sync, mcp).
|
||||||
|
# Remove oikos/* when bin/homelab is ported to Go.
|
||||||
|
backups/
|
||||||
|
.env
|
||||||
|
.infisical-credentials
|
||||||
|
|
||||||
|
# Web UI (Svelte 5) — build artifacts. Ignore built output but keep the
|
||||||
|
# .gitkeep placeholder so `//go:embed all:dist` (web/embed.go) compiles on a
|
||||||
|
# fresh checkout before the UI is built.
|
||||||
|
web/dist/*
|
||||||
|
!web/dist/.gitkeep
|
||||||
|
web/node_modules/
|
||||||
|
|||||||
92
.sops.yaml
92
.sops.yaml
@@ -1,14 +1,12 @@
|
|||||||
# SOPS recipient rules.
|
# SOPS recipient rules.
|
||||||
#
|
#
|
||||||
# Each rule pins one or more age public keys as recipients for files matching
|
# SOPS is DR-fallback only — Infisical is the active secrets backend.
|
||||||
# the path_regex. The build_host_files.py generator doesn't read this file;
|
# Files live in archive/secrets-sops-backup/ for cold recovery.
|
||||||
# `sops` does — to encrypt a new secret, run `sops -e secrets/<name>.yaml`
|
# To encrypt a new DR secret: sops -e archive/secrets-sops-backup/<name>.yaml
|
||||||
# from the repo root and SOPS will pick the matching rule below.
|
|
||||||
#
|
#
|
||||||
# To grant a secret to a new client: add their age public key (from
|
# To grant a DR secret to a new client: add their age public key (from
|
||||||
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
|
# inventory.yaml `hosts.<name>.age_pubkey`) to the relevant rule below, then
|
||||||
# run `sops updatekeys secrets/<name>.yaml` to re-encrypt without rotating
|
# run `sops updatekeys archive/secrets-sops-backup/<name>.yaml` to re-encrypt.
|
||||||
# the ciphertext payload.
|
|
||||||
#
|
#
|
||||||
# To revoke: remove the recipient from the relevant rule and run
|
# To revoke: remove the recipient from the relevant rule and run
|
||||||
# `sops updatekeys` (this is what `homelab client remove` calls). Past
|
# `sops updatekeys` (this is what `homelab client remove` calls). Past
|
||||||
@@ -16,7 +14,7 @@
|
|||||||
# underlying credential if compromise is suspected.
|
# underlying credential if compromise is suspected.
|
||||||
|
|
||||||
creation_rules:
|
creation_rules:
|
||||||
- path_regex: ^secrets/hello\.yaml$
|
- path_regex: ^archive/secrets-sops-backup/hello\.yaml$
|
||||||
# The "hello" secret is encrypted to every enrolled client so the bootstrap
|
# The "hello" secret is encrypted to every enrolled client so the bootstrap
|
||||||
# decrypt test works for everyone. Add each new client's age_pubkey when
|
# decrypt test works for everyone. Add each new client's age_pubkey when
|
||||||
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
|
# they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
|
||||||
@@ -24,9 +22,12 @@ creation_rules:
|
|||||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||||
|
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
|
||||||
|
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||||
|
|
||||||
- path_regex: ^secrets/gitea-pat\.yaml$
|
- path_regex: archive/secrets-sops-backupgitea-pat\.yaml$
|
||||||
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
|
# Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
|
||||||
# since every enrolled client should be able to push (homelab client
|
# since every enrolled client should be able to push (homelab client
|
||||||
# add/remove, wiki edits, etc.).
|
# add/remove, wiki edits, etc.).
|
||||||
@@ -34,19 +35,22 @@ creation_rules:
|
|||||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||||
|
age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4,
|
||||||
|
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||||
|
|
||||||
- path_regex: ^secrets/gitea-tokens\.yaml$
|
- path_regex: archive/secrets-sops-backupgitea-tokens\.yaml$
|
||||||
# Workstations only.
|
# Workstations only.
|
||||||
age: >-
|
age: >-
|
||||||
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris
|
# placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris
|
||||||
|
|
||||||
- path_regex: ^secrets/webhook-hmacs\.yaml$
|
- path_regex: archive/secrets-sops-backupwebhook-hmacs\.yaml$
|
||||||
# LXCs that run a webhook receiver.
|
# LXCs that run a webhook receiver.
|
||||||
age: >-
|
age: >-
|
||||||
# placeholder — fill with age_pubkey of: apps, caddy
|
# placeholder — fill with age_pubkey of: apps, caddy
|
||||||
|
|
||||||
- path_regex: ^secrets/turn-shared-secret\.yaml$
|
- path_regex: archive/secrets-sops-backupturn-shared-secret\.yaml$
|
||||||
# coturn TURN long-term-credential password. Consumed by hubris (which
|
# coturn TURN long-term-credential password. Consumed by hubris (which
|
||||||
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
|
# renders /etc/turnserver.conf + /opt/management.json on the VPS via
|
||||||
# `homelab render-vps-configs`). Other recipients are convenience for
|
# `homelab render-vps-configs`). Other recipients are convenience for
|
||||||
@@ -55,9 +59,11 @@ creation_rules:
|
|||||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||||
|
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||||
|
|
||||||
- path_regex: ^secrets/netbird-authentik-oidc\.yaml$
|
- path_regex: archive/secrets-sops-backupnetbird-authentik-oidc\.yaml$
|
||||||
# Authentik OIDC client secret for the netbird-dashboard provider.
|
# Authentik OIDC client secret for the netbird-dashboard provider.
|
||||||
# Consumed by hubris to render /opt/management.json on the VPS
|
# Consumed by hubris to render /opt/management.json on the VPS
|
||||||
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
|
# (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
|
||||||
@@ -65,9 +71,11 @@ creation_rules:
|
|||||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
|
||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h,
|
||||||
|
age1pwtdws2thdh7vzp2dzttl3zxgcs2tgpcsjsqgw3q04nyml4kvuqq467u4x
|
||||||
|
|
||||||
- path_regex: ^secrets/netbird-pat\.yaml$
|
- path_regex: archive/secrets-sops-backupnetbird-pat\.yaml$
|
||||||
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
|
# NetBird API Personal Access Token. Consumed by the dns-sync job on the
|
||||||
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
|
# `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
|
||||||
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
|
# (When 107 is enrolled, add its age_pubkey here and updatekeys.)
|
||||||
@@ -77,14 +85,50 @@ creation_rules:
|
|||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
||||||
|
|
||||||
- path_regex: ^secrets/openrouter-api-key\.yaml$
|
- path_regex: archive/secrets-sops-backupopenrouter-api-key\.yaml$
|
||||||
# OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
|
# OpenRouter API key consumed by the `nomos` wrapper (bin/nomos) when
|
||||||
# spawning a Goose session. Recipients are any host that should run a
|
# spawning a Goose session. Recipients are any host that should run a
|
||||||
# Nous-Hermes agent. Add a host's age_pubkey here, then
|
# Nomos agent. Add a host's age_pubkey here, then
|
||||||
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
|
# `sops updatekeys -y secrets/openrouter-api-key.yaml`.
|
||||||
# See operations/hermes-agent.md.
|
# See operations/nomos-agent.md.
|
||||||
age: >-
|
age: >-
|
||||||
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||||
|
|
||||||
|
- path_regex: archive/secrets-sops-backupyuvomi-api-token\.yaml$
|
||||||
|
# Named Bearer token for the Yuvomi REST API, consumed by yuvomi-mcp on
|
||||||
|
# LXC 129 (house).
|
||||||
|
age: >-
|
||||||
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||||
|
|
||||||
|
- path_regex: archive/secrets-sops-backuphermes-house-users\.yaml$
|
||||||
|
# Signal number → Yuvomi user_id mapping (PII). Consumed by hermesd on LXC 129.
|
||||||
|
age: >-
|
||||||
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
|
age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
|
||||||
|
age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs,
|
||||||
|
age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
|
||||||
|
|
||||||
|
- path_regex: archive/secrets-sops-backupoikos-approval-hmac\.yaml$
|
||||||
|
# HMAC signing key for Oikos approval-grant tokens (oikos/approve.py).
|
||||||
|
# Recipients: apps (105, runs the approval engine alongside homelab-mcp)
|
||||||
|
# and hubris (admin/debug decrypt). See OIKOS.md "Approval engine".
|
||||||
|
age: >-
|
||||||
|
age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
|
||||||
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
|
||||||
|
|
||||||
|
- path_regex: archive/secrets-sops-backupoikos-console-deploy-secret\.yaml$
|
||||||
|
# Shared HMAC secret for the Gitea deploy webhook (id 14) ->
|
||||||
|
# oikos-console-deploy.service on apps (105). Generated + registered
|
||||||
|
# with Gitea before the apps-side install ran (see
|
||||||
|
# oikos/console/deploy/README.md "Status") — write this exact value
|
||||||
|
# into /etc/oikos-console-deploy/secret rather than letting
|
||||||
|
# webhook/install.sh generate a fresh one.
|
||||||
|
age: >-
|
||||||
|
age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
|
||||||
# webhook noop 2026-05-20T18:16:57+02:00
|
# webhook noop 2026-05-20T18:16:57+02:00
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
175
AGENTS.md
175
AGENTS.md
@@ -4,11 +4,33 @@ You are running on a machine that is part of the **hubris** homelab. The full
|
|||||||
context is in this checkout at `/opt/homelab-context/`. This file is the entry
|
context is in this checkout at `/opt/homelab-context/`. This file is the entry
|
||||||
point. Read it once at start, then keep working.
|
point. Read it once at start, then keep working.
|
||||||
|
|
||||||
|
- **New client?** Read [CLIENTS.md](CLIENTS.md) first.
|
||||||
|
- **Developing on this repo?** Also read [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
|
||||||
|
|
||||||
|
The operating model — OODA loop, risk classes, approval rules, the ontology,
|
||||||
|
and node lifecycle — is defined in [OIKOS.md](.agents/OIKOS.md). Before any mutation,
|
||||||
|
classify the action against `seeds/policy.yaml`; when the class requires
|
||||||
|
approval, stop and ask the operator.
|
||||||
|
|
||||||
|
Agent-facing instruction lives under `.agents/`:
|
||||||
|
`.agents/shared/` holds the conventions every agent applies
|
||||||
|
([writing-style](.agents/shared/writing-style.md), [caveman](.agents/shared/caveman.md),
|
||||||
|
[page-templates](.agents/shared/page-templates.md), [llm-wiki](.agents/shared/llm-wiki.md)), and
|
||||||
|
`.agents/domains/` holds the per-domain schemas
|
||||||
|
([knowledge](.agents/domains/knowledge/schema.md), [operations](.agents/domains/operations/schema.md)).
|
||||||
|
|
||||||
|
**Source of truth:** The Postgres database is the single source of truth for all
|
||||||
|
structured data and knowledge. It is bootstrapped from `seeds/` at deploy time:
|
||||||
|
`seeds/ontology.yaml` (entity types, relationships, lifecycles), `seeds/inventory.yaml`
|
||||||
|
(hosts, services, entities), `seeds/policy.yaml` (risk classes, approval rules), and
|
||||||
|
`seeds/knowledge.yaml` (documents, investigations, runbooks). The old narrative wiki
|
||||||
|
is archived at `archive/knowledge/` for historical reference.
|
||||||
|
|
||||||
## 1. Who you are
|
## 1. Who you are
|
||||||
|
|
||||||
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
|
Run `hostname` (Linux) or `scutil --get LocalHostName` (macOS), then read:
|
||||||
|
|
||||||
/opt/homelab-context/hosts/<your-hostname>.yaml
|
/opt/homelab-context/inventory.yaml
|
||||||
|
|
||||||
That file tells you your role, your peers, what's mounted, and what services
|
That file tells you your role, your peers, what's mounted, and what services
|
||||||
you host. If it does not exist, this client was not enrolled — stop and tell
|
you host. If it does not exist, this client was not enrolled — stop and tell
|
||||||
@@ -17,73 +39,109 @@ the operator to run `homelab client add <hostname>` from an existing client.
|
|||||||
## 2. The topology
|
## 2. The topology
|
||||||
|
|
||||||
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
|
- `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
|
||||||
with their mesh addresses, roles, and service mappings. Treat this file as
|
with their mesh addresses, roles, and service mappings. This is the seed file;
|
||||||
authoritative; anything you read in narrative pages should agree with it.
|
at runtime the DB is authoritative (query via MCP `get_entity` or the REST API).
|
||||||
- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state.
|
- `/opt/homelab-context/seeds/knowledge.yaml` — full narrative knowledge: 36
|
||||||
Both meshes are accepted today; Netbird is preferred for new traffic.
|
documents, 6 investigations, 12 runbooks. Ingested into the DB on deploy.
|
||||||
- `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via
|
- `/opt/homelab-context/.agents/operations/commands.md` — the operator's cheatsheet
|
||||||
dnsmasq on LXC 124. `*.hubris.network` resolves to 192.168.x.x on the LAN
|
for pct, caddy, DNS, and the Oikos command surface.
|
||||||
and to mesh addresses off-LAN.
|
|
||||||
- `/opt/homelab-context/operations/commands.md` — the operator's cheatsheet
|
|
||||||
for pct, caddy, dnsmasq. Use these verbs when you take actions.
|
|
||||||
|
|
||||||
## 3. The MCP server
|
## 3. The MCP server
|
||||||
|
|
||||||
The homelab exposes a Model Context Protocol server with structured tools.
|
The homelab exposes a Model Context Protocol server with structured tools.
|
||||||
Endpoint is in `inventory.yaml` under `services.homelab_mcp.endpoint`.
|
Endpoint: `https://mcp.hubris.network/mcp`.
|
||||||
|
|
||||||
Available tools:
|
Available tools (21 total):
|
||||||
|
|
||||||
Context (pure read):
|
Context — observe + orient:
|
||||||
get_host(name), list_services(), find_service(name_or_role),
|
get_entity(slug), list_entities(type, limit, cursor),
|
||||||
get_topology(), search_docs(query), get_page(path),
|
get_relations(entity), get_blast_radius(entity),
|
||||||
get_changelog(page, since?), whoami(hostname),
|
search_knowledge(query) — ILIKE search over documents, investigations,
|
||||||
list_my_secrets(caller_pubkey?)
|
runbooks in the knowledge_entities table
|
||||||
|
get_patterns(status, entity_type, action) — learned action patterns
|
||||||
|
get_skills(status) — available automation skills
|
||||||
|
|
||||||
Management (read-only):
|
Management — live state:
|
||||||
get_service_status(service), tail_log(service, lines=200),
|
get_service_status(service_slug) — systemctl is-active on target host
|
||||||
list_lxcs(), get_lxc_state(lxc), ping_service(service)
|
tail_log(service_slug, lines=200) — journalctl
|
||||||
|
list_lxcs() — all LXC containers with ID, host, IP, health
|
||||||
|
get_lxc_state(lxc_slug) — pct status from Proxmox host
|
||||||
|
ping_service(service_slug) — HTTP reachability from entity_status
|
||||||
|
|
||||||
Mutations are **not** exposed via MCP. Use the `homelab` CLI for those, with
|
Oikos — decisions:
|
||||||
operator confirmation.
|
explain(service_slug) — compact context card (type, state, health, relations)
|
||||||
|
preflight(service_slug, action) — risk class + approval requirement
|
||||||
|
whoami(hostname) — entity record, peers, health for a client
|
||||||
|
get_change_history(entity_slug, limit=20) — last audit-log entries per entity
|
||||||
|
get_state_snapshot() — fleet health, disk, drift count
|
||||||
|
|
||||||
**When to prefer MCP over grepping the clone:** any time you need to resolve a
|
Operations — observe + act:
|
||||||
name to an address, look up service status, or search the wiki by content.
|
get_health_summary() — fleet health counts (healthy/degraded/down/unknown)
|
||||||
Grep is fine for browsing or when MCP is unreachable.
|
get_signal_history(entity_slug, state, limit) — open + recent signals
|
||||||
|
get_audit_trail(entity_id) — audit log filter + browse
|
||||||
|
get_agent_activity(limit) — agent self-inspection
|
||||||
|
query_metrics(hours=24) — time-series metric bucketed averages
|
||||||
|
get_trend(entity_id, days=7) — metric slope over time
|
||||||
|
get_event_timeline(severity, entity_slug, limit) — recent events
|
||||||
|
|
||||||
## 4. Wiki conventions
|
Execution — the single mutation path:
|
||||||
|
request_execution(target, action, params) — policy-gated.
|
||||||
|
reversible_low (restart, reload, pct_exec, apt audit) runs immediately;
|
||||||
|
config_mutation (systemctl enable/disable, apt upgrade) queues for operator
|
||||||
|
approval via Matrix, then executes on ✅.
|
||||||
|
get_execution_status(execution_id) — poll progress
|
||||||
|
|
||||||
- Pages live under `containers/`, `hosts/`, `vms/`, `infrastructure/`,
|
**When to prefer MCP over grepping the clone:** always for knowledge queries.
|
||||||
`investigations/`, `operations/`. Cross-link liberally; orphans are bugs.
|
`search_knowledge("jellyfin hardware acceleration")` returns ranked results from
|
||||||
- Every page ends with a `## Changelog` section, entries in reverse-chrono
|
the DB with entity links. `get_entity_knowledge("lxc:jellyfin")` returns documents,
|
||||||
order:
|
runbooks, and investigations in one call. Grep the clone only when MCP is
|
||||||
|
unreachable.
|
||||||
|
|
||||||
### YYYY-MM-DD — short title
|
## 4. Knowledge conventions
|
||||||
one or two lines describing what changed and why.
|
|
||||||
|
|
||||||
- Investigation files are dated and slugged: `YYYY-MM-DD-slug.md`.
|
All narrative knowledge (documents, investigations, runbooks) lives in the DB
|
||||||
- Live state takes precedence over docs. If you observe a discrepancy, update
|
(`knowledge_entities` table) and is seeded from `seeds/knowledge.yaml`. Agents
|
||||||
the docs *in the same session* (per the same-session update rule).
|
can register new knowledge via the API:
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/v1/knowledge/{entity_slug}
|
||||||
|
{"title": "...", "content": "...", "tags": ["..."]}
|
||||||
|
```
|
||||||
|
|
||||||
|
The DB is the truth. The old wiki files are in `knowledge/wiki/` pending archive
|
||||||
|
per the DB-as-source-of-truth plan.
|
||||||
|
|
||||||
|
- **Runbook procedures** live as `runbook` entities in the DB and as SKILL.md
|
||||||
|
files under `.agents/skills/<name>/`. They carry `risk_class`, `procedure`
|
||||||
|
(JSON-schema-validated), and are linked to entity types via `applies_to_type`.
|
||||||
|
- **Investigations** are `investigation` entities linked to affected entities
|
||||||
|
via `about` edges.
|
||||||
|
- **Documents** are `document` entities linked to entities via `documents` edges.
|
||||||
|
They carry `at_glance` (structured attributes) and `changelog` (parsed entries).
|
||||||
|
- **Live state precedence.** If you observe a discrepancy between the docs and
|
||||||
|
running state, update the DB *in the same session* via the API. The `oikos export`
|
||||||
|
command regenerates `seeds/knowledge.yaml` for version control.
|
||||||
|
|
||||||
## 5. Acting on the homelab
|
## 5. Acting on the homelab
|
||||||
|
|
||||||
- **Read state**: prefer MCP tools, then files, then shell. Examples:
|
- **Read state**: use MCP tools. Nomos (the AI agent) is the primary
|
||||||
`homelab whoami`, `homelab list`, `homelab status`, `homelab logs caddy`.
|
operator interface — it has 21 MCP tools for observe/orient/decide/act.
|
||||||
- **Cross-host actions** (caddy reload, pct exec, etc.): use the `homelab`
|
- **Actions** (restart, logs, apt, pct exec): Nomos calls `request_execution`
|
||||||
CLI — it resolves hostname → mesh address → ssh / pct path for you. Direct
|
via MCP. `reversible_low` actions execute immediately; `config_mutation`
|
||||||
SSH still works; the CLI just removes the lookup burden.
|
and `destructive` actions are queued for operator approval via Matrix.
|
||||||
- **Secrets**: never hardcode. Call `homelab secret <name>` to decrypt on
|
- **Secrets**: managed by Infisical (`oikos secret` subcommand for migration).
|
||||||
demand using the per-client age key at `/etc/age/key.txt`. Secrets ARE
|
Never hardcode secrets — use env vars from `.env`.
|
||||||
available in this system — `list_my_secrets()` (MCP) shows what you can
|
- **Mutations** (restart, edit configs, etc.): classified against
|
||||||
decrypt.
|
`seeds/policy.yaml`. `reversible_low` actions auto-execute;
|
||||||
- **Mutations** (restart, edit configs, etc.): the `homelab` CLI's mutating
|
`config_mutation`/`destructive` actions require approval.
|
||||||
subcommands ask for confirmation. For ad-hoc work, SSH and edit directly —
|
a valid `--approval-id` from `homelab approval request` — see OIKOS.md.
|
||||||
but commit changes that touch tracked configs (caddy, gitea custom,
|
|
||||||
artifacto, mule-image, etc.; see `infrastructure/auto-deploy.md`).
|
|
||||||
- **Wiki updates**: same-session rule applies to any meaningful state change
|
|
||||||
this client makes.
|
|
||||||
|
|
||||||
## 6. Auto-setup mechanism
|
## 6. Communication mode
|
||||||
|
|
||||||
|
Read and apply `/opt/homelab-context/.agents/shared/caveman.md` (if present). It defines the lab's
|
||||||
|
terse-communication standard — drop filler, keep substance, use fragments.
|
||||||
|
|
||||||
|
## 7. Auto-setup mechanism
|
||||||
|
|
||||||
The homelab-context repo ships tooling that gets automatically installed
|
The homelab-context repo ships tooling that gets automatically installed
|
||||||
on every client after `git pull`. This is handled by `tools/post-pull.sh`
|
on every client after `git pull`. This is handled by `tools/post-pull.sh`
|
||||||
@@ -94,17 +152,18 @@ Currently auto-setup:
|
|||||||
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
|
- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
|
||||||
package, wrapper scripts, and compact output templates for token-efficient
|
package, wrapper scripts, and compact output templates for token-efficient
|
||||||
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
|
CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
|
||||||
- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
|
- **Nomos agent persona** (`tools/setup-nomos-soul.sh`): Provisions
|
||||||
`~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
|
`~/.nomos/SOUL.md` from `NOMOS.md` on Nomos agents. This ensures every
|
||||||
Hermes agent follows the canonical homelab persona (token efficiency, source
|
Nomos agent follows the canonical homelab persona (token efficiency, source
|
||||||
of truth hierarchy). No-op on non-Hermes agents.
|
of truth hierarchy). No-op on non-Nomos agents.
|
||||||
|
|
||||||
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
|
To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
|
||||||
commit and push. All enrolled clients pick it up within 5 minutes.
|
commit and push. All enrolled clients pick it up within 5 minutes.
|
||||||
|
|
||||||
To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
|
To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
|
||||||
|
|
||||||
## 7. When in doubt
|
## 8. When in doubt
|
||||||
|
|
||||||
Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
|
Use MCP tools: `search_knowledge <query>` for narrative context,
|
||||||
The clone is the fallback; MCP is the index.
|
`get_entity <slug>` for structured data, `get_entity_knowledge <slug>` for
|
||||||
|
everything linked to an entity. The clone is the fallback; MCP is the index.
|
||||||
139
CLIENTS.md
Normal file
139
CLIENTS.md
Normal file
@@ -0,0 +1,139 @@
|
|||||||
|
# Oikos — client guide
|
||||||
|
|
||||||
|
If you are a homelab machine, this is what Oikos is and what it gives you.
|
||||||
|
If you are an AI agent running on this machine, also read [AGENTS.md](AGENTS.md).
|
||||||
|
|
||||||
|
## What is Oikos?
|
||||||
|
|
||||||
|
Oikos is the agentic operating system for the **hubris** Proxmox homelab. It
|
||||||
|
observes state, classifies actions against policy, executes approved
|
||||||
|
procedures, learns from outcomes, and notifies the operator. It runs as a
|
||||||
|
Docker stack on mac-mini and exposes an MCP server + REST API.
|
||||||
|
|
||||||
|
## What Oikos provides
|
||||||
|
|
||||||
|
| Capability | How you access it |
|
||||||
|
|------------|-------------------|
|
||||||
|
| Entity query (topology, blast radius) | MCP `get_entity`, `get_blast_radius` |
|
||||||
|
| Full-text knowledge search | MCP `search_knowledge` |
|
||||||
|
| Service status + logs | MCP `get_service_status`, `tail_log` |
|
||||||
|
| LXC inventory + state | MCP `list_lxcs`, `get_lxc_state` |
|
||||||
|
| Context cards | MCP `explain` |
|
||||||
|
| Pre-flight risk classification | MCP `preflight` |
|
||||||
|
| Change history | MCP `get_change_history` |
|
||||||
|
| State snapshot (health, disk, drift) | MCP `get_state_snapshot` |
|
||||||
|
| Secrets (Infisical) | REST API + `oikos secret` CLI |
|
||||||
|
| Approval tokens | Matrix via notifier |
|
||||||
|
|
||||||
|
All MCP tools are read-only. Mutations use the `homelab` CLI with operator
|
||||||
|
approval.
|
||||||
|
|
||||||
|
## Enrollment
|
||||||
|
|
||||||
|
Thin client model — no git clone, no sync timer. `bootstrap.sh` fetches only
|
||||||
|
the agent orientation files and tooling from the raw Gitea URL, then enrolls
|
||||||
|
via the Oikos API.
|
||||||
|
|
||||||
|
To enroll:
|
||||||
|
```bash
|
||||||
|
# Run from any machine with mesh connectivity
|
||||||
|
curl -fsSL https://git.hubris.network/dtoro/oikos/raw/main/bootstrap.sh | sudo bash
|
||||||
|
|
||||||
|
# Or with optional tooling:
|
||||||
|
curl ... | sudo bash -s -- --with-mcp # wire Claude's MCP config
|
||||||
|
curl ... | sudo bash -s -- --with-nomos # install Goose + Nomos
|
||||||
|
```
|
||||||
|
|
||||||
|
This calls `POST /api/v1/clients/enroll` on the Oikos API, which:
|
||||||
|
1. Validates the entity exists in DB (planned or provisioning state)
|
||||||
|
2. Validates mesh IP against expected subnets
|
||||||
|
3. Generates an age keypair and delivers it to the client
|
||||||
|
4. Creates an Infisical machine identity
|
||||||
|
5. Transitions the entity to provisioning state
|
||||||
|
|
||||||
|
## After enrollment
|
||||||
|
|
||||||
|
### What changes on your machine
|
||||||
|
|
||||||
|
- `/opt/homelab/` — agent orientation files (CLIENTS.md, AGENTS.md, OIKOS.md)
|
||||||
|
- `/opt/homelab/tools/` — tooling scripts (caveman, nomos-soul)
|
||||||
|
- `/etc/age/key.txt` — age private key for SOPS decryption (fallback)
|
||||||
|
- `/etc/infisical/identity` — Infisical machine identity (primary secrets)
|
||||||
|
- Context poller — launchd/systemd timer hits `GET /api/v1/clients/{slug}/context` every 5 minutes for agent file updates
|
||||||
|
|
||||||
|
### What's NOT on your machine
|
||||||
|
|
||||||
|
- No git clone of the full repo
|
||||||
|
- No `git pull` sync timer
|
||||||
|
- No `bin/homelab` CLI (replaced by MCP tools + API)
|
||||||
|
- No `.sops.yaml` or SOPS-encrypted backups (served via API context endpoint on demand)
|
||||||
|
|
||||||
|
### Thin client vs control plane
|
||||||
|
|
||||||
|
| | Thin client (workstation) | Control plane (mac-mini) |
|
||||||
|
|---|---|---|
|
||||||
|
| Disk footprint | ~100KB (orientation files + tools) | Full repo clone (~50MB) |
|
||||||
|
| Update mechanism | `GET /context?since=` poll | Git pull + post-pull.sh |
|
||||||
|
| Source of truth | DB via MCP | DB + local seeds + archive |
|
||||||
|
| Secrets access | Infisical (primary), age/SOPS served via API (fallback) | Infisical + local SOPS files |
|
||||||
|
|
||||||
|
### Your identity
|
||||||
|
|
||||||
|
Your identity in the homelab is defined in `inventory.yaml`. Run `hostname`
|
||||||
|
(Linux) or `scutil --get LocalHostName` (macOS), then look up your entry.
|
||||||
|
It tells you your role, what services you host, what's mounted, and your
|
||||||
|
mesh address.
|
||||||
|
|
||||||
|
### Source of truth hierarchy
|
||||||
|
|
||||||
|
1. **Postgres database** (runtime) — authoritative for entities, knowledge,
|
||||||
|
signals, ledger. Query via MCP or REST API.
|
||||||
|
2. **Context poller** — agent files and tooling fetched via API deltas every
|
||||||
|
5 minutes.
|
||||||
|
3. **Never guess.** If data is missing, query MCP. If MCP is down, grep the
|
||||||
|
local `/opt/homelab/` files.
|
||||||
|
|
||||||
|
## The context poller
|
||||||
|
|
||||||
|
Every 5 minutes, launchd (macOS) or systemd (Linux) hits:
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /api/v1/clients/ws:{hostname}/context?since={last_timestamp}
|
||||||
|
```
|
||||||
|
|
||||||
|
The API returns which agent files, tools, and SOPS config changed since the
|
||||||
|
last poll. Only changed files are downloaded. This replaces the old
|
||||||
|
`git pull` with a lightweight HTTP delta.
|
||||||
|
|
||||||
|
To trigger manually: run `/opt/homelab/tools/context-poller.sh`.
|
||||||
|
|
||||||
|
## Making changes
|
||||||
|
|
||||||
|
- **Read state**: use MCP tools or the API
|
||||||
|
- **Mutate state** (restart, edit config, deploy): classify the action against
|
||||||
|
policy (query `preflight` MCP tool):
|
||||||
|
- `read_only` / `reversible_low` — execute directly
|
||||||
|
- `config_mutation` / `destructive` — request operator approval via
|
||||||
|
`POST /api/v1/entities/{slug}/activate` (or equivalent lifecycle endpoint)
|
||||||
|
- **Secrets**: use Infisical (primary) or SOPS (fallback). Never hardcode.
|
||||||
|
- **Knowledge**: if you observe a discrepancy between docs and live state,
|
||||||
|
update the DB via the API in the same session.
|
||||||
|
|
||||||
|
## MCP endpoint
|
||||||
|
|
||||||
|
```
|
||||||
|
https://mcp.hubris.network/mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
Available tools are listed in [AGENTS.md](AGENTS.md#3-the-mcp-server).
|
||||||
|
|
||||||
|
## Communication mode
|
||||||
|
|
||||||
|
Apply [.agents/shared/caveman.md](.agents/shared/caveman.md) — terse,
|
||||||
|
fragment-heavy communication. Drop filler, keep substance.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [AGENTS.md](AGENTS.md) — full agent orientation (read this first)
|
||||||
|
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, risk classes
|
||||||
|
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide for the Oikos codebase
|
||||||
250
CONTRIBUTING.md
250
CONTRIBUTING.md
@@ -1,123 +1,155 @@
|
|||||||
# Contributing to the Homelab Wiki
|
# Contributing to Oikos
|
||||||
|
|
||||||
## Voice
|
Developer guide for the Oikos codebase. If you are a homelab client consuming
|
||||||
|
Oikos, see [CLIENTS.md](CLIENTS.md). If you are an AI agent working on the
|
||||||
|
repo, see [.agents/dev/CONTRIBUTING.md](.agents/dev/CONTRIBUTING.md).
|
||||||
|
|
||||||
Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks.
|
## Dev setup
|
||||||
|
|
||||||
## Page templates
|
- **Go 1.26+** (see `go.mod` for pinned version)
|
||||||
|
- **PostgreSQL with TimescaleDB** — the compose stack includes `timescale/timescaledb:2.17.2-pg16`
|
||||||
|
- **Docker** for the full dev stack
|
||||||
|
|
||||||
### Container page (`containers/<id>-<name>.md`)
|
```bash
|
||||||
|
# Start dependencies (Postgres + Redis)
|
||||||
|
docker compose --profile dev up -d
|
||||||
|
|
||||||
```markdown
|
# Run all tests
|
||||||
# <id> — `<name>`
|
make test
|
||||||
|
|
||||||
One-sentence purpose.
|
# Run integration tests (needs compose Postgres)
|
||||||
|
make test-db
|
||||||
|
|
||||||
## At a glance
|
# Build the binary
|
||||||
- **Hostname:** `<name>`
|
make build
|
||||||
- **IP:** `192.168.8.x`
|
```
|
||||||
- **Privilege:** privileged | unprivileged
|
|
||||||
- **Resources:** N cores / M GiB RAM / D GiB rootfs
|
|
||||||
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any)
|
|
||||||
- **Public hostname:** `<sub>.hubris.network` (if proxied)
|
|
||||||
|
|
||||||
## Role
|
## Project structure
|
||||||
What it does, what it talks to.
|
|
||||||
|
|
||||||
## Service / port map
|
```
|
||||||
| Service | Listen | Notes |
|
cmd/oikos/ Single-binary entry point
|
||||||
|
cmd/nomos/ Nomos MCP client gateway
|
||||||
|
internal/ All Go packages
|
||||||
|
httpapi/ REST + MCP server (OpenAPI-generated)
|
||||||
|
mcp/ MCP tool implementations
|
||||||
|
db/ Connection pool, migrations, seeds, sqlc queries
|
||||||
|
scheduler/ Observe loop, probes, signals
|
||||||
|
actuator/ SSH execution
|
||||||
|
learning/ Pattern recognition, anomaly detection
|
||||||
|
notifier/ Matrix notifications, approval tokens
|
||||||
|
policy/ Risk classifier
|
||||||
|
secrets/ Infisical + SOPS backend
|
||||||
|
domain/ Core types: entities, approvals, signals, patterns
|
||||||
|
ontology/ Type hierarchy, relationship validation
|
||||||
|
knowledge/ Knowledge YAML seed ingestion
|
||||||
|
api/openapi.yaml API contract — the source of truth for endpoints
|
||||||
|
migrations/ Forward-only SQL migrations (TimescaleDB)
|
||||||
|
seeds/ Bootstrap YAML: ontology, inventory, policy, knowledge
|
||||||
|
compose/ Dockerfiles + Caddy config
|
||||||
|
scripts/ Deploy, watchdog, rollback
|
||||||
|
nomos/ Nomos config, persona, skills
|
||||||
|
.agents/ Agent instruction files + skills
|
||||||
|
plans/ Design documents
|
||||||
|
docs/adr/ Architecture decision records
|
||||||
|
```
|
||||||
|
|
||||||
## Storage / config paths
|
## Commands
|
||||||
|
|
||||||
## Auto-deploy
|
| Command | Purpose |
|
||||||
(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
|
|---------|---------|
|
||||||
|
| `make build` | Build `oikos` binary |
|
||||||
|
| `make test` | Run all tests with race detection |
|
||||||
|
| `make test-db` | Run integration tests against compose Postgres |
|
||||||
|
| `make lint` | `go vet` + `golangci-lint` |
|
||||||
|
| `make generate` | Regenerate OpenAPI + sqlc code |
|
||||||
|
| `make generate-check` | CI drift guard — fail if generated code is stale |
|
||||||
|
| `make migrate` | Apply DB migrations |
|
||||||
|
| `make seed` | Ingest seeds into DB |
|
||||||
|
| `make export` | Export DB state to YAML seeds |
|
||||||
|
| `make dev` | Start compose dev stack |
|
||||||
|
| `make clean` | Remove binary + test cache |
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
### APIs are OpenAPI-first
|
||||||
|
|
||||||
|
The REST API is defined in `api/openapi.yaml`. Server code is generated with
|
||||||
|
`oapi-codegen` into `internal/httpapi/gen/`. To add an endpoint:
|
||||||
|
|
||||||
|
1. Add the path + schema to `api/openapi.yaml`
|
||||||
|
2. Run `make generate`
|
||||||
|
3. Implement the handler in `internal/httpapi/impl.go`
|
||||||
|
4. Add tests in `internal/httpapi/api_test.go`
|
||||||
|
|
||||||
|
Never hand-edit `internal/httpapi/gen/api.gen.go`.
|
||||||
|
|
||||||
|
### Database access is sqlc-first
|
||||||
|
|
||||||
|
SQL queries live in `internal/db/queries/*.sql`. Go code is generated with
|
||||||
|
`sqlc` into `internal/db/sqlcgen/`. Config in `sqlc.yaml`.
|
||||||
|
|
||||||
|
- Queries target pgx/v5 with UUID + timestamptz overrides
|
||||||
|
- Never hand-edit generated sqlc code
|
||||||
|
|
||||||
|
### Migrations are forward-only
|
||||||
|
|
||||||
|
SQL migrations live in `migrations/` as `NNN_name.up.sql`. There are no down
|
||||||
|
migrations (see [ADR 0008](docs/adr/0008-forward-only-migrations.md)).
|
||||||
|
Migrations are idempotent where possible (`IF NOT EXISTS`, `DO $$` blocks).
|
||||||
|
|
||||||
|
To add a migration:
|
||||||
|
|
||||||
|
1. Create `migrations/NNN_name.up.sql` with the next sequence number
|
||||||
|
2. Write the DDL
|
||||||
|
3. Run `make migrate` to apply
|
||||||
|
|
||||||
|
### Seeds are DB-generated
|
||||||
|
|
||||||
|
`seeds/*.yaml` are the bootstrap files used by `oikos seed`. After making
|
||||||
|
changes via the API, run `make export` to regenerate the seed files. These
|
||||||
|
files are version-controlled and serve as DR fallback.
|
||||||
|
|
||||||
|
### Writing style
|
||||||
|
|
||||||
|
Follow [.agents/shared/writing-style.md](.agents/shared/writing-style.md).
|
||||||
|
Documentation is reference prose, not marketing. Banned vocabulary includes
|
||||||
|
"robust", "seamless", "leverage", "utilize", "delve", "cutting-edge".
|
||||||
|
|
||||||
|
### Risk classification
|
||||||
|
|
||||||
|
Every mutation is classified against `seeds/policy.yaml` before execution.
|
||||||
|
Four risk classes: `read_only`, `reversible_low`, `config_mutation`,
|
||||||
|
`destructive`. The classifier can only lower autonomy relative to policy,
|
||||||
|
never raise it. When in doubt, escalate.
|
||||||
|
|
||||||
|
## CI
|
||||||
|
|
||||||
|
Gitea Actions runs on push to `main` and pull requests (`ci.yml`):
|
||||||
|
|
||||||
|
1. `go vet` + `golangci-lint` + `govulncheck`
|
||||||
|
2. Generated code drift check (`make generate-check`)
|
||||||
|
3. Build (`go build ./...`)
|
||||||
|
4. Test with race detector + coverage
|
||||||
|
5. Docker build verification (no push)
|
||||||
|
|
||||||
|
Coverage gates: policy + learning packages ≥ 80%, others ≥ 60%.
|
||||||
|
|
||||||
|
## PR workflow
|
||||||
|
|
||||||
|
1. Create a branch from `main`
|
||||||
|
2. Make changes, write tests
|
||||||
|
3. Run `make lint test generate-check`
|
||||||
|
4. Commit with a message following: problem → change → risk → verification
|
||||||
|
5. Push to Gitea; CI gates PRs on green
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
Secrets are managed by Infisical (primary) with SOPS as DR fallback. Never
|
||||||
|
hardcode secrets. Use environment variables from `.env` for local dev.
|
||||||
|
The `.env` and `.infisical-credentials` files are gitignored.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Caddy](121-caddy.md) (if proxied)
|
|
||||||
- [DNS](../infrastructure/dns.md) (if has subdomain)
|
|
||||||
- [Authentik](124-authentik.md) (if SSO)
|
|
||||||
- ...
|
|
||||||
|
|
||||||
## Changelog
|
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
|
||||||
### YYYY-MM-DD — short title
|
- [CLIENTS.md](CLIENTS.md) — for homelab clients consuming Oikos
|
||||||
What changed, why, link to investigation if any.
|
- [docs/adr/](docs/adr/) — architecture decision records
|
||||||
```
|
|
||||||
|
|
||||||
### Cross-cutting page (`infrastructure/<topic>.md`)
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# <Topic>
|
|
||||||
|
|
||||||
One-sentence summary.
|
|
||||||
|
|
||||||
## Why
|
|
||||||
Design rationale — what it replaces, what it solves.
|
|
||||||
|
|
||||||
## Components
|
|
||||||
Where it runs, what files matter.
|
|
||||||
|
|
||||||
## How to apply / use
|
|
||||||
Recipes.
|
|
||||||
|
|
||||||
## Gotchas
|
|
||||||
|
|
||||||
## Related
|
|
||||||
Links to nodes that host or depend on this.
|
|
||||||
|
|
||||||
## Changelog
|
|
||||||
```
|
|
||||||
|
|
||||||
### Plan (`plans/YYYY-MM-DD-slug.md`)
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# YYYY-MM-DD — <title>
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
What this change achieves and why.
|
|
||||||
|
|
||||||
## Current topology / state
|
|
||||||
Diagram or description of what exists now.
|
|
||||||
|
|
||||||
## Target topology / state
|
|
||||||
What it looks like after.
|
|
||||||
|
|
||||||
## Pre-flight checklist
|
|
||||||
|
|
||||||
## Step-by-step procedure
|
|
||||||
|
|
||||||
## Verification
|
|
||||||
|
|
||||||
## Post-migration
|
|
||||||
Changelog entries to write, index status to update.
|
|
||||||
```
|
|
||||||
|
|
||||||
### Investigation (`investigations/YYYY-MM-DD-slug.md`)
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# YYYY-MM-DD — <title>
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
1-3 sentences.
|
|
||||||
|
|
||||||
## Timeline
|
|
||||||
|
|
||||||
## Root cause
|
|
||||||
|
|
||||||
## Mitigations applied
|
|
||||||
|
|
||||||
## Open questions
|
|
||||||
```
|
|
||||||
|
|
||||||
## Linking discipline
|
|
||||||
|
|
||||||
- Every container page links to every cross-cutting page it participates in.
|
|
||||||
- Every cross-cutting page lists the nodes that participate.
|
|
||||||
- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
|
|
||||||
- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
|
|
||||||
|
|
||||||
## Changelog hygiene
|
|
||||||
|
|
||||||
- Reverse-chronological (newest first).
|
|
||||||
- One entry per discrete change, even if you make several in one day.
|
|
||||||
- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
|
|
||||||
- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
|
|
||||||
50
Makefile
Normal file
50
Makefile
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
.PHONY: build test test-db lint generate generate-check dev migrate seed export clean tidy
|
||||||
|
|
||||||
|
BINARY := oikos
|
||||||
|
GO ?= go
|
||||||
|
|
||||||
|
build:
|
||||||
|
$(GO) build -o $(BINARY) -tags timetzdata ./cmd/oikos
|
||||||
|
|
||||||
|
test:
|
||||||
|
$(GO) test -race -cover ./...
|
||||||
|
|
||||||
|
# Integration tests against the compose Postgres (starts it if needed)
|
||||||
|
test-db:
|
||||||
|
docker compose up -d postgres
|
||||||
|
@sleep 3
|
||||||
|
OIKOS_TEST_DATABASE_URL="postgres://oikos:$${OIKOS_DB_PASSWORD:-oikos_dev}@localhost:5432/oikos?sslmode=disable" \
|
||||||
|
$(GO) test -race -count=1 ./internal/db/ ./internal/httpapi/ ./internal/mcp/
|
||||||
|
|
||||||
|
lint:
|
||||||
|
$(GO) vet ./...
|
||||||
|
@command -v golangci-lint >/dev/null 2>&1 && golangci-lint run || echo "golangci-lint not installed, skipping"
|
||||||
|
|
||||||
|
generate:
|
||||||
|
$(GO) run github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1 \
|
||||||
|
-config api/codegen.yaml api/openapi.yaml
|
||||||
|
$(GO) run github.com/sqlc-dev/sqlc/cmd/sqlc@v1.29.0 generate
|
||||||
|
|
||||||
|
# CI drift guard: regenerate and fail if the committed output changed.
|
||||||
|
generate-check: generate
|
||||||
|
@git diff --exit-code -- internal/httpapi/gen internal/db/sqlcgen \
|
||||||
|
|| (echo "generated code is stale — run 'make generate' and commit" && exit 1)
|
||||||
|
|
||||||
|
migrate:
|
||||||
|
$(GO) run ./cmd/oikos migrate
|
||||||
|
|
||||||
|
seed:
|
||||||
|
$(GO) run ./cmd/oikos seed
|
||||||
|
|
||||||
|
export:
|
||||||
|
$(GO) run ./cmd/oikos export
|
||||||
|
|
||||||
|
dev:
|
||||||
|
docker compose --profile dev up -d
|
||||||
|
|
||||||
|
clean:
|
||||||
|
rm -f $(BINARY)
|
||||||
|
$(GO) clean -testcache
|
||||||
|
|
||||||
|
tidy:
|
||||||
|
$(GO) mod tidy
|
||||||
180
README.md
180
README.md
@@ -1,76 +1,136 @@
|
|||||||
# Homelab Wiki — `hubris`
|
# Oikos
|
||||||
|
|
||||||
Living documentation for the **hubris** Proxmox homelab. Every node, every cross-cutting system, and every meaningful incident is its own page; pages are linked so you can start anywhere and walk the graph.
|
Agentic homelab operating system written in Go. Single binary (`cmd/oikos`),
|
||||||
|
Docker-deployed on mac-mini, with a standalone Nomos MCP agent gateway
|
||||||
|
(`cmd/nomos`). Manages the **hubris** Proxmox homelab autonomously — observes
|
||||||
|
state, classifies actions against policy, executes approved procedures over SSH,
|
||||||
|
learns from outcomes, and escalates when uncertain.
|
||||||
|
|
||||||
> Last refreshed against live state: **2026-04-28**.
|
**For agents running on enrolled clients:** start with [AGENTS.md](AGENTS.md).
|
||||||
|
**For client machines:** see [CLIENTS.md](CLIENTS.md).
|
||||||
|
**For developers:** see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||||
|
|
||||||
## Map
|
## Quick start
|
||||||
|
|
||||||
### Hosts
|
```bash
|
||||||
- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`
|
# Dev stack (postgres + api + scheduler + notifier)
|
||||||
|
docker compose --profile dev up -d
|
||||||
|
|
||||||
### VMs
|
# Full stack (adds Nomos agent gateway)
|
||||||
- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation)
|
docker compose --profile full up -d
|
||||||
- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS
|
|
||||||
|
|
||||||
### LXC containers
|
# Build standalone binary
|
||||||
See the full table in [`containers/index.md`](containers/index.md). Quick links:
|
go build -o bin/oikos -tags timetzdata ./cmd/oikos
|
||||||
|
|
||||||
| ID | Name | IP | Role |
|
# Run all roles in one process (dev mode)
|
||||||
| --- | ---------------- | --------------- | --------------------------------------------- |
|
OIKOS_DATABASE_URL="postgres://oikos:oikos_dev@localhost:5432/oikos?sslmode=disable" \
|
||||||
| 101 | [jellyfin](containers/101-jellyfin.md) | 192.168.8.206 | Media server |
|
go run ./cmd/oikos all
|
||||||
| 102 | [nfs-export](containers/102-nfs-export.md) | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS |
|
```
|
||||||
| 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt |
|
|
||||||
| 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server |
|
|
||||||
| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) |
|
|
||||||
| 114 | [nextcloud](containers/114-nextcloud.md) | 192.168.8.224 | Personal cloud |
|
|
||||||
| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse |
|
|
||||||
| 119 | [sophia](containers/119-sophia.md) | 192.168.8.157 | Sophia |
|
|
||||||
| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos |
|
|
||||||
| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy |
|
|
||||||
| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) |
|
|
||||||
| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS |
|
|
||||||
| 126 | [plato](containers/126-plato.md) | 192.168.8.190 | Plato (notes/discovery workspace) |
|
|
||||||
|
|
||||||
### Cross-cutting infrastructure
|
## Architecture
|
||||||
- [DNS — split-horizon](infrastructure/dns.md)
|
|
||||||
- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
|
|
||||||
- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
|
|
||||||
- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
|
|
||||||
- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
|
|
||||||
- [SSH access](infrastructure/ssh-access.md)
|
|
||||||
- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
|
|
||||||
- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md)
|
|
||||||
- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md)
|
|
||||||
- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
|
|
||||||
|
|
||||||
### Investigations
|
```
|
||||||
Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md).
|
┌──────────────────────────────────┐
|
||||||
|
│ mac-mini (Docker) │
|
||||||
|
│ │
|
||||||
|
Workstation ─── │ nomos (8092) ──MCP── api (8090) │
|
||||||
|
(mesh) │ MCP gateway REST + MCP │
|
||||||
|
│ │
|
||||||
|
│ scheduler ── notifier ── postgres │
|
||||||
|
│ (observe) (Matrix) (Timescale)│
|
||||||
|
└──────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
### Operations
|
| Component | Port | Role |
|
||||||
- [Command cheatsheet](operations/commands.md)
|
|-----------|------|------|
|
||||||
- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
|
| `oikos api` | 8090 | REST API + MCP server (15 tools) |
|
||||||
|
| `oikos scheduler` | — | Probe runner, signal lifecycle, metrics |
|
||||||
|
| `oikos notifier` | — | Approval tokens, Matrix alerts |
|
||||||
|
| `nomos serve` | 8092 | MCP client gateway, query routing |
|
||||||
|
|
||||||
## Conventions
|
## Phases
|
||||||
|
|
||||||
- **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format:
|
| Phase | Status | Description |
|
||||||
```
|
|-------|--------|-------------|
|
||||||
### YYYY-MM-DD — short title
|
| 1 — Ontology + DB | ✅ | TimescaleDB, migrations, seeds, blast_radius |
|
||||||
one or two lines on what changed and why.
|
| 2 — API | ✅ | OpenAPI-first REST + MCP, auth, SSE, audit |
|
||||||
```
|
| 3 — Control loop | ✅ | Scheduler, actuator, learning, classifier, notifier |
|
||||||
- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
|
| 4 — Nomos agent | ✅ | Standalone MCP client gateway, agent activity |
|
||||||
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
|
| 5 — Secrets | ✅ | Infisical backend + SOPS fallback, rotation runbooks |
|
||||||
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
|
| 6 — Deploy | ✅ | CI pipeline, cutover checklist, watchdog, rollback |
|
||||||
- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
|
|
||||||
|
|
||||||
## Maintaining this wiki
|
Full plan: [plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md](plans/2026-07-06-consolidate-oikos-control-plane-onto-mac-mini.md).
|
||||||
|
|
||||||
When you change a node:
|
## Operations
|
||||||
1. Update the relevant page (config snapshot, ports, mounts).
|
|
||||||
2. Add a changelog entry at the bottom of that page.
|
|
||||||
3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
|
|
||||||
4. If it's an incident, add an entry to [`investigations/`](investigations/index.md).
|
|
||||||
|
|
||||||
## See also
|
### API endpoints
|
||||||
|
|
||||||
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone
|
```bash
|
||||||
|
curl http://localhost:8090/api/v1/entities?type=service # fleet
|
||||||
|
curl http://localhost:8090/api/v1/health # fleet health
|
||||||
|
curl http://localhost:8090/api/v1/agent-activity # agent log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Nomos queries
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Structured tool call
|
||||||
|
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
|
||||||
|
-d '{"tool":"get_blast_radius","args":{"entity_id":"service:authentik"}}'
|
||||||
|
|
||||||
|
# Natural language
|
||||||
|
curl -X POST localhost:8092/query -H "Content-Type: application/json" \
|
||||||
|
-d '{"query":"what depends on authentik?"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
oikos migrate # apply DB migrations
|
||||||
|
oikos seed # ingest ontology/inventory/policy seeds
|
||||||
|
oikos export # export DB state to YAML
|
||||||
|
oikos api # serve REST + MCP
|
||||||
|
oikos scheduler # run observe loop
|
||||||
|
oikos notifier # run notification loop
|
||||||
|
oikos all # all roles in one process
|
||||||
|
oikos secret list # enumerate SOPS secrets
|
||||||
|
oikos secret migrate # SOPS → Infisical
|
||||||
|
```
|
||||||
|
|
||||||
|
## Repo layout
|
||||||
|
|
||||||
|
```
|
||||||
|
cmd/oikos/ Go entry point — single binary
|
||||||
|
cmd/nomos/ Nomos MCP client gateway
|
||||||
|
internal/ Go packages (httpapi, mcp, scheduler, actuator, learning,
|
||||||
|
notifier, policy, secrets, db, config, ontology, domain,
|
||||||
|
knowledge)
|
||||||
|
api/openapi.yaml API contract (OpenAPI 3.1)
|
||||||
|
migrations/ Forward-only SQL migrations (TimescaleDB)
|
||||||
|
seeds/ Bootstrap YAML (ontology, inventory, policy, knowledge)
|
||||||
|
compose/ Dockerfiles + Caddy config
|
||||||
|
scripts/ Deploy, watchdog, verification, rollback
|
||||||
|
nomos/ Nomos config, persona, skills
|
||||||
|
.agents/ Agent instruction files, shared conventions, skills
|
||||||
|
archive/ Historical reference (legacy wiki, plans, SOPS backups)
|
||||||
|
plans/ Design documents (active + done)
|
||||||
|
docs/adr/ Architecture decision records
|
||||||
|
```
|
||||||
|
|
||||||
|
## For agents
|
||||||
|
|
||||||
|
See [AGENTS.md](AGENTS.md) for the full orientation. Quick reference:
|
||||||
|
|
||||||
|
- **Source of truth:** DB (runtime) then seeds (bootstrap). Old wiki is
|
||||||
|
archived at `archive/knowledge/` — use MCP `search_knowledge` instead.
|
||||||
|
- **Mutations:** classify against policy, request approval for
|
||||||
|
`destructive`/`config_mutation`
|
||||||
|
- **Secrets:** Infisical (primary) or SOPS (fallback) — never hardcode
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [OIKOS.md](.agents/OIKOS.md) — operating model, OODA loop, ontology
|
||||||
|
- [CLIENTS.md](CLIENTS.md) — client onboarding guide
|
||||||
|
- [CONTRIBUTING.md](CONTRIBUTING.md) — developer guide
|
||||||
|
- [plans/](plans/) — design documents and cutover checklist
|
||||||
|
- [docs/adr/](docs/adr/) — architecture decision records
|
||||||
|
|||||||
8
api/codegen.yaml
Normal file
8
api/codegen.yaml
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
# oapi-codegen config — `make generate` regenerates internal/httpapi/gen.
|
||||||
|
package: gen
|
||||||
|
output: internal/httpapi/gen/api.gen.go
|
||||||
|
generate:
|
||||||
|
models: true
|
||||||
|
chi-server: true
|
||||||
|
strict-server: true
|
||||||
|
embedded-spec: true
|
||||||
3303
api/openapi.yaml
Normal file
3303
api/openapi.yaml
Normal file
File diff suppressed because it is too large
Load Diff
7
api/redocly.yaml
Normal file
7
api/redocly.yaml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
# Redocly lint config for api/openapi.yaml (CI runs: redocly lint api/openapi.yaml)
|
||||||
|
extends:
|
||||||
|
- recommended
|
||||||
|
rules:
|
||||||
|
# Every operation declares `default` → RFC 9457 problem+json instead of
|
||||||
|
# enumerating each 4XX (plan R3-3); oapi-codegen handles `default` fine.
|
||||||
|
operation-4xx-response: off
|
||||||
@@ -12,7 +12,7 @@
|
|||||||
|
|
||||||
## Problem statement
|
## Problem statement
|
||||||
|
|
||||||
The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
|
The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
|
||||||
|
|
||||||
| Host | IP | Inside pool? |
|
| Host | IP | Inside pool? |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
358
archive/hermes-plans/2026-07-05_strong-migration-assessment.md
Normal file
358
archive/hermes-plans/2026-07-05_strong-migration-assessment.md
Normal file
@@ -0,0 +1,358 @@
|
|||||||
|
# Assessment: Which nodes can move to `strong`
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
hubris is **memory-starved**: 28 GiB RAM, 71.9 GiB allocated across 18 LXC + 2 VM
|
||||||
|
(2.5× overcommit), 10 GiB swap in active use. strong sits **completely empty** —
|
||||||
|
28 GiB RAM, 25 GiB free, 0 guests, 2.7 TiB unused storage. The single most
|
||||||
|
effective decongestion move is to shift guests off hubris onto strong.
|
||||||
|
|
||||||
|
This document assesses every guest for move-readiness, grouped by constraints
|
||||||
|
(library dependency, GPU, core-infra status), and proposes a phased migration
|
||||||
|
that does **not** require the physical library-SSD move (the blocker of the
|
||||||
|
original plan) — library access from strong is provided via NFS from hubris.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current resource state (live, 2026-07-05)
|
||||||
|
|
||||||
|
### hubris — overloaded
|
||||||
|
|
||||||
|
| Resource | Capacity | Allocated (all guests) | Actual use | Status |
|
||||||
|
|----------|----------|----------------------|------------|--------|
|
||||||
|
| RAM | 28 GiB | 70.2 GiB (2.5× overcommit) | 18 GiB used + 10 GiB swap | ⚠️ heavy swap pressure |
|
||||||
|
| CPU | 12 vCPU (6c/12t) | 43 vCPU (3.6× overcommit) | ~43% scaling MHz | OK (shares) |
|
||||||
|
| local-lvm | 856 GiB | 582 GiB allocated (29% thin) | — | OK |
|
||||||
|
| library (lvmthin) | 3.7 TiB | — | 1.2 TiB used (34%) | OK, 2.3 TiB free |
|
||||||
|
|
||||||
|
### strong — empty, ready
|
||||||
|
|
||||||
|
| Resource | Capacity | Used | Status |
|
||||||
|
|----------|----------|------|--------|
|
||||||
|
| RAM | 28 GiB | 2.3 GiB (host only) | 25 GiB free |
|
||||||
|
| CPU | 16 vCPU (8c/16t) | idle (0.08 load) | 100% free |
|
||||||
|
| local-lvm | 856 GiB | 0 | empty |
|
||||||
|
| ludo-lvm | 1.8 TiB | 0 | empty |
|
||||||
|
| Guests | — | 0 LXC, 0 VM | nothing running |
|
||||||
|
|
||||||
|
### Network topology constraint
|
||||||
|
|
||||||
|
```
|
||||||
|
Fritz!Box (192.168.178.1)
|
||||||
|
└── SODOLA 2.5G switch
|
||||||
|
├── hubris eno1 → vmbr1 (192.168.178.10) → vmbr0 (192.168.8.0/24)
|
||||||
|
│ └── all 20 guests on 192.168.8.x
|
||||||
|
└── strong vmbr0 (192.168.178.181)
|
||||||
|
└── no internal bridge yet, guests would be on 192.168.178.x
|
||||||
|
```
|
||||||
|
|
||||||
|
strong reaches `192.168.8.0/24` via the Fritz static route through hubris.
|
||||||
|
Guests on strong get `192.168.178.x` IPs unless we add an internal bridge
|
||||||
|
on strong (Phase 0 prerequisite — see below).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Per-guest assessment
|
||||||
|
|
||||||
|
### Tier 1 — Move immediately (no library dependency, no core-infra)
|
||||||
|
|
||||||
|
These guests mount **no** `/mnt/library` and are not part of the core
|
||||||
|
infrastructure spine (caddy/dns/auth/mcp). They are the easiest wins.
|
||||||
|
|
||||||
|
| ID | Name | Cores | RAM | Library? | GPU? | Notes |
|
||||||
|
|----|------|-------|-----|----------|------|-------|
|
||||||
|
| 118 | elementsynapse | 2 | 4 GiB | ❌ | ❌ | Matrix homeserver. Public via Caddy (`matrix.hubris.network`). Only change: Caddy backend IP. **Easiest move in the fleet.** |
|
||||||
|
| 129 | house | 2 | 3 GiB | ❌ | ❌ | Yuvomi family planner (Docker). No public Caddy route yet (uses VPS traefik directly). Self-contained. |
|
||||||
|
|
||||||
|
**Combined RAM freed from hubris: 7 GiB.** No NFS, no library, no GPU.
|
||||||
|
|
||||||
|
### Tier 2 — Move with library NFS (high resource consumers)
|
||||||
|
|
||||||
|
These are the heaviest guests and the original migration plan's primary
|
||||||
|
targets. They mount `/mnt/library` and two use the iGPU. Moving them
|
||||||
|
requires an NFS export from hubris → strong (reverse of the original
|
||||||
|
plan's direction, since the physical SSD hasn't moved).
|
||||||
|
|
||||||
|
| ID | Name | Cores | RAM | Library? | GPU? | I/O profile | Notes |
|
||||||
|
|----|------|-------|-----|----------|------|-------------|-------|
|
||||||
|
| 120 | mule-images | 6 | 12 GiB | ✅ mp0 | ✅ iGPU | Write-heavy (photo processing) | **#1 RAM consumer.** Strong has Radeon 680M iGPU (VAAPI works). |
|
||||||
|
| 122 | arriman | 4 | 8 GiB | ✅ mp0 | ❌ | Write-heavy (downloads) | *arr stack + qbit + sab. Mounts library for download writes. |
|
||||||
|
| 101 | jellyfin | 4 | 8 GiB | ✅ mp0 | ✅ iGPU | Read-heavy sequential | Media streaming + transcode. Strong 680M handles VAAPI. |
|
||||||
|
|
||||||
|
**Combined RAM freed: 28 GiB.** This alone would eliminate hubris's swap
|
||||||
|
pressure entirely.
|
||||||
|
|
||||||
|
### Tier 3 — Could move, low urgency
|
||||||
|
|
||||||
|
| ID | Name | Cores | RAM | Library? | Notes |
|
||||||
|
|----|------|-------|-----|----------|-------|
|
||||||
|
| 130 | grimmory | 1 | 2 GiB | ✅ mp0 | Book library (Docker). Migrated from apps LXC recently. |
|
||||||
|
| 131 | teddycloud | 1 | 1 GiB | ✅ mp0 | New (not in inventory.yaml yet). |
|
||||||
|
| 132 | rclone | 1 | 2 GiB | ✅ mp0 (ro) | Backup container. Read-only library mount. |
|
||||||
|
| 128 | trmnl | 1 | 768 MiB | ❌ | TRMNL middleware. No library. Could move but tiny. |
|
||||||
|
| 119 | sophia | 2 | 1 GiB | ✅ mp0 | Workshop. Light use. |
|
||||||
|
|
||||||
|
### Stay on hubris (core infrastructure)
|
||||||
|
|
||||||
|
| ID | Name | Cores | RAM | Why it stays |
|
||||||
|
|----|------|-------|-----|--------------|
|
||||||
|
| 121 | caddy | 1 | 512 MiB | Reverse proxy — terminates all `*.hubris.network`. Must stay on hubris for LAN-side reachability. **Needs backend IP updates** when guests move. |
|
||||||
|
| 107 | dns | 1 | 1 GiB | Technitium DNS, split-horizon. Core. |
|
||||||
|
| 106 | auth-outpost | 1 | 512 MiB | Authentik SSO enforcement. Core. |
|
||||||
|
| 105 | apps | 2 | 4 GiB | homelab MCP + secrets-issuance + artifacto. Core infra. Mounts library. |
|
||||||
|
| 104 | gitea | 1 | 1 GiB | Git server. NFS would hurt git lock/stat perf. Mounts library (bare repos). |
|
||||||
|
| 103 | paperless | 2 | 3 GiB | Document archive. Moderate I/O, OCR writes. Mounts library. |
|
||||||
|
| 102 | nfs-export | 1 | 512 MiB | Exports library to zimaos via NFS. Must stay with the physical library. |
|
||||||
|
| 100 | zimaos | 4 | 8 GiB (VM) | NAS frontend eval. Already NFS-mounts library from 102. |
|
||||||
|
| 108 | haos | 2 | 4 GiB (VM) | Home Assistant OS. Hardware access, low latency. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Constraints & prerequisites
|
||||||
|
|
||||||
|
### 1. Network — strong needs an internal bridge (Phase 0)
|
||||||
|
|
||||||
|
strong currently has only `vmbr0` on `192.168.178.0/24`. Guests created there
|
||||||
|
get household-LAN IPs, not homelab-subnet IPs. Two options:
|
||||||
|
|
||||||
|
- **Option A (recommended):** Add `vmbr1` on strong as a portless internal
|
||||||
|
bridge with a `192.168.8.x/24` address (e.g. `192.168.8.3`). Route between
|
||||||
|
strong's `vmbr0` and `vmbr1` the same way hubris does. Guests go on `vmbr1`
|
||||||
|
and get `192.168.8.x` IPs — transparent to Caddy, DNS, and inter-LXC refs.
|
||||||
|
Requires adding a static route on Fritz (or relying on hubris's existing
|
||||||
|
route — strong would need IP forwarding + a route to 192.168.8.0/24 via vmbr1).
|
||||||
|
|
||||||
|
- **Option B (simpler, messier):** Put guests on `192.168.178.x` directly.
|
||||||
|
Caddy can still reach them (hubris routes to 192.168.178.0/24). But DNS
|
||||||
|
records, inter-LXC references, and firewall rules all assume `192.168.8.x`.
|
||||||
|
More config churn per guest.
|
||||||
|
|
||||||
|
### 2. Storage — rootfs migration (no shared storage)
|
||||||
|
|
||||||
|
`local-lvm` is per-node (not shared). Moving an LXC requires either:
|
||||||
|
- `vzdump` → restore on strong (clean, but needs temp disk space + downtime)
|
||||||
|
- `rsync` the rootfs to a new LXC on strong (faster for large rootfs like 120's 100G)
|
||||||
|
- `pct migrate` only works with shared storage — **not applicable here**
|
||||||
|
|
||||||
|
For VMs (100, 108): `qm migrate` also needs shared storage. Not moving VMs.
|
||||||
|
|
||||||
|
### 3. Library access — NFS from hubris to strong
|
||||||
|
|
||||||
|
Since the physical library SSD is still on hubris, strong's guests that need
|
||||||
|
`/mnt/library` must NFS-mount it from hubris. Options:
|
||||||
|
|
||||||
|
- **Export from hubris host directly** (simplest): add `/mnt/library` to
|
||||||
|
`/etc/exports` on hubris with the same squash params as LXC 102
|
||||||
|
(`rw,all_squash,anonuid=33,anongid=10000,no_subtree_check`). Mount on strong
|
||||||
|
at `/mnt/library`. Strong's guests bind-mount it just like hubris's guests do.
|
||||||
|
|
||||||
|
- **Use existing nfs-export LXC 102**: strong NFS-mounts from `192.168.8.200`
|
||||||
|
(LXC 102). This already has the right squash config. Less host-level change.
|
||||||
|
**This is the path of least resistance.**
|
||||||
|
|
||||||
|
### 4. GPU — iGPU passthrough on strong
|
||||||
|
|
||||||
|
strong has a Ryzen 7 PRO 6850U with Radeon 680M iGPU. For jellyfin (VAAPI
|
||||||
|
transcoding) and mule-images (photo processing), we need:
|
||||||
|
- `/dev/dri/renderD128` passed to the LXC (`lxc.cgroup2.devices.allow` +
|
||||||
|
`lxc.mount.entry` or Proxmox's `dev0:` passthrough)
|
||||||
|
- `video` / `render` group membership inside the container
|
||||||
|
- Confirm `amdgpu` driver loads on strong's host kernel (it should — same APU family)
|
||||||
|
|
||||||
|
### 5. Quorum — 2-node cluster, no QDevice
|
||||||
|
|
||||||
|
Moving guests to strong does NOT fix the quorum issue but **reduces blast
|
||||||
|
radius**: if hubris reboots (its known thermal instability), the guests on
|
||||||
|
strong keep running independently. Consider adding a QDevice as a separate
|
||||||
|
follow-up — it's orthogonal to this migration.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Revised migration phases
|
||||||
|
|
||||||
|
The original plan's NFS-over-LAN approach has been superseded. Instead,
|
||||||
|
**media library data moves to ludo-lvm** on strong so migrated guests access
|
||||||
|
it as a local ext4 mount. Data is split by origin:
|
||||||
|
|
||||||
|
```
|
||||||
|
hubris (stays): library SSD (3.7T, 1.2T used)
|
||||||
|
└── /mnt/library/{documents,images,cloud,homecloud,notes,repos,sophia}
|
||||||
|
↑ user-generated content (docs, photos, cloud sync, notes, repos, workshop)
|
||||||
|
|
||||||
|
strong (moves): ludo-lvm (1.8T, 0 used at start)
|
||||||
|
└── /mnt/media_local ← 1.5T thin volume
|
||||||
|
└── {downloads,movies,music,tv,anime,books}
|
||||||
|
↑ non-user-generated content (media arr stack, book library)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Category | Stays on hubris | Moves to strong |
|
||||||
|
|----------|----------------|-----------------|
|
||||||
|
| Media | — | downloads (25G), movies (51G), music (29G), tv (30G), anime (206G) |
|
||||||
|
| Books | — | books (2.6G) |
|
||||||
|
| Docs/Photos | documents (249M), images (4K) | — |
|
||||||
|
| Cloud sync | cloud (287G), homecloud (367G) | — |
|
||||||
|
| Personal | notes (6.7M), repos (84M), sophia (151G) | — |
|
||||||
|
| **Total** | **~805G** | **~344G** |
|
||||||
|
|
||||||
|
ludo-lvm (1.8T) fits all media + books with ~1.15T headroom for growth.
|
||||||
|
hubris library SSD (3.7T, 1.2T used) retains the user-generated content.
|
||||||
|
Both sides keep their data local — no cross-node NFS needed for daily I/O.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 2a — Prepare ludo-lvm on strong
|
||||||
|
|
||||||
|
1. Create a ext4 filesystem on ludo-lvm for media:
|
||||||
|
```bash
|
||||||
|
lvcreate -n media -L 1.5T ludo-lvm
|
||||||
|
mkfs.ext4 /dev/ludo-lvm/media
|
||||||
|
```
|
||||||
|
2. Mount at `/mnt/media_local` on strong, add to `/etc/fstab`
|
||||||
|
3. rsync media directories from hubris → strong:
|
||||||
|
```bash
|
||||||
|
rsync -av --progress /mnt/library/{movies,tv,anime,downloads,music,books} strong:/mnt/media_local/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Phase 2b — Migrate arriman (122) to strong
|
||||||
|
|
||||||
|
1. Stop arriman on hubris, dump rootfs (24G)
|
||||||
|
2. Restore on strong with IP `192.168.8.245/28` on vmbr1
|
||||||
|
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (downloads land locally)
|
||||||
|
4. Update Caddy: jellyseerr/qbit/sab backends → new IP
|
||||||
|
5. Update inventory.yaml
|
||||||
|
|
||||||
|
### Phase 2c — Migrate jellyfin (101) to strong
|
||||||
|
|
||||||
|
1. Stop jellyfin on hubris, dump rootfs (16G)
|
||||||
|
2. Restore on strong with IP `192.168.8.246/28` on vmbr1
|
||||||
|
3. Pass `/dev/dri/renderD128` + `/dev/dri/card0` (Radeon 680M + RX 7600)
|
||||||
|
4. Mount `/mnt/media_local` → `/mnt/library` via mp0 (media reads locally)
|
||||||
|
5. Update Caddy: `media.hubris.network` → new IP
|
||||||
|
6. Reinstall `sso-inject.js` in web dir (lost on every apt upgrade)
|
||||||
|
7. Test VAAPI transcoding, SSO login, media playback
|
||||||
|
|
||||||
|
### Phase 2d — Migrate grimmory (130) to strong
|
||||||
|
|
||||||
|
1. Stop grimmory on hubris, dump rootfs (16G)
|
||||||
|
2. Restore on strong with IP `192.168.8.247/28` on vmbr1
|
||||||
|
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (books read locally)
|
||||||
|
4. Update Caddy: `books.hubris.network` → new IP
|
||||||
|
5. Update inventory.yaml
|
||||||
|
6. Test: book browsing, calibre-web access
|
||||||
|
|
||||||
|
### No NFS export needed
|
||||||
|
|
||||||
|
With the data split by origin, hubris guests that only need user-generated
|
||||||
|
content (documents, images, cloud, repos, sophia) still access them from the
|
||||||
|
original library SSD — no cross-node NFS required. The two sides are
|
||||||
|
independent.
|
||||||
|
|
||||||
|
**Result after Phase 2: hubris frees 26 GiB RAM (4 migrated guests) + 344G of
|
||||||
|
library I/O burden. Strong becomes the media/books powerhouse.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 3 — Migrate mule-images (120) to strong
|
||||||
|
|
||||||
|
Move photo management (12 GiB RAM, 6 cores, iGPU) last because it needs:
|
||||||
|
- `/mnt/library` access (now NFS from strong — already set up in Phase 2d)
|
||||||
|
- `/dev/dri/renderD128` (Radeon 680M — confirm VAAPI compatibility first)
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
1. Stop mule-images on hubris, rsync the 100G rootfs to strong (faster than vzdump)
|
||||||
|
2. Restore on strong with IP on vmbr1
|
||||||
|
3. Pass Radeon 680M iGPU
|
||||||
|
4. Reconfigure library paths → `/mnt/media_local` (or keep NFS mount)
|
||||||
|
5. Update Caddy: `photos.hubris.network` → new IP
|
||||||
|
6. Test photo import + processing pipeline
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 4 — Tier 3 moves (optional)
|
||||||
|
|
||||||
|
Migrate grimmory (130), teddycloud (131), rclone (132), trmnl (128), sophia (119)
|
||||||
|
as needed — each frees 1–2 GiB. Not urgent; do when convenient.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Phase 5 — Follow-up
|
||||||
|
|
||||||
|
- **QDevice**: add a tiebreaker for 2-node quorum
|
||||||
|
- **Gaming VM**: strong's 6850U has enough cores alongside migrated LXCs
|
||||||
|
- **Hubris library cleanup**: after all guests are confirmed working, decide
|
||||||
|
whether to keep the original library SSD as backup or repurpose it
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resource math after Phase 3 (all Tier 1 + 2 moved)
|
||||||
|
|
||||||
|
| | hubris | strong |
|
||||||
|
|---|--------|--------|
|
||||||
|
| Guests | 11 LXC + 2 VM | 5 LXC |
|
||||||
|
| RAM allocated | ~25 GiB | ~45 GiB |
|
||||||
|
| RAM capacity | 28 GiB | 28 GiB |
|
||||||
|
| Overcommit | 0.9× (under-committed) | 1.6× (manageable) |
|
||||||
|
| Library disk | Local ext4 (3.7T) → NFS client | Local ext4 on ludo-lvm (1.8T) |
|
||||||
|
| GPU | Radeon 760M (idle) | Radeon 680M (jellyfin + mule-images) |
|
||||||
|
|
||||||
|
strong becomes the media/library powerhouse. hubris becomes a lean core-infra
|
||||||
|
node (DNS, auth, git, docs, caddy, HA).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risk register
|
||||||
|
|
||||||
|
| Risk | Impact | Mitigation |
|
||||||
|
|------|--------|------------|
|
||||||
|
| NFS latency for library reads (jellyfin, arriman) | Media playback stutter, slow downloads | Test iperf between strong↔hubris first. If 2.5G link, NFS throughput is fine (~1 Gbit/s). |
|
||||||
|
| GPU passthrough on strong (680M vs 760M) | Transcode quality/compat differences | Both are AMD VAAPI — same driver stack. Test `vainfo` inside LXC before going live. |
|
||||||
|
| Caddy backend IP churn | Service outage if IP wrong | Update Caddyfile in git repo (caddy-conf), test each route before destroying old LXC. |
|
||||||
|
| vzdump/restore downtime | Service unavailable during migration | Schedule off-hours. Use rsync for large rootfs (120's 100G) to minimize freeze window. |
|
||||||
|
| 2-node quorum still fragile | If hubris goes down, strong /etc/pve goes read-only | Guests keep running. Add QDevice as follow-up. |
|
||||||
|
| Library data integrity during NFS transition | Permission drift | NFS `all_squash,anonuid=33,anongid=10000` matches existing LXC 102 config. Verify with `ls -la /mnt/library` after mount. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions for operator
|
||||||
|
|
||||||
|
1. **Internal bridge on strong**: proceed with `vmbr1` on `192.168.8.3/24`
|
||||||
|
(Option A), or use `192.168.178.x` guest IPs (Option B)?
|
||||||
|
2. **Migration method**: `vzdump`/restore (clean, downtime) vs `rsync` rootfs
|
||||||
|
(faster for large disks, needs manual config copy)?
|
||||||
|
3. **Phase 1 priority**: move elementsynapse + house first (quick wins), or
|
||||||
|
go straight to Phase 2 (mule-images/jellyfin/arriman) for maximum relief?
|
||||||
|
4. **Should we add a QDevice now** before moving anything, to protect
|
||||||
|
management plane during the migration?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-05 — Phase 2d complete (grimmory migrated; media NFS to zimaos)
|
||||||
|
grimmory (130) → 192.168.8.247 on strong. Rsync'd /books (2.6G) to ludo-lvm.
|
||||||
|
LXC 102 (nfs-export) now mounts strong's NFS at /mnt/media and exports it as
|
||||||
|
a second share alongside /mnt/library. Zimaos mounts both: /media/library
|
||||||
|
(hubris user-generated) and /media/media (strong media+books).
|
||||||
|
See hosts/strong.md changelog.
|
||||||
|
|
||||||
|
### 2026-07-05 — Phase 2 complete (arriman + jellyfin migrated; library on ludo-lvm)
|
||||||
|
arriman (122) → 192.168.8.245, jellyfin (101) → 192.168.8.246. Created 1.5T
|
||||||
|
thin volume on ludo-lvm, rsync'd 363G of media data. Both containers use local
|
||||||
|
ext4 mount — no NFS. Jellyfin has 680M + RX 7600 GPU passthrough.
|
||||||
|
Caddy backends updated. See hosts/strong.md changelog.
|
||||||
|
|
||||||
|
### 2026-07-05 — Phase 1 complete (elementsynapse + house migrated to strong)
|
||||||
|
Both Tier 1 guests moved: elementsynapse (118) → 192.168.8.242, house (129) → 192.168.8.244.
|
||||||
|
Strong now has vmbr1 at 192.168.8.241/28. Hubris has proxy ARP + /32 routes for strong
|
||||||
|
guest range. DHCP scope narrowed to 192.168.8.100-239 to avoid conflicts.
|
||||||
|
Teddycloud (LXC 131) given static IP 192.168.8.150 due to IP conflict with
|
||||||
|
previous DHCP allocation at 192.168.8.243.
|
||||||
|
See hosts/strong.md changelog for full steps.
|
||||||
|
|
||||||
|
### 2026-07-05 — assessment created
|
||||||
|
Built from live `pct config` + `pvesm status` + `free -h` data pulled from
|
||||||
|
both nodes. Supersedes the storage-migration framing of the original
|
||||||
|
library-SSD plan — this assessment treats the SSD move as optional and
|
||||||
|
focuses on guest relocation via NFS.
|
||||||
33
archive/knowledge/GLOSSARY.md
Normal file
33
archive/knowledge/GLOSSARY.md
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
# Glossary
|
||||||
|
|
||||||
|
Terms and abbreviations used throughout the homelab wiki.
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|------|---------|
|
||||||
|
| **Authentik** | SSO/identity provider. Core runs on the VPS; forward-auth outpost at LXC 106 on hubris |
|
||||||
|
| **Caddy** | Reverse proxy (LXC 121). Terminates TLS for every `*.hubris.network` hostname |
|
||||||
|
| **Caveman** | Terse communication standard for agent responses — no filler, keep substance |
|
||||||
|
| **Forward-auth** | Caddy snippet that delegates authentication to an Authentik outpost. Protects web UIs like qBit, SABnzbd |
|
||||||
|
| **Gitea** | Git server at `git.hubris.network`. Hosts all tracked config repos |
|
||||||
|
| **Gluetun** | WireGuard VPN sidecar on arriman. All \*arr traffic routes through it |
|
||||||
|
| **HAOS** | Home Assistant Operating System. VM 108 on hubris |
|
||||||
|
| **Hubris** | Primary Proxmox VE node (GMKtec NucBox M6 Ultra). PVE hostname, cluster member 1 |
|
||||||
|
| **LXC** | Linux Container (Proxmox). VM-like isolation without a full OS kernel |
|
||||||
|
| **LVM-thin** | Thin-provisioned logical volume manager. Used for all container/VM storage |
|
||||||
|
| **MCP** | Model Context Protocol (MCP server at `mcp.hubris.network`). Structured tools for agents to query homelab state |
|
||||||
|
| **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy |
|
||||||
|
| **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members |
|
||||||
|
| **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows |
|
||||||
|
| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology |
|
||||||
|
| **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong |
|
||||||
|
| **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo |
|
||||||
|
| **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) |
|
||||||
|
| **Traefik** | Reverse proxy on IONOS VPS. Serves `*.hubris.network` to the public internet |
|
||||||
|
| **VAAPI** | Video Acceleration API. Intel/AMD GPU-based hardware transcode for Jellyfin |
|
||||||
|
| **VPS** | Virtual Private Server at IONOS (`82.165.190.79`). Runs Authentik core + Netbird management |
|
||||||
|
| **\\*arr** | Media automation suite: Sonarr (TV), Radarr (movies), Lidarr (music), Prowlarr (indexer), Bazarr (subtitles), Readarr (books — not in use) |
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page
|
||||||
|
- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle
|
||||||
160
archive/knowledge/containers/101-jellyfin.md
Normal file
160
archive/knowledge/containers/101-jellyfin.md
Normal file
@@ -0,0 +1,160 @@
|
|||||||
|
# 101 — `jellyfin`
|
||||||
|
|
||||||
|
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
||||||
|
- **IP:** `192.168.8.246`
|
||||||
|
- **Privilege:** privileged (recreated on strong as priv)
|
||||||
|
- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs
|
||||||
|
- **GPU:** `/dev/dri/renderD128` + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config
|
||||||
|
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
|
||||||
|
- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
|
||||||
|
- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)
|
||||||
|
- **FFmpeg:** jellyfin-ffmpeg7 7.1.4
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
| -------- | ------ | ----- |
|
||||||
|
| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
|
||||||
|
|
||||||
|
## Hardware acceleration (VAAPI)
|
||||||
|
|
||||||
|
GPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993` and
|
||||||
|
`dev1: /dev/dri/card0,gid=44` in
|
||||||
|
`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID 993) and
|
||||||
|
`video` groups inside the container.
|
||||||
|
|
||||||
|
| GPU | Model | Role |
|
||||||
|
|-----|-------|------|
|
||||||
|
| Radeon 680M | iGPU (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |
|
||||||
|
| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1 encoding |
|
||||||
|
|
||||||
|
Encoding settings (`/etc/jellyfin/encoding.xml`):
|
||||||
|
- `HardwareAccelerationType`: `vaapi`
|
||||||
|
- `VaapiDevice`: `/dev/dri/renderD128`
|
||||||
|
- `EnableHardwareEncoding`: `true`
|
||||||
|
- `AllowHevcEncoding`: `true`
|
||||||
|
- `AllowAv1Encoding`: `true`
|
||||||
|
- `EnableTonemapping`: `true`
|
||||||
|
- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1
|
||||||
|
- `EnableThrottling`: `true`
|
||||||
|
- `EnableSegmentDeletion`: `true`
|
||||||
|
|
||||||
|
Trickplay (`/etc/jellyfin/system.xml`):
|
||||||
|
- `EnableHwAcceleration`: `true`
|
||||||
|
- `EnableHwEncoding`: `true`
|
||||||
|
|
||||||
|
## Authentik SSO (OIDC)
|
||||||
|
|
||||||
|
Jellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)
|
||||||
|
v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin
|
||||||
|
handles auth directly via OIDC redirect flow.
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096
|
||||||
|
↓
|
||||||
|
Login page with "Sign in with Authentik" button
|
||||||
|
↓ (click)
|
||||||
|
/sso/OID/start/Authentik
|
||||||
|
↓ (302 redirect)
|
||||||
|
auth.hubris.network OIDC
|
||||||
|
↓ (login)
|
||||||
|
/sso/OID/redirect/Authentik?code=...&state=...
|
||||||
|
↓
|
||||||
|
Jellyfin SSO plugin validates token → logged in
|
||||||
|
```
|
||||||
|
|
||||||
|
### Components
|
||||||
|
|
||||||
|
1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`
|
||||||
|
- Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`
|
||||||
|
- Provider name: `Authentik`
|
||||||
|
- OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`
|
||||||
|
- `SchemeOverride`: `https` (required — without it, plugin generates
|
||||||
|
`http://` redirect URIs that Authentik rejects)
|
||||||
|
- `EnableAuthorization`: `false` (prevents plugin from overwriting admin
|
||||||
|
permissions on each SSO login — see
|
||||||
|
[jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))
|
||||||
|
- `OidScopes`: `["email"]` (openid+profile added by default by the plugin;
|
||||||
|
must be non-null or `OidChallenge()` throws `ArgumentNullException`)
|
||||||
|
|
||||||
|
2. **Authentik OIDC provider** — `Provider for Jellyfin` (PK 6)
|
||||||
|
- Client ID: `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`
|
||||||
|
- Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`
|
||||||
|
- Application slug: `jellyfin`
|
||||||
|
|
||||||
|
3. **SSO button injection** — `/usr/share/jellyfin/web/sso-inject.js`
|
||||||
|
- Injected via `<script defer src="sso-inject.js?v=3">` in `index.html`
|
||||||
|
- Polls for `.readOnlyContent` div on the login page, adds "Sign in with
|
||||||
|
Authentik" button linking to `/sso/OID/start/Authentik`
|
||||||
|
- Cache-busted with `?v=N` parameter (bump on changes)
|
||||||
|
- **Lost on apt upgrade** — re-inject the script tag and copy the JS file
|
||||||
|
after `apt-get upgrade jellyfin*`
|
||||||
|
|
||||||
|
4. **Caddy config** — no forward-auth gate for media.hubris.network:
|
||||||
|
```caddy
|
||||||
|
media.hubris.network {
|
||||||
|
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
|
||||||
|
reverse_proxy 192.168.8.206:8096
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Known issues / pitfalls
|
||||||
|
|
||||||
|
- **`OidScopes` must be non-null** — if the field is missing from the plugin
|
||||||
|
config XML, `OidChallenge()` throws `System.ArgumentNullException`. Always
|
||||||
|
include `OidScopes` in the provider config (even if empty array).
|
||||||
|
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
|
||||||
|
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
|
||||||
|
them with "Redirect URI Error".
|
||||||
|
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
|
||||||
|
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
|
||||||
|
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
|
||||||
|
in `index.html` is the fallback.
|
||||||
|
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
|
||||||
|
incompatible with Caddy's `import authentik` forward-auth. If both are
|
||||||
|
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
|
||||||
|
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
|
||||||
|
- **API key for setup** — a temp API key can be inserted directly into the
|
||||||
|
`ApiKeys` SQLite table for automated configuration:
|
||||||
|
```sql
|
||||||
|
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
|
||||||
|
```
|
||||||
|
|
||||||
|
## Permissions
|
||||||
|
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [Caddy reverse proxy](121-caddy.md)
|
||||||
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
|
- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
|
||||||
|
- [DNS split-horizon](../infrastructure/dns.md)
|
||||||
|
- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-06 — wiki: IP, host, GPU, mount path updated for strong migration
|
||||||
|
Updated At-a-glance: IP 206→246, host hubris→strong, mount /mnt/library→/mnt/media_local, GPU Radeon 760M→680M+RX7600, privilege unpriv→priv. Permissions section updated. Changelog entry in 122-arriman.md updated similarly.
|
||||||
|
|
||||||
|
### 2026-07-04 — VAAPI hardware acceleration + Authentik SSO + resource bump
|
||||||
|
- Upgraded Jellyfin 10.11.8 → 10.11.11 (purge + reinstall to fix DB migration bug)
|
||||||
|
- Enabled VAAPI hardware acceleration (Radeon 760M): h264/hevc/vc1/vp9/av1 decode + encode
|
||||||
|
- Bumped resources: 2→4 cores, 4→8 GiB RAM, 512→1024 MiB swap
|
||||||
|
- Enabled trickplay HW acceleration + throttling + segment deletion
|
||||||
|
- Installed SSO-Auth plugin v4.0.0.4 with Authentik OIDC provider
|
||||||
|
- Configured `SchemeOverride: https`, `OidScopes: ["email"]`, `EnableAuthorization: false`
|
||||||
|
- Injected SSO button via `sso-inject.js` in web `index.html`
|
||||||
|
- Removed Caddy forward-auth gate (incompatible with SSO plugin OIDC flow)
|
||||||
|
- **Database was wiped** during cache relocation attempt — no LVM snapshot
|
||||||
|
existed. All watch states, user accounts, and library configs lost.
|
||||||
|
Libraries re-added via setup wizard.
|
||||||
|
|
||||||
|
### 2026-04-28 — wiki entry created
|
||||||
|
Initial documentation. No config changes.
|
||||||
|
|
||||||
|
### 2026-04-20 — joined the `media` GID 10000 standard
|
||||||
|
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.
|
||||||
@@ -8,7 +8,7 @@ Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clien
|
|||||||
- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
|
- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
|
||||||
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
|
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
|
||||||
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
|
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
|
||||||
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, plato, apps)
|
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
@@ -54,7 +54,7 @@ We considered three options before building this:
|
|||||||
|
|
||||||
| Option | Outcome |
|
| Option | Outcome |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. |
|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
|
||||||
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
|
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
|
||||||
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
|
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
|
||||||
|
|
||||||
@@ -18,7 +18,7 @@ Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/d
|
|||||||
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
|
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
|
||||||
|
|
||||||
## Auth
|
## Auth
|
||||||
Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
|
Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
|
||||||
|
|
||||||
## Storage
|
## Storage
|
||||||
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
|
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
|
||||||
@@ -27,7 +27,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
|
|||||||
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Authentik](124-authentik.md)
|
- [Authentik](106-auth-outpost.md)
|
||||||
- [Caddy](121-caddy.md)
|
- [Caddy](121-caddy.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../infrastructure/monitoring.md)
|
||||||
@@ -45,6 +45,9 @@ LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (r
|
|||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-24 — terminalito deploy webhook (id 12)
|
||||||
|
Push webhook on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
### 2026-04-28 — wiki entry created
|
||||||
Initial documentation.
|
Initial documentation.
|
||||||
|
|
||||||
@@ -52,7 +55,7 @@ Initial documentation.
|
|||||||
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
|
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
|
||||||
|
|
||||||
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
|
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
|
||||||
For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
|
For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
|
||||||
|
|
||||||
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
|
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
|
||||||
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.
|
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# 105 — `apps`
|
# 105 — `apps`
|
||||||
|
|
||||||
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20.
|
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `apps`
|
- **Hostname:** `apps`
|
||||||
@@ -15,7 +15,6 @@ Docker host for everything that doesn't justify its own LXC. Currently runs Arti
|
|||||||
| Hostname | Container | Backend port | Notes |
|
| Hostname | Container | Backend port | Notes |
|
||||||
| --------------------------------- | ---------------- | ------------ | ----- |
|
| --------------------------------- | ---------------- | ------------ | ----- |
|
||||||
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
|
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
|
||||||
| `books.hubris.network` | Booklore | `:6060` | Native OIDC. Redirect URI `/oauth2-callback`. |
|
|
||||||
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
|
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
|
||||||
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
|
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
|
||||||
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
|
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
|
||||||
@@ -46,12 +45,16 @@ Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webh
|
|||||||
### Portainer
|
### Portainer
|
||||||
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
|
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
|
||||||
|
|
||||||
### Booklore
|
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
|
||||||
Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (NOT `/api/oidc`). Container needs `extra_hosts: auth.hubris.network:192.168.8.175`. **Edit via Portainer UI** if it's a Portainer-managed stack.
|
|
||||||
|
|
||||||
> ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning.
|
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
|
||||||
|
> running in Docker on mac-mini. Cutover pending — see
|
||||||
|
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
|
||||||
|
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
|
||||||
|
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
|
||||||
|
> `get_agent_activity`. Source: `internal/mcp/server.go`.
|
||||||
|
|
||||||
### homelab-mcp (`/opt/homelab-mcp/`)
|
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
|
||||||
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
|
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
|
||||||
`/opt/homelab-context/` (this LXC is itself an enrolled
|
`/opt/homelab-context/` (this LXC is itself an enrolled
|
||||||
[homelab-context](../infrastructure/homelab-context.md) client). Source
|
[homelab-context](../infrastructure/homelab-context.md) client). Source
|
||||||
@@ -76,7 +79,10 @@ the server code). Listens on `0.0.0.0:9811`, secret in
|
|||||||
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
|
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
|
||||||
restarts on push.
|
restarts on push.
|
||||||
|
|
||||||
### secrets-issuance (`/opt/secrets-issuance/`)
|
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
|
||||||
|
|
||||||
|
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
|
||||||
|
> are now managed via Infisical (`docker compose --profile infisical up`).
|
||||||
Tiny HTTP service that issues per-client age keypairs the first time
|
Tiny HTTP service that issues per-client age keypairs the first time
|
||||||
each client calls `/issue`. Idempotent: subsequent calls return the
|
each client calls `/issue`. Idempotent: subsequent calls return the
|
||||||
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
|
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
|
||||||
@@ -106,20 +112,23 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h
|
|||||||
## Related
|
## Related
|
||||||
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
|
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
|
||||||
- [Caddy (121)](121-caddy.md)
|
- [Caddy (121)](121-caddy.md)
|
||||||
- [Authentik (124)](124-authentik.md)
|
- [Authentik (124)](106-auth-outpost.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
||||||
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
|
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
|
||||||
|
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
|
||||||
|
|
||||||
### 2026-05-20 — homelab-mcp + secrets-issuance live
|
### 2026-05-20 — homelab-mcp + secrets-issuance live
|
||||||
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
|
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
|
||||||
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
|
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
|
||||||
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
|
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
|
||||||
Caddy fronts both with Let's Encrypt; new vhosts on
|
Caddy fronts both with Let's Encrypt; new vhosts on
|
||||||
[caddy](121-caddy.md), split-horizon DNS entries on
|
[caddy](121-caddy.md), split-horizon DNS entries on
|
||||||
[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire
|
[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire
|
||||||
auto-deploy. LXC is itself an enrolled context client
|
auto-deploy. LXC is itself an enrolled context client
|
||||||
(`/opt/homelab-context/`).
|
(`/opt/homelab-context/`).
|
||||||
|
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# 106 — `auth-outpost`
|
# 106 — `auth-outpost`
|
||||||
|
|
||||||
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `auth-outpost`
|
- **Hostname:** `auth-outpost`
|
||||||
@@ -8,11 +8,11 @@ Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that co
|
|||||||
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
|
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
|
||||||
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
|
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
|
||||||
- **Mounts:** none
|
- **Mounts:** none
|
||||||
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
|
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md)
|
||||||
|
|
||||||
## Role
|
## Role
|
||||||
|
|
||||||
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md).
|
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||||
|
|
||||||
## Service / port map
|
## Service / port map
|
||||||
| Service | Listen | Notes |
|
| Service | Listen | Notes |
|
||||||
@@ -42,15 +42,15 @@ Fix: the LAN outpost gets its **own** domain.
|
|||||||
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
|
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
|
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
|
||||||
- [Caddy (121)](121-caddy.md) — forward-auth consumer
|
- [Caddy (121)](121-caddy.md) — forward-auth consumer
|
||||||
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
|
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
|
||||||
- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md)
|
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-06-06 — Authentik session lifetime extended to 30 days
|
### 2026-06-06 — Authentik session lifetime extended to 30 days
|
||||||
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
|
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
|
||||||
|
|
||||||
### 2026-06-01 — created; forward-auth cut over from LXC 124
|
### 2026-06-01 — created; forward-auth cut over from LXC 124
|
||||||
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).
|
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# 107 — `dns`
|
# 107 — `dns`
|
||||||
|
|
||||||
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
|
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `dns`
|
- **Hostname:** `dns`
|
||||||
@@ -29,7 +29,7 @@ Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recur
|
|||||||
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
|
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
|
||||||
|
|
||||||
## dns-sync (Technitium = authoring source)
|
## dns-sync (Technitium = authoring source)
|
||||||
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
|
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
|
||||||
|
|
||||||
## DHCP
|
## DHCP
|
||||||
|
|
||||||
@@ -42,17 +42,20 @@ Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
|
|||||||
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
|
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
|
- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq
|
||||||
- [DNS split-horizon](../infrastructure/dns.md)
|
- [DNS split-horizon](../infrastructure/dns.md)
|
||||||
- [Mesh](../infrastructure/mesh.md)
|
- [Mesh](../infrastructure/mesh.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
|
||||||
|
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
|
||||||
|
|
||||||
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
|
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
|
||||||
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
|
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
|
||||||
|
|
||||||
### 2026-06-03 — DHCP pool narrowed to `.241–.254`
|
### 2026-06-03 — DHCP pool narrowed to `.241–.254`
|
||||||
Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
|
Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
|
||||||
|
|
||||||
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
|
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
|
||||||
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
|
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
|
||||||
@@ -61,4 +64,4 @@ This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py`
|
|||||||
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
|
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
|
||||||
|
|
||||||
### 2026-06-01 — created; replaced dnsmasq on 124
|
### 2026-06-01 — created; replaced dnsmasq on 124
|
||||||
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).
|
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md).
|
||||||
@@ -11,7 +11,7 @@ Personal cloud / file collaboration. Source-of-truth for the photo libraries sur
|
|||||||
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
|
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
|
||||||
|
|
||||||
## Auth
|
## Auth
|
||||||
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern.
|
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
|
||||||
|
|
||||||
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
|
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
|
||||||
|
|
||||||
@@ -74,7 +74,7 @@ Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` ne
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
|
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
|
||||||
- [Authentik (124)](124-authentik.md)
|
- [Authentik (124)](106-auth-outpost.md)
|
||||||
- [Caddy (121)](121-caddy.md)
|
- [Caddy (121)](121-caddy.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
|
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
|
||||||
@@ -4,7 +4,8 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
|
|||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `elementsynapse`
|
- **Hostname:** `elementsynapse`
|
||||||
- **IP:** `192.168.8.239`
|
- **IP:** `192.168.8.242`
|
||||||
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
||||||
- **Privilege:** **unprivileged**
|
- **Privilege:** **unprivileged**
|
||||||
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
|
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
|
||||||
- **Mounts:** none from `/mnt/library`
|
- **Mounts:** none from `/mnt/library`
|
||||||
@@ -36,7 +37,7 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
|
|||||||
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
|
- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
|
||||||
- [Caddy](121-caddy.md)
|
- [Caddy](121-caddy.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../infrastructure/monitoring.md)
|
||||||
@@ -60,7 +60,7 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
|
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
|
||||||
- [Authentik (124)](124-authentik.md)
|
- [Authentik (124)](106-auth-outpost.md)
|
||||||
- [Caddy (121)](121-caddy.md)
|
- [Caddy (121)](121-caddy.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
||||||
@@ -11,16 +11,16 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
|
|||||||
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
|
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
|
||||||
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
|
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
|
||||||
|
|
||||||
## Sites currently served (live as of 2026-04-28)
|
## Sites currently served (live as of 2026-07-06)
|
||||||
|
|
||||||
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
|
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
|
||||||
- `auth.hubris.network` → [authentik (124)](124-authentik.md) `:9000`
|
|
||||||
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
|
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
|
||||||
- `books.hubris.network` → [apps (105)](105-apps.md) `:6060`
|
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
|
||||||
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
|
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
|
||||||
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
|
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
|
||||||
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
|
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
|
||||||
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
|
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
|
||||||
|
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
|
||||||
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
|
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
|
||||||
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
|
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
|
||||||
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
|
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
|
||||||
@@ -28,15 +28,18 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
|
|||||||
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
|
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
|
||||||
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
|
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
|
||||||
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
|
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
|
||||||
|
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
|
||||||
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
|
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
|
||||||
|
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
|
||||||
|
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
|
||||||
|
|
||||||
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
|
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
|
||||||
|
|
||||||
## Snippet: `(authentik)` forward-auth
|
## Snippet: `(authentik)` forward-auth
|
||||||
|
|
||||||
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
|
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup).
|
||||||
|
|
||||||
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
|
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md).
|
||||||
|
|
||||||
## Caddy environment
|
## Caddy environment
|
||||||
|
|
||||||
@@ -57,7 +60,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
|
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
|
||||||
- [Authentik (124)](124-authentik.md) — forward-auth + IdP
|
- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP
|
||||||
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
||||||
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
|
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
|
||||||
- [Gitea (104)](104-gitea.md) — webhook source
|
- [Gitea (104)](104-gitea.md) — webhook source
|
||||||
@@ -82,7 +85,7 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
|
|||||||
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
|
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
|
||||||
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
|
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
|
||||||
|
|
||||||
Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
|
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
|
||||||
|
|
||||||
### 2026-06-02 — caddy.service unit missing; recreated
|
### 2026-06-02 — caddy.service unit missing; recreated
|
||||||
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
|
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
|
||||||
@@ -4,10 +4,11 @@ Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr
|
|||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Hostname:** `arriman`
|
- **Hostname:** `arriman`
|
||||||
- **IP:** `192.168.8.132`
|
- **IP:** `192.168.8.245`
|
||||||
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
||||||
- **Privilege:** privileged
|
- **Privilege:** privileged
|
||||||
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
|
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
|
||||||
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
|
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
|
||||||
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
|
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
|
||||||
|
|
||||||
## Compose
|
## Compose
|
||||||
@@ -114,7 +115,7 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Caddy (121)](121-caddy.md)
|
- [Caddy (121)](121-caddy.md)
|
||||||
- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass
|
- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Media permissions](../infrastructure/media-permissions.md)
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
- [Hubris host](../hosts/hubris.md)
|
- [Hubris host](../hosts/hubris.md)
|
||||||
48
archive/knowledge/containers/128-trmnl.md
Normal file
48
archive/knowledge/containers/128-trmnl.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# 128 — `trmnl`
|
||||||
|
|
||||||
|
Self-hosted middleware for TRMNL e-ink plugins. TRMNL cloud polls it; it fetches/shapes live data into JSON the plugin's Liquid template renders.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
- **Hostname:** `trmnl`
|
||||||
|
- **IP:** `192.168.8.211`
|
||||||
|
- **Privilege:** unprivileged
|
||||||
|
- **Resources:** 1 core / 768 MiB RAM / 8 GiB rootfs (Debian 13)
|
||||||
|
- **Mounts:** none
|
||||||
|
- **Public hostname:** `trmnl.hubris.network` (via [VPS ingress](../infrastructure/ingress.md))
|
||||||
|
|
||||||
|
## Role
|
||||||
|
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| `trmnl-plugins` | `0.0.0.0:9851` | uvicorn aggregator; `EnvironmentFile=/etc/trmnl-plugins/env` |
|
||||||
|
|
||||||
|
## Storage / config paths
|
||||||
|
- `/opt/terminalito` — git checkout (origin = internal gitea `http://192.168.8.121:3000/dtoro/terminalito.git`)
|
||||||
|
- `/opt/terminalito/server/.venv` — venv
|
||||||
|
- `/etc/trmnl-plugins/env` — `TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
|
||||||
|
- `/etc/systemd/system/trmnl-plugins.service`
|
||||||
|
|
||||||
|
## Auto-deploy
|
||||||
|
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main` → `server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
|
||||||
|
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
|
||||||
|
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
|
||||||
|
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
|
||||||
|
- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
### 2026-06-24 — auto-deploy + LAN DNS wired
|
||||||
|
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
|
||||||
|
|
||||||
|
### 2026-06-24 — public path live
|
||||||
|
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
|
||||||
|
|
||||||
|
### 2026-06-24 — provisioned
|
||||||
|
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public` → `192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.
|
||||||
52
archive/knowledge/containers/129-house.md
Normal file
52
archive/knowledge/containers/129-house.md
Normal file
@@ -0,0 +1,52 @@
|
|||||||
|
# 129 — `house`
|
||||||
|
|
||||||
|
Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modules: calendar, tasks, meals, groceries, budget, documents, notes, contacts, birthdays, housekeeping, recipes, reminders.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Hostname:** `house`
|
||||||
|
- **IP:** `192.168.8.244`
|
||||||
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
||||||
|
- **Privilege:** unprivileged
|
||||||
|
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
|
||||||
|
- **Mounts:** none
|
||||||
|
- **Public hostname:** [`house.hubris.network`](../infrastructure/ingress.md) → VPS traefik → Caddy
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| `oikos` (Yuvomi) | `0.0.0.0:3000` | Docker Compose at `/opt/yuvomi/`, image `ghcr.io/ulsklyc/yuvomi` |
|
||||||
|
|
||||||
|
## Integrations
|
||||||
|
|
||||||
|
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
|
||||||
|
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
|
||||||
|
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
|
||||||
|
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
|
||||||
|
|
||||||
|
## Config paths
|
||||||
|
|
||||||
|
- `/opt/yuvomi/docker-compose.yml` — downloaded from upstream
|
||||||
|
- `/opt/yuvomi/.env` — config including secrets (untracked)
|
||||||
|
- `/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
|
||||||
|
- `/opt/yuvomi/backups/` — auto backups
|
||||||
|
- `/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
|
||||||
|
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
|
||||||
|
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
|
||||||
|
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
|
||||||
|
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
|
||||||
|
- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-27 — Google Calendar OAuth env vars configured
|
||||||
|
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
|
||||||
|
|
||||||
|
### 2026-06-26 — provisioned
|
||||||
|
|
||||||
|
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.
|
||||||
66
archive/knowledge/containers/130-grimmory.md
Normal file
66
archive/knowledge/containers/130-grimmory.md
Normal file
@@ -0,0 +1,66 @@
|
|||||||
|
# 130 — `grimmory`
|
||||||
|
|
||||||
|
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Hostname:** `grimmory`
|
||||||
|
- **IP:** `192.168.8.247`
|
||||||
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
||||||
|
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
|
||||||
|
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
|
||||||
|
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
|
||||||
|
- **Public hostname:** `books.hubris.network`
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| Grimmory | `192.168.8.247:6060` | Docker Compose at `/opt/grimmory/` |
|
||||||
|
| MariaDB | internal only | Sidecar in the same compose stack |
|
||||||
|
|
||||||
|
## Compose
|
||||||
|
|
||||||
|
Located at `/opt/grimmory/docker-compose.yml`. Key points:
|
||||||
|
|
||||||
|
- Image: `ghcr.io/grimmory-tools/grimmory:latest`
|
||||||
|
- Books library: `/mnt/library/books` → `/books` (read-write; media GID 10000 via `GROUP_ID=10000`)
|
||||||
|
- Bookdrop (watched folder for auto-import): `/opt/grimmory/bookdrop` → `/bookdrop`
|
||||||
|
- App data (covers, DB config): `/opt/grimmory/data` → `/app/data`
|
||||||
|
- MariaDB config: `/opt/grimmory/mariadb/config` → `/config` (linuxserver/mariadb image)
|
||||||
|
- `extra_hosts: auth.hubris.network:192.168.8.175` — routes Authentik OIDC discovery to Caddy from inside the container
|
||||||
|
- `FORCE_DISABLE_OIDC=false` — OIDC stays enabled; provider configured via Grimmory admin UI
|
||||||
|
|
||||||
|
Credentials live in `/opt/grimmory/.env` (untracked):
|
||||||
|
- `DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
|
||||||
|
- `MYSQL_ROOT_PASSWORD` — MariaDB root password
|
||||||
|
|
||||||
|
## Authentik OIDC
|
||||||
|
|
||||||
|
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
|
||||||
|
|
||||||
|
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
|
||||||
|
- **Client ID:** `L1u0eFsNhbKgiIvvFeIr2mvZdbtFyzidCq2h6thL`
|
||||||
|
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
|
||||||
|
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
|
||||||
|
- **Scopes:** openid, profile, email, offline_access
|
||||||
|
- **Back-channel logout:** `http://192.168.8.247:6060/api/v1/auth/oidc/backchannel-logout`
|
||||||
|
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
|
||||||
|
|
||||||
|
## Media permissions
|
||||||
|
|
||||||
|
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [apps (105)](105-apps.md) — previous host (Booklore)
|
||||||
|
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060`
|
||||||
|
- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory`
|
||||||
|
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
|
||||||
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-29 — provisioned; Booklore migrated
|
||||||
|
|
||||||
|
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.
|
||||||
62
archive/knowledge/containers/131-teddycloud.md
Normal file
62
archive/knowledge/containers/131-teddycloud.md
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
# 131 — `teddycloud`
|
||||||
|
|
||||||
|
Self-hosted [TeddyCloud](https://github.com/toniebox-reverse-engineering/teddycloud), a
|
||||||
|
reimplementation of the Toniebox cloud backend — lets Tonie figurines play custom/ripped
|
||||||
|
audio content against a local server instead of the official cloud.
|
||||||
|
|
||||||
|
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
|
||||||
|
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
|
||||||
|
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
|
||||||
|
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
|
||||||
|
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
|
||||||
|
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
|
||||||
|
for it — but it never got its own inventory entry or doc page until now.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Hostname:** `teddycloud`
|
||||||
|
- **Host:** hubris (confirmed via `pct config 131` run directly on hubris — the original
|
||||||
|
drift finding's `pct list` source)
|
||||||
|
- **IP:** `192.168.8.150` (static; was briefly `192.168.8.243` via DHCP until the
|
||||||
|
2026-07-05 strong-migration work assigned it a fixed address — see hosts/strong.md)
|
||||||
|
- **Privilege:** unconfirmed (not checked — read-only investigation didn't need it)
|
||||||
|
- **Resources:** 1 core / 1 GiB RAM / 512 MiB swap / 16 GiB rootfs (`local-lvm`), Debian 12
|
||||||
|
(bookworm)
|
||||||
|
- **Mounts:** `/mnt/library` (`mp0`)
|
||||||
|
- **Public hostname:** `teddy.hubris.network` → `192.168.8.150:8443` (see
|
||||||
|
[caddy (121)](121-caddy.md))
|
||||||
|
- **Enrollment:** none — no `age_pubkey`, not a `homelab` CLI client. It's a plain
|
||||||
|
docker-compose app container, not a fleet-managed host. No action needed unless it starts
|
||||||
|
needing secrets.
|
||||||
|
|
||||||
|
## Service
|
||||||
|
|
||||||
|
Runs via `docker compose` at `/opt/teddycloud` — container `teddycloud-teddycloud-1`,
|
||||||
|
image `ghcr.io/toniebox-reverse-engineering/teddycloud:latest`, publishing `80`, `443`,
|
||||||
|
`8080`, and `8443`. Caddy routes `teddy.hubris.network` to the `:8443` port.
|
||||||
|
|
||||||
|
## Risk notes
|
||||||
|
|
||||||
|
**No Caddy forward-auth gate** — unlike `sab.hubris.network` on the same Caddyfile (which
|
||||||
|
is explicitly annotated `(Authentik forward-auth)`), `teddy.hubris.network` has no auth
|
||||||
|
annotation. It's reachable to anyone on the LAN/mesh who can resolve the hostname. Not
|
||||||
|
addressed as part of this doc pass — flagging it here since it's now visible in one place
|
||||||
|
for the first time.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Hubris host](../hosts/hubris.md)
|
||||||
|
- [Caddy (121)](121-caddy.md) — terminates `teddy.hubris.network`
|
||||||
|
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
|
||||||
|
taken by this container
|
||||||
|
- [Containers index](index.md)
|
||||||
|
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-06 — documented for the first time (drift-caught)
|
||||||
|
|
||||||
|
Added to `inventory.yaml` and given this page. Verified live via read-only `pct config 131`
|
||||||
|
+ `pct exec 131 -- ...` on hubris: hostname, IP, resources, and that it runs via
|
||||||
|
`docker compose` (not a raw binary or systemd unit). No changes made to the running
|
||||||
|
container — this is pure documentation catch-up.
|
||||||
199
archive/knowledge/containers/132-rclone.md
Normal file
199
archive/knowledge/containers/132-rclone.md
Normal file
@@ -0,0 +1,199 @@
|
|||||||
|
# 132 — `rclone`
|
||||||
|
|
||||||
|
Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
|
||||||
|
with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
|
||||||
|
and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
|
||||||
|
|
||||||
|
Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
|
||||||
|
so this landed on **132**.)
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Hostname:** `rclone`
|
||||||
|
- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
|
||||||
|
- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
|
||||||
|
incl. `homecloud/` and `documents/`, regardless of owner)
|
||||||
|
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
|
||||||
|
after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
|
||||||
|
- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
|
||||||
|
job must never be able to write into the library
|
||||||
|
- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
|
||||||
|
| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
|
||||||
|
|
||||||
|
## Backup design
|
||||||
|
|
||||||
|
- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
|
||||||
|
- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
|
||||||
|
browsable in Proton's web UI).
|
||||||
|
- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
|
||||||
|
(`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
|
||||||
|
disks once bind-mounted into this LXC.
|
||||||
|
- **Path mapping:** source `S` → `proton:library-backup/<S without leading slash>`
|
||||||
|
(e.g. `/mnt/library/notes` → `proton:library-backup/mnt/library/notes`).
|
||||||
|
- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
|
||||||
|
the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
|
||||||
|
rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
|
||||||
|
so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
|
||||||
|
Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
|
||||||
|
rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
|
||||||
|
query params 404s.
|
||||||
|
- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
|
||||||
|
JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
|
||||||
|
- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
|
||||||
|
**TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
|
||||||
|
|
||||||
|
## rclone + Proton Drive
|
||||||
|
|
||||||
|
- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
|
||||||
|
(`rclone v1.74.3`).
|
||||||
|
- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
|
||||||
|
**This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
|
||||||
|
commit it.** Escrow the Proton account creds in the password manager.
|
||||||
|
- **Config gotchas** (from rclone docs/forum):
|
||||||
|
- Log into Proton via a **browser at least once** first, or key generation fails.
|
||||||
|
- For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
|
||||||
|
codes; obscure with `rclone obscure`.
|
||||||
|
- Passwords with **extended-ASCII** characters are known to break auth.
|
||||||
|
- Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
|
||||||
|
- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
|
||||||
|
to this LXC's age key, so the remote can be rebuilt after a re-provision.
|
||||||
|
|
||||||
|
## The UI (rclone Web GUI)
|
||||||
|
|
||||||
|
`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
|
||||||
|
start). Reach it at **http://192.168.8.214:5572** on the LAN.
|
||||||
|
|
||||||
|
> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
|
||||||
|
> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
|
||||||
|
> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
|
||||||
|
|
||||||
|
## Tracked config (deferred)
|
||||||
|
|
||||||
|
**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
|
||||||
|
directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
|
||||||
|
`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
|
||||||
|
staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
|
||||||
|
[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
|
||||||
|
See [auto-deploy](../infrastructure/auto-deploy.md).
|
||||||
|
|
||||||
|
**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
|
||||||
|
`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
|
||||||
|
a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
|
||||||
|
after this set.
|
||||||
|
|
||||||
|
## Enrollment gotcha: `pct exec` PATH
|
||||||
|
|
||||||
|
`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
|
||||||
|
`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
|
||||||
|
`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
|
||||||
|
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
|
||||||
|
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
|
||||||
|
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
|
||||||
|
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
|
||||||
|
on future LXC bootstraps.
|
||||||
|
|
||||||
|
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
|
||||||
|
|
||||||
|
What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
|
||||||
|
API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
|
||||||
|
OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
|
||||||
|
`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
|
||||||
|
"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
|
||||||
|
is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
|
||||||
|
`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
|
||||||
|
2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
|
||||||
|
hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
|
||||||
|
249M, `repos` 83M) completed cleanly with no further kills.
|
||||||
|
|
||||||
|
**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
|
||||||
|
`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
|
||||||
|
haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
|
||||||
|
this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
|
||||||
|
(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
|
||||||
|
memory pressure returns under a larger folder set later. State kept in
|
||||||
|
`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
|
||||||
|
|
||||||
|
**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
|
||||||
|
1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
|
||||||
|
but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
|
||||||
|
healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
|
||||||
|
`group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
|
||||||
|
an incident; cross-check with unfiltered `core/stats` first.**
|
||||||
|
2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
|
||||||
|
runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
|
||||||
|
any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
|
||||||
|
`systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
|
||||||
|
long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
|
||||||
|
never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
|
||||||
|
*finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
|
||||||
|
Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
|
||||||
|
- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
|
||||||
|
- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
|
||||||
|
- [Containers index](index.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
|
||||||
|
|
||||||
|
What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
|
||||||
|
getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
|
||||||
|
`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
|
||||||
|
Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
|
||||||
|
(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
|
||||||
|
completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
|
||||||
|
stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
|
||||||
|
blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
|
||||||
|
issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
|
||||||
|
bump addresses the actual root cause.
|
||||||
|
|
||||||
|
### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
|
||||||
|
|
||||||
|
The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
|
||||||
|
GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
|
||||||
|
`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
|
||||||
|
`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
|
||||||
|
`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
|
||||||
|
rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
|
||||||
|
as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
|
||||||
|
completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
|
||||||
|
than truncating in place, specifically so it wouldn't risk corrupting the still-running original
|
||||||
|
`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
|
||||||
|
unaffected, kept running to completion under the old in-memory script content). The already-running
|
||||||
|
scheduled sync from before this change is a standalone process and won't retroactively appear in the
|
||||||
|
GUI; every run after this point will.
|
||||||
|
|
||||||
|
### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
|
||||||
|
|
||||||
|
Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
|
||||||
|
entered instead of the TOTP secret — reconfigured with the secret, auth now works
|
||||||
|
(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
|
||||||
|
(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
|
||||||
|
an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
|
||||||
|
plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
|
||||||
|
`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
|
||||||
|
actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
|
||||||
|
rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
|
||||||
|
choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
|
||||||
|
staged at `/root/rclone-repo` for later.
|
||||||
|
|
||||||
|
### 2026-07-01 — provisioned; enrolled
|
||||||
|
|
||||||
|
LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
|
||||||
|
installed from the official binary (`protondrive` backend present). Runner + monthly timer +
|
||||||
|
`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
|
||||||
|
homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
|
||||||
|
secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
|
||||||
|
updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
|
||||||
|
remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
|
||||||
|
operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
|
||||||
|
backup deprecated in the same change.
|
||||||
61
archive/knowledge/containers/133-seanime.md
Normal file
61
archive/knowledge/containers/133-seanime.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
# seanime (LXC 133) — Seanime anime media server
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| VMID | 133 |
|
||||||
|
| Host | strong |
|
||||||
|
| Role | Anime media server (online streaming + torrent client) |
|
||||||
|
| LAN IP | 192.168.8.248/28 (vmbr1) |
|
||||||
|
| Public URL | https://seanime.hubris.network |
|
||||||
|
| Backend | Seanime v3.9.0, systemd service |
|
||||||
|
| Port | 43211 |
|
||||||
|
|
||||||
|
## Mounts
|
||||||
|
|
||||||
|
- `/anime` → `/mnt/media_local/anime` (ludo-lvm, bind mount) — existing anime collection
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
### Seanime (native binary)
|
||||||
|
|
||||||
|
- Binary: `/opt/seanime/bin/seanime`
|
||||||
|
- Data: `/opt/seanime/data/`
|
||||||
|
- Config: `/opt/seanime/data/config.toml`
|
||||||
|
- Service: `systemctl status seanime`
|
||||||
|
- Flags: `--host 0.0.0.0 --port 43211 --datadir /opt/seanime/data --disable-password`
|
||||||
|
- Config highlights:
|
||||||
|
- `secureMode = 'lax'` (allows non-local Caddy proxy)
|
||||||
|
- `trustedProxies = ['192.168.8.0/24']`
|
||||||
|
|
||||||
|
### Torrent client (qBittorrent on arriman)
|
||||||
|
|
||||||
|
- Host: `192.168.8.245:8080`
|
||||||
|
- Auth: subnet whitelist (no password needed from homelab LAN)
|
||||||
|
- qBittorrent config updated: `AuthSubnetWhitelist=192.168.8.0/24`
|
||||||
|
|
||||||
|
### Extensions installed
|
||||||
|
|
||||||
|
**Online streaming (8):**
|
||||||
|
HiAnime, AniWatch, KickAssAnime, Anicrush, Animo, AniNeko, Senshi, Sudatchi
|
||||||
|
|
||||||
|
## Caddy
|
||||||
|
|
||||||
|
- Config: `seanime.hubris.network` block in `/etc/caddy/Caddyfile` (dtoro/caddy-conf repo)
|
||||||
|
- TLS: Let's Encrypt DNS-01 via IONOS API
|
||||||
|
- Backend: `192.168.8.248:43211`
|
||||||
|
|
||||||
|
## DNS
|
||||||
|
|
||||||
|
- `seanime.hubris.network` A record → `192.168.8.175` (Caddy)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-05 — initial creation
|
||||||
|
- LXC 133 created on strong (Debian 13, 2 cores / 2 GiB)
|
||||||
|
- Seanime v3.9.0 installed as native binary + systemd service
|
||||||
|
- /anime bind mount from ludo-lvm
|
||||||
|
- qBittorrent on arriman configured as torrent client
|
||||||
|
- Caddy reverse proxy + DNS records set up
|
||||||
|
- Online streaming extensions installed
|
||||||
82
archive/knowledge/containers/134-romm.md
Normal file
82
archive/knowledge/containers/134-romm.md
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
# 134 — `romm`
|
||||||
|
|
||||||
|
Self-hosted ROM manager ([RomM](https://romm.app)). Browse, search, and play
|
||||||
|
your retro game library from the browser. Runs on Docker Compose with a
|
||||||
|
MariaDB sidecar.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Hostname:** `romm`
|
||||||
|
- **IP:** `192.168.8.249/28` (static, vmbr1 on strong)
|
||||||
|
- **Privilege:** privileged
|
||||||
|
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13, ludo-lvm)
|
||||||
|
- **Mounts:** `/mnt/media_local` → `/mnt/library`
|
||||||
|
- **Public hostname:** `roms.hubris.network`
|
||||||
|
|
||||||
|
## Service / port map
|
||||||
|
|
||||||
|
| Service | Listen | Notes |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| RomM | `192.168.8.249:80` | HTTP (Caddy terminates TLS) |
|
||||||
|
| MariaDB | internal only | Sidecar in the same compose stack |
|
||||||
|
|
||||||
|
## Compose
|
||||||
|
|
||||||
|
Located at `/opt/romm/docker-compose.yml`. Key points:
|
||||||
|
|
||||||
|
- Image: `rommapp/romm:latest`
|
||||||
|
- DB sidecar: `mariadb:latest` with healthcheck
|
||||||
|
- ROM library: `/mnt/library/roms` → `/romm/library` (writable)
|
||||||
|
- Resources (covers, etc.): Docker named volume `romm_resources` → `/romm/resources`
|
||||||
|
- Saves/states: `/opt/romm/assets` → `/romm/assets`
|
||||||
|
- Config: `/opt/romm/config` → `/romm/config`
|
||||||
|
- Auth key: auto-generated, stored in `/opt/romm/.env`
|
||||||
|
|
||||||
|
Environment (`/opt/romm/.env`):
|
||||||
|
- `DB_ROOT_PASSWD` / `DB_PASSWD` — MariaDB credentials
|
||||||
|
- `DB_USER=romm-user` / `DB_NAME=romm`
|
||||||
|
- `ROMM_AUTH_SECRET_KEY` — auto-generated
|
||||||
|
|
||||||
|
## ROM library structure
|
||||||
|
|
||||||
|
RomM expects `/mnt/library/roms/<platform>/<game>/<rom>`. Create platform
|
||||||
|
directories as needed:
|
||||||
|
|
||||||
|
```
|
||||||
|
/mnt/media_local/roms/
|
||||||
|
├── gba/
|
||||||
|
│ └── Pokemon - Emerald/
|
||||||
|
│ └── Pokemon Emerald.gba
|
||||||
|
├── snes/
|
||||||
|
│ └── Super Mario World/
|
||||||
|
│ └── Super Mario World.sfc
|
||||||
|
└── psx/
|
||||||
|
└── Final Fantasy VII/
|
||||||
|
└── Final Fantasy VII.bin
|
||||||
|
```
|
||||||
|
|
||||||
|
## Media permissions
|
||||||
|
|
||||||
|
The `/mnt/media_local/roms` directory is owned `root:media` with mode `2775`
|
||||||
|
(setgid). New files inherit the `media` GID (10000). The LXC is privileged so
|
||||||
|
no idmap block is needed — in-container UID/GID matches the host. Docker
|
||||||
|
containers within the LXC run as-is (read-only mount).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Strong host](../hosts/strong.md)
|
||||||
|
- [Caddy (121)](121-caddy.md) — `roms.hubris.network → 192.168.8.249:80`
|
||||||
|
- [DNS (107)](107-dns.md) — `roms.hubris.network A 192.168.8.175`
|
||||||
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
|
- [RomM docs](https://docs.romm.app)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-05 — provisioned
|
||||||
|
|
||||||
|
LXC 134 created on strong (Debian 13, privileged, `192.168.8.249/28`).
|
||||||
|
Docker + Compose installed. RomM stack deployed at `/opt/romm/`.
|
||||||
|
Created `/mnt/media_local/roms` with setgid `media:GID=10000` mode `2775`.
|
||||||
|
Caddy `roms.hubris.network` → `192.168.8.249:80`.
|
||||||
|
DNS `roms A 192.168.8.175` added to Technitium.
|
||||||
|
Hubris /32 route for `.249` added to `50-strong-route`.
|
||||||
57
archive/knowledge/containers/index.md
Normal file
57
archive/knowledge/containers/index.md
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
# LXC containers — index
|
||||||
|
|
||||||
|
Most containers live on [`hubris`](../hosts/hubris.md). Some have been
|
||||||
|
[migrated to `strong`](../hosts/strong.md) (Phase 1+2, 2026-07-05).
|
||||||
|
|
||||||
|
| ID | Name | Host | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status |
|
||||||
|
| --- | ---------------- | ------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
|
||||||
|
| 101 | [jellyfin](101-jellyfin.md) | **strong** | 192.168.8.246 | priv | 4 | 8 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `media.hubris.network` | running |
|
||||||
|
| 103 | [paperless](103-paperless.md) | hubris | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
|
||||||
|
| 104 | [gitea](104-gitea.md) | hubris | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
|
||||||
|
| 105 | [apps](105-apps.md) | hubris | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
|
||||||
|
| 114 | [nextcloud](114-nextcloud.md) | hubris | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
|
||||||
|
| 118 | [elementsynapse](118-elementsynapse.md) | **strong** | 192.168.8.242 | unpriv | 2 | 4 GiB | 32 GiB | — | `matrix.hubris.network` | running |
|
||||||
|
| 119 | [sophia](119-sophia.md) | hubris | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
|
||||||
|
| 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running |
|
||||||
|
| 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
|
||||||
|
| 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running |
|
||||||
|
| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
|
||||||
|
| 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
|
||||||
|
| 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running |
|
||||||
|
| 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running |
|
||||||
|
| 131 | [teddycloud](131-teddycloud.md) | hubris | 192.168.8.150 | — | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (no auth gate) | running |
|
||||||
|
|| 132 | [rclone](132-rclone.md) | hubris | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running |
|
||||||
|
|| 134 | [romm](134-romm.md) | **strong** | 192.168.8.249 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `roms.hubris.network` | running |
|
||||||
|
|
||||||
|
## Recently destroyed (kept for archaeology)
|
||||||
|
|
||||||
|
| ID | Name | Destroyed | Reason |
|
||||||
|
| --- | ---------------- | --------------- | --------------------------------------------- |
|
||||||
|
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
|
||||||
|
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
|
||||||
|
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
|
||||||
|
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
|
||||||
|
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
|
||||||
|
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) |
|
||||||
|
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
|
||||||
|
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
|
||||||
|
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
|
||||||
|
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
|
||||||
|
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
|
||||||
|
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
|
||||||
|
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
|
||||||
|
|
||||||
|
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
|
||||||
|
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
|
||||||
|
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
|
||||||
|
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
- [Hubris host](../hosts/hubris.md)
|
||||||
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
|
- [Caddy](121-caddy.md) — terminates every public hostname
|
||||||
|
- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain
|
||||||
@@ -1,11 +1,14 @@
|
|||||||
# `hubris` — Proxmox host
|
# `hubris` — Proxmox host
|
||||||
|
|
||||||
Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
|
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
|
||||||
|
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
|
||||||
|
`Homelab` cluster (see [Cluster](#cluster)); the second node is
|
||||||
|
[strong](strong.md), which hosts nothing yet.
|
||||||
|
|
||||||
## At a glance
|
## At a glance
|
||||||
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
||||||
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
||||||
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
|
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
||||||
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
||||||
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
||||||
@@ -22,13 +25,37 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
|
|||||||
|
|
||||||
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
|
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
|
||||||
|
|
||||||
|
## Cluster
|
||||||
|
|
||||||
|
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
|
||||||
|
(cluster/OS hostname `strong`), formed 2026-07-01.
|
||||||
|
|
||||||
|
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
|
||||||
|
strong reaches it via the existing Fritz!Box static route
|
||||||
|
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
|
||||||
|
household LAN. Fine for a home cluster; not latency-isolated.
|
||||||
|
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
|
||||||
|
votes — if either node is down (reboot, maintenance, network hiccup), the
|
||||||
|
survivor's running guests keep working but `/etc/pve` goes read-only:
|
||||||
|
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
|
||||||
|
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
|
||||||
|
Quirks below) make this painful in practice.
|
||||||
|
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
|
||||||
|
(every node has its own, not actually shared). The `library` lvmthin pool
|
||||||
|
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
|
||||||
|
it's a physical thinpool that only exists on this host's hardware.
|
||||||
|
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
|
||||||
|
member so far. See [strong.md](strong.md) and the [library-SSD
|
||||||
|
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
|
||||||
|
for what comes next (physical drive move, service migration — not started).
|
||||||
|
|
||||||
## Tenants
|
## Tenants
|
||||||
|
|
||||||
### VMs
|
### VMs
|
||||||
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
|
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
|
||||||
|
|
||||||
### LXC containers
|
### LXC containers
|
||||||
See [containers/index](../containers/index.md). 13 active (109 syncthing destroyed 2026-05-14).
|
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
|
||||||
|
|
||||||
## Boot-time tuning (load-bearing)
|
## Boot-time tuning (load-bearing)
|
||||||
|
|
||||||
@@ -62,7 +89,7 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
|
|||||||
|
|
||||||
## Quirks
|
## Quirks
|
||||||
|
|
||||||
- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
|
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
|
||||||
- ZFS is **not** in use; storage is LVM-thin + ext4.
|
- ZFS is **not** in use; storage is LVM-thin + ext4.
|
||||||
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
|
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
|
||||||
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
|
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
|
||||||
@@ -73,6 +100,8 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b
|
|||||||
|
|
||||||
- `root@hubris` (self, RSA) — local
|
- `root@hubris` (self, RSA) — local
|
||||||
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
|
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
|
||||||
|
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
|
||||||
|
`pvecm add` could authenticate without a password prompt
|
||||||
|
|
||||||
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
|
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
|
||||||
|
|
||||||
@@ -84,16 +113,20 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
|
|||||||
- [Media permissions](../infrastructure/media-permissions.md)
|
- [Media permissions](../infrastructure/media-permissions.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../infrastructure/monitoring.md)
|
||||||
- [Backups (disabled)](../infrastructure/backups.md)
|
- [Backups (disabled)](../infrastructure/backups.md)
|
||||||
- [Operations cheatsheet](../operations/commands.md)
|
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||||
|
- [strong — Proxmox host](strong.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
|
||||||
|
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
|
||||||
|
|
||||||
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
|
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
|
||||||
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
|
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
|
||||||
|
|
||||||
### 2026-05-14 — LXC 109 (syncthing) decommissioned
|
### 2026-05-14 — LXC 109 (syncthing) decommissioned
|
||||||
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
|
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
|
||||||
|
|
||||||
### 2026-05-14 — network performance baseline captured
|
### 2026-05-14 — network performance baseline captured
|
||||||
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
|
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
|
||||||
@@ -105,7 +138,7 @@ User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`.
|
|||||||
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
||||||
|
|
||||||
### 2026-04-29 — relocated to better-ventilated spot
|
### 2026-04-29 — relocated to better-ventilated spot
|
||||||
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
||||||
|
|
||||||
### 2026-04-28 — Phase 1 WiFi failover
|
### 2026-04-28 — Phase 1 WiFi failover
|
||||||
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
||||||
@@ -114,10 +147,10 @@ Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failov
|
|||||||
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
||||||
|
|
||||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||||
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
||||||
|
|
||||||
### 2026-04-22 — drive removal A/B test
|
### 2026-04-22 — drive removal A/B test
|
||||||
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
|
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
||||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||||
@@ -126,4 +159,4 @@ Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `
|
|||||||
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
||||||
|
|
||||||
### 2026-04-21 — `cpu-epp.service` deployed
|
### 2026-04-21 — `cpu-epp.service` deployed
|
||||||
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).
|
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
9
archive/knowledge/hosts/index.md
Normal file
9
archive/knowledge/hosts/index.md
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
# Hosts
|
||||||
|
|
||||||
|
The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are
|
||||||
|
the generated `hosts/*.yaml` files at the repo root.
|
||||||
|
|
||||||
|
| Document | What it covers |
|
||||||
|
|----------|----------------|
|
||||||
|
| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. |
|
||||||
|
| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. |
|
||||||
159
archive/knowledge/hosts/strong.md
Normal file
159
archive/knowledge/hosts/strong.md
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
# `strong` — Proxmox host
|
||||||
|
|
||||||
|
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
|
||||||
|
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
|
||||||
|
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
|
||||||
|
|
||||||
|
## At a glance
|
||||||
|
|
||||||
|
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
|
||||||
|
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
|
||||||
|
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
|
||||||
|
2026-07-01 — earlier the same day the inventory entry was briefly named
|
||||||
|
`ludo-mini`, but that was renamed so the hostname bootstrap needs
|
||||||
|
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
|
||||||
|
the machine's everyday nickname; some older docs (investigations, the
|
||||||
|
library-SSD migration plan) still refer to it that way — that's fine,
|
||||||
|
those are historical.
|
||||||
|
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
|
||||||
|
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
|
||||||
|
(thinpool) 815G, 8G swap.
|
||||||
|
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
|
||||||
|
existing APFS partition (previously used in a Mac) — wiped via
|
||||||
|
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
|
||||||
|
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
|
||||||
|
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
|
||||||
|
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
|
||||||
|
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
|
||||||
|
was actually created live; not worth renaming), restricted `nodes strong`
|
||||||
|
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
|
||||||
|
Empty so far; this is separate from the [library-SSD migration
|
||||||
|
plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
|
||||||
|
planned drive move from hubris (that hasn't happened) — this is
|
||||||
|
general-purpose VM/CT capacity.
|
||||||
|
- **Network:** `vmbr0` is bridged straight onto the household LAN —
|
||||||
|
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
|
||||||
|
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
|
||||||
|
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
|
||||||
|
via the existing Fritz!Box static route through hubris; no new cabling
|
||||||
|
or routing was needed to join the cluster.
|
||||||
|
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
|
||||||
|
with hubris by convention, actually separate per-node volumes). Hubris's
|
||||||
|
`library` lvmthin pool is restricted to `nodes hubris` in
|
||||||
|
`/etc/pve/storage.cfg` and does not exist here.
|
||||||
|
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
|
||||||
|
workstation had. Reachable today only via LAN routing (confirmed DNS for
|
||||||
|
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
|
||||||
|
Add to Netbird if off-LAN access to this host itself (distinct from any
|
||||||
|
future guests) is needed.
|
||||||
|
- **Homelab-context client enrollment:** done 2026-07-01 via
|
||||||
|
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
|
||||||
|
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
|
||||||
|
5-min sync timer are live; `homelab whoami` resolves correctly. See
|
||||||
|
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
|
||||||
|
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
|
||||||
|
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
|
||||||
|
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
|
||||||
|
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
|
||||||
|
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
|
||||||
|
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
|
||||||
|
|
||||||
|
## Cluster membership
|
||||||
|
|
||||||
|
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
|
||||||
|
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
|
||||||
|
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
|
||||||
|
either node going down freezes management on the survivor).
|
||||||
|
|
||||||
|
## SSH
|
||||||
|
|
||||||
|
Root login via the same key set trusted on hubris (`root@hubris`,
|
||||||
|
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
|
||||||
|
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
|
||||||
|
post cluster-join, so it's cluster-synced same as hubris). No password auth
|
||||||
|
needed going forward.
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [hubris — Proxmox host](hubris.md)
|
||||||
|
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
|
||||||
|
- [Network](../infrastructure/network.md)
|
||||||
|
- [SSH access](../infrastructure/ssh-access.md)
|
||||||
|
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||||
|
|
||||||
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
|
||||||
|
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
|
||||||
|
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
|
||||||
|
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
|
||||||
|
books.hubris.network updated.
|
||||||
|
|
||||||
|
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
|
||||||
|
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
|
||||||
|
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
|
||||||
|
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
|
||||||
|
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
|
||||||
|
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
|
||||||
|
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
|
||||||
|
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
|
||||||
|
Dhcp scope narrowed to 192.168.8.100-239.
|
||||||
|
|
||||||
|
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
|
||||||
|
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
|
||||||
|
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
|
||||||
|
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
|
||||||
|
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
|
||||||
|
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
|
||||||
|
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
|
||||||
|
|
||||||
|
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
|
||||||
|
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
|
||||||
|
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
|
||||||
|
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
|
||||||
|
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
|
||||||
|
Caddy updated: element.hubris.network → 192.168.8.242:8080.
|
||||||
|
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
|
||||||
|
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
|
||||||
|
|
||||||
|
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
|
||||||
|
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
|
||||||
|
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
|
||||||
|
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
|
||||||
|
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
|
||||||
|
|
||||||
|
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
|
||||||
|
|
||||||
|
### 2026-07-01 — enrolled as a homelab-context client
|
||||||
|
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
|
||||||
|
|
||||||
|
### 2026-07-01 — inventory identity renamed ludo-mini → strong
|
||||||
|
Discovered while starting client enrollment: `bootstrap.sh` looks up
|
||||||
|
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
|
||||||
|
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
|
||||||
|
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
|
||||||
|
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
|
||||||
|
(`hosts/ludo-mini.md` → `hosts/strong.md`), README, ssh-access.md all now
|
||||||
|
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
|
||||||
|
|
||||||
|
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
|
||||||
|
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
|
||||||
|
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
|
||||||
|
Used Proxmox's own disk-management API rather than raw LVM commands:
|
||||||
|
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
|
||||||
|
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
|
||||||
|
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
|
||||||
|
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
|
||||||
|
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
|
||||||
|
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
|
||||||
|
pattern — this pool only physically exists here). Empty — no VM/CT disks
|
||||||
|
placed on it yet.
|
||||||
|
|
||||||
|
### 2026-07-01 — Proxmox install; joined Homelab cluster
|
||||||
|
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
|
||||||
|
hubris's trusted set (root password used once, then discarded). Joined the
|
||||||
|
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
|
||||||
|
this node's side, using key-based SSH pre-authorized in both directions —
|
||||||
|
no interactive password prompt needed for the join itself. Cluster now 2
|
||||||
|
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
|
||||||
|
as `strong` and skip a QDevice for now — both revisitable later.
|
||||||
14
archive/knowledge/index.md
Normal file
14
archive/knowledge/index.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Knowledge
|
||||||
|
|
||||||
|
The durable, authoritative current-state documentation of the homelab: one page per node and per
|
||||||
|
cross-cutting system, synthesized from live state and evidence. Structure and rules are in
|
||||||
|
[the knowledge schema](../.agents/domains/knowledge/schema.md).
|
||||||
|
|
||||||
|
| Section | What it covers |
|
||||||
|
|---------|----------------|
|
||||||
|
| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. |
|
||||||
|
| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. |
|
||||||
|
| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. |
|
||||||
|
| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. |
|
||||||
|
| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. |
|
||||||
|
| [GLOSSARY.md](GLOSSARY.md) | Term definitions. |
|
||||||
@@ -23,7 +23,8 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
|||||||
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
|
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
|
||||||
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
||||||
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
||||||
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
|
- ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28)
|
||||||
|
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
|
||||||
|
|
||||||
**Don't strip these when editing app.ini.**
|
**Don't strip these when editing app.ini.**
|
||||||
|
|
||||||
@@ -37,17 +38,19 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
|||||||
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
|
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
|
||||||
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
||||||
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
||||||
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
|
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
|
||||||
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
||||||
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
||||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
|
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
|
||||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
|
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
|
||||||
|
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
|
||||||
|
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
|
||||||
|
|
||||||
> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
|
> Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push.
|
||||||
> Each owns its own clone on LXC 105. They don't conflict because each
|
> Each owns its own clone on LXC 105. They don't conflict because each
|
||||||
> deploy.sh only touches its own service unit + venv.
|
> deploy.sh only touches its own service unit + venv.
|
||||||
|
|
||||||
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
|
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
|
||||||
|
|
||||||
## When you change a tracked config
|
## When you change a tracked config
|
||||||
|
|
||||||
@@ -57,7 +60,7 @@ Always commit + push. Local-only edits drift. Common ones:
|
|||||||
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
|
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
|
||||||
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
||||||
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
||||||
- `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys)
|
- ~~`/opt/plato/app/` ↔ `dtoro/Plato`~~ (destroyed 2026-06-28)
|
||||||
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
|
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
|
||||||
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
||||||
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
|
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
|
||||||
@@ -72,11 +75,6 @@ Always commit + push. Local-only edits drift. Common ones:
|
|||||||
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
|
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
|
||||||
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
|
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
|
||||||
|
|
||||||
### Plato
|
|
||||||
- Shape B (`/opt/plato-deploy/{webhook.py,deploy.sh}`, port `9799`).
|
|
||||||
- The in-LXC checkout's `origin` is `http://192.168.8.121:3000/dtoro/Plato.git` (internal gitea), and git creds are at `/root/.git-credentials` rather than the `/etc/plato-deploy/git-credentials` pattern — the unit doesn't set `ProtectHome` so root's home is reachable.
|
|
||||||
- `/data` is a host bind (`/mnt/library/documents/plato`), so `docker compose up -d --build` rebuilds the image + restarts the container without touching the SQLite db. The [fresh-DB bootstrap workaround](../containers/126-plato.md#fresh-db-bootstrap-workaround) only matters if you blow `plato.db` away.
|
|
||||||
|
|
||||||
### mule-image / Artifacto
|
### mule-image / Artifacto
|
||||||
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
|
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
|
||||||
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
|
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
|
||||||
@@ -119,15 +117,21 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
|
|||||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||||
- [Backups (disabled)](backups.md)
|
- [Backups (disabled)](backups.md)
|
||||||
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-28 — Plato pipeline decommissioned
|
||||||
|
LXC 126 destroyed, webhook id 8 on `dtoro/Plato` removed. `192.168.8.190` removed from gitea `app.ini` `ALLOWED_HOST_LIST`.
|
||||||
|
|
||||||
|
### 2026-06-24 — terminalito pipeline added
|
||||||
|
Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config).
|
||||||
|
|
||||||
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
|
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
|
||||||
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
|
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
|
||||||
|
|
||||||
### 2026-05-13 — Plato pipeline added
|
### 2026-05-13 — Plato pipeline added
|
||||||
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
|
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
### 2026-04-28 — wiki entry created
|
||||||
Initial documentation. Six active pipelines.
|
Initial documentation. Six active pipelines.
|
||||||
@@ -1,6 +1,30 @@
|
|||||||
# Backups — restic on external drive (DISABLED)
|
# Backups — restic on external drive (DEPRECATED — superseded)
|
||||||
|
|
||||||
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
|
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
|
||||||
|
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
|
||||||
|
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
|
||||||
|
> **DISABLED since 2026-04-22** and is not coming back in its old form.
|
||||||
|
|
||||||
|
## Current backup — rclone → Proton Drive (LXC 132)
|
||||||
|
|
||||||
|
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
|
||||||
|
mounted **read-only**.
|
||||||
|
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
|
||||||
|
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
|
||||||
|
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
|
||||||
|
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
|
||||||
|
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
|
||||||
|
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
|
||||||
|
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
|
||||||
|
copy), but strictly better than the previous "no off-host copy at all."
|
||||||
|
|
||||||
|
See [132-rclone](../containers/132-rclone.md) for the full design.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
||||||
|
|
||||||
|
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
@@ -12,7 +36,7 @@ Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabl
|
|||||||
|
|
||||||
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
||||||
|
|
||||||
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
|
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
||||||
|
|
||||||
@@ -81,13 +105,13 @@ Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/R
|
|||||||
|
|
||||||
## Known SPOF
|
## Known SPOF
|
||||||
|
|
||||||
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
|
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
|
||||||
|
|
||||||
## Drive history
|
## Drive history
|
||||||
|
|
||||||
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
||||||
|
|
||||||
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
|
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||||||
|
|
||||||
## Thermal monitoring
|
## Thermal monitoring
|
||||||
|
|
||||||
@@ -95,18 +119,21 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Hubris host](../hosts/hubris.md)
|
- [Hubris host](../hosts/hubris.md)
|
||||||
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||||
- [Monitoring](monitoring.md)
|
- [Monitoring](monitoring.md)
|
||||||
- [Auto-deploy](auto-deploy.md)
|
- [Auto-deploy](auto-deploy.md)
|
||||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
|
||||||
|
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
### 2026-04-28 — wiki entry created
|
||||||
Initial documentation. Status remains DISABLED.
|
Initial documentation. Status remains DISABLED.
|
||||||
|
|
||||||
### 2026-04-22 — DISABLED
|
### 2026-04-22 — DISABLED
|
||||||
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
||||||
|
|
||||||
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
||||||
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
||||||
@@ -7,7 +7,7 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
|
|||||||
## Components
|
## Components
|
||||||
|
|
||||||
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
||||||
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
|
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04).
|
||||||
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
||||||
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
|
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
|
||||||
|
|
||||||
@@ -32,7 +32,6 @@ address=/blog.hubris.network/192.168.8.175
|
|||||||
address=/photos.hubris.network/192.168.8.175
|
address=/photos.hubris.network/192.168.8.175
|
||||||
address=/photos-new.hubris.network/192.168.8.175
|
address=/photos-new.hubris.network/192.168.8.175
|
||||||
address=/artifacto.hubris.network/192.168.8.175
|
address=/artifacto.hubris.network/192.168.8.175
|
||||||
address=/plato.hubris.network/192.168.8.175
|
|
||||||
address=/zimaos.hubris.network/192.168.8.175
|
address=/zimaos.hubris.network/192.168.8.175
|
||||||
address=/nfs-export.hubris.network/192.168.8.200
|
address=/nfs-export.hubris.network/192.168.8.200
|
||||||
```
|
```
|
||||||
@@ -54,7 +53,7 @@ Creating a new Caddyfile site block is necessary but **not sufficient**. Without
|
|||||||
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
||||||
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
||||||
|
|
||||||
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
||||||
|
|
||||||
## Public path — what does and doesn't follow the LAN map
|
## Public path — what does and doesn't follow the LAN map
|
||||||
|
|
||||||
@@ -76,6 +75,9 @@ Either:
|
|||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-28 — `plato.hubris.network` removed
|
||||||
|
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
|
||||||
|
|
||||||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
|
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
|
||||||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
|
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
|
||||||
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
||||||
@@ -92,7 +94,7 @@ The "delete NetBird managed zone → forward everything to Technitium" plan was
|
|||||||
|
|
||||||
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
|
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
|
||||||
|
|
||||||
> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
|
> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
|
||||||
|
|
||||||
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
|
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
|
||||||
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
|
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
|
||||||
@@ -105,10 +107,10 @@ Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backe
|
|||||||
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
|
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
|
||||||
|
|
||||||
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
|
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
|
||||||
Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
|
||||||
|
|
||||||
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
|
||||||
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
|
||||||
|
|
||||||
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
|
||||||
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
|
||||||
@@ -117,7 +119,7 @@ NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.
|
|||||||
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
|
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
|
||||||
|
|
||||||
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
|
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
|
||||||
New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
|
New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
|
||||||
|
|
||||||
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
|
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
|
||||||
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
|
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
|
||||||
@@ -5,7 +5,7 @@ Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
|||||||
self-locating and able to read the same source of truth.
|
self-locating and able to read the same source of truth.
|
||||||
|
|
||||||
Operational walkthrough for enrolling a new client lives in
|
Operational walkthrough for enrolling a new client lives in
|
||||||
[operations/agent-enrollment.md](../operations/agent-enrollment.md); this
|
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
|
||||||
page is the architecture reference.
|
page is the architecture reference.
|
||||||
|
|
||||||
## What's where
|
## What's where
|
||||||
@@ -121,7 +121,7 @@ The MCP server and secrets-issuance each have their own clone
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Operations: agent enrollment](../operations/agent-enrollment.md) — the
|
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
|
||||||
step-by-step for adding a new client
|
step-by-step for adding a new client
|
||||||
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
|
||||||
pipelines (and the rest of the lab's webhook pipelines)
|
pipelines (and the rest of the lab's webhook pipelines)
|
||||||
@@ -133,7 +133,7 @@ The MCP server and secrets-issuance each have their own clone
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-05-20 — system live across hubris, apps, republic-laptop
|
### 2026-05-20 — system live across hubris, apps, republic-laptop
|
||||||
Phase 1 of the [cross-client context plan](../README.md) merged. Three
|
Phase 1 of the [cross-client context plan](../../../README.md) merged. Three
|
||||||
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
|
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
|
||||||
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
|
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
|
||||||
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
|
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
|
||||||
66
archive/knowledge/infrastructure/index.md
Normal file
66
archive/knowledge/infrastructure/index.md
Normal file
@@ -0,0 +1,66 @@
|
|||||||
|
# Infrastructure — cross-cutting systems
|
||||||
|
|
||||||
|
The homelab's shared infrastructure: systems that span multiple nodes and
|
||||||
|
are documented in their own pages. Each system below links to its full doc.
|
||||||
|
|
||||||
|
## Network
|
||||||
|
|
||||||
|
- **[Network](network.md)** — physical topology, subnets, routing, DHCP.
|
||||||
|
Homelab `192.168.8.0/24` isolated from household `192.168.178.0/24`.
|
||||||
|
Proxmox hubris acts as subnet router.
|
||||||
|
- **[DNS — split-horizon](dns.md)** — Technitium DNS on LXC 107,
|
||||||
|
`192.168.8.2:53`. `*.hubris.network` resolves to LAN IPs on the homelab
|
||||||
|
network and to mesh addresses off-LAN.
|
||||||
|
|
||||||
|
## Connectivity / mesh
|
||||||
|
|
||||||
|
- **[Mesh — Tailscale → Netbird migration](mesh.md)** — overlay networking.
|
||||||
|
Netbird is the preferred path; Tailscale is legacy.
|
||||||
|
- **[SSH access](ssh-access.md)** — dual-server SSH (OpenSSH + Netbird SSH)
|
||||||
|
on hubris, key distribution.
|
||||||
|
|
||||||
|
## Public ingress
|
||||||
|
|
||||||
|
- **[Public ingress — VPS traefik + cert mirror](ingress.md)** — how home
|
||||||
|
services reach the open internet. Two-stage: VPS traefik (IONOS) terminates
|
||||||
|
TLS, proxies over Netbird to home Caddy.
|
||||||
|
- **[Caddy reverse proxy](../containers/121-caddy.md)** — LAN endpoint.
|
||||||
|
Terminates TLS for every `*.hubris.network` hostname, forwards to backends.
|
||||||
|
|
||||||
|
## Storage
|
||||||
|
|
||||||
|
- **[Media permissions — GID 10000 standard](media-permissions.md)** — shared
|
||||||
|
group permission model across all LXCs that read/write the media library.
|
||||||
|
- **[Backups — rclone → Proton Drive](backups.md)** — off-host backup strategy.
|
||||||
|
LXC 132 handles rclone to Proton Drive; restic-on-USB deprecated.
|
||||||
|
|
||||||
|
## Identity & access
|
||||||
|
|
||||||
|
- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider.
|
||||||
|
Core server runs on the VPS; LAN forward-auth outpost at LXC 106.
|
||||||
|
OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent,
|
||||||
|
Yuvomi, and more.
|
||||||
|
|
||||||
|
## Management & automation
|
||||||
|
|
||||||
|
- **[Homelab context distribution](homelab-context.md)** — `/opt/homelab-context`
|
||||||
|
clone, MCP server, secrets issuance, cross-client sync.
|
||||||
|
- **[Auto-deploy — gitea-webhook pipelines](auto-deploy.md)** — push-to-deploy
|
||||||
|
for Caddy config, mule-image, and other tracked repos.
|
||||||
|
- **[Monitoring](monitoring.md)** — health checks, watchdogs, alerting
|
||||||
|
(migrated from claudio-bot to Hermes cron).
|
||||||
|
- **[VPS hardening](vps-hardening.md)** — IONOS netbird VPS: fail2ban,
|
||||||
|
nftables, OIDC SSH, security posture.
|
||||||
|
|
||||||
|
## Topology
|
||||||
|
|
||||||
|
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
|
||||||
|
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
|
||||||
|
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [README](../../../README.md) — entry point
|
||||||
|
- [Containers index](../containers/index.md)
|
||||||
|
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||||||
|
- [OIKOS operating model](../../../.agents/OIKOS.md)
|
||||||
@@ -42,12 +42,14 @@ LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8
|
|||||||
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
|
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
|
||||||
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
|
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
|
||||||
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
|
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
|
||||||
|
| `trmnl.hubris.network` | whole host | `192.168.8.211:9851` ([trmnl 128](../containers/128-trmnl.md)) | `trmnl-ratelimit` (20 rps / 40 burst) | `trmnl.fullchain.crt` / `trmnl.privkey.key` |
|
||||||
|
| `house.hubris.network` | whole host | `192.168.8.212:3000` ([house 129](../containers/129-house.md)) | `house-ratelimit` (30 rps / 60 burst) | `house.fullchain.crt` / `house.privkey.key` |
|
||||||
|
|
||||||
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
|
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
|
||||||
|
|
||||||
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
|
||||||
|
|
||||||
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
|
||||||
|
|
||||||
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
|
||||||
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
|
||||||
@@ -85,8 +87,11 @@ No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `
|
|||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-06-24 — `trmnl.hubris.network` exposed
|
||||||
|
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
|
||||||
|
|
||||||
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
|
||||||
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
### 2026-04-28 — wiki entry created
|
||||||
Initial documentation.
|
Initial documentation.
|
||||||
@@ -37,7 +37,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
|
|||||||
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
|
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
|
||||||
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
|
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
|
||||||
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
|
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
|
||||||
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore
|
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / grimmory
|
||||||
- `downloads` — \*arr stack output
|
- `downloads` — \*arr stack output
|
||||||
- `images` — photoprism / immich / mulita
|
- `images` — photoprism / immich / mulita
|
||||||
- `roms` — emu frontends
|
- `roms` — emu frontends
|
||||||
@@ -61,6 +61,8 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with
|
|||||||
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
|
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
|
||||||
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
|
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
|
||||||
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
|
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
|
||||||
|
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
|
||||||
|
| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed |
|
||||||
|
|
||||||
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
|
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
|
||||||
|
|
||||||
@@ -68,7 +70,7 @@ Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*` (109 destroyed 20
|
|||||||
|
|
||||||
## Gotchas
|
## Gotchas
|
||||||
|
|
||||||
- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container.
|
- **[apps (105)](../containers/105-apps.md) and [grimmory (130)](../containers/130-grimmory.md) are Docker hosts.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000`, `user: "<uid>:10000"`, or `GROUP_ID=10000` (linuxserver images) in compose. Grimmory, audiobookshelf-in-docker, etc. need this per-container.
|
||||||
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
|
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
|
||||||
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
|
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
|
||||||
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
|
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
|
||||||
@@ -109,7 +109,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [DNS split-horizon](dns.md)
|
- [DNS split-horizon](dns.md)
|
||||||
- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
|
- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides
|
||||||
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
|
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
|
||||||
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
|
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
|
||||||
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
|
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
|
||||||
@@ -117,7 +117,7 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
|
||||||
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
|
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
|
||||||
|
|
||||||
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
|
||||||
|
|
||||||
@@ -139,11 +139,11 @@ The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Ca
|
|||||||
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
|
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
|
||||||
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
|
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
|
||||||
|
|
||||||
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
|
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
|
||||||
|
|
||||||
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
|
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
|
||||||
|
|
||||||
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
|
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
|
||||||
|
|
||||||
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):
|
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):
|
||||||
|
|
||||||
@@ -64,7 +64,7 @@ No NAT on Proxmox — traffic flows without double-NAT.
|
|||||||
|
|
||||||
## Remote access
|
## Remote access
|
||||||
|
|
||||||
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../vps/).
|
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/).
|
||||||
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
|
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
@@ -79,10 +79,10 @@ No NAT on Proxmox — traffic flows without double-NAT.
|
|||||||
|
|
||||||
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
|
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
|
||||||
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
|
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
|
||||||
Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
|
Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
|
||||||
|
|
||||||
### 2026-06-02 — Executed migration; Proxmox as subnet router
|
### 2026-06-02 — Executed migration; Proxmox as subnet router
|
||||||
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
|
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
|
||||||
|
|
||||||
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
|
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
|
||||||
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
|
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
|
||||||
@@ -97,19 +97,23 @@ When onboarding a new machine:
|
|||||||
|
|
||||||
## Hosts
|
## Hosts
|
||||||
|
|
||||||
### Hubris (PVE host)
|
### Hubris + strong (PVE cluster: `Homelab`)
|
||||||
|
|
||||||
| Detail | Value |
|
Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
|
||||||
|--------|-------|
|
cluster-synced, so a key added on either node is authorized on both.
|
||||||
| LAN IP | `192.168.8.77` |
|
|
||||||
| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
|
|
||||||
| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
|
|
||||||
| SSH user | `root` |
|
|
||||||
| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
|
|
||||||
|
|
||||||
Authorized root keys currently deployed:
|
| Detail | hubris | strong |
|
||||||
|
|--------|--------|-----------|
|
||||||
|
| LAN IP | `192.168.8.77` | `192.168.178.181` |
|
||||||
|
| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
|
||||||
|
| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
|
||||||
|
| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
|
||||||
|
| SSH user | `root` | `root` |
|
||||||
|
|
||||||
|
Authorized root keys currently deployed (cluster-wide):
|
||||||
- `root@hubris` (self, RSA)
|
- `root@hubris` (self, RSA)
|
||||||
- `d.toro.v@pm.me` (ed25519) — mac-mini
|
- `d.toro.v@pm.me` (ed25519) — mac-mini
|
||||||
|
- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join
|
||||||
|
|
||||||
### LXCs
|
### LXCs
|
||||||
|
|
||||||
@@ -131,7 +135,6 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|
|||||||
| 120 | mule-images | `192.168.8.136` | photo-management |
|
| 120 | mule-images | `192.168.8.136` | photo-management |
|
||||||
| 121 | caddy | `192.168.8.175` | reverse-proxy |
|
| 121 | caddy | `192.168.8.175` | reverse-proxy |
|
||||||
| 122 | arriman | `192.168.8.132` | arr-stack |
|
| 122 | arriman | `192.168.8.132` | arr-stack |
|
||||||
| 126 | plato | `192.168.8.190` | app |
|
|
||||||
|
|
||||||
### Workstations
|
### Workstations
|
||||||
|
|
||||||
@@ -139,7 +142,8 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|
|||||||
|------|----|--------|--------------|----------|
|
|------|----|--------|--------------|----------|
|
||||||
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
|
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
|
||||||
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
|
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
|
||||||
| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
|
|
||||||
|
strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.
|
||||||
|
|
||||||
### VPS (external)
|
### VPS (external)
|
||||||
|
|
||||||
@@ -174,11 +178,14 @@ done
|
|||||||
|
|
||||||
- [Mesh migration](mesh.md)
|
- [Mesh migration](mesh.md)
|
||||||
- [VPS hardening](vps-hardening.md)
|
- [VPS hardening](vps-hardening.md)
|
||||||
- [Agent enrollment](../operations/agent-enrollment.md)
|
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
|
||||||
- [Homelab CLI](../bin/homelab)
|
- [Homelab CLI](../../../bin/homelab)
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
|
||||||
|
strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
|
||||||
|
|
||||||
### 2026-06-02 — universal SSH reachability
|
### 2026-06-02 — universal SSH reachability
|
||||||
|
|
||||||
Replaced ad-hoc per-workstation SSH configs with inventory-generated
|
Replaced ad-hoc per-workstation SSH configs with inventory-generated
|
||||||
96
archive/knowledge/infrastructure/topology.md
Normal file
96
archive/knowledge/infrastructure/topology.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
<!-- Generated by oikos/gen-topology.py from inventory.yaml. -->
|
||||||
|
<!-- Do NOT edit by hand - your changes will be overwritten. -->
|
||||||
|
|
||||||
|
# Topology (generated)
|
||||||
|
|
||||||
|
Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services.
|
||||||
|
Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md).
|
||||||
|
|
||||||
|
## Compute & ingress
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph hubris_sub["hubris (Proxmox)"]
|
||||||
|
trmnl["trmnl<br/>LXC 128<br/>trmnl-middleware<br/>192.168.8.211"]
|
||||||
|
nfs_export["nfs-export<br/>LXC 102<br/>storage-export<br/>192.168.8.200"]
|
||||||
|
paperless["paperless<br/>LXC 103<br/>document-archive<br/>192.168.8.130"]
|
||||||
|
gitea["gitea<br/>LXC 104<br/>git-server<br/>192.168.8.121"]
|
||||||
|
apps["apps<br/>LXC 105<br/>docker-apps<br/>192.168.8.205"]
|
||||||
|
auth_outpost["auth-outpost<br/>LXC 106<br/>authentik-gateway<br/>192.168.8.6"]
|
||||||
|
dns["dns<br/>LXC 107<br/>dns-server<br/>192.168.8.2"]
|
||||||
|
nextcloud["nextcloud<br/>LXC 114<br/>file-sync<br/>192.168.8.224"]
|
||||||
|
sophia["sophia<br/>LXC 119<br/>workshop<br/>192.168.8.109"]
|
||||||
|
mule_images["mule-images<br/>LXC 120<br/>photo-management<br/>192.168.8.136"]
|
||||||
|
caddy["caddy<br/>LXC 121<br/>reverse-proxy<br/>192.168.8.175"]
|
||||||
|
teddycloud["teddycloud<br/>LXC 131<br/>teddycloud<br/>192.168.8.150"]
|
||||||
|
zimaos["zimaos<br/>VM 100<br/>nas-frontend-eval<br/>192.168.8.195"]
|
||||||
|
haos["haos<br/>VM 108<br/>home-automation<br/>192.168.8.101"]
|
||||||
|
end
|
||||||
|
subgraph strong_sub["strong (Proxmox)"]
|
||||||
|
house["house<br/>LXC 129<br/>family-planner<br/>192.168.8.244"]
|
||||||
|
jellyfin["jellyfin<br/>LXC 101<br/>media-server<br/>192.168.8.246"]
|
||||||
|
elementsynapse["elementsynapse<br/>LXC 118<br/>matrix-server<br/>192.168.8.242"]
|
||||||
|
arriman["arriman<br/>LXC 122<br/>arr-stack<br/>192.168.8.245"]
|
||||||
|
grimmory["grimmory<br/>LXC 130<br/>book-library<br/>192.168.8.247"]
|
||||||
|
seanime["seanime<br/>LXC 133<br/>anime-media-server<br/>192.168.8.248"]
|
||||||
|
romm["romm<br/>LXC 134<br/>rom-manager<br/>192.168.8.249"]
|
||||||
|
end
|
||||||
|
rclone["rclone<br/>lxc<br/>backup"]
|
||||||
|
republic_laptop([republic-laptop<br/>workstation<br/>primary-dev])
|
||||||
|
mac_mini([mac-mini<br/>workstation<br/>dev<br/>192.168.178.182])
|
||||||
|
netbird_vps[[netbird-vps<br/>external<br/>netbird-mgmt]]
|
||||||
|
url_artifacto(["artifacto.hubris.network"]) -->|routes-to| apps
|
||||||
|
url_authentik(["auth.hubris.network"]) -->|routes-to| netbird_vps
|
||||||
|
url_gitea(["git.hubris.network"]) -->|routes-to| gitea
|
||||||
|
url_homelab_mcp(["mcp.hubris.network"]) -->|routes-to| apps
|
||||||
|
url_jellyfin(["media.hubris.network"]) -->|routes-to| jellyfin
|
||||||
|
url_matrix(["matrix.hubris.network"]) -->|routes-to| elementsynapse
|
||||||
|
url_nextcloud(["cloud.hubris.network"]) -->|routes-to| nextcloud
|
||||||
|
url_paperless(["paperless.hubris.network"]) -->|routes-to| paperless
|
||||||
|
url_photos(["photos.hubris.network"]) -->|routes-to| mule_images
|
||||||
|
url_proxmox_ui(["proxmox.hubris.network"]) -->|routes-to| hubris_sub
|
||||||
|
url_secrets_issuance(["secrets.hubris.network"]) -->|routes-to| apps
|
||||||
|
url_teddycloud(["teddy.hubris.network"]) -->|routes-to| teddycloud
|
||||||
|
url_trmnl(["trmnl.hubris.network"]) -->|routes-to| trmnl
|
||||||
|
url_zimaos(["zimaos.hubris.network"]) -->|routes-to| zimaos
|
||||||
|
```
|
||||||
|
|
||||||
|
## Storage (mounts)
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
mnt_library[("/mnt/library")]
|
||||||
|
mnt_media_local[("/mnt/media_local")]
|
||||||
|
mnt_media_local_anime[("/mnt/media_local/anime")]
|
||||||
|
apps["apps"] -->|mounts| mnt_library
|
||||||
|
arriman["arriman"] -->|mounts| mnt_media_local
|
||||||
|
gitea["gitea"] -->|mounts| mnt_library
|
||||||
|
grimmory["grimmory"] -->|mounts| mnt_media_local
|
||||||
|
hubris["hubris"] -->|mounts| mnt_library
|
||||||
|
jellyfin["jellyfin"] -->|mounts| mnt_media_local
|
||||||
|
mule_images["mule-images"] -->|mounts| mnt_library
|
||||||
|
nextcloud["nextcloud"] -->|mounts| mnt_library
|
||||||
|
paperless["paperless"] -->|mounts| mnt_library
|
||||||
|
romm["romm"] -->|mounts| mnt_media_local
|
||||||
|
seanime["seanime"] -->|mounts| mnt_media_local_anime
|
||||||
|
sophia["sophia"] -->|mounts| mnt_library
|
||||||
|
teddycloud["teddycloud"] -->|mounts| mnt_library
|
||||||
|
```
|
||||||
|
|
||||||
|
## Archaeology (destroyed nodes)
|
||||||
|
|
||||||
|
| Node | ID | Destroyed | Reason |
|
||||||
|
|---|---|---|---|
|
||||||
|
| plato | 126 | 2026-06-28 | notes workspace decommissioned; data retained at /mnt/library/documents/plato |
|
||||||
|
| claudio-bot | 123 | 2026-06-04 | replaced by Hermes Agent on mac-mini; monitoring moved to homelab-health-watchdog cron |
|
||||||
|
| mule-photos-new | 127 | 2026-05-22 | PhotoPrism test stack promoted to LXC 120 (Mulimage 2.0 merge) |
|
||||||
|
| heaper | 116 | 2026-05-14 | decommissioned; data retained at /mnt/library/heaper |
|
||||||
|
| syncthing | 109 | 2026-05-14 | decommissioned; library subtree was empty |
|
||||||
|
| seafile | 125 | 2026-05-13 | Seafile Pro evaluation rejected; files.hubris.network removed from caddy + dns |
|
||||||
|
| arr-yunohost | 100 | 2026-04-28 | migrated to docker stack on arriman (LXC 122) |
|
||||||
|
| flaresolverr | 106 | 2026-04-28 | folded into the arriman docker compose |
|
||||||
|
| marimo | 107 | 2026-04-28 | decommissioned |
|
||||||
|
| photoprism | 110 | 2026-04-28 | replaced by mule-images (LXC 120) |
|
||||||
|
| karakeep | 111 | 2026-04-28 | decommissioned |
|
||||||
|
| immich | 112 | 2026-04-28 | replaced by mule-images (LXC 120) |
|
||||||
|
| reticulum | 115 | 2026-04-28 | decommissioned |
|
||||||
@@ -2,12 +2,12 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
|
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
|
||||||
|
|
||||||
## Timeline
|
## Timeline
|
||||||
|
|
||||||
### 2026-04-19 — drive attached
|
### 2026-04-19 — drive attached
|
||||||
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
|
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
|
||||||
|
|
||||||
### 2026-04-19 → 2026-04-21 — first crashes
|
### 2026-04-19 → 2026-04-21 — first crashes
|
||||||
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
|
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
|
||||||
@@ -24,13 +24,13 @@ Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly
|
|||||||
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
|
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
|
||||||
|
|
||||||
### 2026-04-22 — recurrence after 30 h 37 m
|
### 2026-04-22 — recurrence after 30 h 37 m
|
||||||
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
|
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
|
||||||
|
|
||||||
### 2026-04-22 — `cpu-epp.service` design bug fixed
|
### 2026-04-22 — `cpu-epp.service` design bug fixed
|
||||||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
|
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
|
||||||
|
|
||||||
### 2026-04-22 — drive removed (A/B test)
|
### 2026-04-22 — drive removed (A/B test)
|
||||||
User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
|
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
|
||||||
|
|
||||||
### 2026-04-23 — SSD cooling + thermal pads installed
|
### 2026-04-23 — SSD cooling + thermal pads installed
|
||||||
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
|
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
|
||||||
@@ -78,9 +78,9 @@ Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the Nu
|
|||||||
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
|
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
|
||||||
|
|
||||||
## Affected nodes
|
## Affected nodes
|
||||||
- [Hubris host](../hosts/hubris.md)
|
- [Hubris host](../../../wiki/hosts/hubris.md)
|
||||||
- [Backups (disabled)](../infrastructure/backups.md)
|
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
|
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
|
||||||
@@ -2,9 +2,9 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
|
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
|
||||||
|
|
||||||
Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
|
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
|
||||||
|
|
||||||
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
|
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
|
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
|
||||||
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
|
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
|
||||||
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
|
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
|
||||||
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
|
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
|
||||||
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
|
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
|
||||||
|
|
||||||
@@ -51,7 +51,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
| | Before | After |
|
| | Before | After |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
|
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
|
||||||
| Version | `2026.2.2` | `2026.5.2` |
|
| Version | `2026.2.2` | `2026.5.2` |
|
||||||
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
|
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
|
||||||
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
|
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
|
||||||
@@ -76,7 +76,7 @@ The real reason the browser kept hitting the *old* Authentik even after the VPS
|
|||||||
|
|
||||||
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
|
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
|
||||||
|
|
||||||
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
|
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
|
||||||
|
|
||||||
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
|
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
|
||||||
|
|
||||||
@@ -99,11 +99,11 @@ The policy was orphaned (no matched type data or had incompatible evaluation). R
|
|||||||
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
|
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
|
||||||
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
|
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
|
||||||
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
|
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
|
||||||
- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
|
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
|
||||||
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
|
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [124 — authentik](../containers/124-authentik.md)
|
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
|
||||||
- [DNS split-horizon](../infrastructure/dns.md)
|
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
|
||||||
- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
|
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
|
||||||
- [Mesh migration](../infrastructure/mesh.md)
|
- [Mesh migration](../../../wiki/infrastructure/mesh.md)
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
|
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
|
||||||
|
|
||||||
## Timeline
|
## Timeline
|
||||||
|
|
||||||
@@ -90,9 +90,9 @@ print("session_duration:", stage.session_duration) # → "days=30"
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
|
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
|
||||||
- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
|
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
|
||||||
- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
|
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
|
||||||
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
|
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
@@ -54,8 +54,8 @@ This is the same class of drift as the June 5th incidents (paperless, HAOS, apps
|
|||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
|
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
|
||||||
- [Caddy (121)](../containers/121-caddy.md)
|
- [Caddy (121)](../../wiki/containers/121-caddy.md)
|
||||||
- [elementsynapse (118)](../containers/118-elementsynapse.md)
|
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
|
||||||
- [dns-sync script](../scripts/dns-sync.py)
|
- [dns-sync script](../../../scripts/dns-sync.py)
|
||||||
- [check-caddy-backends script](../scripts/check-caddy-backends.sh)
|
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)
|
||||||
29
archive/knowledge/investigations/index.md
Normal file
29
archive/knowledge/investigations/index.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# Investigations
|
||||||
|
|
||||||
|
Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
|
||||||
|
|
||||||
|
## Active / recent
|
||||||
|
|
||||||
|
| Date | Title | Status |
|
||||||
|
| ------------ | ------------------------------------------------------------------ | ------------- |
|
||||||
|
| 2026-06-06 | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
|
||||||
|
| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
|
||||||
|
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
|
||||||
|
| 2026-06-01 | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
|
||||||
|
|
||||||
|
## Resolved (archived)
|
||||||
|
|
||||||
|
See [`archive/`](archive/):
|
||||||
|
|
||||||
|
| Date | Title |
|
||||||
|
| ------------ | ------------------------------------------------------------------ |
|
||||||
|
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
|
||||||
|
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
|
||||||
|
- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
|
||||||
|
- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
|
||||||
|
- Link back from every node's changelog that's affected.
|
||||||
|
- Move to `archive/` when the incident is fully resolved and no longer actively referenced.
|
||||||
10
archive/knowledge/log.md
Normal file
10
archive/knowledge/log.md
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
# Knowledge — operations log
|
||||||
|
|
||||||
|
Append-only record of documentation-maintenance operations on the knowledge wiki (restructures,
|
||||||
|
source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in
|
||||||
|
each page's `## Changelog` and the Oikos change ledger, not here.
|
||||||
|
|
||||||
|
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
|
||||||
|
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
|
||||||
|
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
|
||||||
|
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.
|
||||||
351
archive/knowledge/references/cert-sync-and-traefik-config.md
Normal file
351
archive/knowledge/references/cert-sync-and-traefik-config.md
Normal file
@@ -0,0 +1,351 @@
|
|||||||
|
# Current cert sync script + traefik dynamic config
|
||||||
|
|
||||||
|
Snapshot of the two artifacts that control public service exposure as of
|
||||||
|
2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP
|
||||||
|
(192.168.8.206) to new strong IP (192.168.8.246).
|
||||||
|
|
||||||
|
## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
|
||||||
|
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
|
||||||
|
# and hot-reloads.
|
||||||
|
#
|
||||||
|
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
|
||||||
|
# challenges before traefik's allowACMEByPass can respond, so traefik can't
|
||||||
|
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
|
||||||
|
# we just mirror what it already has.
|
||||||
|
#
|
||||||
|
# Runs daily via hubris-public-cert-sync.timer.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
CADDY_LXC=121
|
||||||
|
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
|
||||||
|
VPS_HOST=root@100.122.165.149
|
||||||
|
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data
|
||||||
|
|
||||||
|
# Map: source hostname -> "crt_filename key_filename" on the VPS.
|
||||||
|
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
|
||||||
|
declare -A HOSTS=(
|
||||||
|
[artifacto.hubris.network]="fullchain.crt privkey.key"
|
||||||
|
[blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
|
||||||
|
[trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
|
||||||
|
[sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
|
||||||
|
[media.hubris.network]="media.fullchain.crt media.privkey.key"
|
||||||
|
[paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
|
||||||
|
)
|
||||||
|
|
||||||
|
TMP=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$TMP"' EXIT
|
||||||
|
|
||||||
|
for host in "${!HOSTS[@]}"; do
|
||||||
|
read -r crt_name key_name <<< "${HOSTS[$host]}"
|
||||||
|
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
|
||||||
|
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"
|
||||||
|
|
||||||
|
if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
|
||||||
|
echo "hubris-public-cert-sync: $host unchanged"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
|
||||||
|
echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding a new host
|
||||||
|
|
||||||
|
1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`)
|
||||||
|
2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"`
|
||||||
|
3. `systemctl start hubris-public-cert-sync.service` to sync immediately
|
||||||
|
4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"`
|
||||||
|
5. Add matching `tls.certificates` entry in traefik dynamic config
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## traefik-dynamic.yaml (VPS, `/opt/`)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
tcp:
|
||||||
|
serversTransports:
|
||||||
|
pp-v2:
|
||||||
|
proxyProtocol:
|
||||||
|
version: 2
|
||||||
|
|
||||||
|
tls:
|
||||||
|
certificates:
|
||||||
|
- certFile: /letsencrypt/fullchain.crt
|
||||||
|
keyFile: /letsencrypt/privkey.key
|
||||||
|
- certFile: /letsencrypt/blog.fullchain.crt
|
||||||
|
keyFile: /letsencrypt/blog.privkey.key
|
||||||
|
- certFile: /letsencrypt/trmnl.fullchain.crt
|
||||||
|
keyFile: /letsencrypt/trmnl.privkey.key
|
||||||
|
- certFile: /letsencrypt/sso.fullchain.crt
|
||||||
|
keyFile: /letsencrypt/sso.privkey.key
|
||||||
|
- certFile: /letsencrypt/media.fullchain.crt
|
||||||
|
keyFile: /letsencrypt/media.privkey.key
|
||||||
|
- certFile: /letsencrypt/paperless.fullchain.crt
|
||||||
|
keyFile: /letsencrypt/paperless.privkey.key
|
||||||
|
|
||||||
|
http:
|
||||||
|
routers:
|
||||||
|
artifacto-public:
|
||||||
|
rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- artifacto-strip-sso
|
||||||
|
- artifacto-ratelimit
|
||||||
|
service: artifacto-public
|
||||||
|
|
||||||
|
blog-public:
|
||||||
|
rule: 'Host(`blog.hubris.network`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- blog-ratelimit
|
||||||
|
service: blog-public
|
||||||
|
|
||||||
|
trmnl-public:
|
||||||
|
rule: 'Host(`trmnl.hubris.network`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- trmnl-ratelimit
|
||||||
|
service: trmnl-public
|
||||||
|
|
||||||
|
matrix-public:
|
||||||
|
rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- matrix-ratelimit
|
||||||
|
service: matrix-public
|
||||||
|
|
||||||
|
matrix-wellknown:
|
||||||
|
rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 20
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
service: matrix-wellknown-svc
|
||||||
|
|
||||||
|
house-public:
|
||||||
|
rule: Host(`house.hubris.network`)
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- house-ratelimit
|
||||||
|
service: house-public
|
||||||
|
|
||||||
|
sso-public:
|
||||||
|
rule: 'Host(`sso.hubris.network`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- sso-ratelimit
|
||||||
|
service: sso-public
|
||||||
|
|
||||||
|
media-public:
|
||||||
|
rule: 'Host(`media.hubris.network`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- media-ratelimit
|
||||||
|
service: media-public
|
||||||
|
|
||||||
|
paperless-api-public:
|
||||||
|
rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 20
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- paperless-ratelimit
|
||||||
|
service: paperless-public
|
||||||
|
|
||||||
|
paperless-public:
|
||||||
|
rule: 'Host(`paperless.hubris.network`)'
|
||||||
|
entryPoints:
|
||||||
|
- websecure
|
||||||
|
priority: 10
|
||||||
|
tls:
|
||||||
|
certResolver: letsencrypt
|
||||||
|
middlewares:
|
||||||
|
- authentik-forwardauth
|
||||||
|
- paperless-ratelimit
|
||||||
|
service: paperless-public
|
||||||
|
|
||||||
|
middlewares:
|
||||||
|
artifacto-strip-sso:
|
||||||
|
headers:
|
||||||
|
customRequestHeaders:
|
||||||
|
X-Artifacto-Gateway: ""
|
||||||
|
X-Authentik-Username: ""
|
||||||
|
X-Authentik-Groups: ""
|
||||||
|
X-Authentik-Email: ""
|
||||||
|
X-Authentik-Name: ""
|
||||||
|
X-Authentik-Uid: ""
|
||||||
|
X-Authentik-Jwt: ""
|
||||||
|
X-Authentik-Meta-Jwks: ""
|
||||||
|
X-Authentik-Meta-Outpost: ""
|
||||||
|
X-Authentik-Meta-Provider: ""
|
||||||
|
X-Authentik-Meta-App: ""
|
||||||
|
X-Authentik-Meta-Version: ""
|
||||||
|
|
||||||
|
admin-allowlist:
|
||||||
|
ipAllowList:
|
||||||
|
sourceRange:
|
||||||
|
- "5.61.168.0/24"
|
||||||
|
|
||||||
|
artifacto-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 50
|
||||||
|
period: 1s
|
||||||
|
burst: 100
|
||||||
|
blog-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 100
|
||||||
|
period: 1s
|
||||||
|
burst: 200
|
||||||
|
trmnl-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 20
|
||||||
|
period: 1s
|
||||||
|
burst: 40
|
||||||
|
matrix-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 30
|
||||||
|
period: 1s
|
||||||
|
burst: 60
|
||||||
|
house-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 30
|
||||||
|
period: 1s
|
||||||
|
burst: 60
|
||||||
|
sso-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 30
|
||||||
|
period: 1s
|
||||||
|
burst: 60
|
||||||
|
media-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 30
|
||||||
|
period: 1s
|
||||||
|
burst: 60
|
||||||
|
paperless-ratelimit:
|
||||||
|
rateLimit:
|
||||||
|
average: 20
|
||||||
|
period: 1s
|
||||||
|
burst: 40
|
||||||
|
authentik-forwardauth:
|
||||||
|
forwardAuth:
|
||||||
|
address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
|
||||||
|
trustForwardHeader: true
|
||||||
|
authResponseHeaders:
|
||||||
|
- X-authentik-username
|
||||||
|
- X-authentik-groups
|
||||||
|
- X-authentik-email
|
||||||
|
- X-authentik-name
|
||||||
|
- X-authentik-uid
|
||||||
|
- X-authentik-jwt
|
||||||
|
- X-authentik-meta-jwks
|
||||||
|
- X-authentik-meta-outpost
|
||||||
|
- X-authentik-meta-provider
|
||||||
|
- X-authentik-meta-app
|
||||||
|
- X-authentik-meta-version
|
||||||
|
|
||||||
|
services:
|
||||||
|
artifacto-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.205:3100'
|
||||||
|
blog-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.205:8080'
|
||||||
|
trmnl-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.211:9851'
|
||||||
|
matrix-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.242:8008'
|
||||||
|
matrix-wellknown-svc:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://matrix-wellknown:80'
|
||||||
|
house-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.244:3000'
|
||||||
|
sso-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.6:9000'
|
||||||
|
media-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.246:8096'
|
||||||
|
paperless-public:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: 'http://192.168.8.130:8000'
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adding a new service — four blocks needed
|
||||||
|
|
||||||
|
1. **Router** — `http.routers.<name>-public` with `tls: {}` (not
|
||||||
|
`certResolver`)
|
||||||
|
2. **Middleware** — rate limit, one per service
|
||||||
|
3. **Service** — `http.services.<name>-public` with the backend URL
|
||||||
|
4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the
|
||||||
|
cert sync HOSTS entry
|
||||||
|
|
||||||
|
### Key file naming convention
|
||||||
|
|
||||||
|
| Cert name | Host | Convention |
|
||||||
|
|-----------|------|------------|
|
||||||
|
| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix |
|
||||||
|
| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` |
|
||||||
|
| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` |
|
||||||
|
| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` |
|
||||||
|
| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` |
|
||||||
|
| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` |
|
||||||
|
|
||||||
|
### ⚠️ Critical — keep backends in sync after LXC migrations
|
||||||
|
|
||||||
|
When moving an LXC between Proxmox nodes, update **both**:
|
||||||
|
1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121)
|
||||||
|
2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH)
|
||||||
|
|
||||||
|
Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy
|
||||||
|
but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused
|
||||||
|
Bad Gateway for off-LAN users. Use Python-based editing (see
|
||||||
|
`references/traefik-config-editing.md`) for accurate surgical fixes.
|
||||||
9
archive/knowledge/sources/index.md
Normal file
9
archive/knowledge/sources/index.md
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
# Sources
|
||||||
|
|
||||||
|
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
|
||||||
|
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
|
||||||
|
incident time, then linked from the changelogs of the nodes they implicate).
|
||||||
|
|
||||||
|
| Slug | Reference | Summary |
|
||||||
|
|------|-----------|---------|
|
||||||
|
| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. |
|
||||||
@@ -46,7 +46,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
|
|||||||
## Open items
|
## Open items
|
||||||
|
|
||||||
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
|
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
|
||||||
- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
|
- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
|
||||||
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
|
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
|
||||||
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`.
|
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`.
|
||||||
|
|
||||||
@@ -59,7 +59,7 @@ The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g.
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
|
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
|
||||||
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
|
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
|
||||||
|
|
||||||
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
|
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
|
||||||
|
|
||||||
@@ -87,4 +87,4 @@ Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs mod
|
|||||||
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
|
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
|
||||||
|
|
||||||
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
|
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
|
||||||
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.
|
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.
|
||||||
@@ -7,7 +7,7 @@ Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC,
|
|||||||
- **HAOS version:** 16.3 (last verified)
|
- **HAOS version:** 16.3 (last verified)
|
||||||
- **IP:** `192.168.8.101`
|
- **IP:** `192.168.8.101`
|
||||||
- **Resources:** 4 GiB RAM, 32 GiB boot disk
|
- **Resources:** 4 GiB RAM, 32 GiB boot disk
|
||||||
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123`
|
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123`
|
||||||
|
|
||||||
## Auth
|
## Auth
|
||||||
|
|
||||||
@@ -18,7 +18,7 @@ Key gotchas:
|
|||||||
```
|
```
|
||||||
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
|
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
|
||||||
```
|
```
|
||||||
so OIDC discovery resolves internally to [authentik (124)](124-authentik.md).
|
so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md).
|
||||||
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
|
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
|
||||||
- YAML config:
|
- YAML config:
|
||||||
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
|
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
|
||||||
@@ -30,8 +30,8 @@ Key gotchas:
|
|||||||
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
|
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
- [Authentik (124)](124-authentik.md)
|
- [Authentik (124)](../containers/106-auth-outpost.md)
|
||||||
- [Caddy (121)](121-caddy.md)
|
- [Caddy (121)](../containers/121-caddy.md)
|
||||||
- [DNS](../infrastructure/dns.md)
|
- [DNS](../infrastructure/dns.md)
|
||||||
- [Monitoring](../infrastructure/monitoring.md)
|
- [Monitoring](../infrastructure/monitoring.md)
|
||||||
|
|
||||||
14
archive/knowledge/vms/index.md
Normal file
14
archive/knowledge/vms/index.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# VMs — index
|
||||||
|
|
||||||
|
Two QEMU VMs running on [hubris](../hosts/hubris.md):
|
||||||
|
|
||||||
|
| ID | Name | Role | IP | Public hostname |
|
||||||
|
|----|------|------|----|-----------------|
|
||||||
|
| 100 | [zimaos](100-zimaos.md) | NAS frontend eval (ZimaOS) | `192.168.8.195` | [`zimaos.hubris.network`](../infrastructure/dns.md) |
|
||||||
|
| 108 | [haos-16.3](108-haos.md) | Home automation (HAOS) | `192.168.8.101` | [`home.hubris.network`](../infrastructure/dns.md) |
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [Hubris host](../hosts/hubris.md) — both VMs run here
|
||||||
|
- [Containers index](../containers/index.md) — LXCs on both nodes
|
||||||
|
- [README](../../../README.md)
|
||||||
5
archive/ledger/2026-07.jsonl
Normal file
5
archive/ledger/2026-07.jsonl
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
{"ts": "2026-07-06T11:05:35+00:00", "agent": "mac-mini", "entity": "host:teddycloud", "action": "activate", "risk": "config_mutation", "verification": "homelab node teddycloud relations", "result": "ok"}
|
||||||
|
{"ts": "2026-07-06T11:15:04+00:00", "agent": "mac-mini", "entity": "repo:Homelab-Docs", "action": "register-webhook", "risk": "config_mutation", "result": "ok", "notes": "webhook id 14 for oikos-console deploy"}
|
||||||
|
{"ts": "2026-07-06T11:29:56+00:00", "agent": "mac-mini", "entity": "service:caddy", "action": "add-site-block", "risk": "config_mutation", "verification": "curl -s https://git.hubris.network (unrelated route still healthy after reload)", "result": "ok", "notes": "oikos.hubris.network -> 192.168.8.205:8091, Authentik-gated, in dtoro/caddy-conf@c195142"}
|
||||||
|
{"ts": "2026-07-06T11:40:28+00:00", "agent": "mac-mini", "entity": "host:dns", "action": "add-record", "risk": "config_mutation", "verification": "dig @192.168.8.2 +short oikos.hubris.network", "result": "ok", "notes": "oikos.hubris.network A -> 192.168.8.175 (Caddy LAN IP), via Technitium API, no token persisted"}
|
||||||
|
{"ts": "2026-07-06T11:57:21+00:00", "agent": "mac-mini", "entity": "host:apps", "action": "deploy-oikos-console", "risk": "config_mutation", "verification": "curl http://127.0.0.1:8091/ on apps -> 200; https://oikos.hubris.network/ -> 302 (Authentik gate)", "result": "ok"}
|
||||||
21
archive/oikos-cards/cards/host-apps.md
Normal file
21
archive/oikos-cards/cards/host-apps.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# apps (host:apps)
|
||||||
|
|
||||||
|
- kind: lxc (LXC 105)
|
||||||
|
- state: active
|
||||||
|
- runs-on: host:hubris
|
||||||
|
- role: docker-apps
|
||||||
|
- address: 192.168.8.205 (mesh: tailscale:apps)
|
||||||
|
- mounts: /mnt/library
|
||||||
|
- doc: knowledge/wiki/containers/105-apps.md
|
||||||
|
- secrets: enrolled (age key present)
|
||||||
|
|
||||||
|
## Blast radius
|
||||||
|
- impacts: service:artifacto, service:homelab_mcp, service:secrets_issuance
|
||||||
|
- affected by: host:hubris, mount:/mnt/library, repo:dtoro/Artifacto, repo:dtoro/Homelab-Docs
|
||||||
|
- full blast radius: service:artifacto, service:homelab_mcp, service:secrets_issuance
|
||||||
|
|
||||||
|
## Safe actions
|
||||||
|
- see the services this host runs for action-level risk classes
|
||||||
|
|
||||||
|
## Recent changes
|
||||||
|
- 2026-07-06T11:57:21+00:00 deploy-oikos-console (config_mutation) — ok
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user