dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
Files
e32b21072d4ac1a58a1c9a812e8f5548c32d2392
oikos/infrastructure/auto-deploy.md
dtoro e32b21072d claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes 
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core
- Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only)
- Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor
- Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift
- Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options)
- Wire Matrix (matrix:dtoro) as health alert delivery platform
- Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml
- Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
2026-06-04 10:14:56 +02:00

146 lines
12 KiB
Markdown
Raw Blame History

# Auto-deploy — gitea-webhook pipelines
Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
## Two shapes
### Shape A — checkout IS the working tree (config repos)
`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
### Shape B — receiver outside the app repo (compose stacks)
The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
## Common
- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
**Don't strip these when editing app.ini.**
- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
## Pipelines
| Repo | Target | Shape | Receiver | Webhook id | Reload action |
| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
## When you change a tracked config
Always commit + push. Local-only edits drift. Common ones:
- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
- `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys)
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
## Per-pipeline notes / gotchas
### caddy-conf
- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
### gitea-customizations
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
### Plato
- Shape B (`/opt/plato-deploy/{webhook.py,deploy.sh}`, port `9799`).
- The in-LXC checkout's `origin` is `http://192.168.8.121:3000/dtoro/Plato.git` (internal gitea), and git creds are at `/root/.git-credentials` rather than the `/etc/plato-deploy/git-credentials` pattern — the unit doesn't set `ProtectHome` so root's home is reachable.
- `/data` is a host bind (`/mnt/library/documents/plato`), so `docker compose up -d --build` rebuilds the image + restarts the container without touching the SQLite db. The [fresh-DB bootstrap workaround](../containers/126-plato.md#fresh-db-bootstrap-workaround) only matters if you blow `plato.db` away.
### mule-image / Artifacto
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
### backup-library
- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
### homelab-mcp / secrets-issuance
- Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent
Shape-B target with its own webhook receiver.
- The deploy script restarts the service it just updated. Because the
webhook receiver itself is a separate systemd unit (`*-deploy.service`),
it does NOT restart itself — but `deploy.sh` running `systemctl
restart homelab-mcp-deploy.service` (or the secrets-issuance one)
would create a kill-self loop. The current `deploy.sh` is careful
to only restart the main service.
- Both services consume `/opt/homelab-context` for their runtime data
(inventory, secret recipient lookup). That clone is **the same clone
every other client has** — kept fresh by `homelab-context-sync.timer`,
not by these webhooks.
## Custom-built binaries that overlap apt-managed paths
If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
## Related
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
## Changelog
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
### 2026-05-13 — Plato pipeline added
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
### 2026-04-28 — wiki entry created
Initial documentation. Six active pipelines.
### 2026-04-22 — Artifacto pipeline added
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
### 2026-06-04 — claudio-bot pipeline decommissioned
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
### 2026-04-21 — mule-image + claudio-bot pipelines added
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
Initial three. Set the conventions everything else follows.
Reference in New Issue View Git Blame Copy Permalink