# Auto-deploy — gitea-webhook pipelines Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes. ## Two shapes ### Shape A — checkout IS the working tree (config repos) `/etc/` or `/var/lib//...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap. ### Shape B — receiver outside the app repo (compose stacks) The app repo at `/opt/` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout. ## Common - All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/-deploy/secret`. - All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded. - Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently: - `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md)) - `192.168.8.175` ([caddy (121)](../containers/121-caddy.md)) - `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto) - ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~ - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md)) - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library) - `192.168.8.190` ([plato (126)](../containers/126-plato.md)) - `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito) **Don't strip these when editing app.ini.** - Git creds for root-run deploy services live in `/etc/-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`. ## Pipelines | Repo | Target | Shape | Receiver | Webhook id | Reload action | | ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- | | `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` | | `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed | | `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` | | `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` | | `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` | | `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed | | `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) | | `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart | | `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart | | `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` | > Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push. > Each owns its own clone on LXC 105. They don't conflict because each > deploy.sh only touches its own service unit + venv. > **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md). ## When you change a tracked config Always commit + push. Local-only edits drift. Common ones: - `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys) - `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys) - `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys) - `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys) - `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys) - ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04) - `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys) - `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md)) ## Per-pipeline notes / gotchas ### caddy-conf - Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits. - The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). ### gitea-customizations - Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP. - Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run. ### Plato - Shape B (`/opt/plato-deploy/{webhook.py,deploy.sh}`, port `9799`). - The in-LXC checkout's `origin` is `http://192.168.8.121:3000/dtoro/Plato.git` (internal gitea), and git creds are at `/root/.git-credentials` rather than the `/etc/plato-deploy/git-credentials` pattern — the unit doesn't set `ProtectHome` so root's home is reachable. - `/data` is a host bind (`/mnt/library/documents/plato`), so `docker compose up -d --build` rebuilds the image + restarts the container without touching the SQLite db. The [fresh-DB bootstrap workaround](../containers/126-plato.md#fresh-db-bootstrap-workaround) only matters if you blow `plato.db` away. ### mule-image / Artifacto - Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec -- journalctl -u -deploy-webhook -f`. - **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL. - Manual deploy: `pct exec -- /opt/-deploy/deploy.sh`. - Health: `pct exec -- curl -s http://127.0.0.1:/health` → `ok`. - Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated. ### backup-library - Currently the only deploy that targets the host directly (`192.168.8.77:9798`). - `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run. ### homelab-mcp / secrets-issuance - Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105 (`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent Shape-B target with its own webhook receiver. - The deploy script restarts the service it just updated. Because the webhook receiver itself is a separate systemd unit (`*-deploy.service`), it does NOT restart itself — but `deploy.sh` running `systemctl restart homelab-mcp-deploy.service` (or the secrets-issuance one) would create a kill-self loop. The current `deploy.sh` is careful to only restart the main service. - Both services consume `/opt/homelab-context` for their runtime data (inventory, secret recipient lookup). That clone is **the same clone every other client has** — kept fresh by `homelab-context-sync.timer`, not by these webhooks. ## Custom-built binaries that overlap apt-managed paths If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile. Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary: 1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: `, `Version: 1:-hubris`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)). 2. **Hold the apt package.** `apt-mark hold ` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten. If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns. ## Related - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these - [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets - [Backups (disabled)](backups.md) - [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference ## Changelog ### 2026-06-24 — terminalito pipeline added Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config). ### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo. ### 2026-05-13 — Plato pipeline added Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`. ### 2026-04-28 — wiki entry created Initial documentation. Six active pipelines. ### 2026-04-22 — Artifacto pipeline added Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended. ### 2026-06-04 — claudio-bot pipeline decommissioned LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead. ### 2026-04-21 — mule-image + claudio-bot pipelines added Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot. ### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped Initial three. Set the conventions everything else follows.