oikos/knowledge/wiki/infrastructure/auto-deploy.md

Auto-deploy — gitea-webhook pipelines

Several configs and apps in the lab live in dtoro/* repos on gitea (104) and auto-redeploy on push. All pipelines follow one of two shapes.

Two shapes

Shape A — checkout IS the working tree (config repos)

/etc/<thing> or /var/lib/<thing>/... is itself a git clone. Push triggers git pull + a reload command. Used for pure-config repos where re-cloning is cheap.

Shape B — receiver outside the app repo (compose stacks)

The app repo at /opt/<thing> is the working tree, but the deploy tooling (webhook.py, deploy.sh, systemd unit) lives in a sibling /opt/<thing>-deploy/ so the app repo stays portable. Push triggers git pull + docker compose up -d --build. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.

Common

• All receivers validate X-Gitea-Signature HMAC-SHA256 against a per-pipeline secret in /etc/<thing>-deploy/secret.

• All filter to refs/heads/main (or master for older repos). Gitea's "test delivery" button sends ref=main (without refs/heads/) — those will log "ignoring ref main" and 204. Real pushes work. Don't "fix" the ref filter to accept both — it'd also accept PR merges from side branches that got fast-forwarded.

• Gitea's app.ini [webhook] ALLOWED_HOST_LIST must include every receiver IP. Currently:

  • 127.0.0.1 (gitea customizations on LXC 104)
  • 192.168.8.175 (caddy (121))
  • 192.168.8.205 (apps (105) — Artifacto)
  • 192.168.8.230 (claudio-bot — destroyed 2026-06-04)
  • 192.168.8.136 (mule-images (120))
  • 192.168.8.77 (hubris host — backup-library)
  • 192.168.8.190 (plato (126)) (destroyed 2026-06-28)
  • 192.168.8.211 (trmnl (128) — terminalito)

  Don't strip these when editing app.ini.

• Git creds for root-run deploy services live in /etc/<thing>-deploy/git-credentials (mode 600) and are wired via credential.helper = store --file=/etc/<thing>-deploy/git-credentials in the repo's .git/config. Necessary because the unit typically runs with ProtectHome=true, which blocks /root.

Pipelines

Repo	Target	Shape	Receiver	Webhook id	Reload action
dtoro/caddy-conf	caddy (121) /etc/caddy/	A	http://192.168.8.175:9797/deploy	2	caddy validate + systemctl reload caddy
dtoro/gitea-customizations	gitea (104) /var/lib/gitea/custom/	A	http://127.0.0.1:9797/deploy (loopback)	(orig)	systemctl restart gitea if templates changed
dtoro/mule-image	mule-images (120) /opt/mule-image/	B	http://192.168.8.136:9797/deploy	6	docker compose up -d --build
dtoro/Artifacto	apps (105) /opt/artifacto/	B	http://192.168.8.205:9798/deploy	7	docker compose up -d --build
dtoro/Plato	plato (126) /opt/plato/app/ (destroyed 2026-06-28)	⊘	http://192.168.8.190:9799/deploy (dead)	8 (removed)	Repo archived — LXC destroyed
dtoro/claudio-bot	claudio-bot (123) (destroyed 2026-06-04)	⊘	http://192.168.8.230:9797/deploy (dead)	(archived)	Repo archived — LXC destroyed
dtoro/backup-library	hubris host /opt/backup-library/	A	http://192.168.8.77:9798/deploy	(orig)	runs deploy.sh (preserves admin-edited /etc/restic/include-*.list)
dtoro/Homelab-Docs → homelab-mcp	apps (105) /opt/homelab-mcp/	B	http://192.168.8.205:9811/deploy	10	reinstalls homelab-mcp.service + restart
dtoro/Homelab-Docs → secrets-issuance	apps (105) /opt/secrets-issuance/	B	http://192.168.8.205:9821/deploy	11	reinstalls secrets-issuance.service + restart
dtoro/terminalito	trmnl (128) /opt/terminalito/	B	http://192.168.8.211:9797/deploy	12	reinstalls units + systemctl restart trmnl-plugins
dtoro/Homelab-Docs → oikos-console	apps (105) /opt/oikos-console/	B	http://192.168.8.205:9831/deploy	14	reinstalls oikos-console.service + restart — see oikos/console/deploy/README.md

Note: dtoro/Homelab-Docs has three webhooks firing on the same push. Each owns its own clone on LXC 105. They don't conflict because each deploy.sh only touches its own service unit + venv.

Not yet wired: dtoro/claudio-monitor (push, then /opt/claudio-monitor/scripts/deploy.sh manually). The former authentik LXC (124) is destroyed — Authentik runs on the VPS. DNS moved to Technitium on dns (107).

When you change a tracked config

Always commit + push. Local-only edits drift. Common ones:

• /etc/caddy/Caddyfile ↔ dtoro/caddy-conf (auto-deploys)
• /var/lib/gitea/custom/ ↔ dtoro/gitea-customizations (auto-deploys)
• /opt/artifacto/ ↔ dtoro/Artifacto (auto-deploys)
• /opt/mule-image/ ↔ dtoro/mule-image (auto-deploys)
• /opt/plato/app/ ↔ dtoro/Plato (destroyed 2026-06-28)
• /opt/claudio-bot/ ↔ dtoro/claudio-bot (destroyed 2026-06-04)
• /opt/backup-library/ ↔ dtoro/backup-library (auto-deploys)
• /opt/homelab-mcp/ + /opt/secrets-issuance/ ↔ dtoro/Homelab-Docs (auto-deploys both, see homelab-context)

Per-pipeline notes / gotchas

caddy-conf

• Repo includes scripts/webhook/install.sh. Editing the systemd unit inside the repo does not auto-reinstall — re-run install.sh manually after unit edits.
• The unit has ReadWritePaths=/etc/caddy — load-bearing (ProtectSystem=full would otherwise block git pull).

gitea-customizations

• Receiver is on loopback (127.0.0.1:9797), not the LXC IP.
• Online3DViewer binary assets are NOT tracked; deploy.sh fetches them on first run.

mule-image / Artifacto

• Async deploy (returns 202) — gitea would otherwise time out the request. Logs: pct exec <id> -- journalctl -u <thing>-deploy-webhook -f.
• Cloning from inside the LXC must use the internal gitea IP (http://192.168.8.121:3000/...). https://git.hubris.network hits a connection reset from inside apps (105) (Caddy routing / TLS hairpin not configured for this LXC). Configured origin on the in-LXC checkout is the internal URL.
• Manual deploy: pct exec <id> -- /opt/<thing>-deploy/deploy.sh.
• Health: pct exec <id> -- curl -s http://127.0.0.1:<port>/health → ok.
• Setup tokens used to register the webhook (e.g., artifacto-deploy-setup, artifacto-deploy-setup-2, artifacto-cleanup on user dtoro) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's /users/{u}/tokens endpoints require basic auth (not bearer), so cleanup couldn't be automated.

backup-library

• Currently the only deploy that targets the host directly (192.168.8.77:9798).
• deploy.sh is careful to preserve admin edits to /etc/restic/include-*.list — canonical source is config/ in the repo, but the install path is treated as authoritative once deploy.sh has run.

homelab-mcp / secrets-issuance

• Both ride a single push to dtoro/Homelab-Docs. Two clones on LXC 105 (/opt/homelab-mcp, /opt/secrets-issuance) — each is an independent Shape-B target with its own webhook receiver.
• The deploy script restarts the service it just updated. Because the webhook receiver itself is a separate systemd unit (*-deploy.service), it does NOT restart itself — but deploy.sh running systemctl restart homelab-mcp-deploy.service (or the secrets-issuance one) would create a kill-self loop. The current deploy.sh is careful to only restart the main service.
• Both services consume /opt/homelab-context for their runtime data (inventory, secret recipient lookup). That clone is the same clone every other client has — kept fresh by homelab-context-sync.timer, not by these webhooks.

Custom-built binaries that overlap apt-managed paths

If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly /usr/bin/<name> — then the next apt upgrade of the corresponding package will silently clobber the custom build. That's exactly how LXC 121 caddy went down for ~10 min on 2026-05-21: an xcaddy build with caddy-dns/ionos lived at /usr/bin/caddy and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.

Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:

1. Ship the build as a .deb with an epoch-bumped version. Use dpkg-deb --build (or nfpm) to package the binary as Package: <name>, Version: 1:<upstream>-hubris<n>. The epoch (1:) means it beats any non-epoch upstream version regardless of point bumps, so apt upgrade is a no-op for that package. Used by caddy: caddy 1:2.11.3-hubris1 (see commit 2026-05-21 in 121-caddy.md).

2. Hold the apt package. apt-mark hold <pkg> on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than .deb packaging but: (a) the hold flag isn't preserved by dpkg -i of a new version, (b) it's invisible unless you check apt-mark showhold, (c) you have to remember to apt-mark unhold when you intentionally want a new version. The new homelab apt-audit subcommand surfaces holds across the fleet so they don't get forgotten.

If you're not sure what's already lurking, run homelab apt-audit --fleet and look at the NONAPT column — that's a count of binaries in /usr/bin/{caddy,docker,jellyfin} + /usr/local/bin/* that no apt package owns.

Related

• Gitea (104) — webhook source for all of these
• Caddy (121), apps (105), mule-images (120), hubris host — webhook targets
• Backups (disabled)
• Operations cheatsheet — homelab apt-audit / homelab apt-upgrade reference

Changelog

2026-06-28 — Plato pipeline decommissioned

LXC 126 destroyed, webhook id 8 on dtoro/Plato removed. 192.168.8.190 removed from gitea app.ini ALLOWED_HOST_LIST.

2026-06-24 — terminalito pipeline added

Webhook id 12 on dtoro/terminalito → http://192.168.8.211:9797/deploy on trmnl (128). Shape B (server/deploy/webhook.py receiver, in-repo server/deploy/deploy.sh; secret /etc/terminalito-deploy/secret). app.ini ALLOWED_HOST_LIST extended with 192.168.8.211. Verified end-to-end with a push. Repo-local credential.helper in /opt/terminalito/.git/config (the unit can't read root's global git config).

2026-05-20 — homelab-mcp + secrets-issuance pipelines added

Webhook ids 10 + 11 on dtoro/Homelab-Docs (ports 9811 + 9821 on apps (105)). Two webhooks on one repo — each owns its own clone (/opt/homelab-mcp, /opt/secrets-issuance) and only restarts its own service. See homelab-context for why both services live in one repo.

2026-05-13 — Plato pipeline added

Webhook id 8 on dtoro/Plato (port 9799 on plato (126)). app.ini ALLOWED_HOST_LIST extended to include 192.168.8.190.

2026-04-28 — wiki entry created

Initial documentation. Six active pipelines.

2026-04-22 — Artifacto pipeline added

Webhook id 7 on dtoro/Artifacto (port 9798 on apps). app.ini ALLOWED_HOST_LIST extended.

2026-06-04 — claudio-bot pipeline decommissioned

LXC 123 destroyed, dtoro/claudio-bot archived. Webhook port 9797 dead.

2026-04-21 — mule-image + claudio-bot pipelines added

Webhook id 6; receiver on apps' sibling /opt/mule-deploy/. Same shape used for claudio-bot.

2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped

Initial three. Set the conventions everything else follows.