f5cd320433
Initial documentation of the hubris Proxmox homelab as a cross-linked markdown wiki. Per-node pages, cross-cutting infrastructure pages, an investigation log, and an operations cheatsheet. Each node and topic ends with a Changelog section so changes can be tracked in-place going forward. Refreshed against live state on 2026-04-28 — 14 active LXCs (109 syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test state of the 2026-04-21 hubris crash-loop investigation.
94 lines
7.1 KiB
Markdown
94 lines
7.1 KiB
Markdown
# Auto-deploy — gitea-webhook pipelines
|
|
|
|
Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
|
|
|
|
## Two shapes
|
|
|
|
### Shape A — checkout IS the working tree (config repos)
|
|
|
|
`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
|
|
|
|
### Shape B — receiver outside the app repo (compose stacks)
|
|
|
|
The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
|
|
|
|
## Common
|
|
|
|
- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
|
|
- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
|
|
- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
|
|
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
|
|
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
|
|
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
|
|
- `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
|
|
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
|
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
|
|
|
**Don't strip these when editing app.ini.**
|
|
|
|
- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
|
|
|
|
## Pipelines
|
|
|
|
| Repo | Target | Shape | Receiver | Webhook id | Reload action |
|
|
| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
|
|
| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
|
|
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
|
|
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
|
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
|
| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
|
|
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
|
|
|
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
|
|
|
|
## When you change a tracked config
|
|
|
|
Always commit + push. Local-only edits drift. Common ones:
|
|
|
|
- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
|
|
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
|
|
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
|
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
|
- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
|
|
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
|
|
|
## Per-pipeline notes / gotchas
|
|
|
|
### caddy-conf
|
|
- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
|
|
- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
|
|
|
|
### gitea-customizations
|
|
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
|
|
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
|
|
|
|
### mule-image / Artifacto
|
|
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
|
|
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
|
|
- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
|
|
- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
|
|
- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
|
|
|
|
### backup-library
|
|
- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
|
|
- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
|
|
|
|
## Related
|
|
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
|
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
|
|
- [Backups (disabled)](backups.md)
|
|
|
|
## Changelog
|
|
|
|
### 2026-04-28 — wiki entry created
|
|
Initial documentation. Six active pipelines.
|
|
|
|
### 2026-04-22 — Artifacto pipeline added
|
|
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
|
|
|
|
### 2026-04-21 — mule-image + claudio-bot pipelines added
|
|
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
|
|
|
|
### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
|
|
Initial three. Set the conventions everything else follows.
|