dtoro
4efddb8bed
Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative
links that predated this session — a container rename, incident/plan docs
that moved into archive/done subfolders without their inbound links being
updated, and a handful of relative-depth bugs in files nested under
containers/archive/ and plans/done/.
Fixes applied, by category:
- 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs).
- investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md
-> archive/ prefix (both moved to investigations/archive/ previously).
- plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot,
2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix.
- Depth bugs in files nested one level deeper than their siblings assumed
(investigations/archive/*, knowledge/wiki/containers/archive/*,
plans/done/*) — corrected relative-path depth.
- Destroyed containers with no surviving page (126-plato) delinked to the
containers/index.md archaeology row instead of a 404.
- ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine).
- netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate
record, matching the existing convention for hosts without a wiki page).
- runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md
(missed in the phase-4 runbook move because the referencing files used a
bare filename, not a runbooks/ prefix).
- One dangling forward-reference to a never-written investigation delinked
to the actual incident record it was describing.
Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into
devops/homelab-authentik-admin/ — an intentional reference to a sibling repo,
not present in this checkout.
Verification: broken-link count 126 -> 2 (real remainder is the cross-repo
reference above); gen-topology.py --check still exit 0; build_host_files.py
still idempotent; all inventory.yaml doc_page targets still resolve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
112 lines
8.4 KiB
Markdown
112 lines
8.4 KiB
Markdown
# 114 — `nextcloud`
|
||
|
||
Personal cloud / file collaboration. Source-of-truth for the photo libraries surfaced by [mulita (120)](120-mule-images.md).
|
||
|
||
## At a glance
|
||
- **Hostname:** `nextcloud`
|
||
- **IP:** `192.168.8.224`
|
||
- **Privilege:** privileged
|
||
- **Resources:** 4 cores / 6 GiB RAM / 25 GiB rootfs
|
||
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (Nextcloud user files under `/mnt/library/homecloud`)
|
||
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
|
||
|
||
## Auth
|
||
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
|
||
|
||
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
|
||
|
||
## DNS workaround (load-bearing)
|
||
Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied:
|
||
- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`.
|
||
- `pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order.
|
||
- `/etc/hosts` override kept as belt-and-suspenders.
|
||
|
||
Also needs `allow_local_remote_servers=true` in `config.php`.
|
||
|
||
> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md).
|
||
|
||
## Storage
|
||
Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md).
|
||
|
||
## Web stack (Apache event MPM + php-fpm)
|
||
Apache 2.4 with **`mpm_event`** as the MPM and PHP served via **php-fpm 8.4** over the Unix socket `/run/php/php8.4-fpm.sock` (mod_php disabled). Routing to FPM is via the distro's `/etc/apache2/conf-available/php8.4-fpm.conf` (gated by `<IfModule !mod_php8.c>`, so just `a2dismod php8.4` flips Apache onto FPM).
|
||
|
||
Why this matters: with mod_php every Apache worker is forced onto `mpm_prefork` (one heavy process per connection, full PHP interpreter loaded in each). On FPM + event, Apache workers are ~5–8 MB and async; PHP work is done by a separate FPM pool that keeps opcache hot. Concurrent upload throughput jumps significantly because Apache no longer blocks per-connection on a heavy PHP process.
|
||
|
||
### PHP-FPM pool
|
||
- Socket: `/run/php/php8.4-fpm.sock` (owner/group `www-data`)
|
||
- `pm = dynamic`, `pm.max_children = 30`, `pm.start_servers = 6`, `pm.min/max_spare_servers = 4/12`, `pm.max_requests = 500`
|
||
- Steady-state ~6 idle workers at ~80 MB each — bump `max_children` only if peak concurrency exceeds 30.
|
||
|
||
### PHP limits (FPM + CLI both)
|
||
Set in `/etc/php/8.4/fpm/php.ini` **and** `/etc/php/8.4/cli/php.ini` (the latter so `occ`/cron see the same ceilings):
|
||
|
||
| key | value |
|
||
|---|---|
|
||
| `upload_max_filesize` | `16G` |
|
||
| `post_max_size` | `16G` |
|
||
| `memory_limit` | `512M` |
|
||
| `max_execution_time` | `3600` |
|
||
| `max_input_time` | `3600` |
|
||
| `output_buffering` | `Off` |
|
||
|
||
After edits: `systemctl reload php8.4-fpm`. Caddy in front (`cloud.hubris.network`) imposes no body-size limit, so these are the effective ceiling.
|
||
|
||
### Opcache + JIT
|
||
`/etc/php/8.4/mods-available/opcache.ini` — full config maintained, not commented stubs:
|
||
- `opcache.enable=1`, `opcache.memory_consumption=256`, `opcache.interned_strings_buffer=32`, `opcache.max_accelerated_files=20000`
|
||
- `opcache.jit=tracing`, `opcache.jit_buffer_size=64M`
|
||
|
||
### Apache mod_reqtimeout
|
||
`/etc/apache2/mods-available/reqtimeout.conf` — relaxed body trickle so slow-WAN uploads don't get killed:
|
||
- `RequestReadTimeout header=20-40,MinRate=500`
|
||
- `RequestReadTimeout body=20,MinRate=100` (was `body=10,MinRate=500`)
|
||
|
||
## MariaDB tuning
|
||
Overrides in `/etc/mysql/mariadb.conf.d/99-nextcloud-tuning.cnf`:
|
||
- `innodb_buffer_pool_size = 1G` (default was 128M)
|
||
- `innodb_log_file_size = 256M`
|
||
- `innodb_flush_log_at_trx_commit = 2` (group-commit; trades a few ms of durability for write throughput — acceptable for NC)
|
||
- `innodb_flush_method = O_DIRECT`, `innodb_io_capacity = 2000` / `..._max = 4000`
|
||
- `max_allowed_packet = 256M`, `tmp_table_size = 64M`, `max_heap_table_size = 64M`
|
||
|
||
Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` needs a clean restart).
|
||
|
||
## Related
|
||
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
|
||
- [Authentik (124)](106-auth-outpost.md)
|
||
- [Caddy (121)](121-caddy.md)
|
||
- [DNS](../infrastructure/dns.md)
|
||
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
|
||
- [Media permissions](../infrastructure/media-permissions.md)
|
||
|
||
## Changelog
|
||
|
||
### 2026-05-13 — Apache keepalive bumped (and TCP BBR on host)
|
||
Follow-up to the FPM/MPM/MariaDB round earlier the same day. After a WiFi-bound Linux client kept seeing ~2 MB/s downloads despite a 152 Mbps speed test (so the WiFi medium wasn't the cap), bumped Apache `KeepAliveTimeout 5 → 60` and `MaxKeepAliveRequests 100 → 1000` so the mirall desktop client reuses one TLS+TCP connection across many sequential file fetches instead of paying handshake + slow-start each time. Paired with TCP BBR + `fq` qdisc enabled on the [hubris host](../hosts/hubris.md#2026-05-13--tcp-bbr--fq-qdisc-enabled), which directly addresses single-stream throughput on jittery links. Server-side loopback baseline post-BBR is ~404 MB/s for a single 23 MB CR2 download via Caddy. Client-side diagnosis pending (curl single-stream + parallel tests) to determine if the remaining gap is mirall config (bandwidth limit / `parallelNetworkRequests`) or path shaping.
|
||
|
||
### 2026-05-13 — throughput tuning: FPM + event MPM + opcache JIT + MariaDB
|
||
Investigated upload/download speed. Bottlenecks were inside this LXC — LAN (33.9 Gbit/s caddy→NC iperf3) and `/mnt/library` NVMe (714 MB/s sustained) were not the limit. Changes:
|
||
- **Apache mod_php → php-fpm 8.4** (`a2dismod php8.4`). Flips PHP from in-process to FPM-over-Unix-socket. Apache workers now ~5–8 MB instead of ~80 MB each.
|
||
- **MPM prefork → event** (`a2dismod mpm_prefork && a2enmod mpm_event`). Async I/O, far better concurrency for parallel uploads.
|
||
- **opcache JIT enabled** (`tracing`, 64M buffer), `memory_consumption=256`, `interned_strings_buffer=32`, `max_accelerated_files=20000`.
|
||
- **MariaDB**: `innodb_buffer_pool_size 128M → 1G`, `innodb_log_file_size 96M → 256M`, `flush_log_at_trx_commit 1 → 2`, `O_DIRECT`, higher I/O capacity. New file `99-nextcloud-tuning.cnf`.
|
||
- **Apache mod_reqtimeout**: body trickle relaxed from `body=10,MinRate=500` → `body=20,MinRate=100` so slow-WAN uploads aren't killed.
|
||
- **PHP-FPM ini** synced to match what the apache2/cli inis already had (FPM was stuck at 2M/8M because FPM wasn't serving traffic before, but now it does).
|
||
|
||
Backups: `/etc/php/8.4/fpm/php.ini.bak.20260513`, `/etc/apache2/mods-available/reqtimeout.conf.bak.20260513`, `/etc/php/8.4/mods-available/opcache.ini.bak.20260513`. MariaDB tuning is a new file (rollback = `rm`).
|
||
|
||
Verified: `apache2ctl configtest` Syntax OK, `occ status` clean, `occ setupchecks` shows ✓ on memory_limit / output_buffering / DB indices, `status.php` returns 200 in <40 ms via Caddy with HTTP/2.
|
||
|
||
### 2026-05-10 — PHP upload limits raised
|
||
Apache + CLI php.ini bumped from distro defaults (`upload_max_filesize=2M`, `post_max_size=8M`, `memory_limit=128M`, `max_execution_time=30`) to NC-recommended (`16G/16G/512M/3600`). Backups at `/etc/php/8.4/{apache2,cli}/php.ini.bak-20260510-*`. Was causing slow/failing uploads via web UI.
|
||
|
||
### 2026-04-28 — wiki entry created
|
||
Initial documentation.
|
||
|
||
### 2026-04-26 — Nextcloud-rooted libraries shipped (mulita)
|
||
[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`.
|
||
|
||
### 2026-04-21 — wired into Authentik (native OIDC + nc_uid override)
|
||
Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`.
|