oikos/investigations/archive/2026-04-21-hubris-crash-loop.md

2026-04-21 — Hubris crash loop (thermal + USB drive)

Summary

hubris hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the performance governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via cpu-epp.service; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; cpu-epp remains as belt-and-suspenders.

Timeline

2026-04-19 — drive attached

External Silicon Motion Portable SSD (vid:pid 090c:2320) attached for the new restic backup pipeline. Pre-attach uptime had been 33 days stable.

2026-04-19 → 2026-04-21 — first crashes

Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by uas_eh_abort_handler storms and xHCI resets on port 6-1.

2026-04-21 — crash cadence escalates

23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once cpu-epp.service started applying EPP=balance_power.

2026-04-21 — first set of mitigations applied

• cpu-epp.service deployed. Sets scaling_governor=powersave + EPP=balance_power at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
• Crash capture: /etc/sysctl.d/60-crash-capture.conf (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), /etc/modprobe.d/softdog.conf (soft_panic=1 soft_margin=60), /etc/systemd/system.conf.d/watchdog.conf (RuntimeWatchdogSec=15s).
• rasdaemon installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to /var/lib/rasdaemon/ras-mc_event.db. Query ras-mc-ctl --summary.
• UAS blacklist for the drive: /etc/modprobe.d/usb-storage-quirks.conf → options usb-storage quirks=090c:2320:u. Forces BOT instead of UAS for the SMI bridge.
• Mount-on-demand for the drive: /usr/local/sbin/backup-usb.sh attach|detach|status toggles /sys/bus/usb/devices/*/authorized so the drive is de-authorized when no backup is running.

2026-04-22 — recurrence after 30 h 37 m

Same silent-cutoff signature at 18:42:08. Much longer than any pre-cpu-epp crash (12 h max), so cpu-epp helps but is not sufficient on its own. claudio-monitor showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.

2026-04-22 — cpu-epp.service design bug fixed

Was After=multi-user.target + WantedBy=multi-user.target — queued behind pve-guests.service. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the performance governor. Fixed: now After=sysinit.target + Before=pve-guests.service.

2026-04-22 — drive removed (A/B test)

User physically removed the external USB drive. Backup timers disabled, fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.

2026-04-23 — SSD cooling + thermal pads installed

Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) 53 °C; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.

2026-04-23 thermal-pad verdict

Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. The load-bearing signal is the lifetime warning-time counter, not absolute sensor1 — watch that going forward, not sensor1 threshold.

2026-04-28 — A/B test passing so far

3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.

2026-04-29 — physical relocation

User moved the host to a spot with better airflow. Post-move idle baseline (45 min uptime, light load): Tctl 47.2 °C, GPU 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C. Notably nvme0 sensor1 is 27 °C cooler than the 2026-04-23 thermal-pad steady-state of 60–61 °C. Watch lifetime warning-time counters (nvme smart-log → "Warning Comp. Temperature Time") over the next week — flat trend confirms the move helped. Doesn't change the drive-removal A/B test, but reduces residual thermal headroom risk independently.

Root cause

Two-contributor:

1. Thermal: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under performance governor. Mitigated by cpu-epp.service. In amd-pstate=active mode the governor MUST be powersave for EPP values to take effect — performance silently ignores them.
2. Storage: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).

Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so fwupdmgr can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.

Diagnostic notes

• Pure hardware hangs leave no trace. pstore stays empty across reboots despite panic_on_oops/hardlockup/softlockup/rcu + softdog soft_panic=1 being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
• First check if hubris crash-loops again with the same silent-cutoff signature: cpu-epp.service is active and /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference is balance_power.
• Non-ECC RAM means memory faults are silent. If cpu-epp is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out pcie_aspm=off pci=nomsi workarounds in /etc/default/grub.
• Grub escalation (pcie_aspm=off pcie_port_pm=off pci=nomsi) deliberately NOT applied yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
• mcelog is not in Debian 13 repos; rasdaemon is the replacement.

BIOS update path

Checked 2026-04-21. GMKtec is not on LVFS, so fwupdmgr can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a .exe flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email service@gmktec.com with the S/N to confirm a newer BIOS / AGESA level actually exists. fwupd is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.

Mitigations applied

Mitigation	Status	Notes
cpu-epp.service (powersave + balance_power)	Active	Unit ordering fixed 2026-04-22 to run before guests
Crash capture (sysctl + softdog + watchdog)	Active	Catches everything but pure silicon hangs
rasdaemon	Active	Persistent telemetry for MCE / memory / PCIe AER
UAS blacklist for 090c:2320	Active (drive currently absent)	/etc/modprobe.d/usb-storage-quirks.conf
Mount-on-demand (backup-usb.sh)	Active (drive currently absent)	Backup unit ExecStartPre/ExecStopPost hooks
Drive physically removed	Active	A/B test in progress
Thermal pads on NVMes	Active 2026-04-23	Steady-state composite 47 °C / sensor1 60 °C
pcie_aspm=off pci=nomsi	NOT applied	Reserved for if crashes recur without the drive

Affected nodes

• Hubris host
• Backups (disabled)
• Monitoring

Open questions

• Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
• If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
• Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
• Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.