Stress testing (the standing stress-fleet cluster)
High load + randomized testing across a concurrent QEMU cluster, catching every anomaly class β design doc docs/test/stress-fleet.md
The standing stress cluster targets: wild writes (armed capture on the hunt chains), panics/hangs, leaks, fake-OOM recurrences, scheduler pathologies, perf-under-load drift (vs docs/development/perf-gate-baseline.md), and data corruption. Probes live in test/stressng and test/cowrace.
Β§1Cluster composition
| Member | Kind | Load |
|---|---|---|
| chains Γ2 | wild-write hunt chains | self-advancing hunt kernel + load chains (frozen, never restarted) |
| ltp-a/b/c/d | LTP sweep instances | riscv64 TCG smp2/2G, random-start full LTP sweeps |
| monkey-a/b | randomized-testing instances | x86_64 KVM 4G, usbless OH image + randomized churn (storage hotplug, hilog flood/rotate, appspawn kill-restart, bm/hidumper/param walks, sync) |
| memstorm | memory storm | mmap/munmap/MAP_FIXED/mprotect/fork storms, dfx=periodic,memwatch,watchdog |
| fsfuzz | filesystem fuzzing | random-syscall fuzzer + file/pipe/socket/madvise/fork sequences |
| ltpm | control member | LTP sweeps on the current main kernel (worktree-vs-main drift control) |
- Round mechanics: 1,800 s per round, serial to file, a classification line (decisive CRITICAL > WARN > WEDGE > clean); wedge rule: no serial growth for β₯300 s β kill, mark WEDGE, keep the tail.
- Host budget discipline: adaptive fleet size, a hard floor of β₯10 GB free host memory (checked via
freebefore every launch/ramp); QEMU launches niced +10; shared images only-snapshot/snapshot=on+file.locking=off; writable images are private reflink copies. - Standing shape: 2 chains + 9 members = 11 QEMUs, guarded by a 30-minute central sweeper.
Β§2Root-caused classes (fleet wins)
- The SIGILL family (kernel bug, fixed): read_counter() read the time CSR with the write-flavored form
csrrw a0, time, zeroβ the whole instruction is illegal-instruction-prone; the mount hammer loop hit it within ~100 s on every build (toolchain zcb exonerated by paired trials). Fixed with the canonical read formcsrr. - Seven user-length round-up overflows (fixed): fsfuzz r172 drove madvise length=β1 into
attempt to add with overflow(syscall/memory.rs); all seven user-length round-ups (madvise/mprotect/msync/mremap/mincore/mlock/munlock) gained wrap checks (EINVAL, Linux end-wrap semantics); post-fix the deterministic seed stream ran 150 s+ with zero panics (v4 died within 60 s). - The zero-serve family (excluded): cluster LTP sweeps were round-for-round identical across the virtio/bio completion rewrite (1253/1254 TPASS both rounds) β the rewrite cost nothing on that surface.
Β§3Known classes under watch
- The fsfuzz WATCHDOG family (a perf-registry static+heap lock pair): r282 recorded zero DEADLOCK prints (a first for this home, 45,858 lines), r283 returned 118 β same family, same ra, the static lock moved with the new binary; adjudicated as a known transient class until the fix lands on main (closure waits for the c5640ccc class).
- oom01 fake-OOM (root cause identified): the direct-reclaim leg landed (the page_alloc slowpath's deep final round, 39a515a5), but the kills fire from the kswapd escalation path (16 consecutive zero-reclaim balance_pgdat passes β out_of_memory), which has no deep drain. Evidence: OOM-FORENSIC shows in_direct_reclaim=1 with 172 MB of clean page cache one list away and zero OOM-DEEP-PASS lines all round; the control side improved nothing. The s17 design (one deep drain at the escalation point, reset the failure counter on success) awaits the mm/perf owner.
- The W3 stall class: oom01 SIGKILL delivered, then serial silence β the s17 shape, owned by the W3/OH work lane.
- monkey-b / memstorm instance status: monkey-b x86 rounds all WARN with zero critical flags, margin growth 141β178 MB within criterion; memstorm r34 PREEMPT-UNDERFLOW=8 (the trio family band 6β10), beats 86 vs the 91β97 band β under continued observation before adjudication.
Β§4Fleet queue
- fsfuzz accumulation; a closure round once the fix class reaches main;
- the kswapd deep-drain leg handed to mm/perf;
- census r20+ round-over-round stability + the monkey-b big-heap watch;
- the memstorm beat-band verdict at r35/r36.