Stress testing (the standing stress-fleet cluster)

High load + randomized testing across a concurrent QEMU cluster, catching every anomaly class β€” design doc docs/test/stress-fleet.md

The standing stress cluster targets: wild writes (armed capture on the hunt chains), panics/hangs, leaks, fake-OOM recurrences, scheduler pathologies, perf-under-load drift (vs docs/development/perf-gate-baseline.md), and data corruption. Probes live in test/stressng and test/cowrace.

Β§1Cluster composition

MemberKindLoad
chains Γ—2wild-write hunt chainsself-advancing hunt kernel + load chains (frozen, never restarted)
ltp-a/b/c/dLTP sweep instancesriscv64 TCG smp2/2G, random-start full LTP sweeps
monkey-a/brandomized-testing instancesx86_64 KVM 4G, usbless OH image + randomized churn (storage hotplug, hilog flood/rotate, appspawn kill-restart, bm/hidumper/param walks, sync)
memstormmemory stormmmap/munmap/MAP_FIXED/mprotect/fork storms, dfx=periodic,memwatch,watchdog
fsfuzzfilesystem fuzzingrandom-syscall fuzzer + file/pipe/socket/madvise/fork sequences
ltpmcontrol memberLTP sweeps on the current main kernel (worktree-vs-main drift control)
  • Round mechanics: 1,800 s per round, serial to file, a classification line (decisive CRITICAL > WARN > WEDGE > clean); wedge rule: no serial growth for β‰₯300 s β†’ kill, mark WEDGE, keep the tail.
  • Host budget discipline: adaptive fleet size, a hard floor of β‰₯10 GB free host memory (checked via free before every launch/ramp); QEMU launches niced +10; shared images only -snapshot/snapshot=on + file.locking=off; writable images are private reflink copies.
  • Standing shape: 2 chains + 9 members = 11 QEMUs, guarded by a 30-minute central sweeper.

Β§2Root-caused classes (fleet wins)

  • The SIGILL family (kernel bug, fixed): read_counter() read the time CSR with the write-flavored form csrrw a0, time, zero β€” the whole instruction is illegal-instruction-prone; the mount hammer loop hit it within ~100 s on every build (toolchain zcb exonerated by paired trials). Fixed with the canonical read form csrr.
  • Seven user-length round-up overflows (fixed): fsfuzz r172 drove madvise length=βˆ’1 into attempt to add with overflow (syscall/memory.rs); all seven user-length round-ups (madvise/mprotect/msync/mremap/mincore/mlock/munlock) gained wrap checks (EINVAL, Linux end-wrap semantics); post-fix the deterministic seed stream ran 150 s+ with zero panics (v4 died within 60 s).
  • The zero-serve family (excluded): cluster LTP sweeps were round-for-round identical across the virtio/bio completion rewrite (1253/1254 TPASS both rounds) β€” the rewrite cost nothing on that surface.

Β§3Known classes under watch

  • The fsfuzz WATCHDOG family (a perf-registry static+heap lock pair): r282 recorded zero DEADLOCK prints (a first for this home, 45,858 lines), r283 returned 118 β€” same family, same ra, the static lock moved with the new binary; adjudicated as a known transient class until the fix lands on main (closure waits for the c5640ccc class).
  • oom01 fake-OOM (root cause identified): the direct-reclaim leg landed (the page_alloc slowpath's deep final round, 39a515a5), but the kills fire from the kswapd escalation path (16 consecutive zero-reclaim balance_pgdat passes β†’ out_of_memory), which has no deep drain. Evidence: OOM-FORENSIC shows in_direct_reclaim=1 with 172 MB of clean page cache one list away and zero OOM-DEEP-PASS lines all round; the control side improved nothing. The s17 design (one deep drain at the escalation point, reset the failure counter on success) awaits the mm/perf owner.
  • The W3 stall class: oom01 SIGKILL delivered, then serial silence β€” the s17 shape, owned by the W3/OH work lane.
  • monkey-b / memstorm instance status: monkey-b x86 rounds all WARN with zero critical flags, margin growth 141–178 MB within criterion; memstorm r34 PREEMPT-UNDERFLOW=8 (the trio family band 6–10), beats 86 vs the 91–97 band β€” under continued observation before adjudication.

Β§4Fleet queue

  • fsfuzz accumulation; a closure round once the fix class reaches main;
  • the kswapd deep-drain leg handed to mm/perf;
  • census r20+ round-over-round stability + the monkey-b big-heap watch;
  • the memstorm beat-band verdict at r35/r36.