Performance debugging
§1perf_event_open (syscall 241)
Rux implements Linux's perf_event_open ABI (asm-generic number 241), software events:
- Event types: CPU_CLOCK / TASK_CLOCK / PAGE_FAULTS / CONTEXT_SWITCHES; counting and sampling modes.
- Sampling mode: an mmap ring emits records with the field order ID/IDENTIFIER/IP/TID/TIME/ID/STREAM_ID/CPU/PERIOD/CALLCHAIN matching Linux; sideband records COMM/FORK/EXIT/MMAP2/LOST; the ioctl set; the
/proc/sys/kernel/perf_event_paranoidgate (default 2). - Sampling granularity (a recorded divergence): software sampling rides the tick — a
sample_periodbelow one tick (10 ms at 100 Hz) coalesces to tick-rate samples; Linux's per-event hrtimer swevent is the follow-up. Sufficient for hotspot profiles (second-scale windows); insufficient for per-call precise profiling — the use boundary is stated as such. - How it was validated: building Linux's own profiler against this ABI exposed parity gaps no synthetic test had covered (inherit semantics, ring-fd O_RDWR, MISC cpumode byte order), then delivered the campaign's hotspot tables (80,961 → 244,516 samples, kernel 1-5 / user 2-16+ frame chains) — an ABI is validated by a real consumer.
Usage
# inside the guest (perf_event_paranoid=2 default: same-process-group counting + own-user sampling)
# counting: perf_event_open(attr.disabled=1, type=PERF_TYPE_SOFTWARE,
# config=PERF_COUNT_SW_CPU_CLOCK) → ioctl ENABLE → read the mmap counter slot → ioctl DISABLE
# sampling: attr.sample_period=N + the mmap ring → read PERF_RECORD_SAMPLE(IP, CALLCHAIN)
cat /proc/sys/kernel/perf_event_paranoid
§2syscost: the boot census collector
- Arming: the kernel cmdline
syscost=1; zero cost when off (one cached atomic load per syscall exit; two relaxed adds in the tick hooks). It consumes no dfx= switch — the probe owns its own cmdline key. - What it measures: per-native-syscall-number {count, wall cycles, max} (NR_SLOTS=512) + a slow-op ring of >4 ms outliers with the caller comm (SLOW_RING=64 / SLOW_CYC=40000 ticks) + tick-mode per-CPU user/kernel-task/idle counters + a 256-slot kernel-text page histogram (RIP_SLOTS=256, symbolized offline against the built ELF's nm output); dumps every 60 s from the timer softirq (SYSCOST_DUMP_SECS=60).
- The IF=0 EFLAGS-pair probe (v3): edge-triggered + self-healing; every hook reads the real IF flag and adjudicates the model against it — a modeled-open segment found with IF=1 is a leak, closed into HEAL; the HARDIRQ/SOFTIRQ/UART zones split separately; REST call sites attribute via #[track_caller] into a 96-slot table; segments carry raw TSC stamps. This is the official IF=0-share measurement (Tier 3), not a proxy.
- Gate readings (Tier 1, a quiet syscost boot): census TERM-DESCHED 8 (band 8±2) / WAKE-REFUSED 32 (band 32±4); tick ≥99.5/s; zero panics/OOM; syscall volume by profile class (debug 15.5M ±80%, release 50M ±60%); clock_gettime 3-4 µs; nanosleep ≤244 ms.
§3pbench: the seven microbenchmark groups
test/pbench.c: one static musl x86_64 binary that runs identically on Rux and the same-test platform Linux — every number is a same-binary pair. Runs as init (init=/test/pbench) for quiet probe boots. Timing: per-op rdtsc deltas calibrated against CLOCK_MONOTONIC_RAW; the output line is PBENCH <name> n=<N> med=<us> p99=<us> min=<us> max=<us> avg=<us> wall=<us>, terminated by PBENCH DONE rc=0; unsupported primitives SKIP themselves.
| Group | Measures | Protocol notes |
|---|---|---|
| P1 | p1_null (SYS_getuid loop N=1e6) / p1_clock | the empty-syscall and clock-read baselines |
| P2 | p2_ctxsw | 1-byte pipe ping-pong pinned to CPU0/CPU1, N=1e5, half roundtrip (one switch) |
| P3 | p3a_fault_anon (64MB) / p3b_fault_cow (32MB) / p3c_fault_file (16MB) | the three fault classes |
| P4 | p4_fork_exit / p4_fork_exec (N=1e3) | the exec target is a self re-exec |
| P5 | p5a_pipe / p5b_dgram+recv_peek / p5c_stream_write / p5e-p5f flow-control discriminators | 74 B (the hilogd protocol shape); p5a N=1e5, p5b N=2e4 |
| P6 | p6_stat/openclose/getdents/read_cached/write_fsync/append/mkdrmdir/unlink | a 1000-file × 4KB tree |
| P7 | p7_sleep1ms | nanosleep 1 ms ×1e4, recording OVERSHOOT |
The gate anchor column (pb1, KVM debug class): p1_null 0.300 / p1_clock 0.371 / p2_ctxsw 1.242 / p3b_cow 8.075 / p6_write_fsync 8442.6 / p6_mkdrmdir 216.7 / p6_unlink 54.6 / p6_append 5.19 µs; the Tier-2 criterion is median ±20% or improved. The p2_ctxsw anchor is declared stale (the pre-nohz parent already drifted +60%; a re-measurement is on record).
§4The performance gate: three tiers and guarded domains
Merge gating runs in three tiers (docs/development/perf-gate-baseline.md is the authoritative table; perf-gate.sh automates it):
- Tier 1 (all kernel changes): one quiet syscost boot (KVM q35 smp2/4G, syscost=1, 780 s) — census exact, tick rate, zero panics/OOM, volume band, per-op average bands.
- Tier 2 (hot-path changes: mm/sched/fs/ipc/binder/timer/arch trap): one pbench boot added; P1-P7 medians ±20% or improved; the profile-class rule: cross-class comparison is meaningless (debug vs release-strong).
- Tier 3 (allocator or IRQ-path changes): the IF=0 share (the official EFLAGS-pair measurement, 20.4 ±3pp).
| Guarded domain | Key rows |
|---|---|
| Allocation/reclaim | small_alloc global class-lock sections ≤25M/boot; REST share ≤10%; no reclaim slowpath emergence (drct=0 class) |
| Scheduling | GRQ average hold 2786 ns; census 8±2/32±4; p2 ±20%; the p4 fork open regression on the watch row |
| Filesystem | stat/openclose/read_cached/append/mkdrmdir/unlink median bands; buffer hit rate ≥99.1% |
| Interrupts/IRQ | IF=0 20.4±3pp; max cli REST ≤10 ms; SOFTIRQ 3.1±2pp; no new IRQ-OFF print classes |
| IPC | the p5 band table: 18.3 / 23–41 / 24–27 / 25 / 41–46 µs |
| binder | the SLOW-TXN class: queue ≤10 ms everywhere; {1,64} exec-side 4.5-9 s keyed bands; FAIL at queue ≥50 ms |
§5Specialized counters
- fsw (cmdline
fsw=1): the fs sync-write-class census — an FSW1 row (br/rd/wr/sd/ev/sa/pb/ps/pc/ph/pb2/pm/lvl/es) + an FSW2 row (ri/wi/lk/ls/ni/ar/fr/j=jstart.jstop.jlazy.jcommit_fast.jrevoke/bl/biglock/w walks.hits.misses); dumped at :30 past each minute (avoiding the console-emit race with syscost). The fs gate reads "no new sync-write class outside the provisioning phase". - mmprof / PageCacheOps: memory accounting cross-checks and the page-cache hit rate (a meminfo row), inputs to OOM forensics.
- census / wakewatch: the scheduling-stall census and wake-to-run latency, belonging to the scheduling domain (see scheduler §8).
§6The standard workflow
- Detect: a per-op average leaving its band in the Tier-1 syscost dump → locate the syscall number or census class;
- Attribute: pbench locates the group (P2 switch / P5 IPC / P6 fs) → the syscost RIP histogram or REST table names the segment → trace events (rux:schedtrace, lock:) capture the micro-sequence;
- Fix and verify: same-test platform interleaved A/B (≥3, median) → perf-gate.sh three-tier verdict → the baseline table refresh rides the merge.
Tool philosophy: every gated domain has a corresponding measurement in the tool set; if a change affects system performance and no existing tool can measure it, adding the DFX tool for it is part of the change, not an afterthought (the merge-gate policy). All new observability goes through tracepoints (see tracepoints).
kernel/src/perf/mod.rs, kernel/src/dfx/syscost.rs, test/pbench.c. Authoritative table: docs/development/perf-gate-baseline.md (three tiers + six guarded domains + all bands); automation: perf-gate.sh (worktree run/).