Performance debugging

The perf_event_open syscall surface, the syscost boot census, the pbench microbenchmarks, the IF=0 probe and the performance gate — each tool's design, data shape and usage protocol · Sources: kernel/src/perf/, kernel/src/dfx/syscost.rs, test/pbench.c, docs/development/perf-gate-baseline.md

§1perf_event_open (syscall 241)

Rux implements Linux's perf_event_open ABI (asm-generic number 241), software events:

  • Event types: CPU_CLOCK / TASK_CLOCK / PAGE_FAULTS / CONTEXT_SWITCHES; counting and sampling modes.
  • Sampling mode: an mmap ring emits records with the field order ID/IDENTIFIER/IP/TID/TIME/ID/STREAM_ID/CPU/PERIOD/CALLCHAIN matching Linux; sideband records COMM/FORK/EXIT/MMAP2/LOST; the ioctl set; the /proc/sys/kernel/perf_event_paranoid gate (default 2).
  • Sampling granularity (a recorded divergence): software sampling rides the tick — a sample_period below one tick (10 ms at 100 Hz) coalesces to tick-rate samples; Linux's per-event hrtimer swevent is the follow-up. Sufficient for hotspot profiles (second-scale windows); insufficient for per-call precise profiling — the use boundary is stated as such.
  • How it was validated: building Linux's own profiler against this ABI exposed parity gaps no synthetic test had covered (inherit semantics, ring-fd O_RDWR, MISC cpumode byte order), then delivered the campaign's hotspot tables (80,961 → 244,516 samples, kernel 1-5 / user 2-16+ frame chains) — an ABI is validated by a real consumer.

Usage

# inside the guest (perf_event_paranoid=2 default: same-process-group counting + own-user sampling)
# counting: perf_event_open(attr.disabled=1, type=PERF_TYPE_SOFTWARE,
#   config=PERF_COUNT_SW_CPU_CLOCK) → ioctl ENABLE → read the mmap counter slot → ioctl DISABLE
# sampling: attr.sample_period=N + the mmap ring → read PERF_RECORD_SAMPLE(IP, CALLCHAIN)
cat /proc/sys/kernel/perf_event_paranoid

§2syscost: the boot census collector

  • Arming: the kernel cmdline syscost=1; zero cost when off (one cached atomic load per syscall exit; two relaxed adds in the tick hooks). It consumes no dfx= switch — the probe owns its own cmdline key.
  • What it measures: per-native-syscall-number {count, wall cycles, max} (NR_SLOTS=512) + a slow-op ring of >4 ms outliers with the caller comm (SLOW_RING=64 / SLOW_CYC=40000 ticks) + tick-mode per-CPU user/kernel-task/idle counters + a 256-slot kernel-text page histogram (RIP_SLOTS=256, symbolized offline against the built ELF's nm output); dumps every 60 s from the timer softirq (SYSCOST_DUMP_SECS=60).
  • The IF=0 EFLAGS-pair probe (v3): edge-triggered + self-healing; every hook reads the real IF flag and adjudicates the model against it — a modeled-open segment found with IF=1 is a leak, closed into HEAL; the HARDIRQ/SOFTIRQ/UART zones split separately; REST call sites attribute via #[track_caller] into a 96-slot table; segments carry raw TSC stamps. This is the official IF=0-share measurement (Tier 3), not a proxy.
  • Gate readings (Tier 1, a quiet syscost boot): census TERM-DESCHED 8 (band 8±2) / WAKE-REFUSED 32 (band 32±4); tick ≥99.5/s; zero panics/OOM; syscall volume by profile class (debug 15.5M ±80%, release 50M ±60%); clock_gettime 3-4 µs; nanosleep ≤244 ms.

§3pbench: the seven microbenchmark groups

test/pbench.c: one static musl x86_64 binary that runs identically on Rux and the same-test platform Linux — every number is a same-binary pair. Runs as init (init=/test/pbench) for quiet probe boots. Timing: per-op rdtsc deltas calibrated against CLOCK_MONOTONIC_RAW; the output line is PBENCH <name> n=<N> med=<us> p99=<us> min=<us> max=<us> avg=<us> wall=<us>, terminated by PBENCH DONE rc=0; unsupported primitives SKIP themselves.

GroupMeasuresProtocol notes
P1p1_null (SYS_getuid loop N=1e6) / p1_clockthe empty-syscall and clock-read baselines
P2p2_ctxsw1-byte pipe ping-pong pinned to CPU0/CPU1, N=1e5, half roundtrip (one switch)
P3p3a_fault_anon (64MB) / p3b_fault_cow (32MB) / p3c_fault_file (16MB)the three fault classes
P4p4_fork_exit / p4_fork_exec (N=1e3)the exec target is a self re-exec
P5p5a_pipe / p5b_dgram+recv_peek / p5c_stream_write / p5e-p5f flow-control discriminators74 B (the hilogd protocol shape); p5a N=1e5, p5b N=2e4
P6p6_stat/openclose/getdents/read_cached/write_fsync/append/mkdrmdir/unlinka 1000-file × 4KB tree
P7p7_sleep1msnanosleep 1 ms ×1e4, recording OVERSHOOT

The gate anchor column (pb1, KVM debug class): p1_null 0.300 / p1_clock 0.371 / p2_ctxsw 1.242 / p3b_cow 8.075 / p6_write_fsync 8442.6 / p6_mkdrmdir 216.7 / p6_unlink 54.6 / p6_append 5.19 µs; the Tier-2 criterion is median ±20% or improved. The p2_ctxsw anchor is declared stale (the pre-nohz parent already drifted +60%; a re-measurement is on record).

§4The performance gate: three tiers and guarded domains

Merge gating runs in three tiers (docs/development/perf-gate-baseline.md is the authoritative table; perf-gate.sh automates it):

  • Tier 1 (all kernel changes): one quiet syscost boot (KVM q35 smp2/4G, syscost=1, 780 s) — census exact, tick rate, zero panics/OOM, volume band, per-op average bands.
  • Tier 2 (hot-path changes: mm/sched/fs/ipc/binder/timer/arch trap): one pbench boot added; P1-P7 medians ±20% or improved; the profile-class rule: cross-class comparison is meaningless (debug vs release-strong).
  • Tier 3 (allocator or IRQ-path changes): the IF=0 share (the official EFLAGS-pair measurement, 20.4 ±3pp).
Guarded domainKey rows
Allocation/reclaimsmall_alloc global class-lock sections ≤25M/boot; REST share ≤10%; no reclaim slowpath emergence (drct=0 class)
SchedulingGRQ average hold 2786 ns; census 8±2/32±4; p2 ±20%; the p4 fork open regression on the watch row
Filesystemstat/openclose/read_cached/append/mkdrmdir/unlink median bands; buffer hit rate ≥99.1%
Interrupts/IRQIF=0 20.4±3pp; max cli REST ≤10 ms; SOFTIRQ 3.1±2pp; no new IRQ-OFF print classes
IPCthe p5 band table: 18.3 / 23–41 / 24–27 / 25 / 41–46 µs
binderthe SLOW-TXN class: queue ≤10 ms everywhere; {1,64} exec-side 4.5-9 s keyed bands; FAIL at queue ≥50 ms
Comparability rules: both sides of a comparison must use the same accelerator (KVM vs KVM); debug and release-strong never cross classes; TCG p2 readings are dominated by host load; baseline updates happen only via a merge that passed its own gate.

§5Specialized counters

  • fsw (cmdline fsw=1): the fs sync-write-class census — an FSW1 row (br/rd/wr/sd/ev/sa/pb/ps/pc/ph/pb2/pm/lvl/es) + an FSW2 row (ri/wi/lk/ls/ni/ar/fr/j=jstart.jstop.jlazy.jcommit_fast.jrevoke/bl/biglock/w walks.hits.misses); dumped at :30 past each minute (avoiding the console-emit race with syscost). The fs gate reads "no new sync-write class outside the provisioning phase".
  • mmprof / PageCacheOps: memory accounting cross-checks and the page-cache hit rate (a meminfo row), inputs to OOM forensics.
  • census / wakewatch: the scheduling-stall census and wake-to-run latency, belonging to the scheduling domain (see scheduler §8).

§6The standard workflow

  1. Detect: a per-op average leaving its band in the Tier-1 syscost dump → locate the syscall number or census class;
  2. Attribute: pbench locates the group (P2 switch / P5 IPC / P6 fs) → the syscost RIP histogram or REST table names the segment → trace events (rux:schedtrace, lock:) capture the micro-sequence;
  3. Fix and verify: same-test platform interleaved A/B (≥3, median) → perf-gate.sh three-tier verdict → the baseline table refresh rides the merge.

Tool philosophy: every gated domain has a corresponding measurement in the tool set; if a change affects system performance and no existing tool can measure it, adding the DFX tool for it is part of the change, not an afterthought (the merge-gate policy). All new observability goes through tracepoints (see tracepoints).

Code: kernel/src/perf/mod.rs, kernel/src/dfx/syscost.rs, test/pbench.c. Authoritative table: docs/development/perf-gate-baseline.md (three tiers + six guarded domains + all bands); automation: perf-gate.sh (worktree run/).