Tracepoints

kernel/src/trace/ Β· The unified trace-event system: a single declaration table derives the full event, a byte-exact tracefs, two-level gating and per-CPU rings β€” design, the complete event inventory and usage Β· Observability policy (2026-11-02): all new kernel observability enters through this framework

Β§1Design: one declaration row derives everything

Each event appears exactly once in the tracepoints! declaration table β€” one row declares the event's full identity: group/name/id/verbatim print fmt/field list/typed call signature/renderer. From that single row the macro derives the three artifacts a hand-written site used to duplicate:

  1. The static table row (group/name/id/print_fmt/fields/enabled) β€” the table is the catalog;
  2. The record function trace_<name>(field, ...) -> bool β€” typed arguments pack into argument slots with the armed check inside (when the event is off, the call site pays only argument evaluation);
  3. The renderer <name>_fmt β€” shared by the text output and the format file.

Design points:

  • The print fmt is captured verbatim from the Linux host's live files β€” the text format and field offsets are byte-identical to Linux's, so any host tool (event analyzers parsing the format files) works unmodified.
  • One declaration = one edit: the table row, the call signature and the file shapes are a single edit β€” the "format and data drift" defect class is structurally unrepresentable (the same derive pattern as procfile! D-4).
  • An argument cap of 10 u64 slots (TRACE_ARGS=10): a comm costs two slots; exceeding it is unrepresentable at compile time (the PackArg trait declares slot usage per type).
  • Two-level gating: recording requires both the event's own enable switch and the global tracing_on; the tracing_on check sits inside record, after argument marshalling.

Β§2Event inventory (69 events / 17 groups)

Group#Events
rux20The rux: own family (Linux has no counterparts): epoll_ctl_insert/epoll_ctl_remove/epoll_wait_scan/epoll_wait_exit/epoll_delivery, futex_miss, fuxstuck, hilog_census/hilog_kmsg/hilog_top, itimer_arm, memwatch, mmforensics, ohhunt, schedtrace, sigtrace, spinwatch/spinwatch_fd, sysring, taskdump, txnhang
sched9sched_switch/sched_waking/sched_wakeup/sched_wakeup_new/sched_migrate_task/sched_process_fork/sched_process_exit/sched_process_free/sched_process_wait
ext47ext4_sync_file_enter/exit, ext4_ext_map_blocks_enter/exit, ext4_da_write_begin/end, ext4_free_blocks
net / binder / input5+5+4net: udp_deliver/udp_wake/lo_xmit/sock_wait/pkt_drop; binder: binder_transaction/binder_transaction_received/binder_transaction_alloc_buf/binder_transaction_buffer_release/binder_wait_for_work; input: input_event/evdev_read/evdev_open/evdev_wake
kmem4kmalloc/kfree/mm_page_alloc/mm_page_free
raw_syscalls / irq / exceptions / block / jbd22+2+2+2+2raw_syscalls: sys_enter/sys_exit; irq: irq_handler_entry/exit; exceptions: page_fault_user/kernel; block: block_rq_issue/complete; jbd2: jbd2_start_commit/jbd2_end_commit
vmscan / timer / signal / printk / oom1 eachmm_vmscan_lru_shrink_inactive, itimer_arm (counted under rux), signal_generate, printk:console, oom:mark_victim

The rux: family is a registered divergence shape: Linux has no epoll/input-specific events; it is an own-event family β€” semantically aligned with Linux's event model (group/name/id/format), with content specific to this kernel.

Β§3The tracefs tree and usage

Once mounted, the file tree matches Linux file-for-file (standard names, standard modes, byte formats captured from the live host's /sys/kernel/tracing):

mount -t tracefs none /sys/kernel/tracing
cd /sys/kernel/tracing
ls
# tracing_on  trace  trace_pipe  available_tracers  current_tracer
# available_events  set_event  buffer_size_kb  snapshot  options/
# events/<group>/{enable,filter,<event>/{enable,filter,id,format}}

cat events/sched/sched_wakeup/format     # field list + print fmt (byte-exact vs Linux)
cat events/sched/sched_wakeup/id         # the event id

A typical session

echo 1 > tracing_on                                # global master switch (the second gating level)
echo 1 > events/sched/sched_switch/enable          # arm one event
echo 'sched:*' > set_event                          # arm by group (wildcards and negation !event)
echo 'ext4:* jbd2:*' > set_event                    # multiple groups
cat trace                                           # read the buffer (non-consuming; write 1 to trace to clear)
cat trace_pipe                                      # consuming read, no header
echo 4096 > buffer_size_kb                          # per-CPU ring size (KiB)
echo 1 > snapshot                                   # archive the live ring and restart it; read it back with cat snapshot
echo 0 > tracing_on                                 # stop recording (arming retained)

Output line format

%16s-%-7d [%03d] %c%c%c%c%c %5lu.%06lu: %s: %s
#      task      pid  cpu  irq flags     timestamp        group:  content
# e.g. launcher-327  [002] d..1 1234.567890: sched:sched_switch: ...

Rings and consistency

  • Per-CPU rings with per-CPU sequence numbers and read-side timestamp merging β€” tail latency p99 1.403β†’0.741 Β΅s (βˆ’47%; the observability's own cost is measured).
  • A 600 s fully-armed soak survived 1.56M ring overwrites with zero panics β€” the ring-full overwrite path is verified.

Β§3bCost model

StateCostVerification
Event disabledOne load + test + branch (the call site pays only argument evaluation)Per-site release disassembly verification (all 39 kernel sites one-load: movzbl+test+jcc)
Event enabled+13–19 ns/record (pure-write loop); +28–35 ns (epoll ping-pong context)On/off A/B microbenchmarks
Image cost+24,576 bytes (+0.182%)Build size comparison
Soak600 s fully armed, zero panics1.56M ring overwrites verified

Gate criterion: the presence of trace sites does not move the gate bands β€” the armed-off state is an unreachable path at the disassembly level; census and pbench anchors are measured on un-armed configurations.

Β§4Mapping the legacy dfx= switches to events

The old ad-hoc switch table is frozen (see the debugging guide); its instrumentation migrated into trace events, and the boot parameter dfx=<name> is now an alias arming the corresponding event family:

dfx= aliasrux: events armed
dfx=taskdumprux:taskdump (periodic task snapshots)
dfx=watchdogspinlock warnings + rux:taskdump linkage
dfx=memwatchrux:memwatch (allocation-site accounting)
dfx=mmforensicsrux:mmforensics (PTE ledger/double-teardown detection)
dfx=schedtracerux:schedtrace (scheduler stepping)
dfx=sysringrux:sysring (syscall dual-recording)
dfx=sigtracerux:sigtrace (signal chains)
dfx=ohhuntrux:ohhunt (drm ioctl/mmap/prime-import)
dfx=fuxstuck / txnhang / spinwatch / hilogcensusthe corresponding rux: events (hunt-specific observation points)

The complete retired-token table and the mapping implementation live in kernel/src/dfx/switches.rs (the trace alias table); /sys/kernel/debug/dfx/ mirrors the same control surface in debugfs.

Β§5How to add an event

  1. Add one row to the tracepoints! table in kernel/src/trace/mod.rs: group/name/id/print fmt (captured verbatim from Linux's live files, or self-declared for the rux: family)/field list/call signature/renderer.
  2. Call trace_<name>(args...) at the site β€” no hand-written armed checks, packing or rendering.
  3. Guest-side verification: cat events/<group>/<name>/format must be byte-identical to the Linux format (Linux families), or self-consistent with the declaration (rux: family).
  4. Cost verification: confirm one-load call sites in the release disassembly; an on/off A/B microbenchmark for the enabled cost.

Policy basis: the observability policy (2026-11-02) β€” the old dfx= ad-hoc switch table is frozen at 49 entries and must not grow; any new observation point must enter this framework as a tracepoint. This is an interface decision aligned with Linux (the debugfs/tracefs ecosystem works directly), not an internal preference.

Code: kernel/src/trace/mod.rs (the macro + declaration table), kernel/src/trace/files.rs (the tracefs tree). Related: performance debugging (syscost/pbench), lock debugging (lock: observation), the debugging guide (the legacy switch table).