Tracepoints
Β§1Design: one declaration row derives everything
Each event appears exactly once in the tracepoints! declaration table β one row declares the event's full identity: group/name/id/verbatim print fmt/field list/typed call signature/renderer. From that single row the macro derives the three artifacts a hand-written site used to duplicate:
- The static table row (group/name/id/print_fmt/fields/enabled) β the table is the catalog;
- The record function
trace_<name>(field, ...) -> boolβ typed arguments pack into argument slots with the armed check inside (when the event is off, the call site pays only argument evaluation); - The renderer
<name>_fmtβ shared by the text output and the format file.
Design points:
- The print fmt is captured verbatim from the Linux host's live files β the text format and field offsets are byte-identical to Linux's, so any host tool (event analyzers parsing the format files) works unmodified.
- One declaration = one edit: the table row, the call signature and the file shapes are a single edit β the "format and data drift" defect class is structurally unrepresentable (the same derive pattern as procfile! D-4).
- An argument cap of 10 u64 slots (TRACE_ARGS=10): a comm costs two slots; exceeding it is unrepresentable at compile time (the PackArg trait declares slot usage per type).
- Two-level gating: recording requires both the event's own enable switch and the global tracing_on; the tracing_on check sits inside record, after argument marshalling.
Β§2Event inventory (69 events / 17 groups)
| Group | # | Events |
|---|---|---|
| rux | 20 | The rux: own family (Linux has no counterparts): epoll_ctl_insert/epoll_ctl_remove/epoll_wait_scan/epoll_wait_exit/epoll_delivery, futex_miss, fuxstuck, hilog_census/hilog_kmsg/hilog_top, itimer_arm, memwatch, mmforensics, ohhunt, schedtrace, sigtrace, spinwatch/spinwatch_fd, sysring, taskdump, txnhang |
| sched | 9 | sched_switch/sched_waking/sched_wakeup/sched_wakeup_new/sched_migrate_task/sched_process_fork/sched_process_exit/sched_process_free/sched_process_wait |
| ext4 | 7 | ext4_sync_file_enter/exit, ext4_ext_map_blocks_enter/exit, ext4_da_write_begin/end, ext4_free_blocks |
| net / binder / input | 5+5+4 | net: udp_deliver/udp_wake/lo_xmit/sock_wait/pkt_drop; binder: binder_transaction/binder_transaction_received/binder_transaction_alloc_buf/binder_transaction_buffer_release/binder_wait_for_work; input: input_event/evdev_read/evdev_open/evdev_wake |
| kmem | 4 | kmalloc/kfree/mm_page_alloc/mm_page_free |
| raw_syscalls / irq / exceptions / block / jbd2 | 2+2+2+2+2 | raw_syscalls: sys_enter/sys_exit; irq: irq_handler_entry/exit; exceptions: page_fault_user/kernel; block: block_rq_issue/complete; jbd2: jbd2_start_commit/jbd2_end_commit |
| vmscan / timer / signal / printk / oom | 1 each | mm_vmscan_lru_shrink_inactive, itimer_arm (counted under rux), signal_generate, printk:console, oom:mark_victim |
The rux: family is a registered divergence shape: Linux has no epoll/input-specific events; it is an own-event family β semantically aligned with Linux's event model (group/name/id/format), with content specific to this kernel.
Β§3The tracefs tree and usage
Once mounted, the file tree matches Linux file-for-file (standard names, standard modes, byte formats captured from the live host's /sys/kernel/tracing):
mount -t tracefs none /sys/kernel/tracing
cd /sys/kernel/tracing
ls
# tracing_on trace trace_pipe available_tracers current_tracer
# available_events set_event buffer_size_kb snapshot options/
# events/<group>/{enable,filter,<event>/{enable,filter,id,format}}
cat events/sched/sched_wakeup/format # field list + print fmt (byte-exact vs Linux)
cat events/sched/sched_wakeup/id # the event id
A typical session
echo 1 > tracing_on # global master switch (the second gating level)
echo 1 > events/sched/sched_switch/enable # arm one event
echo 'sched:*' > set_event # arm by group (wildcards and negation !event)
echo 'ext4:* jbd2:*' > set_event # multiple groups
cat trace # read the buffer (non-consuming; write 1 to trace to clear)
cat trace_pipe # consuming read, no header
echo 4096 > buffer_size_kb # per-CPU ring size (KiB)
echo 1 > snapshot # archive the live ring and restart it; read it back with cat snapshot
echo 0 > tracing_on # stop recording (arming retained)
Output line format
%16s-%-7d [%03d] %c%c%c%c%c %5lu.%06lu: %s: %s
# task pid cpu irq flags timestamp group: content
# e.g. launcher-327 [002] d..1 1234.567890: sched:sched_switch: ...
Rings and consistency
- Per-CPU rings with per-CPU sequence numbers and read-side timestamp merging β tail latency p99 1.403β0.741 Β΅s (β47%; the observability's own cost is measured).
- A 600 s fully-armed soak survived 1.56M ring overwrites with zero panics β the ring-full overwrite path is verified.
Β§3bCost model
| State | Cost | Verification |
|---|---|---|
| Event disabled | One load + test + branch (the call site pays only argument evaluation) | Per-site release disassembly verification (all 39 kernel sites one-load: movzbl+test+jcc) |
| Event enabled | +13β19 ns/record (pure-write loop); +28β35 ns (epoll ping-pong context) | On/off A/B microbenchmarks |
| Image cost | +24,576 bytes (+0.182%) | Build size comparison |
| Soak | 600 s fully armed, zero panics | 1.56M ring overwrites verified |
Gate criterion: the presence of trace sites does not move the gate bands β the armed-off state is an unreachable path at the disassembly level; census and pbench anchors are measured on un-armed configurations.
Β§4Mapping the legacy dfx= switches to events
The old ad-hoc switch table is frozen (see the debugging guide); its instrumentation migrated into trace events, and the boot parameter dfx=<name> is now an alias arming the corresponding event family:
| dfx= alias | rux: events armed |
|---|---|
| dfx=taskdump | rux:taskdump (periodic task snapshots) |
| dfx=watchdog | spinlock warnings + rux:taskdump linkage |
| dfx=memwatch | rux:memwatch (allocation-site accounting) |
| dfx=mmforensics | rux:mmforensics (PTE ledger/double-teardown detection) |
| dfx=schedtrace | rux:schedtrace (scheduler stepping) |
| dfx=sysring | rux:sysring (syscall dual-recording) |
| dfx=sigtrace | rux:sigtrace (signal chains) |
| dfx=ohhunt | rux:ohhunt (drm ioctl/mmap/prime-import) |
| dfx=fuxstuck / txnhang / spinwatch / hilogcensus | the corresponding rux: events (hunt-specific observation points) |
The complete retired-token table and the mapping implementation live in kernel/src/dfx/switches.rs (the trace alias table); /sys/kernel/debug/dfx/ mirrors the same control surface in debugfs.
Β§5How to add an event
- Add one row to the
tracepoints!table inkernel/src/trace/mod.rs: group/name/id/print fmt (captured verbatim from Linux's live files, or self-declared for the rux: family)/field list/call signature/renderer. - Call
trace_<name>(args...)at the site β no hand-written armed checks, packing or rendering. - Guest-side verification:
cat events/<group>/<name>/formatmust be byte-identical to the Linux format (Linux families), or self-consistent with the declaration (rux: family). - Cost verification: confirm one-load call sites in the release disassembly; an on/off A/B microbenchmark for the enabled cost.
Policy basis: the observability policy (2026-11-02) β the old dfx= ad-hoc switch table is frozen at 49 entries and must not grow; any new observation point must enter this framework as a tracepoint. This is an interface decision aligned with Linux (the debugfs/tracefs ecosystem works directly), not an internal preference.
kernel/src/trace/mod.rs (the macro + declaration table), kernel/src/trace/files.rs (the tracefs tree). Related: performance debugging (syscost/pbench), lock debugging (lock: observation), the debugging guide (the legacy switch table).