Debugging

The debugging hub — every major debugging feature has its own design-and-usage page; this page gives the overview, the legacy-switch quick table and the symptom routes

§1The section map

PageContent
Tracepointsthe unified trace-event system (D-15): the single declaration table, 69 events/17 groups, a byte-exact tracefs, two-level gating, the cost model, the dfx= alias mapping, how to add an event
Performance debuggingthe perf_event_open surface (syscall 241), the syscost census + the IF=0 EFLAGS-pair probe, pbench P1-P7, the perf-gate tiers and six guarded domains, fsw
Lock debuggingfour defense lines: typed lock keys (compile time)/the spin watchdog/SPIN·Kani·loom model checking/hunt-wedge automated capture; the lockdep-lite status stated honestly
Memory forensicsthe ledger master ledger, mmforensics, memwatch/allocwatch, the OOM forensic block, the USEGV family, scribble, hung_task/softlockup/sysrq, offline forensics
Panic & symbolsthe panic report structure, the compile-time symbol table D-3, oops forms, line-atomic dumps, REMOTE-BT, the symbolization workflow

§2Kernel logging

  • The serial port is the primary log (console=ttyS0): everything prints to the UART during boot; after the scheduler starts the default level drops to emerg, and dmesg -n 7 reopens the full console output.
  • Levels 0–7 match Linux; the macros pr_emerg!/pr_err!/pr_warn!/pr_info!/pr_debug! (println! is a pr_info! alias); release builds compile out pr_debug.
  • The 1 MB kmsg ring: read via /proc/kmsg, syslog(2), dmesg; the devkmsg policy (/proc/sys/kernel/printk_devkmsg: on|ratelimit|off) only decides how much the CONSOLE shows — the ring always records everything (our alignment with Linux's devkmsg semantics).
  • Persistent log (default off): panic logs survive a reboot in /var/log/kmsg; after a crash, extract offline with debugfs (see memory forensics §8).

§3The trace system (summary)

All new observability goes through the unified tracepoints (the observability policy, 2026-11-02). Currently 69 events / 17 groups; the tracefs tree matches Linux file-for-file and byte-for-byte; disabled sites are one load+test+branch, enabled +13–19 ns/record, the image +0.18%. Full design and usage on Tracepoints.

§4The dfx= runtime switches (the frozen legacy surface, 49 entries)

The boot parameter is dfx=switch1,switch2; the authoritative table is the dfx_switch_table! in kernel/src/dfx/switches.rs (49 entries, frozen and not growing — new observability goes to tracepoints); debugfs mirrors the same table at /sys/kernel/debug/dfx/; most tokens have migrated into arming aliases of rux: events (the mapping on Tracepoints §4). High-frequency items:

SwitchPurpose
watchdogspinlock deadlock warnings + a whole-task snapshot (SBI-direct, independent of printk)
taskdumpperiodic/on-demand task snapshot hooks
memwatchper-call-site heap/page accounting (leak hunting; memory forensics §3)
ledgerthe physical-page master ledger — true exhaustion vs accounting drift (memory forensics §1)
mmforensicsthe PTE install ledger, double-teardown detection, page-table stamp rings (memory forensics §2; costs on fork/exit — investigation only)
scribble / scribbleparkbyte-wise control-word scribble hunting; parks on hit, pairs with a gdb watchpoint (memory forensics §6)
steptrace / writetraceinstruction stepping / write-path per-event tracing (steptrace once wrote a 2.7 GB serial log in one boot — the cost is stated as such)
sysforensic / sysringper-syscall forensics (dual recording, failure ring, comm scan) — must stay off in production (ungated, p1_null paid 12 atomics/syscall)
ohhuntthe OH graphics probe (drm ioctl/mmap/prime-import; now trace-migrated)
claiminject-classdeliberate wedge reproducers — never in production builds

Compile-time features (Cargo.toml)

cargo build --target riscv64gc-unknown-none-elf --features riscv64,dfx-lock-owner
# dfx-lock-owner: one atomic store per lock/unlock; deadlock warnings carry holder=<cpu> + REMOTE-BT stack capture

§5GDB and QEMU-level debugging

  • GDB: make debug (after make build; attaches to QEMU's gdbstub — breakpoints/single-step/memory).
  • Deterministic reproduction: THREAD=single make run (single-threaded TCG); or ACCEL=tcg to disable KVM.
  • The QEMU monitor: attach via a unix socket, screendump, register/memory dumps (hunt-wedge uses this automatically).
  • Serial to file: -serial file:xxx.log for offline grepping after long soaks.

§6Symptom routes

SymptomRoute
Kernel hang, no outputTHREAD=single reproduction → dfx=watchdog deadlock warnings → hunt-wedge automated capture → addr2line symbolization
Kernel page fault / illegal instructionread the panic block (panic §1): scause classifies, stval addresses → symbolize the Call trace → add mmforensics for scene
User program crashthe USEGV-L line names pid/rip/cr2 → USEGV-FOR attributes it (memory forensics §5)
Filesystem problemsthe fsw counters for new sync-write classes → debugfs offline log extraction → e2fsck -f for integrity
Suspected leak / OOM in doubtmemory forensics: memwatch accounting → the ledger decides true exhaustion vs drift
Lock contention/orderinglock debugging: lock: events for latency → key the pair at compile time → the watchdog holder line cross-locates
Performance regressionperformance debugging: the perf-gate.sh three tiers → syscost bands → pbench locates the group → trace captures the micro-sequence
Canonical repo docs: docs/guides/debugging.md, docs/architecture/lock-ordering.md, docs/development/perf-gate-baseline.md, kernel/src/dfx/ and kernel/src/trace/.