Debugging
The debugging hub — every major debugging feature has its own design-and-usage page; this page gives the overview, the legacy-switch quick table and the symptom routes
§1The section map
| Page | Content |
|---|---|
| Tracepoints | the unified trace-event system (D-15): the single declaration table, 69 events/17 groups, a byte-exact tracefs, two-level gating, the cost model, the dfx= alias mapping, how to add an event |
| Performance debugging | the perf_event_open surface (syscall 241), the syscost census + the IF=0 EFLAGS-pair probe, pbench P1-P7, the perf-gate tiers and six guarded domains, fsw |
| Lock debugging | four defense lines: typed lock keys (compile time)/the spin watchdog/SPIN·Kani·loom model checking/hunt-wedge automated capture; the lockdep-lite status stated honestly |
| Memory forensics | the ledger master ledger, mmforensics, memwatch/allocwatch, the OOM forensic block, the USEGV family, scribble, hung_task/softlockup/sysrq, offline forensics |
| Panic & symbols | the panic report structure, the compile-time symbol table D-3, oops forms, line-atomic dumps, REMOTE-BT, the symbolization workflow |
§2Kernel logging
- The serial port is the primary log (
console=ttyS0): everything prints to the UART during boot; after the scheduler starts the default level drops to emerg, anddmesg -n 7reopens the full console output. - Levels 0–7 match Linux; the macros pr_emerg!/pr_err!/pr_warn!/pr_info!/pr_debug! (println! is a pr_info! alias); release builds compile out pr_debug.
- The 1 MB kmsg ring: read via /proc/kmsg, syslog(2), dmesg; the devkmsg policy (/proc/sys/kernel/printk_devkmsg: on|ratelimit|off) only decides how much the CONSOLE shows — the ring always records everything (our alignment with Linux's devkmsg semantics).
- Persistent log (default off): panic logs survive a reboot in /var/log/kmsg; after a crash, extract offline with debugfs (see memory forensics §8).
§3The trace system (summary)
All new observability goes through the unified tracepoints (the observability policy, 2026-11-02). Currently 69 events / 17 groups; the tracefs tree matches Linux file-for-file and byte-for-byte; disabled sites are one load+test+branch, enabled +13–19 ns/record, the image +0.18%. Full design and usage on Tracepoints.
§4The dfx= runtime switches (the frozen legacy surface, 49 entries)
The boot parameter is dfx=switch1,switch2; the authoritative table is the dfx_switch_table! in kernel/src/dfx/switches.rs (49 entries, frozen and not growing — new observability goes to tracepoints); debugfs mirrors the same table at /sys/kernel/debug/dfx/; most tokens have migrated into arming aliases of rux: events (the mapping on Tracepoints §4). High-frequency items:
| Switch | Purpose |
|---|---|
| watchdog | spinlock deadlock warnings + a whole-task snapshot (SBI-direct, independent of printk) |
| taskdump | periodic/on-demand task snapshot hooks |
| memwatch | per-call-site heap/page accounting (leak hunting; memory forensics §3) |
| ledger | the physical-page master ledger — true exhaustion vs accounting drift (memory forensics §1) |
| mmforensics | the PTE install ledger, double-teardown detection, page-table stamp rings (memory forensics §2; costs on fork/exit — investigation only) |
| scribble / scribblepark | byte-wise control-word scribble hunting; parks on hit, pairs with a gdb watchpoint (memory forensics §6) |
| steptrace / writetrace | instruction stepping / write-path per-event tracing (steptrace once wrote a 2.7 GB serial log in one boot — the cost is stated as such) |
| sysforensic / sysring | per-syscall forensics (dual recording, failure ring, comm scan) — must stay off in production (ungated, p1_null paid 12 atomics/syscall) |
| ohhunt | the OH graphics probe (drm ioctl/mmap/prime-import; now trace-migrated) |
| claiminject-class | deliberate wedge reproducers — never in production builds |
Compile-time features (Cargo.toml)
cargo build --target riscv64gc-unknown-none-elf --features riscv64,dfx-lock-owner
# dfx-lock-owner: one atomic store per lock/unlock; deadlock warnings carry holder=<cpu> + REMOTE-BT stack capture
§5GDB and QEMU-level debugging
- GDB:
make debug(after make build; attaches to QEMU's gdbstub — breakpoints/single-step/memory). - Deterministic reproduction:
THREAD=single make run(single-threaded TCG); orACCEL=tcgto disable KVM. - The QEMU monitor: attach via a unix socket,
screendump, register/memory dumps (hunt-wedge uses this automatically). - Serial to file: -serial file:xxx.log for offline grepping after long soaks.
§6Symptom routes
| Symptom | Route |
|---|---|
| Kernel hang, no output | THREAD=single reproduction → dfx=watchdog deadlock warnings → hunt-wedge automated capture → addr2line symbolization |
| Kernel page fault / illegal instruction | read the panic block (panic §1): scause classifies, stval addresses → symbolize the Call trace → add mmforensics for scene |
| User program crash | the USEGV-L line names pid/rip/cr2 → USEGV-FOR attributes it (memory forensics §5) |
| Filesystem problems | the fsw counters for new sync-write classes → debugfs offline log extraction → e2fsck -f for integrity |
| Suspected leak / OOM in doubt | memory forensics: memwatch accounting → the ledger decides true exhaustion vs drift |
| Lock contention/ordering | lock debugging: lock: events for latency → key the pair at compile time → the watchdog holder line cross-locates |
| Performance regression | performance debugging: the perf-gate.sh three tiers → syscost bands → pbench locates the group → trace captures the micro-sequence |
Canonical repo docs: docs/guides/debugging.md, docs/architecture/lock-ordering.md, docs/development/perf-gate-baseline.md, kernel/src/dfx/ and kernel/src/trace/.