Panic & symbolization

The kernel panic report structure, the compile-time kallsyms symbol table, oops forms and line-atomic dumps β€” the complete path from a serial line to a source line Β· Sources: kernel/src/dfx/ (backtrace/oops/ksyms), arch/*/trap.rs, design-divergences.md (D-3)

Β§1Panic report structure

The panic handler bypasses console log levels and writes the UART directly, then halts with wfi. A complete report contains:

  1. The PANIC message + file:line (panic!'s arguments and source location);
  2. The CSR quartet (riscv64): sstatus / scause / stval / sepc; x86_64's counterparts are cr2/rip/rsp/error code;
  3. All 31 general registers (riscv64) β€” saved by inline asm before any stack unwinding;
  4. The stack backtrace: a frame-pointer chain walk (s0/fp on riscv64, rbp on x86_64), up to 32 frames under Call trace:.
FieldHow to read it
scausebit 63 set = interrupt; exception codes: 0 misaligned fetch, 2 illegal instruction, 8 U-mode ecall, 12 instruction page fault, 13 load fault, 15 store/AMO fault; interrupt codes: 5 timer, 9 external
sepcthe trapping instruction address
stvalon a page fault = the faulting virtual address; on an illegal instruction = the instruction bits
Call tracethe first address is the panic function's return address (ra); each following entry is the caller's return address

Β§2The compile-time symbol table (D-3)

  • Build: at build time, nm + addr2line -C (demangled) run over the linked ELF, compacted into a NUL-separated name blob plus an (offset, address, size, type) index placed in .rodata; text symbols only, a two-pass build.
  • Scale: ~22.6k symbols, 2.4 MB of names + 0.7 MB of index; the image grows +3.1 MB (riscv64 once had to move its boot window/heap base/RAM reserve 14β†’18 MB together for it; x86_64 derives its mapping from the runtime __kernel_end and needed no move).
  • Boot cost: zero β€” the data is in the image, no runtime scan.
  • Consumers: /proc/kallsyms (standard addr type name lines, 22,636 lines, coverage through _etext), native symbolization for oopses/backtraces (the %ps form name+0xoff/0xsize, non-text symbols marked with ?), and the backtracer.
  • Guards: scripts/test/kallsyms-smoke.sh pins the symbol coverage; cross builds ignore a stale table by target-triple.

Β§3Oops forms

  • The kernel-mode fault order: exception table β†’ fixup (rip = the fixup address) β†’ KernelPanic; oops printing carries a re-entrancy latch (against recursive dumps from wild frame pointers).
  • The USEGV family (x86_64): structured lines for user-space segfaults β€” see memory forensics Β§5; the PF-NOCTX beacon names a raw user-pointer dereference before the oops machinery mangles context (in the SMAP context).
  • %ps symbolization: backtrace lines carry symbol+0xoff/0xsize directly β€” a first judgment needs no offline addr2line; exact line numbers still go through addr2line.

Β§4Line-atomic dumps (the printk emit discipline)

  • Per-CPU staged batched TX with bounded printk-emit-lock acquisition (Full 50 ms / Brief 100 Β΅s) β€” measured: mangled task lines under concurrent dumps went 1.7% β†’ 0 (1506/1506 well-formed).
  • Motivation: interleaved concurrent dumps once truncated forensic lines against each other β€” an untrustworthy scene equals no scene.

Β§5REMOTE-BT: cross-CPU stack capture

  • Under the dfx-lock-owner feature, the watchdog/forensic paths can request a remote CPU's stack snapshot via an IRQ_WORK IPI β€” a holder stuck at IF=0 has a dead tick of its own and can only be observed by another CPU.
  • Together with allocwatch's cross-CPU holder naming it forms the complete "who holds it" forensic pair.

Β§6The symbolization workflow

# 1. symbolize address by address (pick the toolchain per arch)
riscv64-linux-gnu-addr2line -e <kernel.elf> -f -C <addr> [addr2 addr3 ...]
x86_64-linux-gnu-addr2line -e <the unstripped ELF for the bzImage> -f -C <addr>
# 2. feed the whole Call trace's addresses in one command
# 3. cross-check against /proc/kallsyms symbol boundaries (if the guest is alive)
  • Symbolizing a bzImage: keep the unstripped ELF from the build and point addr2line -e at it; the runtime /proc/kallsyms addresses match the ELF symbol table (link address = run address).
  • Release builds default to a 371-byte ksyms stub (kallsyms_generated.rs) β€” diagnostic builds that need symbols re-enable it via the feature; the two image-size figures (12.2 MB vs 16.0 MB) differ exactly by this.

Β§7The complete path from serial to root cause

  1. Capture: serial to file (-serial file:) or hunt-wedge's automated capture;
  2. Classify: scause/cr2 names the exception class; the panic message + file:line give the first read;
  3. Locate: the whole Call trace's addresses through addr2line in one command β†’ the function and line;
  4. Zoom: more scene needed β†’ re-run with dfx=mmforensics / dfx=watchdog / REMOTE-BT;
  5. Establish the root cause: deterministic reproduction (THREAD=single) + intersecting across boots.
Code: kernel/src/dfx/backtrace.rs (the frame-pointer walk, force-frame-pointers=yes), kernel/src/dfx/oops.rs, kernel/src/dfx/ksyms.rs + build/gen-ksyms.py (the generator). Divergence: D-3. Related: memory forensics (scene deepening), lock debugging (the watchdog linkage).