Memory forensics
Dedicated forensic instruments for leaks, corruption, OOM and user-space segfaults β the question each instrument answers, how to arm it and how to read it Β· Sources: kernel/src/dfx/, docs/guides/debugging.md
Β§1dfx=ledger: the physical-page master ledger
- The question it answers: "where did the RAM go?" β every managed page is classified into exactly one bucket via a single descriptor pass, in precedence order (free_linked / free_orphan[β€LEAK] / reserved / page_tables[the PT_LEDGER exact bitmap] / anon / cow / mapped_file / lru_file / other_used), the buckets are summed back against the zone counters, and any gap prints as an explicit mismatch line.
- Reading it: mismatch = 0 β true exhaustion; mismatch > 0 β accounting drift (a leak or a classification gap). This is the tool that separated the fake-OOM family (accounting drift masquerading as exhaustion) from true exhaustion.
- Arming:
dfx=ledger; it also emits automatically with the OOM forensic block; the UART magic "LEDGER!" triggers on demand.
Β§2dfx=mmforensics: page-table and frame forensics
- Trigger sites: the RAW-FREE watchpoint (Zone::free_pages), live-page-table release refusal, PTLEDGER DOUBLE-FREE refusal, the FUT-STALE-TREE census.
- Output: the page-descriptor state at the event, the PT-ledger bit, the frame's "incarnation history" (the forensic rings: PT_STAMP_RING β including the stamping tree's ROOT β, PT_FREE_RING, ZTRACE_ALLOC/ZTRACE_FREE, PTEI_RING data-page installs), per-CPU pageset rows, and compensation for the kernel's missing reverse map: a bounded scan of every live task's page-table tree for links pointing at the pfn.
- Cost: accounting overhead on the fork/exit hot paths β arm during investigations only.
Β§3dfx=memwatch and allocwatch
- memwatch: per-size-class alloc/free counters + a call-site histogram for large objects (2048<sizeβ€4096, hashed from the first 4 frame-pointer returns); periodic dumps ride the taskdump raw-UART path β the first choice for leak localization.
- allocwatch: cross-CPU holder naming β a waiter cannot report itself (printing allocates, and it is stuck inside the allocator); scheduler_tick scans all stamps cross-CPU and reports anything held >1 s with the holder's task state. Design note: the allocator-freeze family's "instrument mirage" verdict (da27be84) β this instrument once served to clear an accusation rather than to confirm it; it works in both directions.
Β§4The OOM forensic block
- Automatic output at OOM: the SP-7 VscanCensus (per-page skip reasons), the direct-reclaim gate state (holder pid + the RECLAIM_GATE_EXIT_RELEASES counter), the ledger bucket sums, and the oom_control details (chosen/chosen_points/totalpages).
- Reading path: ledger mismatch first β for drift, the PT_LEDGER bitmap and the anon/cow buckets locate the source; for true exhaustion, VscanCensus's rc2/rc3/rc4 rows locate the failing reclaim branch.
Β§5User-space segfault forensics: the USEGV family
- USEGV-L: a structured line for user-space SIGSEGVs (pid/rip/cr2) β the first read when a user process crashes without a kernel panic.
- USEGV-FOR: attributing the cr2 landing β e.g. "cr2 INSIDE BINDER BUFFER of proc β¦ β stale daemon mapping dropped at fork (DONTFORK)": the cr2 sits inside a process's binder buffer, annotated with the fork-time stale-daemon-mapping discard, naming the defect family directly.
- Adjudicating test-self-defects like shmat1 depends on this line (the host-baseline cross-table plus the cr2 landing).
Β§6The scribble family: control-word scribble hunting
- Shadow canaries over thread.{fs_base,gs_base,sp,callee.ret_addr} and the parked pt_regs control words; verified at both quiesce points and ticks β a wild write reports on hit.
- scribblepark / scribblepid1 / scribblesweep: park on hit and pair with a GDB watchpoint to walk back to the writer (the pty deep-nesting stack-overflow incident was located this way: a memcpy at sp below the stack bottom).
Β§7Hang and soft-lockup detection
- hung_task (khungtaskd): tasks in TASK_UNINTERRUPTIBLE with an unchanged context-switch count; HUNG_TASK_TIMEOUT_SECS = 120, check interval = timeout/2.
- softlockup: per-CPU timestamps updated in scheduler_tick, threshold 10 s.
- sysrq (/proc/sysrq-trigger): implemented c (deliberate crash)/d (show held locks)/i (kill-all except init)/t (taskdump)/s (sync); unimplemented items print explicit notices (SAK/RT-nice etc.).
Β§8Offline image forensics (no boot)
debugfs -R "cat /var/log/kmsg" <copied.img> # extract the kernel log after a crash
e2fsck -f <copied.img> # image integrity
- With the persistent log enabled, panic logs survive in /var/log/kmsg (a one-line switch in kernel/src/main.rs); with no live machine after a crash, this is the only log source.
- Copy images with reflink (host cp --reflink) to avoid mutating the original; all boots run -snapshot + file.locking=off.
Β§9Reading quick-reference
| Symptom | Instrument path |
|---|---|
| Suspected leak | dfx=memwatch per-site accounting β allocwatch to name a stuck holder β ledger to confirm totals |
| Is the OOM report trustworthy | ledger mismatch decides true exhaustion vs accounting drift β VscanCensus locates the failing reclaim branch |
| Double allocation/frame corruption | dfx=mmforensics: the PT ledger + stamp rings give the frame's incarnation history + the all-task reverse scan finds the holder |
| Wild write | scribble canary parks on hit β GDB watchpoint walks to the writer |
| User-space crash | the USEGV-L line names pid/rip/cr2 β USEGV-FOR attributes the cr2 landing |
| Heap corruption (brk class) | the TRIMQ-class isolation instruments live on their hunt work lanes (branch-only); on main combine mmforensics + scribble |
| Crash with no live machine | debugfs extracts /var/log/kmsg offline β e2fsck -f validates the image |
Code:
kernel/src/dfx/ (ledger/mmforensics/memwatch/scribble/oops/hung_task/softlockup). Related: panic & symbols, memory management. Policy: the trace-event-migrated parts of this page (rux:mmforensics/rux:memwatch) are on tracepoints; the rest is the frozen dfx= legacy surface.