Memory management

kernel/src/mm/ Β· Address spaces, page tables, the buddy/slab two-tier allocator, page cache and writeback, reclaim, fault paths and TLB management β€” mechanisms, invariants and measured numbers Β· Sources: docs/architecture/memory.md, design-divergences.md (D-2/D-7/D-16/D-20)

Β§1Address-space layout and page tables

All three architectures share one address-space partitioning aligned with Linux at constant granularity; the page-table layer follows each architecture's specification:

Itemriscv64 (Sv39)x86_64 (4-level)aarch64 (4-level)
User space256 GB (0x0–0x3FFF_FFFF_FFFF)128 TB (up to 0x0000_7fff_ffff_f000)256 GB (TTBR0 L0 entry 0)
Direct map PAGE_OFFSET0xFFFFFFD6_000000000xffff_8880_0000_00000xffff_c000_0000_0000
Kernel link address0xFFFFFFFF800000000xffff_ffff_8000_00000xffff_ffff_8008_0000
vmemmap / VMALLOCPer-architecture constant regions (VMALLOC 64 GB below PAGE_OFFSET, vmemmap 4 GB below VMALLOC; riscv64 matches Linux constant-for-constant)
  • Sv39 three-level page tables: 9/9/9 + 12-bit offset; PTE bit fields PPN[53:10], D/A/G/U/X/W/R/V, software COW at bit 8, SVPBMT memory type at bits 62:61; satp MODE=8/ASID[59:44]/PPN[43:0].
  • ASID management: Sv39 ASIDs 0–511 (0 = kernel, 1 reserved, 2–511 giving 510 user ASIDs), atomic bitmap with CAS retry, linear rescan on exhaustion; switch_mm() does not embed the ASID in satp (ASID=0) β€” ASID-tagged targeted flushes are issued separately.
  • x86_64 PCID layer (dual CPUID gating, PCID+INVPCID): every mm owns a 12-bit PCID and the CR3 write carries tag+NOFLUSH; INVPCID type 0 precise single-page invalidation / type 3 full flush; ASIDs are synchronously flushed on all CPUs before returning to the pool.
  • Huge pages: Sv39 supports 2 MB megapages (PMD) and 1 GB gigapages; kernel regions are mapped with 2 MB pages at boot; no THP daemon or defragmentation exists (hugepage.rs is explicit huge-page support plus compaction data structures, and is entirely #![forbid(unsafe_code)]).
  • aarch64 specifics: TCR_EL1.IPS = 44 bits (with the 32-bit default, translations with output β‰₯4 GB produce a level-2 address-size abort, DFSC=2, distinct from a translation fault); TPIDR_EL0 is the task state for user TLS, saved/installed by __switch_to at the commit point.

Β§2Three-stage boot-time initialization

Page-table allocation and memory initialization proceed in three stages keyed to allocator availability; the page-table operations table switches between stages via pt_ops_set_* hooks:

  1. Early: static BSS arrays (EARLY_PMD: [PageTable; 8], EARLY_PTE: [PageTable; 128]); boot.S builds the MMU trampoline (identity mapping + kernel VA), and rust_main constructs ROOT_PAGE_TABLE, mapping the kernel image with 2 MB huge pages.
  2. Fixmap: memblock_init() parses DTB memory regions and memblock_reserve()s kernel/heap/slab β†’ setup_linear_mapping() β†’ allocation moves to memblock.
  3. Late: init_heap()+init_slab() β†’ init_vmemmap() β†’ init_page_descriptors() (all Page structs zeroed) β†’ init_zone_system() (ZoneNormal built from memblock free ranges) β†’ pt_ops_set_late() β†’ setup_device_mappings().

memblock uses dual region lists (memory + reserved, [MemBlockRegion; 128]), with available = memory βˆ’ reserved. x86_64 boot order: memblock_init β†’ init_slab β†’ init_vmemmap β†’ init_page_descriptors β†’ init_zone_system β†’ setup_device_mappings β†’ sched::init β†’ kswapd::init β†’ (after the shell starts) enable_oom.

Β§3vmemmap and the page descriptor

  • vmemmap layout: page descriptors are mapped in a dedicated VA region, addressed as VMEMMAP_START + (pfn βˆ’ base_pfn) Γ— sizeof(Page); pfn_to_page / page_to_pfn / pfn_valid are all O(1).
  • struct Page: #[repr(C, align(64))] β€” 64-byte cache-line aligned; fields include flags (4B atomic), _mapcount (AtomicI32, βˆ’1 = unmapped), _refcount (AtomicI32, 0 = free), private (buddy order / slab data), mapping, index, and the buddy free-list pointer next_free.
  • Page flag set: Locked/Writeback/Referenced/UpToDate/Dirty/Lru/Head/Waiters/Active/Reserved/Private/Reclaim/SwapBacked/Unevictable/Cow/Anonymous; the PageType enum is Normal/Buddy/Slab/PageCache/Anonymous.
  • COW refcount rules: fork calls get_page() (+2 for a writable share: share + COW); exit's put_page() frees only when the count reaches zero.

Β§4zone / buddy / per-CPU pageset

  • Zone model: ZoneDma/ZoneDma32/ZoneNormal/ZoneMovable; the riscv64 QEMU configuration has only ZoneNormal. A Zone holds free_area: [FreeArea; MAX_ORDER+1], MAX_ORDER = 10 (max 1024 pages = 4 MB); buddy split/merge follows the standard form (buddy_pfn = pfn ^ (1<<order)).
  • Watermarks: WMARK_MIN/LOW/HIGH; setup_per_zone_wmarks follows the __setup_per_zone_wmarks() shape: min_free_kbytes = int_sqrt(managed_kbΒ·16) clamped to [128, 262144] (a historical "/4" made every watermark four times smaller than Linux's β€” fixed); watermark_gap = max(wmark_min/4, managedΒ·10/10000).
  • PCP (per-CPU pageset): order-0 frames are served from the local CPU with zero zone-lock acquisition; the zone lock is paid once per PCP_BATCH=16 batch. Rows are 64-byte aligned. Three invariants are on record: INV-PCP-1 (the three-state frame lifecycle: OnFreelist rc==0 / OnPcp rc==0 / in-use rcβ‰₯1; violations leak and taint), INV-PCP-2 (zone free_pages counts both buddy and pcp β€” Linux semantics), INV-PCP-3 (a row is touched only by its CPU with IRQs off). Motivation: before the PCP, every order-0 alloc/free was a full zone-lock round trip, with the page-descriptor init loop inside the lock.
  • Migration types: Unmovable/Movable/Reclaimable.

Β§5small_alloc: the small-object allocator (magazine caches)

Origin: at OpenHarmony first-frame composition, ~213,000 net-live small objects in the 1B–1KB classes (measured by CR3 walks: 23.2M allocs / 23.0M frees) each pinned a whole 4 KiB buddy block β€” 852 MB of the 992 MB window β€” triggering an ALLOCTHROW panic during first-frame composition. The packed small-object allocator was built for this.

  • Class structure: 8 power-of-two classes (S = 8..1024, SMALL_MAX = 1024); a slab page is a raw order-0 buddy block (bypassing GlobalAlloc, eliminating recursion), objects are S-aligned, and each page carries a 16-byte PageHdr with magic/free_head/free_count/page linkage.
  • 1-bit free routing: one static bit per heap page (SLAB_PAGE_BITMAP, 262144 pages = 1 GiB window = 32 KiB .bss); the bit is cleared BEFORE an empty page returns to the buddy β€” a stale pointer into a reclaimed page falls through to the buddy-side tripwires (explicit error) instead of silently corrupting slab data; is_slab_ptr is a lock-free routing check and object_span() provides a forensic mapping.
  • Per-CPU magazines (depth 16): motivation measured β€” the class locks were 12.6% of the whole machine (732M irqsave sections/boot at ~250 ns each, the largest IF=0 contributor). The magazine bank was the first D-7 CpuLocal conversion (MAGAZINES: CpuLocal<[Magazine; 8], MAX_CPUS>); fast paths pop/push under local preemption-off only with IRQs enabled (interrupt context never touches a row and routes to the class-lock bodies); the class lock amortizes over batches (refill/flush β‰ˆ MAG_DEPTH objects).
  • Correctness: reclaim requires free_count == objs_per_page under the class lock β€” a magazine entry keeps free_count short, so reclaiming a magazine-resident page is unreachable by construction; push-time screening re-reads the page header inside the irq-off window (magic/class match/aligned slot), and doubtful pointers route to the locked consume-and-taint path β€” a bad pointer can never park in a magazine. Counters are bumped only in locked bodies; magazine residency biases the live estimate by at most MAX_CPUSΓ—8Γ—MAG_DEPTH.
  • Measured: global class-lock sections/boot 19.7M (gate ≀25M; latest board 6.0M), REST share of machine 0.1–1.2% (gate ≀10%), 0.4% tier with section count 6.0M (down from 18.8M).

Β§6Packed kernel-stack slab (D-2, x86_64)

  • Three 80 KB slots carved from each 256 KB order-6 block, plus a side-table slot map and empty-block reclaim; 85.3 KB per stack versus 128 KB power-of-two allocation β€” about 115 MB saved at the launcher-floor task population (stacks were 42% of the 904 MB floor); measured slot packing 99.9% (kb=981/ks=2941); the deepest boot, 3445 live tasks, had zero canary hits.
  • Why the arch gate: packing only pays for non-power-of-two request sizes, and the 80 KB slot's ~15 KB depth margin is an x86_64-measured figure β€” the riscv64 test kernel nests deeper (the ungated first attempt corrupted adjacent heap data on riscv64: getpriority PRIO_PGRP read a trashed nice, suite 1094/1096). riscv64 and unmeasured arches compile the slab out entirely and keep the 128 KB base; Kernel.toml kernel_stack_size_x86_64 = 81920 + per-arch pick in build.rs + a compile-time slot-fit assert. With the gate, the riscv64 suite is 1096/1096.
  • Known costs: the side-table leaf lock sits on the fast path; 6.7% per-block tail waste.

Β§7Page cache and writeback

  • Organization: 4 KB pages keyed by (fs_id, inode_number, page_index) β€” fs_id was added after a u32 inode truncation cross-collided filesystems (review 5.3).
  • Lock model (two-stage evolution): the old shape was one global BTreeMap spinlock β€” cold start faulted 100 MB of libraries page-by-page on 4 CPUs with every request serialized. Current shape: global lookup lock-free β€” a fixed-capacity open-addressed slot table (PC_TABLE_CAP = 32768, linear probing), readers claim their CPU's reader bank (SeqCst RMW), and a hit probes with Acquire loads while claiming a strong count; cross-inode fault contention is zero (per-inode spinlock); writers (wlock: insert/remove/eviction/iteration) nest into per-inode locks one-way only; no allocation under either lock (R-WRITESTORM-1). Whole-entry removal uses a park-and-drain protocol (tombstone under wlock, released after every reader bank observes zero) β€” a false miss is always safe (the miss path re-reads; insert dedups).
  • Capacity policy: soft cap = half of managed RAM, floor 32 MB (MIN_CACHED_PAGES = 8192). Under the old fixed 32 MB cap: during OH shared-library loading, ~580 tasks faulted libraries page-by-page, each fault a miss+evict+disk refetch+ext4 big-lock round trip measured at ~1.5 s per page β€” the evidence that drove the capacity change.
  • Writeback and the dirty watermark (D-20): once the Normal zone's free count falls below half its managed pages (zone_below_dirty_writeback_mark), the buffered writer flushes its own inode's dirty range and then runs a system-wide recovery pass (PageCache::writeback_recover: free already-clean pages first, collect dirty inodes via a full-table rotating walk, flush each inode through the batched engine, and directly evict flushed pages up to the cap + 1/16 managed). Rux keeps no per-BDI dirty accounting, so the mark is a zone free-fraction (Linux uses dirty_background_threshold / balance_dirty_pages dirty fractions) β€” the discipline, not the constant, is the contract. Measured: eliminated the delayed-allocation dirty-balloon wedge (243k dirty pages growing to RAM scale, the zone draining to zero, with reclaim's own nested io-lock re-entering flush as the last producer); LTP fallocate05/06 on three fresh boots RC=0, 16/24 TPASS, zero TFAIL; riscv64 make test 1584/1584.
  • O_DIRECT coherency: bypasses the page cache and serves straight from blocks; an O_DIRECT read after buffered writes flushes the dirty page-cache range first (99ea87e3); diotest2/5 cover it.
  • Observability: /proc/meminfo gains a PageCacheOps counter row (hits/misses/evictions ratio).

Β§8Reclaim: kswapd, vmscan, swap and the direct-reclaim gate

  • kswapd: wakes when a zone falls below the high watermark and runs balance_pgdat; 16 consecutive reclaim failures (KSWAPD_MAX_FAILURES) hand off to the OOM killer; OOM is disabled until boot completes (enable_oom at main.rs:1192).
  • vmscan: priority-based scanning (DEF_PRIORITY=12 descending to 1); reclaim sources are the page-cache shrinker (clean, unreferenced) and the anonymous scan (try_to_unmap, freed on success). The SP-7 per-page skip-reason census (VscanCensus: calls/want/got/null_page/not_anon/not_sb/unmapped/rc2-rc4+...) β€” one relaxed fetch_add per filter branch, read by the OOM forensic dump and a 1 Hz heartbeat, zero output on standard boots. Post-fix measurements: flag-clear handouts + head insertion + head rotation (each the opposite of the pre-fix policy); the reclaim check rate went 35%β†’82–89% across three repetitions (1,240/3,589 β†’ 2,163/2,629 β†’ 2,990/3,346), swap writes +75%.
  • Swap: entry encoding with the bit-62 signature, type [9:8] (MAX_SWAP_DEVICES=4), offset [53:10] (2^44 pages/device), V=0; Kernel.toml swap_size_mb = 256.
  • Direct-reclaim recursion gate: the PF_MEMALLOC equivalent; the gate holds the holder's PID (0 = free) rather than a 0/1 flag β€” when the holder is SIGKILLed inside the section (do_exit does not unwind), reclaim_gate_exit_release(pid) clears the gate by identity via CAS; otherwise every later allocation takes the no-drain skip path (the fake-OOM family's root cause). The SP-5 gate census verified reclaim still progressing across 4+ OOM dumps with the holder pid constant.

Β§9Fault paths and COW

  • Anonymous forward fault-around: a 32-page window (ANON_FAULTAROUND_PAGES = 32), write faults only, restricted to VMAs β‰₯ 1 MB (ANON_FAULTAROUND_MIN_VMA) β€” small stack/brk mappings are unaffected; on sequential streams the trap count drops ~32Γ—; batch fills carry a no-TLB-flush contract (batch frames are freshly installed); aarch64 ports the same shape.
  • File-backed path: the page-fault file path warms the page cache and then pins; D-12's read-miss window batch fill lives on the filesystem page.
  • COW invariants (INV-COW-1..5, x86_64): COW set β‡’ W clear; COW β‡’ refcount β‰₯ 2; a refcount==1 fault restores W without copying; refcount>1 β†’ allocate/copy/decrement/map with W=1 COW=0; after fork every writable child PTE is downgraded W=0 COW=1. riscv64: PTE bit 8 is the COW software flag; fork walks only VPN2[0..255]; a refcount==1 sole owner takes the fast restore.
  • Fault latency measurements (mm_probe PID-1 test platform, both kernels): anon sequential alloc+touch+free 503.6 vs 513.9 ns/fault (0.98Γ—); anon random-order write fault 613.9 vs 323.2 ns (1.90Γ— β†’ 1.7Γ— after walk consolidation); file-backed random touch amortized 34.6 vs 70.2 (0.49Γ—); fork+exit 606 vs 1207 Β΅s (0.50Γ—).

Β§10mmap / VMA management and TLB

  • User-space layout (riscv64): null-page guard β†’ ELF segments β†’ heap (brk from 512 MB upward, capped at TASK_SIZE/3) β†’ guard region β†’ mmap area top-down from TASK_SIZEβˆ’64 GB β†’ user stack at TASK_SIZEβˆ’8MB growing down, capped at 8 MB. brk authority is per-mm (start_brk/brk atomics) β€” a per-TASK brk once let a thread's shrink release the main thread's live pages (the pango/FcInit SIGSEGV family). VmaManager = BTreeMap<start, Vma>, O(log n).
  • per-mm PTE lock split mode: GlobalPteLock is a RwSpinlock epoch (exclusive = the old global single-lock semantics; shared = per-mm sections); PTE writers on different mms run concurrently; split mode is enabled on all architectures. The mm also carries mm_surgery: Rwsem (a whole-syscall surgery lock, Linux's mmap_lock(write) role) closing the mremap-move vs sibling-munmap race; vma_manager is a Rwsem (sleepable, replacing a pure spinner that let slow readers monopolize CPUs).
  • Leaf-table direct unmap walk: allocation-free per-page walking; the upper chain resolves once per 2 MiB block and leaf PTEs are iterated directly (removing a self-inflicted per-chunk cond_resched, 132 Β΅s per call Γ— n=213k). mmap average 72 Β΅s β†’ 27–32 Β΅s; munmap body 177–185 β†’ 52 Β΅s; >16 MiB mmap median 5.6 ms β†’ 415–427 Β΅s; the vm-op family's share of the machine under OH load went 15 s/CPU-min β†’ 5 s/CPU-min.
  • Whole-VMA munmap fast path (2026-11-15): when the range carries no page-table state (glibc mmap/munmap churn), one authoritative emptiness scan under the PTE section ("a same-mm fault cannot install while pte_lock is held") is followed by whole-VMA removal with the zap skipped. munmap 64 MB 23.2 β†’ 4.3 Β΅s (61Γ— β†’ 5.3Γ—); 16 MB 6.2 β†’ 1.4 Β΅s. The fault-entry walk consolidation (one leaf walk per fault): B row 613.9 β†’ 558–565 ns. The mprotect block-hoisted walker was measured and reverted (4/4 boots wedged in a soft lockup inside the timer tick) β€” the per-page walk tolerates a concurrent upper-level mutation the hoist does not; recorded as-is.
  • TLB flush and the lazy mm_cpumask: before the fix, every munmap/fork-COW/mprotect/COW fault broadcast a full TLB-flush IPI to all 8 CPUs β€” /proc/interrupts on a pristine OH boot counted 4,031,356 shootdown IPIs by t=200s (~20k/s; each a KVM-intercepted LAPIC write plus a remote full flush). The per-mm cpu_mask (set at switch-in before the serializing CR3 write; cleared at switch-away where the plain CR3 write's full flush makes it sound; grow-only under PCID) drops the count to 10,209 β€” a 395Γ— reduction (βˆ’99.7%), with flush semantics unchanged (the mask only decides who may hold entries).

Β§11The deferred-free timer ring (D-16)

  • POOL_CAP = 16384 entries with a DEFER_MS = 1500 wall-clock drain; frees matching a registered size (the MmStruct ArcInner block, the Task page itself) enter the ring before returning to the heap.
  • Rationale: procfs walkers hold unaccounted &MmStruct borrows; even with vma_write walker exclusion and an RCU grace wait, a reader preempted mid-syscall can resume after the last Arc drop and lock a garbage rwlock word (the #GP-in-btree-navigate family, evidenced at OH log pid 554 / exitwalk pids 314-315). It is isomorphic to Linux freeing mm_struct via call_rcu, but wall-clock based β€” this implementation's RCU counts preemption points as quiescent states, and a mid-syscall borrower is not quiescent, so the wall clock covers a superset.
  • Bounded: a fixed ring, overflow frees immediately; draining is driven opportunistically from the alloc path (idle systems never accumulate). Revisit condition: once unaccounted borrows become RcuHead, the ring can be deleted.

Β§12The OOM selector

  • Scoring follows the oom_badness shape: points = resident_pages + oom_score_adj Γ— totalpages / 1000 (review 4.14 upgraded the baseline from total_vm to resident pages β€” under a pure total_vm baseline, a process that mmap()ed a huge region once exhausted memory before being killed; total_vm remains only as the fallback for counter-less mms); kernel threads / PID 1 / oom_score_adj ≀ βˆ’1000 are skipped.
  • OOM_LOCK serializes out_of_memory (review 4.13: without the lock, concurrent OOM triggers from kswapd and direct reclaim each SIGKILLed their own victim and raced the OomControl); the mm is pinned via address_space_arc() (the Arc drop must stay outside pid-hash bucket locks β€” the last-ref drop runs mm teardown, which may schedule).
  • Forensics: dfx=ledger classifies every managed page (free linked / orphan / reserved / page tables [the PT_LEDGER exact bitmap] / anon / cow / mapped_file / lru_file) and sums the buckets back against the zone counters, printing any gap as an explicit mismatch line β€” the tool that distinguishes true exhaustion from accounting drift.

Β§13Typed per-CPU storage CpuLocal<T,N> (D-7)

  • Rows are 64-byte aligned; with(f) is the only access surface: IRQs off β†’ cpu id read inside the window β†’ the guard is !Send and Drop restores IRQ state; an out-of-range cpu id takes the None branch (callers fall back to locked bodies); same-slot re-entry is refused (in_use flag, counted and tainted).
  • The landing incident (recorded in D-7): the first version was a non-mut static without interior mutability, and rustc emitted the LLVM global as constant (.rodata) β€” debug builds booted, optβ‰₯1 deleted every store and hoisted loads, the refill loop never terminated, and release boots died at the first c64 miss (ALLOCTHROW at t=3 s, 15.9M objects carved inside 2 lock sections, 99% of the heap); after a revert it re-landed with rows: UnsafeCell<...>. Verification: MAGAZINES resides in .bss and the optβ‰₯1 disassembly contains the refill-loop stores.
  • The incident's value: the interaction of the Rust type system with LLVM constant propagation is a compile-time risk class that does not exist in C β€” closed out with invariants and a disassembly verification procedure.

Β§14/proc surfaces

  • meminfo and vmstat are derived by the procfile! macro (D-4): the kv/pad/cols/kvtab declaration forms; vmstat is a byte-identical snapshot and meminfo is column-exact against Linux; /proc/[pid]/status STATUS_FIELDS is pinned against a live render snapshot.
  • pagemap, vmstat counters (note_fault feeds pgfault/min_flt from the fault path), and the MemoryInfo struct (mem_total/free/available/used, heap, slab_pages, pages_free/used/dirty/cow/anon).
  • Zone free counters include PCP caches (INV-PCP-2).

Β§15Benchmark numbers

Metric (rigs in docs/test/)RuxSame-test platform LinuxRatio
Geekbench 5.5.1 single-core (9950X KVM q35 smp4 4G)24652165113%
Geekbench 5.5.1 multi-core75047027105%
Page-fault latency (file-backed random touch, amortized)34.6 ns70.2 ns0.49Γ—
munmap 64 MB (after the whole-VMA fast path)4.3 Β΅s0.38 Β΅s5.3Γ— (was 61Γ—)
fork+exit606 Β΅s1207 Β΅s0.50Γ—
Anon random-write fault558–565 ns323.2 ns1.7Γ—
TLB shootdown IPIs (OH boot, t=200s)10,209β€”βˆ’99.7% (from 4,031,356)

Honest gaps on record: mprotect 4 MB 2.36 vs 0.23 Β΅s (10Γ—); anon random-write faults 1.7Γ— (post-consolidation); pthread create+join ~6Γ— (reading is load-polluted); single-core deltas within Β±15% are noise (the report's own criterion).

Code: kernel/src/mm/ (allocator / buddy_allocator / page_alloc / pcp / small_alloc / slab / page_cache / vmscan / kswapd / oom_kill / vma / swap etc.). Divergences: D-2/D-7/D-16/D-20. Invariants: INV-PCP-1..3, INV-COW-1..5. Instruments: dfx=mmforensics, dfx=ledger, fsw, VscanCensus (see memory forensics).