Memory management
Β§1Address-space layout and page tables
All three architectures share one address-space partitioning aligned with Linux at constant granularity; the page-table layer follows each architecture's specification:
| Item | riscv64 (Sv39) | x86_64 (4-level) | aarch64 (4-level) |
|---|---|---|---|
| User space | 256 GB (0x0β0x3FFF_FFFF_FFFF) | 128 TB (up to 0x0000_7fff_ffff_f000) | 256 GB (TTBR0 L0 entry 0) |
| Direct map PAGE_OFFSET | 0xFFFFFFD6_00000000 | 0xffff_8880_0000_0000 | 0xffff_c000_0000_0000 |
| Kernel link address | 0xFFFFFFFF80000000 | 0xffff_ffff_8000_0000 | 0xffff_ffff_8008_0000 |
| vmemmap / VMALLOC | Per-architecture constant regions (VMALLOC 64 GB below PAGE_OFFSET, vmemmap 4 GB below VMALLOC; riscv64 matches Linux constant-for-constant) | ||
- Sv39 three-level page tables: 9/9/9 + 12-bit offset; PTE bit fields PPN[53:10], D/A/G/U/X/W/R/V, software COW at bit 8, SVPBMT memory type at bits 62:61; satp MODE=8/ASID[59:44]/PPN[43:0].
- ASID management: Sv39 ASIDs 0β511 (0 = kernel, 1 reserved, 2β511 giving 510 user ASIDs), atomic bitmap with CAS retry, linear rescan on exhaustion;
switch_mm()does not embed the ASID in satp (ASID=0) β ASID-tagged targeted flushes are issued separately. - x86_64 PCID layer (dual CPUID gating, PCID+INVPCID): every mm owns a 12-bit PCID and the CR3 write carries tag+NOFLUSH; INVPCID type 0 precise single-page invalidation / type 3 full flush; ASIDs are synchronously flushed on all CPUs before returning to the pool.
- Huge pages: Sv39 supports 2 MB megapages (PMD) and 1 GB gigapages; kernel regions are mapped with 2 MB pages at boot; no THP daemon or defragmentation exists (hugepage.rs is explicit huge-page support plus compaction data structures, and is entirely
#![forbid(unsafe_code)]). - aarch64 specifics: TCR_EL1.IPS = 44 bits (with the 32-bit default, translations with output β₯4 GB produce a level-2 address-size abort, DFSC=2, distinct from a translation fault); TPIDR_EL0 is the task state for user TLS, saved/installed by __switch_to at the commit point.
Β§2Three-stage boot-time initialization
Page-table allocation and memory initialization proceed in three stages keyed to allocator availability; the page-table operations table switches between stages via pt_ops_set_* hooks:
- Early: static BSS arrays (
EARLY_PMD: [PageTable; 8],EARLY_PTE: [PageTable; 128]); boot.S builds the MMU trampoline (identity mapping + kernel VA), and rust_main constructs ROOT_PAGE_TABLE, mapping the kernel image with 2 MB huge pages. - Fixmap:
memblock_init()parses DTB memory regions andmemblock_reserve()s kernel/heap/slab βsetup_linear_mapping()β allocation moves to memblock. - Late:
init_heap()+init_slab()βinit_vmemmap()βinit_page_descriptors()(all Page structs zeroed) βinit_zone_system()(ZoneNormal built from memblock free ranges) βpt_ops_set_late()βsetup_device_mappings().
memblock uses dual region lists (memory + reserved, [MemBlockRegion; 128]), with available = memory β reserved. x86_64 boot order: memblock_init β init_slab β init_vmemmap β init_page_descriptors β init_zone_system β setup_device_mappings β sched::init β kswapd::init β (after the shell starts) enable_oom.
Β§3vmemmap and the page descriptor
- vmemmap layout: page descriptors are mapped in a dedicated VA region, addressed as
VMEMMAP_START + (pfn β base_pfn) Γ sizeof(Page);pfn_to_page / page_to_pfn / pfn_validare all O(1). - struct Page:
#[repr(C, align(64))]β 64-byte cache-line aligned; fields include flags (4B atomic), _mapcount (AtomicI32, β1 = unmapped), _refcount (AtomicI32, 0 = free), private (buddy order / slab data), mapping, index, and the buddy free-list pointer next_free. - Page flag set: Locked/Writeback/Referenced/UpToDate/Dirty/Lru/Head/Waiters/Active/Reserved/Private/Reclaim/SwapBacked/Unevictable/Cow/Anonymous; the PageType enum is Normal/Buddy/Slab/PageCache/Anonymous.
- COW refcount rules: fork calls get_page() (+2 for a writable share: share + COW); exit's put_page() frees only when the count reaches zero.
Β§4zone / buddy / per-CPU pageset
- Zone model: ZoneDma/ZoneDma32/ZoneNormal/ZoneMovable; the riscv64 QEMU configuration has only ZoneNormal. A Zone holds
free_area: [FreeArea; MAX_ORDER+1], MAX_ORDER = 10 (max 1024 pages = 4 MB); buddy split/merge follows the standard form (buddy_pfn = pfn ^ (1<<order)). - Watermarks: WMARK_MIN/LOW/HIGH;
setup_per_zone_wmarksfollows the__setup_per_zone_wmarks()shape: min_free_kbytes = int_sqrt(managed_kbΒ·16) clamped to [128, 262144] (a historical "/4" made every watermark four times smaller than Linux's β fixed); watermark_gap = max(wmark_min/4, managedΒ·10/10000). - PCP (per-CPU pageset): order-0 frames are served from the local CPU with zero zone-lock acquisition; the zone lock is paid once per PCP_BATCH=16 batch. Rows are 64-byte aligned. Three invariants are on record: INV-PCP-1 (the three-state frame lifecycle: OnFreelist rc==0 / OnPcp rc==0 / in-use rcβ₯1; violations leak and taint), INV-PCP-2 (zone free_pages counts both buddy and pcp β Linux semantics), INV-PCP-3 (a row is touched only by its CPU with IRQs off). Motivation: before the PCP, every order-0 alloc/free was a full zone-lock round trip, with the page-descriptor init loop inside the lock.
- Migration types: Unmovable/Movable/Reclaimable.
Β§5small_alloc: the small-object allocator (magazine caches)
Origin: at OpenHarmony first-frame composition, ~213,000 net-live small objects in the 1Bβ1KB classes (measured by CR3 walks: 23.2M allocs / 23.0M frees) each pinned a whole 4 KiB buddy block β 852 MB of the 992 MB window β triggering an ALLOCTHROW panic during first-frame composition. The packed small-object allocator was built for this.
- Class structure: 8 power-of-two classes (S = 8..1024,
SMALL_MAX = 1024); a slab page is a raw order-0 buddy block (bypassing GlobalAlloc, eliminating recursion), objects are S-aligned, and each page carries a 16-byte PageHdr with magic/free_head/free_count/page linkage. - 1-bit free routing: one static bit per heap page (SLAB_PAGE_BITMAP, 262144 pages = 1 GiB window = 32 KiB .bss); the bit is cleared BEFORE an empty page returns to the buddy β a stale pointer into a reclaimed page falls through to the buddy-side tripwires (explicit error) instead of silently corrupting slab data; is_slab_ptr is a lock-free routing check and object_span() provides a forensic mapping.
- Per-CPU magazines (depth 16): motivation measured β the class locks were 12.6% of the whole machine (732M irqsave sections/boot at ~250 ns each, the largest IF=0 contributor). The magazine bank was the first D-7 CpuLocal conversion (
MAGAZINES: CpuLocal<[Magazine; 8], MAX_CPUS>); fast paths pop/push under local preemption-off only with IRQs enabled (interrupt context never touches a row and routes to the class-lock bodies); the class lock amortizes over batches (refill/flush β MAG_DEPTH objects). - Correctness: reclaim requires free_count == objs_per_page under the class lock β a magazine entry keeps free_count short, so reclaiming a magazine-resident page is unreachable by construction; push-time screening re-reads the page header inside the irq-off window (magic/class match/aligned slot), and doubtful pointers route to the locked consume-and-taint path β a bad pointer can never park in a magazine. Counters are bumped only in locked bodies; magazine residency biases the live estimate by at most MAX_CPUSΓ8ΓMAG_DEPTH.
- Measured: global class-lock sections/boot 19.7M (gate β€25M; latest board 6.0M), REST share of machine 0.1β1.2% (gate β€10%), 0.4% tier with section count 6.0M (down from 18.8M).
Β§6Packed kernel-stack slab (D-2, x86_64)
- Three 80 KB slots carved from each 256 KB order-6 block, plus a side-table slot map and empty-block reclaim; 85.3 KB per stack versus 128 KB power-of-two allocation β about 115 MB saved at the launcher-floor task population (stacks were 42% of the 904 MB floor); measured slot packing 99.9% (kb=981/ks=2941); the deepest boot, 3445 live tasks, had zero canary hits.
- Why the arch gate: packing only pays for non-power-of-two request sizes, and the 80 KB slot's ~15 KB depth margin is an x86_64-measured figure β the riscv64 test kernel nests deeper (the ungated first attempt corrupted adjacent heap data on riscv64: getpriority PRIO_PGRP read a trashed nice, suite 1094/1096). riscv64 and unmeasured arches compile the slab out entirely and keep the 128 KB base; Kernel.toml
kernel_stack_size_x86_64 = 81920+ per-arch pick in build.rs + a compile-time slot-fit assert. With the gate, the riscv64 suite is 1096/1096. - Known costs: the side-table leaf lock sits on the fast path; 6.7% per-block tail waste.
Β§7Page cache and writeback
- Organization: 4 KB pages keyed by (fs_id, inode_number, page_index) β fs_id was added after a u32 inode truncation cross-collided filesystems (review 5.3).
- Lock model (two-stage evolution): the old shape was one global BTreeMap spinlock β cold start faulted 100 MB of libraries page-by-page on 4 CPUs with every request serialized. Current shape: global lookup lock-free β a fixed-capacity open-addressed slot table (
PC_TABLE_CAP = 32768, linear probing), readers claim their CPU's reader bank (SeqCst RMW), and a hit probes with Acquire loads while claiming a strong count; cross-inode fault contention is zero (per-inode spinlock); writers (wlock: insert/remove/eviction/iteration) nest into per-inode locks one-way only; no allocation under either lock (R-WRITESTORM-1). Whole-entry removal uses a park-and-drain protocol (tombstone under wlock, released after every reader bank observes zero) β a false miss is always safe (the miss path re-reads; insert dedups). - Capacity policy: soft cap = half of managed RAM, floor 32 MB (
MIN_CACHED_PAGES = 8192). Under the old fixed 32 MB cap: during OH shared-library loading, ~580 tasks faulted libraries page-by-page, each fault a miss+evict+disk refetch+ext4 big-lock round trip measured at ~1.5 s per page β the evidence that drove the capacity change. - Writeback and the dirty watermark (D-20): once the Normal zone's free count falls below half its managed pages (
zone_below_dirty_writeback_mark), the buffered writer flushes its own inode's dirty range and then runs a system-wide recovery pass (PageCache::writeback_recover: free already-clean pages first, collect dirty inodes via a full-table rotating walk, flush each inode through the batched engine, and directly evict flushed pages up to the cap + 1/16 managed). Rux keeps no per-BDI dirty accounting, so the mark is a zone free-fraction (Linux uses dirty_background_threshold / balance_dirty_pages dirty fractions) β the discipline, not the constant, is the contract. Measured: eliminated the delayed-allocation dirty-balloon wedge (243k dirty pages growing to RAM scale, the zone draining to zero, with reclaim's own nested io-lock re-entering flush as the last producer); LTP fallocate05/06 on three fresh boots RC=0, 16/24 TPASS, zero TFAIL; riscv64 make test 1584/1584. - O_DIRECT coherency: bypasses the page cache and serves straight from blocks; an O_DIRECT read after buffered writes flushes the dirty page-cache range first (99ea87e3); diotest2/5 cover it.
- Observability: /proc/meminfo gains a PageCacheOps counter row (hits/misses/evictions ratio).
Β§8Reclaim: kswapd, vmscan, swap and the direct-reclaim gate
- kswapd: wakes when a zone falls below the high watermark and runs balance_pgdat; 16 consecutive reclaim failures (KSWAPD_MAX_FAILURES) hand off to the OOM killer; OOM is disabled until boot completes (enable_oom at main.rs:1192).
- vmscan: priority-based scanning (DEF_PRIORITY=12 descending to 1); reclaim sources are the page-cache shrinker (clean, unreferenced) and the anonymous scan (try_to_unmap, freed on success). The SP-7 per-page skip-reason census (VscanCensus: calls/want/got/null_page/not_anon/not_sb/unmapped/rc2-rc4+...) β one relaxed fetch_add per filter branch, read by the OOM forensic dump and a 1 Hz heartbeat, zero output on standard boots. Post-fix measurements: flag-clear handouts + head insertion + head rotation (each the opposite of the pre-fix policy); the reclaim check rate went 35%β82β89% across three repetitions (1,240/3,589 β 2,163/2,629 β 2,990/3,346), swap writes +75%.
- Swap: entry encoding with the bit-62 signature, type [9:8] (MAX_SWAP_DEVICES=4), offset [53:10] (2^44 pages/device), V=0; Kernel.toml swap_size_mb = 256.
- Direct-reclaim recursion gate: the PF_MEMALLOC equivalent; the gate holds the holder's PID (0 = free) rather than a 0/1 flag β when the holder is SIGKILLed inside the section (do_exit does not unwind),
reclaim_gate_exit_release(pid)clears the gate by identity via CAS; otherwise every later allocation takes the no-drain skip path (the fake-OOM family's root cause). The SP-5 gate census verified reclaim still progressing across 4+ OOM dumps with the holder pid constant.
Β§9Fault paths and COW
- Anonymous forward fault-around: a 32-page window (
ANON_FAULTAROUND_PAGES = 32), write faults only, restricted to VMAs β₯ 1 MB (ANON_FAULTAROUND_MIN_VMA) β small stack/brk mappings are unaffected; on sequential streams the trap count drops ~32Γ; batch fills carry a no-TLB-flush contract (batch frames are freshly installed); aarch64 ports the same shape. - File-backed path: the page-fault file path warms the page cache and then pins; D-12's read-miss window batch fill lives on the filesystem page.
- COW invariants (INV-COW-1..5, x86_64): COW set β W clear; COW β refcount β₯ 2; a refcount==1 fault restores W without copying; refcount>1 β allocate/copy/decrement/map with W=1 COW=0; after fork every writable child PTE is downgraded W=0 COW=1. riscv64: PTE bit 8 is the COW software flag; fork walks only VPN2[0..255]; a refcount==1 sole owner takes the fast restore.
- Fault latency measurements (mm_probe PID-1 test platform, both kernels): anon sequential alloc+touch+free 503.6 vs 513.9 ns/fault (0.98Γ); anon random-order write fault 613.9 vs 323.2 ns (1.90Γ β 1.7Γ after walk consolidation); file-backed random touch amortized 34.6 vs 70.2 (0.49Γ); fork+exit 606 vs 1207 Β΅s (0.50Γ).
Β§10mmap / VMA management and TLB
- User-space layout (riscv64): null-page guard β ELF segments β heap (brk from 512 MB upward, capped at TASK_SIZE/3) β guard region β mmap area top-down from TASK_SIZEβ64 GB β user stack at TASK_SIZEβ8MB growing down, capped at 8 MB. brk authority is per-mm (start_brk/brk atomics) β a per-TASK brk once let a thread's shrink release the main thread's live pages (the pango/FcInit SIGSEGV family). VmaManager = BTreeMap<start, Vma>, O(log n).
- per-mm PTE lock split mode: GlobalPteLock is a RwSpinlock epoch (exclusive = the old global single-lock semantics; shared = per-mm sections); PTE writers on different mms run concurrently; split mode is enabled on all architectures. The mm also carries
mm_surgery: Rwsem(a whole-syscall surgery lock, Linux's mmap_lock(write) role) closing the mremap-move vs sibling-munmap race; vma_manager is a Rwsem (sleepable, replacing a pure spinner that let slow readers monopolize CPUs). - Leaf-table direct unmap walk: allocation-free per-page walking; the upper chain resolves once per 2 MiB block and leaf PTEs are iterated directly (removing a self-inflicted per-chunk cond_resched, 132 Β΅s per call Γ n=213k). mmap average 72 Β΅s β 27β32 Β΅s; munmap body 177β185 β 52 Β΅s; >16 MiB mmap median 5.6 ms β 415β427 Β΅s; the vm-op family's share of the machine under OH load went 15 s/CPU-min β 5 s/CPU-min.
- Whole-VMA munmap fast path (2026-11-15): when the range carries no page-table state (glibc mmap/munmap churn), one authoritative emptiness scan under the PTE section ("a same-mm fault cannot install while pte_lock is held") is followed by whole-VMA removal with the zap skipped. munmap 64 MB 23.2 β 4.3 Β΅s (61Γ β 5.3Γ); 16 MB 6.2 β 1.4 Β΅s. The fault-entry walk consolidation (one leaf walk per fault): B row 613.9 β 558β565 ns. The mprotect block-hoisted walker was measured and reverted (4/4 boots wedged in a soft lockup inside the timer tick) β the per-page walk tolerates a concurrent upper-level mutation the hoist does not; recorded as-is.
- TLB flush and the lazy mm_cpumask: before the fix, every munmap/fork-COW/mprotect/COW fault broadcast a full TLB-flush IPI to all 8 CPUs β /proc/interrupts on a pristine OH boot counted 4,031,356 shootdown IPIs by t=200s (~20k/s; each a KVM-intercepted LAPIC write plus a remote full flush). The per-mm cpu_mask (set at switch-in before the serializing CR3 write; cleared at switch-away where the plain CR3 write's full flush makes it sound; grow-only under PCID) drops the count to 10,209 β a 395Γ reduction (β99.7%), with flush semantics unchanged (the mask only decides who may hold entries).
Β§11The deferred-free timer ring (D-16)
POOL_CAP = 16384entries with aDEFER_MS = 1500wall-clock drain; frees matching a registered size (the MmStruct ArcInner block, the Task page itself) enter the ring before returning to the heap.- Rationale: procfs walkers hold unaccounted &MmStruct borrows; even with vma_write walker exclusion and an RCU grace wait, a reader preempted mid-syscall can resume after the last Arc drop and lock a garbage rwlock word (the #GP-in-btree-navigate family, evidenced at OH log pid 554 / exitwalk pids 314-315). It is isomorphic to Linux freeing mm_struct via call_rcu, but wall-clock based β this implementation's RCU counts preemption points as quiescent states, and a mid-syscall borrower is not quiescent, so the wall clock covers a superset.
- Bounded: a fixed ring, overflow frees immediately; draining is driven opportunistically from the alloc path (idle systems never accumulate). Revisit condition: once unaccounted borrows become RcuHead, the ring can be deleted.
Β§12The OOM selector
- Scoring follows the oom_badness shape:
points = resident_pages + oom_score_adj Γ totalpages / 1000(review 4.14 upgraded the baseline from total_vm to resident pages β under a pure total_vm baseline, a process that mmap()ed a huge region once exhausted memory before being killed; total_vm remains only as the fallback for counter-less mms); kernel threads / PID 1 / oom_score_adj β€ β1000 are skipped. - OOM_LOCK serializes out_of_memory (review 4.13: without the lock, concurrent OOM triggers from kswapd and direct reclaim each SIGKILLed their own victim and raced the OomControl); the mm is pinned via address_space_arc() (the Arc drop must stay outside pid-hash bucket locks β the last-ref drop runs mm teardown, which may schedule).
- Forensics: dfx=ledger classifies every managed page (free linked / orphan / reserved / page tables [the PT_LEDGER exact bitmap] / anon / cow / mapped_file / lru_file) and sums the buckets back against the zone counters, printing any gap as an explicit mismatch line β the tool that distinguishes true exhaustion from accounting drift.
Β§13Typed per-CPU storage CpuLocal<T,N> (D-7)
- Rows are 64-byte aligned;
with(f)is the only access surface: IRQs off β cpu id read inside the window β the guard is !Send and Drop restores IRQ state; an out-of-range cpu id takes the None branch (callers fall back to locked bodies); same-slot re-entry is refused (in_use flag, counted and tainted). - The landing incident (recorded in D-7): the first version was a non-mut static without interior mutability, and rustc emitted the LLVM global as
constant(.rodata) β debug builds booted, optβ₯1 deleted every store and hoisted loads, the refill loop never terminated, and release boots died at the first c64 miss (ALLOCTHROW at t=3 s, 15.9M objects carved inside 2 lock sections, 99% of the heap); after a revert it re-landed withrows: UnsafeCell<...>. Verification: MAGAZINES resides in .bss and the optβ₯1 disassembly contains the refill-loop stores. - The incident's value: the interaction of the Rust type system with LLVM constant propagation is a compile-time risk class that does not exist in C β closed out with invariants and a disassembly verification procedure.
Β§14/proc surfaces
- meminfo and vmstat are derived by the procfile! macro (D-4): the kv/pad/cols/kvtab declaration forms; vmstat is a byte-identical snapshot and meminfo is column-exact against Linux; /proc/[pid]/status STATUS_FIELDS is pinned against a live render snapshot.
- pagemap, vmstat counters (note_fault feeds pgfault/min_flt from the fault path), and the MemoryInfo struct (mem_total/free/available/used, heap, slab_pages, pages_free/used/dirty/cow/anon).
- Zone free counters include PCP caches (INV-PCP-2).
Β§15Benchmark numbers
| Metric (rigs in docs/test/) | Rux | Same-test platform Linux | Ratio |
|---|---|---|---|
| Geekbench 5.5.1 single-core (9950X KVM q35 smp4 4G) | 2465 | 2165 | 113% |
| Geekbench 5.5.1 multi-core | 7504 | 7027 | 105% |
| Page-fault latency (file-backed random touch, amortized) | 34.6 ns | 70.2 ns | 0.49Γ |
| munmap 64 MB (after the whole-VMA fast path) | 4.3 Β΅s | 0.38 Β΅s | 5.3Γ (was 61Γ) |
| fork+exit | 606 Β΅s | 1207 Β΅s | 0.50Γ |
| Anon random-write fault | 558β565 ns | 323.2 ns | 1.7Γ |
| TLB shootdown IPIs (OH boot, t=200s) | 10,209 | β | β99.7% (from 4,031,356) |
Honest gaps on record: mprotect 4 MB 2.36 vs 0.23 Β΅s (10Γ); anon random-write faults 1.7Γ (post-consolidation); pthread create+join ~6Γ (reading is load-polluted); single-core deltas within Β±15% are noise (the report's own criterion).
kernel/src/mm/ (allocator / buddy_allocator / page_alloc / pcp / small_alloc / slab / page_cache / vmscan / kswapd / oom_kill / vma / swap etc.). Divergences: D-2/D-7/D-16/D-20. Invariants: INV-PCP-1..3, INV-COW-1..5. Instruments: dfx=mmforensics, dfx=ledger, fsw, VscanCensus (see memory forensics).