Architecture & module design

Mechanisms, data structures and measured numbers per subsystem — numbers come from the repository docs; unmeasured items are marked as such

§0Overview

Rux is one source tree serving three architectures (riscv64 / x86_64 / aarch64), ~90% arch-independent. Externally it matches the Linux ABI across 349 syscalls; internally it is a clean-room design — every internal decision that differs from the mainstream is registered in the design-divergence register with compatibility evidence.

SubsystemLinesShareRole
process/ + sched/ + sync/60,32116.7%tasks / scheduling / synchronization
fs/ (ext4 · VFS · page cache · jbd2)76,88621.3%filesystem stack and caches
arch/ (three port layers)45,17512.5%boot / traps / context switch / TLB
syscall/41,04111.3%syscall boundary (the capsule edge)
dfx/ + trace/ (syscost · kallsyms · watchdog · trace-event)23,7376.6%diagnostics and observability
drivers/ (virtio · DRM · input)26,2807.3%device drivers
ipc/ (binder · SysV · MQ)22,1096.1%inter-process communication
net/ (TCP/IP · unix · netlink)20,6375.7%network stack
mm/20,1265.6%memory management
others (top-level files · interrupt · timer · perf · io_uring · safety · security · module)25,5617.1%infrastructure and security modules

The layering model (docs/architecture/kernel-layers.md): the Framework layer (arch/mm primitives/interrupt/virtio core/trace) allows unsafe with per-block // SAFETY: comments; the service layer targets 100% forbid(unsafe_code) (47 files already forbid it at compile time, 56 more contain none); the syscall entry is the capsule boundary, validating and translating user pointers in one place.

§1Scheduler (kernel/src/sched/) · deep-dive →

A single global run queue — GRQ (divergence D-1). One GlobalRunQueue serves every CPU with three per-class subqueues: DL (BTreeMap, EDF), RT (bitmap + per-priority lists), CFS (BTreeMap by vruntime); atomic nr_running, an idle_cpus bitmap, and per-CPU current/idle/stop pointers only. No per-CPU queues, no load balancer — idle CPUs pull work on demand via IPI.

  • CFS (fair.rs): vruntime = runtime ×(NICE_0_LOAD/weight); cfs_min_granularity_ns = 0.7 ms (also the wakeup granularity), a 6 ms latency window, 100 Hz ticks; pick lifts non-runnable entries out of the timeline and reinserts them; min_vruntime takes the whole tree's leftmost (never erasing a sleeper's credit).
  • The measured-wait wakeup preemption gate (D-14): a wakeup with no idle CPU pokes the busy CPU only when the wakee's measured wait ≥ 6 ms — replacing Linux's vruntime-credit comparison (which Rux's vruntime adjustments would fool every time).
  • wakewatch: a wake→run latency forensics instrument (1024-slot side table + a pick-skip reason table).
  • The GRQ lock is protected by typed lock keys (D-11 — nesting is a compile-time fact).

Key numbers

  • GRQ critical section (post measurement-correction): production ≈60 ns, minimal-bracket measured 75–80 ns. Tree work ~20–30 ns per mutation, zero allocation under the lock. The 70–84 ns lock wait is the KVM tax on cross-vCPU locked instructions (Linux's per-rq lock pays it too — it is not the GRQ being slow). The gate band tracks the call-site IF=0 span average (irqsave→restore, including lock-acquisition wait and critical-section work): 2786 ns @ n=6.3M; flat across a 126.0M-syscall boot → size-coupled, not contention-bound (at ≤4 vCPUs).
  • Boot census reads exactly, on every qualified quiet boot: TERM-DESCHED 8 (band 8±2), WAKE-REFUSED 32 (band 32±4).
  • Three context-switch levers (FSGSBASE 393 ns/switch + lazy FPU + CPUID-gated PCID): lat_ctx -s64 2 4.97→2.07 µs (−58%); same-test platform lmbench 1.98 vs 6.71 µs (3.39×).
  • With D-14: the 74-byte unix-stream wake/preempt/switch round trip ~275 µs → ~2.5 µs; the p5 IPC family 10–18×.

Compatibility: pure internal policy, no ABI surface. Guarded by the performance gate's scheduler domain (hold band, census, P2 ±20%).

§2Memory management (kernel/src/mm/) · deep-dive →

Deep dive in the memory page; summary:

  • Three-stage boot: memblock (dual-region table) → fixmap → zone/buddy; Sv39 layout constants match Linux (PAGE_OFFSET/VMALLOC/vmemmap/TASK_SIZE).
  • vmemmap: a 64-byte cache-line-aligned Page per physical page, pfn↔page O(1); buddy MAX_ORDER=10, order-0 through the per-CPU pageset (PCP) with zero zone locks.
  • small_alloc (magazine): 8 power-of-two classes (8..1024B) + a per-CPU magazine (depth 16); class locks are taken only for batched refill/flush; one free bit per slab page, cleared before the page returns — a stale pointer lands on the buddy-side sentinel instead of corrupting silently.
  • Packed kernel-stack slab (D-2, x86_64): 80KB slots in 256KB blocks + a side table; 85.3KB/stack vs 128KB, 99.9% slot packing, 3,445 live tasks (deepest recorded) with zero canary hits.
  • Page cache: lockless global slot-table reads + per-inode spinlocks; floor 32MB, soft cap half of managed RAM (a fixed 32MB cap once made OH library loads cost ~1.5 s per fault).
  • Deferred-free ring (D-16): MmStruct/Task free into a 16,384-slot global ring drained on a 1500 ms wall clock — bounded reclamation latency independent of scheduling.
  • Per-mm PTE lock: a global epoch (read side of an RwSpinlock) + per-mm spinlock split mode, replacing a global PTE lock that let fork-walks monopolize the machine; leaf-table direct unmap: mmap 72 µs→27–32 µs, munmap body 177→52 µs.
  • CpuLocal<T,N> (D-7): typed per-CPU storage — "the IRQ-off window a per-CPU access relies on = the guard's lifetime" is a type-level fact.

Key numbers

  • small_alloc global class-lock sections/boot 19.7M (gate ≤25M; latest board 6.0M), REST machine share 0.1–1.2% (gate ≤10%).
  • Anon reclaim check rate 35%→82–89% (SP-7), swap writes +75%.
  • Negative-lookup memo oracle: 475k negative hits, 963k stale normalizations, 0 stale negative deliveries.

Compatibility: /proc/meminfo, pagemap, mmap/munmap/mprotect/mremap/mlock/brk semantics aligned item by item (msync wrap checks, mmap offset alignment EINVAL and the rest).

§3File systems (kernel/src/fs/) · deep-dive →

  • The capability-declarative FileOps builder (D-6): one method per slot, duplicate installs are compile errors; errno mapping collapses to a single point. Paid incident: read() on a valid epoll fd returned EBADF and the OH event loop spun 155,991 times.
  • Lockless dcache reads (WalkClaim reader banks): removed two spinlock pairs per path component — stat 0.643→0.423 µs, open 1.128→0.889 µs; the negative memo invalidates by parent-directory generation.
  • ext4 (12 files, 17,486 lines, plus jbd2 3,899 lines; deep-dive on the filesystem page): mount feature gates (unknown incompat → refuse; ENCRYPT is a capability bit and accepted — refusing it once took down OH's stock userdata); read-side window batch fill (D-12, ≤64 blocks/128KiB merged requests), write-side batched metadata publication + discard-on-free (D-13).
  • jbd2 direct-write commit (D-8): commit writes transaction buffers straight to their final locations and advances the log tail/head with the commit — a completed commit leaves nothing to replay, and the 8–12 synchronous round trips per unlink/rmdir disappear; on-disk format and recovery (e2fsck) unchanged.
  • bio/buffer cache: buffer_head five states + LRU; write planning once re-scanned all pending quadratically (fixed with a 3-round cap + conservative linear fallback; 252 samples of one fsync wedging 1,092 s pinned it).
  • The procfile! macro (D-4): /proc formatters derived from one field-declaration table, killing the format-drift defect class (vmstat/meminfo/status byte-identical to a live host).
  • pipe: 64 KiB capacity, PIPE_BUF=4096 atomic writes (Linux defaults). io_uring: the 5.1+ ABI (SQ/CQ ring mmap), operations executed synchronously in the enter context.

Key numbers

  • Metadata re-measurement (2026-11-01): stat 200.9→71.1 µs, openclose 512.9→156.3 µs, read_cached 546.8→203.7 µs, append 657.2→77.4 µs, mkdrmdir 22.8→2.76 ms, unlink 1,141→808 µs.
  • D-13 end to end: p6_mkdrmdir median 29,796→2,107 µs, p99 54→16.4 ms (vs 153 µs for Linux — the residue is shared syscall/lookup overhead, not fs).
  • buffer cache hit rate 99.1% (gate ≥ baseline).

Compatibility: the jbd2 on-disk format is unchanged (image mounting/recovery identical); POSIX does not promise per-operation directory persistence (that is fsync(dirfd)); LTP change families green.

§4IPC (kernel/src/ipc/) · deep-dive →

  • binder (14,144 lines, the largest single file; deep-dive on the IPC page): the load-bearing implementation of the OpenHarmony service mesh. A misc device /dev/binder; mmap buffer silently truncated at 4 MiB (same as Linux); protocol version 8; the full BC_*/BR_* set, flat_binder_object cross-process translation, deferred TRANSACTION_COMPLETE for sync transactions. Lock model: one global spinlock, no user-memory copies and no sleeping under it.
  • The access_token registry: /dev/access_token_id + BINDER_GET_SENDER_INFO — samgr's permission checks depend on it; its absence once caused the OH display blackout (the b98–b112 root cause).
  • SysV trio + POSIX MQ: structs compile-time asserted against Linux's layouts (ShmidDsUapi 112B, MqAttr 64B); SysV objects scoped per ipc_namespace.
  • The binder poll wake/availability fix converged four semantic divergences in one diff (Linux's three OR conditions in binder_has_work_ilocked).

Key numbers

  • With /data mounted and the service mesh at ~89k txns/s, binder lock REST sections averaged 9.2 µs (single-transaction max 73 ms — a watch row; not reproduced on re-runs).
  • pipe round trip p5a 18.3 µs, unix dgram p5b 23–41 µs (was 318.6 µs before D-14, −93%).

Compatibility: the UAPI is byte-aligned with Linux's binder header (_IOC encoding, packed layouts); LTP sem/msg/shm families + the full OH service set are the guards.

§5Network (kernel/src/net/, 18 files)

  • Bottom-up: SkBuff → ethernet → arp → ipv4 → icmp/udp/tcp (+ipv6); a socket layer with SOCK_STREAM/DGRAM/RAW + unix + netlink.
  • TCP (4,841 lines): ISN from a four-tuple hash + jiffies; MSS 1460, window 65535, RTO 200 ms–120 s, DELACK 40 ms, zero-window persist probes at 5 s (Linux initial values).
  • AF_UNIX (3,707 lines): STREAM/SEQPACKET/DGRAM, the abstract namespace, SCM_RIGHTS fd passing, kernel-filled SCM_CREDENTIALS; peers are mutual Weaks (the fd is the only strong owner — "peer closed" = Weak::upgrade failure, no reference cycles); a single-lock send path with no AB-BA.
  • netlink set operations gate on NET_ADMIN, raw/packet on NET_RAW (errno precedence pinned from the Linux source).

Key numbers

  • Same-test platform lmbench: lat_pipe 3.74 vs 13.30 µs (3.56×).
  • unix stream p5c 24.9–26.8 µs (was 276.5 µs, −90%).
  • TCP throughput/congestion benchmarks: not yet measured (stated plainly; covered by the LTP socket family + nettest).

§6Drivers (kernel/src/drivers/)

  • virtio: VirtIO 1.0 register layouts; VirtQueue (1,246 lines) + a PCI transport layer (2,146 lines, modern + transitional); the blk device-ID table was once mis-shifted (a reserved base address read as a device ID, zero matches with disable-legacy) — guarded by run.sh disable-legacy=on.
  • virtio-net lock order (INV-LOCK-6): {rx|tx}_queue → stats, one-way; an AB-BA where poll held stats across refill was caught by the watchdog (two CPUs spinning against each other).
  • gpu/drm: a framebuffer core + virtio-gpu + DRM/PRIME (the GEM_CLOSE shared-keep semantics fix); the display path carries GNOME and OH rendering.
  • intc: PLIC/CLINT (riscv64), APIC (x86_64), GICv3 (aarch64); pci: ECAM enumeration.
  • Device MMIO discipline (D-17): everything through the kernel-side alias arena, the bootstrap identity mapping UNLINKed after the boot-stack switch — no low identity window at runtime, and uaccess has no identity hole.

§7Observability (kernel/src/trace/ + dfx/) · tracepoint deep-dive →

  • The unified trace-event framework (D-15): one declaration-table row = group/name/id/print-fmt/fields/typed record function; the tracefs (/sys/kernel/tracing) tree and byte formats align exactly (tracing_on/trace/trace_pipe/set_event/snapshot/events/*/enable…). Currently 69 events / 17 groups (sched 9, rux 20, ext4 7, net and binder 5 each, input 4, kmem 4, raw_syscalls/irq/exceptions/block/jbd2 2 each, vmscan/timer/signal/printk/oom 1 each); the complete inventory and usage live on Tracepoints.
  • Measured gate cost: disabled sites compile to one load+test+branch (verified point by point in release disassembly); enabling costs +13–19 ns/record; image cost +0.18%.
  • kallsyms (D-3): a compile-time symbol table in .rodata (2.4MB names + 0.7MB index @ ~22.6k symbols), zero boot cost, standard /proc/kallsyms lines.
  • census + heal: per-task per-field tick observation + phantom-preempt adjudication + a 20 ms starvation bound; IF=0 share 25.7%→20.4%, longest cli segment 594–599 ms one-shot → 6–9 ms, WAKE-STRANDED 233→31.
  • The dfx= switch table is frozen at 49 entries (legacy, partly migrated into event-arming aliases): new observability is tracepoint-only.

§8The three port layers (kernel/src/arch/) · boot deep-dive → / interrupts & timers →

Details in the porting guide and boot flow. Highlights:

  • riscv64: OpenSBI M→S mode, Sv39; a 288-byte PtRegs (orig_a0 rollback); SUM-bit uaccess + an exception table; Sstc direct stimecmp writes; a 9-bit ASID atomic bitmap allocator.
  • x86_64: multiboot1 + the bzImage boot protocol; FSGSBASE/lazy-FPU/PCID levers (switch −58%); SMAP via pushfq EFLAGS.AC brackets (D-18, ~55 ns — dodging the TCG STAC/CLAC #UD); userspace W^X.
  • aarch64: QEMU virt GICv3 (requires -M virt,gic-version=3); an embedded DTB blob; TPIDR_EL0 as the task TLS state; EL0 FP traps capturing NEON state; TASK_SIZE 256GB, TCR IPS=44-bit.

§9The design-divergence register (29 entries)

Source: docs/architecture/design-divergences.md. Each entry must pass three gates — external compatibility intact, affected LTP families green, zero upstream breakage; entries are never deleted, superseded ones keep history.

IDDesign · benefit · key numbers · known cost
D-1The global run queue: no per-CPU queues, no balancer — one source of truth for scheduling order; all forensics reason about one structure. Critical section ≈60 ns (production) / 75–80 ns (minimal bracket; tree work ~20–30 ns + bookkeeping)) 2786 ns (band 2786). Cost: a contention ceiling at high core counts + locality loss (registered).
D-2Packed kernel-stack slab (x86_64): 85.3KB/stack vs 128KB, 99.9% slot packing, 3,445 live tasks zero canary; non-gated archs compile it out. Cost: side-table locks + 6.7% tail waste.
D-3Compile-time symbol table in .rodata: zero boot cost, 2.4MB names + 0.7MB index @22.6k symbols, shared by kallsyms/backtraces/%ps. Cost: image +3.1MB.
D-4The procfile! /proc formatter macro: one declaration table casts the renderer + shape tests, killing the format-drift class (vmstat/meminfo/status byte-identical). Cost: ~18 hand-written files remain.
D-5The tracepoint! declaration macro — superseded by D-15, kept for history.
D-6Capability-checked file-ops declarations: duplicate installs are compile errors, errno mapping in one point (paid incident: an epoll fd's EBADF spun the OH loop 155,991 times). Migration complete.
D-7Typed per-CPU CpuLocal<T,N>: accessible only via with(f); guard lifetime = the IRQ-off window; out-of-range ids take the survival branch. Guard: changes require a release-strong pbench boot.
D-8Direct-write jbd2 commit (no replay window, no revoke-cancel): the 8–12 synchronous round trips per unlink/rmdir are gone (p6_mkdrmdir 52–58 ms → single-digit ms); on-disk format unchanged, recovery = e2fsck.
D-9LD_* environment scrubbing for secure-exec: at AT_SECURE the kernel deletes all LD_* before installing envp (libc-independent defense in depth). Visible difference: /proc/<pid>/environ omits LD_*.
D-10Result-typed internal error API (an Errno enum, negative errno mapped exactly once per call chain): file-locks, xattr and the VFS inode-op tables converted.
D-11Typed lock keys: nesting = a trait-impl table; acquiring in reverse does not compile, zero runtime. Used by the GRQ, futex buckets and wait slots.
D-12ext4 read-miss window batch fill: one round trip fills the whole miss window, contiguous segments merged into ≤64-block/128KiB requests — the prerequisite for desktop-scale images (per-page synchronous commits cost ~1.8 ms under TCG). Cost: the miss path holds EXT4_BIG_LOCK (correctness first).
D-13Batched metadata publication + discard-on-free: PUB_BATCH=64 batching + forgetting dead blocks + full-block overwrite without reads. p6_mkdrmdir 29,796→2,107 µs, p99 54→16.4 ms.
D-14The measured-wait wakeup preemption gate: poke a busy CPU only when the wakee's measured wait ≥6 ms. 74-byte unix streams 275 µs→~2.5 µs, p5 family 10–18×. Cost: sub-6ms ping-pong wakeups wait for slice expiry.
D-15The single-declaration tracepoint table: one row = the whole event, armed() a single disabled-path branch, typed arguments are compile errors; tracefs byte-aligned. Cost: 10 u64 slots per record.
D-16The deferred-free timer ring: 16,384 slots + a 1500 ms wall-clock drain (covering QSBR-invisible unaccounted borrows). Revisit condition: once borrows become RcuHead, delete the ring.
D-17The kernel-side device alias arena: no identity MMIO window at runtime, no uaccess identity hole (a bad user pointer faults instead of reading physical memory). Guards: PF-NOCTX/SMAP-VIOL beacons stay silent.
D-18SMAP via the pushfq EFLAGS.AC sequence: STAC/CLAC unconditionally #UD under TCG (established by paired trials); ~55 ns per bracket, boots on hardware and TCG alike.
D-19The typed timer wheel: clock domains cannot be mixed, periodic requires NonZeroU64, WakeArmed/WakeSpent scope guards. Six historical bug classes became compile errors; same-test platform A/B: arm lock 924→697 ns, nanosleep 772.6→662.7 ms, p2_ctxsw 12.198 vs 15.400 µs.
D-20The writer-side dirty-writeback watermark on a free-fraction mark (no global dirty accounting): once the Normal zone's free count falls below half its managed pages (zone_below_dirty_writeback_mark), the buffered writer flushes its own inode's dirty range and runs the system-wide recovery pass (writeback_recover: free already-clean pages first, collect dirty inodes via a full-table rotating walk and flush each, directly evict flushed pages). The difference from Linux: Linux throttles by per-BDI dirty thresholds / the balance_dirty_pages dirty fraction, while Rux keeps no per-BDI accounting and uses the zone free-fraction as the mark — the discipline, not the constant, is the contract. It eliminated the delayed-allocation dirty-balloon wedge (243k dirty pages growing to RAM scale, the zone draining to zero); LTP fallocate05/06 on three fresh boots RC=0; riscv64 make test 1584/1584. Cost: no per-BDI precision — with multi-device loads the watermark granularity is coarser than Linux's. Note: PSI (/proc/pressure) is not in main; it is recorded on the debug-alignment resume queue.

§10Rust-native engineering (register highlights)

Thesis: externally Linux-compatible, internally not a C→Rust translation — correctness and speed come from the type system, ownership, or internal architectures Linux does not have. Every entry carries numbers measured by this repository's instruments (entries without evidence are placeholders in the G bucket).

  • The type system turns bug classes into compile errors: D-19's six misuse classes caught by rustc; the S-1 concurrency-primitive model check (loom interleavings + Kani proofs of INV-M1..M5) caught a real pid-0 recursion bug within its first 90 seconds; D-11 lock keys, D-7 CpuLocal, the WalkClaim lockless dcache (stat 0.643→0.423 µs).
  • Safe-by-construction shapes are themselves optimizations: D-1's single-structure forensics (how the OH campaign actually found the scheduling bug family); the leaf-table direct unmap (vm-op family 15→5 s/CPU-min); the negative-lookup memo (oracle 475k/963k/0).
  • Semantic divergences are found by diffing the Linux tree and fixed by adopting its behavior (root causes, not symptoms): execve ETXTBSY, SysV IPC per namespace, shared-futex keys resolved to the physical word, tmpfs MAP_SHARED coherence through the page cache, the binder poll triple-condition, O_DIRECT reads flushing the dirty page-cache range first, SYSRET TF|RF fallback, UTF-8 validation on every user-writable string.
  • Safety properties C kernels rent from tools: constructive memory safety + SMAP/CR0.WP/no identity window/userspace W^X/seccomp/capabilities + a designed panic surface (every instance retired with a regression test).
  • Redesigns that landed as performance (same-test platform A/B): lmbench four-row leads (lat_ctx 3.4×, lat_pipe 2.6×, exec 1.39×, fstat 1.19×), switch −58%, p5 family −90~93%, the fs metadata family −65~−88%; Geekbench multi-core 7504 (4-CPU pinned).
  • Honest placeholders (bucket G): the typed-binder transaction FSM, an ext4/jbd2 host-side test slice, an internal completion-ring writeback base — no A/B numbers yet; the only way out is a measured row.

§11Security and performance together (ledger highlights)

Thesis: security checks happen at compile time and vanish from the fast paths — the fast design and the safe design are the same design. Source: docs/architecture/safety-and-performance.md.

  • Bug taxonomy v1 (3,358 deduplicated commits adjudicated one by one): of 33 memory-corruption defects, 23 live in explicit unsafe blocks/arch assembly and 10 in safe code — all ownership/refcount/lock-protocol classes safe Rust does not type-check today; zero pointer-expression defects (OOB/iterator invalidation/wild-pointer arithmetic/format strings) in safe code. 6 of 33 were caught by review before any boot.
  • The unsafe surface is enumerated and contracted: 7.1% of lines (14.0% on the dual series); 47 service files compile-time forbid(unsafe_code), 63 more already unsafe-free; SAFETY comments cover 63%; 164 Kani harnesses prove the core unsafe code for all inputs.
  • Every performance row is paired with its safety sentence: e.g. the switch path carries typed per-CPU storage + keyed locks; lmbench/Geekbench numbers and the census-adjudicated structure share a table; the cost side (futex wake→run ~10 ms behind a running task, tick-quantized) is recorded as-is.
  • The honest gap register v1: KASLR (no boot entropy source), kernel text/rodata W^X tightening, compiler stack protectors, KASAN-style poisoning breadth, KCSAN watchpoints, kallsyms masking — each with what closing it looks like.