Interrupts & timers
kernel/src/interrupt/ + drivers/intc/ + drivers/timer/ + timer/ Β· The IRQ framework, softirqs, the preemption model, IF=0 discipline, the typed timer wheel, NOHZ and the vDSO β mechanisms and measured numbers Β· Sources: design-divergences.md (D-19), docs/development/perf-gate-baseline.md (the interrupt guarded domain)
Β§1The interrupt framework
- The irq_desc model (kernel/src/interrupt/): one descriptor per IRQ holding the action chain; the irq_chip function-pointer table abstracts controller operations; irq_domain performs the hwirqβvirq mapping; request_irq/free_irq/request_nmi form the registration surface.
- Dispatch: riscv64 claims the highest-priority pending IRQ from the PLIC and enters the IRQ framework, with EOI performed by the flow handler via irq_chip.irq_eoi; x86_64 enters trap_handler directly by vector; aarch64 uses the GICv3 claim/EOI pair.
Β§2Vector layout (per architecture)
| Arch | Layout |
|---|---|
| x86_64 | LAPIC timer 0xE0; dedicated IPI vectors 0xE1β0xE5 (CALL_FUNCTION/RESCHEDULE/TLB_FLUSH/STOP/IRQ_WORK β one vector per IPI type, skipping the pending-bit handshake); PIC remapped master 0x20-0x27 / slave 0x28-0x2F (IRQ0=32); syscall 0x80; spurious 0xFF. External interrupts stay on the 8259 (virtual wire): LVT LINT0 = ExtINT β every EOI must call both the APIC eoi and pic_send_eoi (a leftover in-service state wedges all further interrupt delivery). |
| riscv64 | PLIC at 0x0c000000 (accessed through the kernel-side MMIO alias; the identity low window is not cloneable by user MAP_FIXED); two contexts per hart, S-mode = 2Β·hart+1; the priority layering: UART (irq 10) = 5, virtio-mmio (1..=8) = 3, everything else at base β pre-fix, every IRQ had priority 1 and the console could not preempt bulk device traffic. The CLINT is M-mode only (OpenSBI v1.3+); S-mode goes through SBI; IPIs use the SBI IPI extension. |
| aarch64 | GICv3: INTIDs 0-15 are SGIs (INTID 0 carries the IPI bitmap), 16-31 PPIs (INTID 27 = PPI 11 = the EL1 virtual timer), 32+ SPIs (PL011 = SPI 1); the CPU interface uses the ICC_* system registers; GICD_CTLR under DS=1: bit 1 = Grp1NS enable (writing the secure-world encoding leaves Group-1 forwarding off β zero IRQ exceptions, the timer PPI latched in GICR_ISPENDINGR); SGIs go via ICC_SGI1R (TargetList/INTID). |
Β§3Preemption model and the IRQ-off discipline
- preempt_count bit layout: PREEMPT [0:7], SOFTIRQ [8:15], HARDIRQ [16:19], NMI [20:23], PREEMPT_ACTIVE bit 26; the count lives on the Task struct;
preemptible()= count==0. - irq_enter/exit spans a context switch: HARDIRQ_OFFSET is booked by the task that entered β the tick handler may switch inside the bracket, so the per-CPU booked-task stack releases on the booked task; a split is the producer of the "leaked 0x10000 face" (established as the root cause on the 119th epoll-pwait run).
- Kernel preemption defaults off (explicit schedule() voluntary points); the
kpreempt=boot param arms the irq-exit preemption hook; the A/B reads flat for hog regimes and the default stays off. - The IF=0 discipline and the official measurement: the EFLAGS-pair probe (syscost v3) reads the real IF flag at every hook and adjudicates the model against it β a modeled-open segment found with IF=1 is a leak, closed into the HEAL counters; the HARDIRQ/SOFTIRQ/UART zones split separately; REST call sites attribute via #[track_caller] into a 96-slot table. Official share: IF=0 = 20.4 Β±3pp (pa115debt1; the historical anchor pa27 = 25.7%, the drop's mechanism being phantom-bits moving softirq dispatch out of the irq_exit IF=0 window into schedulable ksoftirqd); the longest closed cli segment in the REST zone is 6β9 ms (band β€10 ms; the former 594β599 ms one-shots are gone); the 210 ms hardirq-frame one-shot is a named watch row (the starvation bound of inline dispatch).
Β§4The softirq model
- 10 types: Hi/Timer/NetTx/NetRx/Block/IrqPoll/Tasklet/Sched/Hrtimer/Rcu; a per-CPU pending bitmap plus per-CPU/per-vector raise counters (feeding /proc/softirqs).
- Dispatch: raised from hardirq; processed in
__do_softirq()at irq_exit, overflowing into ksoftirqd. A double budget bound:MAX_SOFTIRQ_RESTART = 10rounds +2 msof wall clock (the Linux MAX_SOFTIRQ_TIME shape) β rationale: an unbounded irq-exit budget once let a timer/IO storm pin both CPUs in IRQ context (93% si_busy) with zero scheduling picks. - ksoftirqd: one per CPU; schedules on need_resched inside the drain loop (the phantom-bits fairness rule: otherwise a feedback-heavy backlog lets the kthread monopolize its CPU through thousands of 2 ms windows); the re-check-before-schedule lost-wake pattern (the run_ksoftirqd shape).
- Window progress records: SOFTIRQ_WIN packed {open|handler bit|restart_count|pending} + SOFTIRQ_WIN_T0, cross-CPU readable β external forensics for wedged windows; a softirq handler booking SOFTIRQ_OFFSET plus a heal landing mid-window wedges the machine (VW-WEDGE established as the root cause, 29 completions lost).
Β§5The timer subsystem and the typed wheel (D-19)
- Time base: nominal 10 MHz (100 ns/cycle); the x86_64 timekeeper is the TSC via multiply-shift with HPET calibration (a 1 Hz resync gated), all state in the vDSO data page; the riscv64 time CSR is naturally 10 MHz; aarch64 uses the EL1 virtual timer (CNTV_CVAL).
- Wheel shape: 64 buckets per level, three levels (granularity 1/64/4096 jiffies, total span 262144 jiffies β 43.7 min at HZ=100) plus a FAR list for deeper horizons (rescanned every 4096 jiffies β 41 s); per-pass work is O(due entries); plain fixed .bss storage with the single TIMER_TABLE lock; delivery outside the lock (R12-3).
- Six historical bug classes eliminated by the type system: mixed clock-domain comparison (Deadline<JiffyClock>/<HresClock> mixing = compile error), one-shot action leak (the Action enum is exclusive: Wake/Signal/TimerFd), periodicity misuse (Option<NonZeroU64>), the silent if/else-if delivery chain, slot-lifecycle misuse (FiredToken by value), and delivery-context overreach (a SoftirqCtx ZST pins delivery to softirq context). On the real tree: six misuses were all rejected by rustc (E0308/E0599/E0061/E0382).
- Caller-side guards: WakeArmed owns one registration id (not Copy/Clone β a double cancel is unrepresentable); cancel(self) consumes into the WakeSpent tombstone; Drop cancels an unconsumed guard (the pool-exhaustion wedge class); hres arms bake in the hart re-program (an hres sleep whose caller forgets the re-program waits for the next 10 ms grid tick).
- Same-test platform A/B (s5-boot, OH image, KVM q35 smp2/4G, syscost=1, 780 s; A = the old flat-table kernel, B = the typed wheel): timer arm lock hold 924β697 ns, pending 1288β868 ns, del 1291β862 ns (volume matched within 0.1%); nanosleep average 772.6β662.7 ms; pbench p2_ctxsw 12.198 vs 15.400 Β΅s. Cost: slot 56β64 B (a 66 KB .bss pool).
- Superseded shape: a flat 1024-slot array whose expiry was re-checked on every softirq pass, with timer classes encoded as runtime flag combinations.
Β§6NOHZ / tick-sched
- A function-for-function mirror of Linux's tick-sched.c (check_tick_dependency, tick_nohz_stop/restart_sched_tick, tick_nohz_irq_exit, the nohz_full= parameter); per-CPU TickSched{inidle, tick_stopped, tick_dep_mask, next_tick}; tick_dep bits POSIX_TIMER/PERF_EVENTS/SCHED/CLOCK_UNSTABLE/RCU/RCU_EXP.
- Structural mappings (recorded in the rust-native register): per-rq β GRQ (the SCHED dependency is evaluated on the fly by can_stop_full_tick; the 0β1 kick kept as a resched IPI from the enqueue edge); clockevent KTIME_MAX arm β timer_arm_oneshot(u64::MAX) (the u32 count clamp parks the LAPIC at max delta); tick_do_timer_cpu β the boot CPU with sleep capped at TIMEKEEPING_MAX_DEFERMENT (10 s); perf: the dep tracks any armed sampling event β Rux software sampling rides the tick, recorded as a divergence (Linux's per-event hrtimer swevent is the follow-up).
- Wired on x86_64; the other architectures keep their grid-glue set_next_trigger. Non-nohz_full CPUs keep the byte-identical grid path.
- Measured (4+4 interleaved lmbench lat_syscall,null runs, 6 hostile external QEMUs): the main tip passes 0/10; nohz_full=1 β 11/12 (92%); p1_null median 0.050 vs 0.060 Β΅s (β17%); both census bands hold exactly; GRQ average hold 699 ns (a release-class cross-class reading, not comparable to the debug band); a pre-nohz parent control confirmed the p2 drift belongs to board spread, not the nohz change.
Β§7The vDSO
- Structure: every execve installs a two-page user mapping β a data page (RW, the kernel-updated time base) and a code page (RX, a minimal ELF with three symbols); AT_SYSINFO_EHDR points at the code page; glibc/musl call __vdso_clock_gettime/gettimeofday/clock_getres directly.
- Clock set: REALTIME(0)/MONOTONIC(1)/MONOTONIC_RAW(4)/BOOTTIME(7)/TAI(11 = REALTIME + 37); on x86_64 any other clockid falls back inside the vDSO to the real syscall.
- Bit-equivalence design (x86_64, the unified track): read_time()'s TSC branch and the vDSO functions compute the same value from the same words with the same integer operations β nom(t) = base_mono_nom + ((tsc(t) β cycle_last) Γ mult) >> 32, integer-associative so the two tracks are bit-identical (what LTP clock_gettime04 demands); base_mono_nom is nominal-domain and monotone; the seq word doubles as the writer lock (cmpxchg evenβodd), closing the multi-writer publication tear; the data page is user-RO β precisely because the kernel trusts its words.
- The rejected fast path (on record): per-CPU correction publication carried the TSC_CPU_OFF spread β tsleep2 β356 ms at t<1 s; the syscall-vs-vdso spread to +1.2 ms; a clock_gettime04 residual of β4582 ns. The unified track replaces the correction chain with bit-equivalence.
- The riscv64 lesson: the seq word must be read THROUGH the page pointer β a former separate VDSO_SEQ static was never aliased to the page, so a reader descheduled between its base_tsc and base_mono_ns loads paired a stale tsc with a fresh mono (a vDSO clock ahead of the syscall clock by the whole preemption gap; LTP clock_gettime04 "Time travelled backwards β454618735 ns").
- Measured (name-only same-test platform A/B, tok-r4fix vehicle, KVM q35 SMP2 4G, 780 s): clock_gettime syscalls per OH boot 8,259,773 β 0 (β29.4 CPU-s = β2.0% of the machine, β26% of total syscall volume), the nanosleep average and the GRQ lock-site average flat, zero panics on both sides; LTP clock_gettime04 6/6 including the vdso variants and both COARSE clocks.
Β§8IPI costs and TLB shootdowns
- The resched IPI sender side alone is 935β1139 ns; IPI+halt exit ~2β5 Β΅s of KVM exit; wakes inside the 120 Β΅s polling idle window elide the IPI (Dekker SeqCst pairing for lost-wake safety).
- The TLB-shootdown lazy mm_cpumask: pre-fix, every munmap/fork-COW/mprotect/COW fault broadcast a full-flush IPI to all CPUs β a pristine OH boot counted 4,031,356 by t=200s (~20k/s); the per-mm cpu_mask (set at switch-in, cleared at switch-away, grow-only under PCID) reduces this to 10,209 (β99.7%) with flush semantics unchanged. Mechanism details on the memory-management Β§10.
Code:
kernel/src/interrupt/ (preempt/softirq/ksoftirqd/irqchip), kernel/src/drivers/intc/ (apic/plic/clint/gicv3), kernel/src/drivers/timer/, kernel/src/timer/ (wheel/oneshot/nohz), kernel/src/mm/vdso.rs. Divergence: D-19 (the typed wheel). Gate bands (the interrupt guarded domain): IF=0 20.4Β±3pp, max cli REST β€10 ms, SOFTIRQ 3.1Β±2pp. Instruments: the syscost IF=0 EFLAGS-pair probe, tick-mode shares, rux:schedtrace (see performance debugging).