Skip to content

ADR 0004: CPU Affinity and Pinned Threading Model

Status

Approved

Context

OS schedulers balance execution threads dynamically across CPU cores. Moving a low-latency thread between cores invalidates cache locality (L1/L2), resulting in performance degradation and latency spikes.

Decision

Aegis will assign each pipe thread to a dedicated physical CPU core and pin it using operating system CPU affinity calls.

Design Constraints:

  • Threads (Ingestion, Load Balancer, Workers, Writer) are pinned during initialization.
  • We prioritize physical cores over hyperthreaded cores to prevent resource sharing (ALU/FPU bottlenecks).
  • Platform preprocessor checks will compile OS-specific pinning functions (e.g. pthread_setaffinity_np on Linux) and fallback cleanly on Windows.

Options Considered

  1. Dynamic OS Scheduling (No Affinity): Simple, but causes thread migrations, cache pollution, and high tail latencies.
  2. Cooperative Multitasking (Fiber loops): Good for thousands of connections, but introduces user-space scheduling overhead; not suitable for linear pipeline processing.
  3. Core Pinning: Guarantees absolute cache locality and zero thread migration overhead.

Consequences

  • Positive: Warm caches, minimal context switching, and highly deterministic packet latencies.
  • Negative: Consumes physical CPU cores exclusively. Pinning cores to 100% load (due to polling) limits the host system's capability to run other application processes.