Skip to content

ADR 0003: Contiguous Memory Pool for Packet Buffers

Status

Approved

Context

Raw network packet frames vary in size and must be processed by the ingestion, load balancing, worker, and writer threads. Re-allocating buffers dynamically for every packet frame causes CPU locks and heap fragmentation, limiting engine throughput.

Decision

We will implement a contiguous PacketBufferPool allocated once during system startup.

Design Constraints:

  • Fixed Size Blocks: Each packet buffer block is fixed to 2048 bytes (to accommodate standard 1500-byte MTU ethernet frames plus protocol headers).
  • Reference Counting: Each block contains an atomic reference counter. A block is only returned to the pool once its reference count drops to 0 (e.g. after being drop-discarded or written to output by the writer thread).
  • Free Index Tracking: Available block indices are tracked in a lock-free queue.

Options Considered

  1. Dynamic std::vector<uint8_t> Allocation: High allocation latency, frequent memory fragmentation.
  2. Object Pooling using standard mutexes: Simple, but introduces synchronization bottlenecks on the buffer acquisition/release paths.
  3. Lock-Free Contiguous Block Pool: Fastest block acquisition (<15ns), zero heap allocations, zero locks.

Consequences

  • Positive: Constant-time buffer allocation/deallocation, zero heap allocations on the packet path, safe multi-threaded buffer sharing via atomic reference counts.
  • Negative: Fixed capacity. System limits must be sized appropriately (e.g. 65,536 buffers) to avoid packet drops under extreme network queues.