ADR 0007: Round-Robin Polling SPSC Array for the Output Writer

Status

Approved

Context

When multiple worker threads finish processing packets, they must pass accepted packets to the single Output Writer thread. A shared Multi-Producer Single-Consumer (MPSC) queue requires multiple threads to contend for the write pointer, necessitating atomic Compare-And-Swap (CAS) instructions. Under high packet rates, cache invalidation and bus locking cause thread stalls and latency spikes.

Decision

We will establish dedicated Single-Producer Single-Consumer (SPSC) queues from each worker thread to the Output Writer thread. The Output Writer thread will poll these queues in a round-robin loop.

Design Constraints:

• Each Fast Path worker has a private SPSC queue pointing to the Writer.
• The Output Writer thread runs a busy-wait poll across all worker queues.
• When a queue contains a packet, the Writer consumes it, writes it to disk/file, and releases the buffer block back to the pool.

Options Considered

1. Shared MPSC Queue: Single queue, but suffers from cache-line bouncing and CAS retries under high contention.
2. Mutex-Guarded Single Queue: High lock contention and context switch overhead.
3. Worker-Dedicated SPSC Queues with Polling Writer: Zero producer contention, lock-free, zero CAS loops.

Consequences

• Positive: Worker threads never experience contention when forwarding packets, resulting in predictable p99 processing latencies.
• Negative: Polling multiple queues increases CPU utilization on the Writer thread, but is appropriate for dedicated low-latency hardware.