Skip to content

Concurrency & Lock-Free Design: Aegis Packet Engine

1. Lock-Free Queue Topology

To achieve high throughput and predictable low-latency, Aegis utilizes lock-free queues to pass packet references between pipeline stages.

 [Ingestion] ---> (SPSC) ---> [Load Balancer] ---> (SPSC Array) ---> [Workers] ---> (SPSC Array) ---> [Writer]

Queue Selection:

  1. Single-Producer Single-Consumer (SPSC): Bounded ring buffer. Very fast, uses atomic index updates with relaxed/acquire-release memory ordering. No CAS (Compare-And-Swap) loop required.
  2. Worker-to-Writer SPSC Array: Instead of an MPSC queue, each worker thread has a private SPSC queue connected to the Output Writer. The Output Writer thread polls these queues in a round-robin loop. This architecture ensures absolute lock-freedom and eliminates producer-producer contention on the write path.

2. Bounded SPSC Lock-Free Queue Details

An SPSC queue has exactly one writer thread and one reader thread. This simplifies synchronization, allowing us to implement it without a CAS loop.

SPSC Implementation Strategy:

  • Storage: Pre-allocated array of type T of size Capacity (must be a power of 2 to allow fast bitwise modulo: index & (Capacity - 1)).
  • Write Index: Updated only by the producer. Exposed to the consumer via std::atomic<size_t> write_idx_.
  • Read Index: Updated only by the consumer. Exposed to the producer via std::atomic<size_t> read_idx_.
  • Memory Ordering:
  • Producer writes element, then stores write_idx_ using std::memory_order_release.
  • Consumer loads write_idx_ using std::memory_order_acquire, reads element, then stores read_idx_ using std::memory_order_release.
  • Producer loads read_idx_ using std::memory_order_acquire to verify space availability.
  • Cache Alignment: 
    alignas(64) std::atomic<size_t> write_idx_{0};
alignas(64) std::atomic<size_t> read_idx_{0};
    This ensures the read and write indexes reside on separate cache lines. The producer thread repeatedly writes to write_idx_ and reads read_idx_, while the consumer thread repeatedly writes to read_idx_ and reads write_idx_. Separating them avoids cache line bouncing (false sharing).

3. Thread Pinning (CPU Affinity)

Pinning threads prevents the operating system scheduler from moving execution context between cores, which invalidates CPU caches (L1/L2).

Implementation Pattern (Linux/POSIX):

#ifdef __linux__
#include <pthread.h>
#include <sched.h>

bool pinThreadToCore(std::thread& thread, int core_id) {
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(core_id, &cpuset);

    pthread_t native_handle = thread.native_handle();
    int rc = pthread_setaffinity_np(native_handle, sizeof(cpu_set_t), &cpuset);
    return rc == 0;
}
#else
// Fallback for non-Linux platforms (e.g. Windows)
bool pinThreadToCore(std::thread&, int) {
    return false; // Dynamic scheduling fallback
}
#endif

4. Backpressure and Queue Full Strategies

Since our queues are bounded to avoid memory bloating: 1. Ingestion Queue Full: The PCAP reader stalls. This is acceptable as offline PCAP processing is rate-limited by reader speeds. For live captures, we track buffer drops. 2. Worker Queue Full: If a worker's SPSC queue is full, the Load Balancer thread can perform a spin-lock (busy wait with asm volatile("pause" ::: "memory") or std::this_thread::yield()) for a short duration before trying again. 3. Writer Queue Full: If a worker's dedicated Output SPSC queue is full, workers block or drop the packet with a "buffer drop" stat if it's live traffic.