Skip to content

Architecture & Concurrency Model: Aegis Packet Engine

1. System Topology

The system leverages a lock-free pipeline architecture designed for multi-core CPUs. In a high-throughput systems design, we isolate responsibilities to prevent cache trashing and lock contention.

                  +-----------------------+
                  |  Ingestion / Reader   |
                  +-----------+-----------+
                              | (Reads packet into Pool Buffer)
                              | (Assigns index)
                              v
                  +-----------+-----------+
                  |     Load Balancer     |
                  +-----/     |     \-----+
                       /      |      \
     (Hash % Workers) /       |       \ (SPSC Queues)
                     v        v        v
                  +----+    +----+    +----+
                  |FP0 |    |FP1 |    |FPk | (Fast Path Workers)
                  +--+-+    +--+-+    +--+-+
                     |         |         |
                     |         |         | (Private SPSC Queues)
                     v         v         v
                  +-----------+-----------+
                  |   Output Writer / IO  | (Round-Robin Polling)
                  +-----------------------+

2. Component Details

2.1 Ingestion / Reader

  • Function: Ingests packets from PCAP file or live interface.
  • Details:
  • Runs on a dedicated thread.
  • Aquires a free buffer from the PacketBufferPool.
  • Reads raw packet bytes into the buffer.
  • Constructs a lightweight PacketJob (containing timestamps, offsets, and length).
  • Pushes the PacketJob index/reference to the Load Balancer's SPSC queue.

2.2 Load Balancer (LB)

  • Function: Dispatches jobs to worker threads based on the 5-tuple.
  • Details:
  • Computes a hash of the packet's FiveTuple (IPs, Ports, Protocol).
  • Selects the target Fast Path (FP) worker using hash % NumWorkers.
  • Pushes the job to that worker's SPSC queue.
  • Consistent hashing guarantees that all packets belonging to the same connection are processed by the same worker thread.

2.3 Fast Path Workers (FP)

  • Function: Parse protocol headers, track connection state, run DPI, and apply filtering.
  • Details:
  • Thread-isolated: Each FP thread has its own ConnectionTracker map.
  • Parses Ethernet, IPv4/IPv6, TCP/UDP headers.
  • Performs SNI (TLS Client Hello) or Host (HTTP) extraction.
  • Consults the local cache of blocking rules.
  • If a packet is allowed, it is pushed to the worker's dedicated Output SPSC queue. If blocked, the packet buffer is released, and stats are updated.

2.4 Output Writer

  • Function: Serializes filtered packets to an output PCAP file or discards them.
  • Details:
  • Runs on a dedicated thread.
  • Polls all worker-to-writer SPSC queues in a round-robin loop.
  • Writes the packet header and payload back to disk if a packet is present.
  • Releases the buffer back to the PacketBufferPool after write completion.

3. Concurrency & Synchronization Model

3.1 Lock-Free Bounded Ring Buffer

We implement a lock-free Single-Producer Single-Consumer (SPSC) queue based on a ring buffer of fixed capacity. * Head/Tail Counters: Uses std::atomic<size_t> for head and tail indexes. * Cache Line Padding: Separates head and tail counters by 64 bytes (alignas(64)) to prevent false sharing between the producer thread and the consumer thread.

3.2 Thread Allocation & Core Pinning

A typical thread topology for a 4-core physical CPU (with Hyperthreading/SMT disabled or handled):

Core ID Thread Type Description
Core 0 Ingestion / Reader Network ingestion, packet buffer acquisition.
Core 1 Load Balancer Computes 5-tuple hash and distributes to worker queues.
Core 2 Fast Path Worker 0 Deep inspection, flow tracking, rules matching.
Core 3 Fast Path Worker 1 Deep inspection, flow tracking, rules matching.
Core 4 Output Writer / IO PCAP output serialization and pool buffer release.

On systems with fewer cores, thread pinning falls back to standard scheduler assignment, but Aegis prioritizes hardware isolation where possible.

3.3 Memory Reclamation and Atomic Counters

  • Packets are stored in a fixed pool of blocks.
  • The PacketJob contains a reference to the buffer's block.
  • When a packet is dropped, the FP worker increments/releases it.
  • When a packet is accepted, it is sent to the writer, which writes and releases it.
  • A thread-safe atomic reference counter tracks block ownership, preventing double-frees and leaks.