Skip to content

Technical Requirements Document (TRD): Aegis Packet Engine

1. System Architecture Overview

Aegis is composed of a multi-stage pipeline designed to minimize cache misses, eliminate dynamic memory allocations during packet processing, and run without lock contention.

Pipeline Stages:

  [Ingestion / PCAP Reader]
             │
             ▼  (SPSC Lock-Free Queue 1)
     [Load Balancer]
       /     │     \
      /      │      \ (SPSC Lock-Free Queues 2..N)
     ▼       ▼       ▼
   [FP0]   [FP1]   [FPk] (Fast Path Workers: Flow Track, DPI, Rules)
     \       │       /
      \      │      / (Individual SPSC Worker-to-Writer Queues)
       ▼     ▼     ▼
    [Output / Exporter (Round-Robin Polling)]

2. Low-Latency Concurrency Design

2.1 Lock-Free Bounded Queues

  • Single Producer Single Consumer (SPSC): The ingestion thread communicates with the Load Balancer via a dedicated SPSC queue. The Load Balancer communicates with each worker (Fast Path) thread using private SPSC queues.
  • Worker-to-Writer SPSC Array: Instead of worker threads competing on a shared Multi-Producer Single-Consumer (MPSC) queue, each worker thread forwards accepted packets to the Output Writer via a private SPSC queue. The Output Writer polls these SPSC queues in a round-robin loop. This guarantees lock-freedom and eliminates producer-producer contention on the write path.
  • Fallback Strategy: If lock-free structures are unsupported (rare on modern x86/ARM), compilation will fall back to using atomic flags or mutexes with diagnostic logs.

2.2 Thread Pinning & CPU Affinity

  • To prevent OS context-switch overhead, threads are pinned to specific CPU cores.
  • Configurable topology via a YAML/JSON file or CLI options (e.g., pin Ingestion to Core 0, Load Balancer to Core 1, Workers to Cores 2-7).
  • Fallback to OS scheduler on platforms that do not support thread pinning (e.g., default Windows).

2.3 False Sharing Prevention

  • All shared queues and thread-local status structures must be padded to the CPU cache line size (64 bytes on x86/x64).
  • Use alignas(64) for atomic pointers, write heads, and read heads.

3. Zero-Copy Packet Lifecycle

3.1 Pre-allocated Packet Pool

  • Memory for packets is allocated once at startup.
  • A PacketBufferPool maintains a ring of pre-allocated buffers of fixed size (2048 bytes).
  • The PCAP Reader writes directly into the pre-allocated buffer.
  • The packet metadata struct (PacketJob) stores:
  • A raw pointer to the buffer: const uint8_t* raw_data.
  • Offsets for headers: eth_offset, ip_offset, tcp_offset, payload_offset.
  • Lengths: packet_length, payload_length.
  • A reference-counting mechanism (atomic counter) to return the buffer to the pool once processed/written.

3.2 String Views

  • IP addresses, MAC addresses, and domain names are never instantiated as std::string inside the critical path.
  • Use std::string_view for SNI or Host sub-strings, referencing the packet payload memory directly.
  • IPs are represented as uint32_t (IPv4) or std::array<uint8_t, 16> (IPv6).

4. State Management (Connection Tracker)

4.1 Partitioned Flow Tables

  • Each Fast Path worker thread owns a private flow table.
  • Zero Contention: Because packets are dispatched using a consistent hash of their 5-tuple, all packets for a connection are guaranteed to go to the same worker. No locks or atomic synchronizations are needed to access or modify the flow table.
  • Data Structure: Cache-efficient hash map with open addressing or closed hashing with pre-allocated nodes to avoid memory allocation.

4.2 TCP Stream Reassembly (Lightweight)

  • Aegis tracks TCP sequence numbers for the handshake phase.
  • Extracts the TLS Client Hello even if it is fragmented across two adjacent TCP segments by copying segments to a small flow-specific scratch buffer.
  • Expired or terminated flows are reclaimed using a timer-based thread-local sweeper.

5. Optimized Rule Engine

5.1 Matching Algorithms

  • IP / Subnet Matching: CIDR ranges are parsed into a Prefix Tree (Trie) or checked using bitmask checks sorted by subnet length (longest prefix match first).
  • Domain / SNI Matching: Fast substring search or Aho-Corasick automaton for multiple string matching.

5.2 Hot-Reloading Rules

  • Rules can be reloaded at runtime.
  • Uses double-buffering (read-copy-update pattern) with a shared pointer or atomic swap. The worker threads swap to the new rule set atomically without blocking.

6. Build & Portability Requirements

  • Target compiler support: GCC >= 8.1, Clang >= 7.0, MSVC >= 2019.
  • Compile flags:
  • Release: -O3 -march=native -ffast-math -flto
  • Debug/Profiling: -O2 -g -ggdb -fno-omit-frame-pointer
  • Static Analysis check: Integration with clang-tidy and compile with -Wall -Wextra -Wpedantic -Werror.