Memory Architecture: Aegis Packet Engine

1. Zero-Allocation Pipeline Principles

In low-latency systems, dynamic memory management (malloc, free, new, delete) is prohibited during execution. Aegis achieves zero heap allocation on the fast path by leveraging pre-allocated memory pools.

       Startup (Heap allocation permitted)
                     │
                     ▼
  +-------------------------------------+
  | PacketBufferPool    FlowEntryPool   |
  | (Flat arrays of pre-allocated blocks|
  +-------------------------------------+
                     │
                     ▼
         Packet Processing Ingestion
  +-------------------------------------+
  | Acquire index -> Parse -> Track     |
  | (No new/malloc, uses indices/views) |
  +-------------------------------------+
                     │
                     ▼
            Packet Reclamation
  +-------------------------------------+
  | Release index back to free list     |
  +-------------------------------------+

---

2. Pre-allocated Packet Buffer Pool & Memory Mapping

Aegis defines a PacketBufferPool that manages a single contiguous block of memory divided into fixed-size segments.

Contiguous Buffer Pool (Mapped at startup):
+---------------------------------------------------------------------------------+
| PacketBlock 0      | PacketBlock 1      | PacketBlock 2      | ...              |
| [2048 Bytes]       | [2048 Bytes]       | [2048 Bytes]       |                  |
+--------------------+--------------------+--------------------+------------------+
   ^                    ^
   |                    |
   | (Ref Pointer)      | (Ref Pointer)
   |                    |
+--------------------+  |
| PacketJob 0        |  |
| - block_idx = 0    |  |
| - eth_offset = 0   |  |
| - ip_offset = 14   |  |
+--------------------+  |
                        |
                     +--------------------+
                     | PacketJob 1        |
                     | - block_idx = 1    |
                     | - eth_offset = 0   |
                     | - ip_offset = 14   |
                     +--------------------+

Key Data Structures:

• Buffer Block:

  struct PacketBlock {
      alignas(64) uint8_t data[2048]; // Max MTU sizes fit within 2KB
      std::atomic<uint32_t> ref_count{0}; // Atomic reference count for multi-thread sharing
  };
  

• Free Index Queue:
• A lock-free SPSC/MPMC queue storing index IDs of free blocks: 0, 1, 2, ..., PoolSize-1.
• Acquire a block: size_t idx = free_indices.pop();
• Release a block:

  if (block[idx].ref_count.fetch_sub(1, std::memory_order_acq_rel) == 1) {
      free_indices.push(idx);
  }
  

---

3. Flat Cache-Friendly Flow Table

Rather than using a pointer-heavy node-based hash map (like std::unordered_map which does a heap allocation for every insertion), Aegis implements a dense bucket hash table or a flat pre-allocated map.

Design Choices:

• Pre-allocated Buckets:

  struct FlowNode {
      FiveTuple tuple;
      ConnectionState state;
      AppType app_type;
      std::array<char, 64> sni; // Fixed-size char array instead of std::string
      size_t sni_len;
      uint64_t packets;
      uint64_t bytes;
      uint32_t next_node_idx; // For collision chaining in a flat array
      bool in_use;
  };
  

• Collision Resolution:
• Chaining is done via indices pointing to other nodes in the flat FlowPool array, ensuring all data is sequentially and contiguously stored in memory. This improves cache-line prefetching.
• Fast Path threads run with zero allocation when tracking new connections. If the map runs out of capacity, it drops the flow or evicts the oldest via a circular list (LRU tracking).

---

4. Avoiding std::string Allocations

The original ParsedPacket structure uses std::string src_ip; and std::string dest_ip; to represent parsed IP addresses. This is extremely slow because it requires a heap allocation for the string copy.

Aegis Representation:

• IPv4 Address: uint32_t (4 bytes).
• IPv6 Address: std::array<uint8_t, 16> (16 bytes).
• MAC Address: std::array<uint8_t, 6> (6 bytes).
• Formating: Formatting to human-readable strings (e.g. 192.168.1.1) is only done during reporting, logging, or CLI output. The core engine passes and matches these fields using raw integer operations.
• Payload Views: Parsers expose payload fields via std::string_view which points directly into the raw packet buffer block memory.