Aegis Deep Packet Inspection & Networking Tutorial
This tutorial provides a walkthrough of fundamental networking concepts, the mechanics of Deep Packet Inspection (DPI), Server Name Indication (SNI) extraction, and flow-based packet filtering as implemented in the Aegis Framework.
1. What is DPI?
Deep Packet Inspection (DPI) is a network packet filtering technology that examines the data field (payload) of a packet as it passes an inspection checkpoint, unlike conventional packet filtering which only analyzes packet headers (e.g., source and destination IP addresses or port numbers).
Real-World Use Cases
- Quality of Service (QoS): Traffic shaping and bandwidth allocation for specific protocols (e.g., deprioritizing file-sharing streams).
- Security & Intrusion Detection: Inspecting payloads for known malware signatures or protocol exploits.
- Access Control: Blocking or redirecting traffic to restricted resources or websites on enterprise/ISP levels.
Traffic Stream (PCAP/Wire) ────► [ Aegis DPI Core ] ────► Forwarded Stream (Permitted)
│
▼
- Protocol Classification
- Policy & Rules Check
- Flow Statistics Tracking
2. Networking Background
The Network Stack (Layers)
Packets processed by Aegis pass through multiple encapsulation layers:
┌────────────────────────────────────────────────────────────────────────┐
│ Layer 7: Application (HTTP Host, TLS Client Hello, DNS Name) │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 4: Transport (TCP / UDP Headers, Ports, Sequence Numbers) │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 3: Network (IPv4 / IPv6 Headers, Source & Destination IPs) │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 2: Data Link (Ethernet Frame, MAC Addresses, EtherType) │
└────────────────────────────────────────────────────────────────────────┘
Packet Layout & Encapsulation
Each layer wraps the next, similar to a nested structure. For example, a TLS Client Hello packet over TCP looks like:
┌──────────────────────────────────────────────────────────────────┐
│ Ethernet Header (14 bytes) │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ IP Header (20 bytes) │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ TCP Header (20 bytes) │ │ │
│ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ Payload (Application Data) │ │ │ │
│ │ │ │ e.g., TLS Client Hello containing plaintext SNI │ │ │ │
│ │ │ └──────────────────────────────────────────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
The 5-Tuple Definition
A packet's flow (connection) is uniquely identified by the 5-Tuple:
1. Source IP: The address of the sender.
2. Destination IP: The address of the receiver.
3. Source Port: The port number representing the sending socket.
4. Destination Port: The port number representing the destination service (e.g., 443 for HTTPS, 80 for HTTP).
5. Protocol: The transport layer protocol (6 for TCP, 17 for UDP).
All packets sharing the same 5-tuple (or its reverse for bidirectional traffic) belong to the same flow.
3. Server Name Indication (SNI) Extraction
Server Name Indication (SNI) is an extension to the TLS protocol that indicates what hostname the client is attempting to connect to at the start of the handshaking process.
The TLS Handshake Flow
Because the TLS handshake begins before the secure channel is established, the client sends its destination host in plaintext:
┌──────────┐ ┌──────────┐
│ Client │ │ Server │
└────┬─────┘ └────┬─────┘
│ │
│ ──── Client Hello (Contains plain SNI) ────────────►│
│ │
│ ◄─── Server Hello (Contains Certificate) ───────────│
│ │
│ [ Key Exchange & Negotiation ] │
│ │
│ ◄═══ Encrypted Session Stream ════════════════════► │
TLS Client Hello Byte Structure
To extract the SNI, the parser scans the TLS record bytes sequentially:
Byte 0: Content Type = 0x16 (Handshake Record)
Bytes 1-2: Protocol Version = 0x0301, 0x0302, or 0x0303 (TLS)
Bytes 3-4: Record Length
-- Handshake Sub-layer --
Byte 5: Handshake Type = 0x01 (Client Hello)
Bytes 6-8: Handshake Length
-- Client Hello Body --
Bytes X-Y: Random Values, Session ID, Cipher Suites, and Compressions (skipped by offset)
Bytes Z-Z+1: Extensions Length
-- Loop Extensions to find SNI (Type 0x0000) --
Bytes E-E+1: Extension Type (0x0000 = SNI)
Bytes E+2-3: Extension Data Length
Bytes E+4-5: Server Name List Length
Byte E+6: Server Name Type (0x00 = Hostname)
Bytes E+7-8: Hostname Length (K)
Bytes E+9...: Hostname Value (Plaintext string, e.g. "www.youtube.com")
4. How Access Control & Blocking Works
Aegis filters packets by associating access control policies with tracked flows.
Flow-Based Blocking Lifecycle
Because protocol metadata (like the SNI or HTTP host) appears after the connection handshake, Aegis applies rules statefully:
Connection to blocked.com:
1. TCP SYN ──► No SNI visible yet ──► FORWARD (Track Flow)
2. TCP SYN-ACK ──► No SNI visible yet ──► FORWARD (Track Flow)
3. TCP ACK ──► Connection active ──► FORWARD (Track Flow)
4. Client Hello ──► SNI "blocked.com" ──► DROP (Mark Flow as Blocked)
5. Subsequent ──► Flow is BLOCKED ──► DROP immediately
- Bidirectional Blocking: Once a flow is marked as blocked, all future packets in both directions (client-to-server and server-to-client) are dropped.
- Resource Reclamation: Closed or timed-out flows are pruned periodically from the connection tracker to reclaim memory slots.
5. Packet Processing Pipeline Lifecycles
Aegis supports two operation modes:
A. Single-Threaded Mode (Educational Reference)
Designed to illustrate basic protocol parsing and flow lookup without thread concurrency:
1. Read Frame: Read raw bytes and timestamp from PCAP file.
2. Parse Headers: Extract MACs, IPs, Ports, and Protocol.
3. Flow Lookup: Match 5-tuple against flow table.
4. Extract Payload Metadata: If destination is port 443, parse TLS Client Hello; if port 80, parse HTTP Host.
5. Check Rules: Consult the IP, Application, or Domain blacklist.
6. Action: If permitted, write packet directly to output file; if blocked, increment stats and ignore.
B. High-Performance Multi-Threaded Mode (Aegis Core)
Optimized for high-throughput, low-latency execution:
1. Ingestion Core: PCAP Reader thread reads packets directly into pre-allocated memory slots in the PacketBufferPool and pushes a lightweight PacketView descriptor.
2. Load Balancers: Distribute PacketViews to target workers using a 5-tuple consistent hash ring.
3. Worker Cores: Each worker thread polls its own lock-free SPSC queue, updates its local ConnectionTracker flow states, checks RCU-managed rules, and routes permitted packets to its worker-to-writer SPSC queue.
4. Polling Output Writer: Scans all worker-to-writer queues in a lock-free round-robin loop, writes permitted frames to disk, and releases memory block indices back to the pool.