Network Performance & Optimization

Latency is the single most impactful variable in distributed system performance. Understanding the hardware realities — from CPU cache to transatlantic fiber — helps engineers make correct architectural tradeoffs.

L1 Cache hit0.5 ns

L2 Cache hit7 ns

L3 Cache hit20 ns

Main memory (RAM)100 ns

SSD random read (NVMe)0.1 ms

Same-DC network round trip0.5 ms

HDD sequential read1 ms

Cross-region cloud (AWS us-east → eu-west)80 ms

Transatlantic round trip (NY → London)130 ms

Transcontinental (LA → Tokyo)200 ms

CDN Anycast Edge PoP & Origin Shield Architecture

Request Status at Local Edge PoP (Singapore):

⚡ EDGE HIT (Served in 5ms directly from local Edge RAM cache)

Key Metrics

Metric	Definition	Typical Values
Latency (RTT)	Round-trip time for a packet	Same datacenter: <1ms; Cross-continent: 100–200ms
Bandwidth	Maximum data rate	1 Gbps server NIC; 100 Mbps home broadband
Throughput	Actual data transferred per second	Always ≤ bandwidth (overhead reduces it)
Packet loss	% of packets not delivered	>1% causes severe TCP performance degradation
Jitter	Variance in latency	High jitter = poor VoIP/video quality
TTFB	Time To First Byte	Time from request → first byte of response

The Latency Hierarchy (Physical Limits & Speed of Light)

Understanding physical latency constraints prevents impossible architecture SLAs. Network latency across distance is strictly bound by the speed of light in optical fiber ( $\approx 200{,}000\text{ km/s}$ , or $\sim 5\mu\text{s per km}$ round-trip):

Operation	Typical Latency	Scale Analogy (1 CPU Cycle = 1 sec)
L1 CPU Cache Reference	$0.5\text{ – }1\text{ ns}$	1 second
L2 CPU Cache Reference	$3\text{ – }4\text{ ns}$	3 – 4 seconds
L3 CPU Cache Reference	$10\text{ – }20\text{ ns}$	10 – 20 seconds
Main Memory (DRAM) Access	$50\text{ – }100\text{ ns}$	1 – 2 minutes
NVMe SSD Random Read	$10\text{ – }30\ \mu\text{s}$	3 – 8 hours
Same-Rack Datacenter Round-Trip (100GbE)	$10\text{ – }50\ \mu\text{s}$	3 – 14 hours
Cross-AZ in Same AWS Region (e.g., us-east-1a to 1b)	$0.5\text{ – }1.2\text{ ms}$	6 – 14 days
US East to US West (WAN ~4,000 km fiber)	$60\text{ – }75\text{ ms}$	2 – 2.5 years
US to Europe (Trans-Atlantic cable ~6,500 km)	$75\text{ – }90\text{ ms}$	2.5 – 3 years
US to Asia-Pacific (Trans-Pacific cable ~12,000 km)	$150\text{ – }220\text{ ms}$	5 – 7 years
Geostationary Satellite (GEO ~36,000 km orbit)	$500\text{ – }700\text{ ms}$	16 – 22 years
Starlink Low Earth Orbit (LEO ~550 km orbit)	$25\text{ – }45\text{ ms}$	10 – 18 months

QUIC and HTTP/3

QUIC solves TCP's fundamental limitations for HTTP:

Problem 1: TCP head-of-line blocking
  HTTP/2 on TCP: 1 lost packet blocks ALL streams until retransmitted
  QUIC: each stream is independent; one lost packet only blocks its own stream

Problem 2: Connection establishment latency
  TCP + TLS 1.3: 2 RTT before data
  QUIC: 1 RTT first connection; 0-RTT for resumption (known server)

Problem 3: Connection migration (mobile)
  TCP: connection identified by 4-tuple (src_ip:port, dst_ip:port)
       IP change (Wi-Fi → mobile) = connection broken, full reconnect
  QUIC: connection ID is independent of IP — survives network handoffs

0-RTT Resumption:
  First connection:  1 RTT handshake + server sends session ticket
  Later connections: client sends data + session ticket in first packet
                     0 RTT before data starts flowing!

Compression

Response Body Compression

# nginx: gzip compression
gzip on;
gzip_types text/plain application/json application/javascript text/css;
gzip_min_length 1024;         # only compress if > 1KB
gzip_comp_level 6;            # 1-9, tradeoff: cpu vs ratio
gzip_vary on;                 # add Vary: Accept-Encoding header

# Brotli (better than gzip, ~15% smaller)
brotli on;
brotli_types text/plain application/json;
brotli_comp_level 6;

// Spring Boot: enable gzip
server:
  compression:
    enabled: true
    mime-types: application/json,text/html,text/plain
    min-response-size: 1024

Protocol Buffers vs JSON Size

JSON:     {"userId":42,"total":99.90,"status":"pending","items":[{"productId":1,"qty":2}]}
          → ~85 bytes

Protobuf: binary encoding of same message
          → ~25 bytes (3.4x smaller)
          → faster to serialize/deserialize

TCP Tuning for High Throughput

# Increase socket buffers for high-bandwidth-delay-product paths
# BDP = bandwidth × RTT (bytes "in flight" for max throughput)
# 1 Gbps × 100ms RTT = 12.5 MB BDP → buffers must be ≥ 12.5 MB

sysctl -w net.core.rmem_max=16777216         # max receive buffer (16 MB)
sysctl -w net.core.wmem_max=16777216         # max send buffer
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Enable TCP window scaling (needed for BDP > 64KB)
sysctl -w net.ipv4.tcp_window_scaling=1

# Enable SACK (selective ack — only retransmit lost segments)
sysctl -w net.ipv4.tcp_sack=1

# BBR congestion control (better on high-latency / lossy paths)
sysctl -w net.ipv4.tcp_congestion_control=bbr

# Increase connection backlog
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# TIME_WAIT tuning (high-throughput servers)
sysctl -w net.ipv4.tcp_tw_reuse=1            # reuse TIME_WAIT sockets
sysctl -w net.ipv4.ip_local_port_range="1024 65535"  # more ephemeral ports

DNS Pre-Resolution & Connection Pre-Warming

<!-- Browser hints: pre-resolve DNS before user clicks link -->
<link rel="dns-prefetch" href="//api.example.com">

<!-- Pre-connect (DNS + TCP + TLS) -->
<link rel="preconnect" href="https://api.example.com">

<!-- Prefetch entire resource -->
<link rel="prefetch" href="/api/products">

Measuring Network Performance

# Ping: basic latency (ICMP)
ping -c 100 api.example.com    # 100 samples → see jitter
ping -i 0.2 -c 100 google.com  # 200ms interval

# traceroute: path + per-hop latency
traceroute -n api.example.com    # -n = no DNS reverse lookup
mtr api.example.com              # continuous, interactive traceroute

# TCP connection timing
curl -w "\n
DNS:     %{time_namelookup}s
Connect: %{time_connect}s
TLS:     %{time_appconnect}s
TTFB:    %{time_starttransfer}s
Total:   %{time_total}s\n" \
     -o /dev/null -s https://api.example.com

# iperf3: measure actual bandwidth
iperf3 -s                        # server mode
iperf3 -c server-ip -t 30        # client: 30 second test

# ss: socket statistics (Linux)
ss -s                            # summary
ss -tn state established         # TCP established connections
ss -tn sport = :8080             # connections on port 8080

Bandwidth-Delay Product (BDP)

BDP = bandwidth × RTT

Example: 1 Gbps link, 100ms RTT
BDP = 1,000,000,000 bits/s × 0.1s = 100,000,000 bits = 12.5 MB

This is how much data can be "in flight" at once.
TCP window must be ≥ BDP for full link utilization.

Default TCP window: 64KB = ~5 Mbps on 100ms link!
With window scaling (up to 1 GB window): ≥ 1 Gbps on 100ms link ✅

Linux Kernel Ingestion Path & NIC Ring Buffers

To scale beyond 10 Gbps and millions of packets per second (pps), engineers must optimize the physical packet reception path inside the Linux kernel:

[ Physical Wire ]
       │
       ▼
[ NIC Hardware RX FIFO ] ──► [ Direct Memory Access (DMA) ]
                                      │
                                      ▼
                           [ RX Ring Buffer (Descriptor Ring) ]
                                      │
                               [ Hard IRQ (CPU) ]
                                      │
                         [ NAPI Polling Routine ]
                                      │
                         [ SoftIRQ: ksoftirqd/core_id ]
                                      │
                    [ Allocate sk_buff (Socket Buffer) ]
                                      │
             ┌────────────────────────┴────────────────────────┐
             ▼                                                 ▼
      [ Gro / TSO Assembly ]                           [ eBPF / XDP Bypass ]
             │                                                 │
             ▼                                         (Drop / Redirect
   [ netif_receive_skb ]                                without sk_buff)
             │
             ▼
   [ IP / Routing / Netfilter ]
             │
             ▼
    [ TCP Socket Recv-Q ] ──► [ Application: sys_read / epoll ]

1. Diagnosing Kernel Ingestion Drops (`/proc/net/softnet_stat`)

# Each line corresponds to one CPU core
cat /proc/net/softnet_stat

# Formatted interpretation:
# Col 1: Total processed frame count
# Col 2: Squeezed count (NAPI poll ran out of time budget before draining ring)
# Col 3: Dropped frames (netdev_max_backlog queue full!)

Kernel Tuning Parameters:

# 1. Expand NIC hardware ring buffers (prevent packet drops during micro-bursts)
ethtool -g eth0                              # View current and max ring sizes
ethtool -G eth0 rx 4096 tx 4096              # Maximize hardware descriptor rings

# 2. Increase NAPI budget (allow SoftIRQ to process more frames per interrupt)
sysctl -w net.core.netdev_budget=600         # Default: 300
sysctl -w net.core.netdev_budget_usecs=4000  # Default: 2000 us

# 3. Increase backlog queue between driver and IP stack
sysctl -w net.core.netdev_max_backlog=10000  # Default: 1000

# 4. Enable Receive Side Scaling (RSS) to distribute 4-tuple flows across CPU cores
ethtool -k eth0 | grep receive-hashing       # Ensure 'on'

2. eBPF / XDP (eXpress Data Path) Wire-Speed Bypass

Traditional Linux packet processing allocates a heavy kernel sk_buff data structure ( $\sim 240\text{ bytes}$ ) per packet. Under a 10M pps volumetric flood, memory allocation alone exhausts CPU cache and memory bandwidth.

XDP Solution: Runs an eBPF bytecode program inside the network driver before sk_buff allocation. It inspects packet headers directly inside the DMA ring buffer:

XDP_DROP: Discards malicious packets at wire speed ( $\sim 24\text{ – }40\text{ million pps}$ on a single CPU core).
XDP_TX: Bounces packet back out the same interface (wire-speed load balancer).
XDP_PASS: Hands benign traffic to the standard Linux TCP/IP stack.

Bufferbloat & Active Queue Management (AQM)

Bufferbloat is the excessive buffering of packets inside intermediate network devices (home routers, switches, DSLAMs).

The Problem

Traditional routers use Drop-Tail FIFO queues. When a large file transfer saturates the link, the TCP sender's congestion window keeps growing until the queue is 100% full. Because buffers are sized in megabytes, packets sit in queues for hundreds of milliseconds:

Interactive traffic (SSH keystrokes, DNS lookups, video game UDP packets) is stuck behind the elephant flow's queued packets.
Latency (RTT) inflates from $15\text{ ms}$ to $> 1500\text{ ms}$ under load!

The Solution: FQ-CoDel & CAKE

Modern Linux kernels replace simple FIFO with FQ-CoDel (Fair Queueing Controlled Delay):

Flow Isolation (FQ): Hashes packets into separate queues by 5-tuple. Small interactive packets (mice flows) bypass bulk transfers (elephant flows).
Controlled Delay (CoDel): Measures how long packets dwell in the queue (sojourn time). If the minimum sojourn time over a 100ms interval exceeds $5\text{ ms}$ , CoDel drops packets or sets ECN (Explicit Congestion Notification) bits, forcing the sender to slow down before buffers inflate.

# Enable FQ-CoDel as the default queueing discipline across all interfaces
sysctl -w net.core.default_qdisc=fq_codel

Congestion Control: BBR vs CUBIC

Dimension	CUBIC (Loss-Based)	BBR (Model-Based)
Underlying Premise	Packet loss = congestion. Grows window until a drop occurs.	Bufferbloat $\ne$ congestion. Models the physical pipe capacity.
Operating Point	Fills the entire link and intermediate router buffers.	Kleinrock optimal point: $BDP = \text{BtlBw} \times \text{RTprop}$ .
Lossy Networks (Wireless/WAN)	Catastrophic throughput drop. A $1\%$ random loss cuts throughput by up to $80\%$ .	Immune to non-congestive loss. Maintains line rate up to $15\text{–}20\%$ packet loss.
Bufferbloat Impact	High. Causes persistent queue standing delay.	Low. Drains queue regularly by pacing transmissions.
Linux Configuration	`sysctl -w net.ipv4.tcp_congestion_control=cubic`	`sysctl -w net.ipv4.tcp_congestion_control=bbr`

Kernel TLS (kTLS) & Zero-Copy I/O

Traditional user-space HTTPS servers (OpenSSL / BoringSSL) cannot use Linux zero-copy primitives like sendfile():

Traditional HTTPS:
[ Disk ] ──► [ Page Cache ] ──► [ User Space: Decrypt / Encrypt ] ──► [ Kernel Socket Buffer ] ──► [ NIC ]
(4 context switches + 2 CPU memory copies per send)

Kernel TLS (kTLS):
[ Disk ] ──► [ Page Cache ] ────────────────────────────────────────► [ Kernel kTLS Engine ] ──► [ NIC DMA ]
(2 context switches + 0 CPU copies via sendfile() + Hardware Crypto offload)

With kTLS (Linux 4.13+):

User-space initiates the TLS handshake (OpenSSL/BoringSSL) to negotiate session keys.
The symmetric cipher keys (AES-128-GCM / AES-256-GCM) are installed into the kernel socket via setsockopt(fd, SOL_TLS, TLS_TX, ...).
The application simply invokes sendfile(socket_fd, file_fd, offset, count).
The kernel encrypts data on the fly directly inside kernel space—or offloads encryption entirely to SmartNIC crypto processors—cutting CPU utilization by up to $60\%$ .

Interview Questions

Q1. What is the difference between latency, bandwidth, and throughput?

Latency: time for a packet to travel from source to destination (one-way) or round-trip (RTT). Bandwidth: maximum capacity of the channel (bits per second). Throughput: actual useful data transferred per second — always less than bandwidth due to overhead, protocol inefficiency, and latency-induced idle time. Analogy: a highway (bandwidth), travel time (latency), actual cars passing per hour (throughput).

Q2. Why does connection establishment add significant overhead, and how is it minimized?

TCP 3-way handshake + TLS 1.3 = 2 RTTs before the first byte of HTTP data. On a 100ms RTT link, this is 200ms just for setup. Minimized by: HTTP keep-alive (reuse connections for many requests), connection pooling (pre-established pool of connections), HTTP/2 multiplexing (one connection, many concurrent requests), QUIC 0-RTT (resume previous sessions with zero handshake latency).

Q3. What is head-of-line blocking and how does HTTP/3 solve it?

TCP delivers data in order — if one packet is lost, all subsequent packets queue up waiting for the retransmission (even if they belong to unrelated HTTP/2 streams). HTTP/3 uses QUIC over UDP, where each stream's byte sequence is independent. A lost packet for stream 3 only blocks stream 3, not streams 1, 2, 4. This is critical for lossy networks where HTTP/2's TCP HoL blocking degrades performance significantly.

Q4. What is the Bandwidth-Delay Product and why does it matter for TCP?

BDP = bandwidth × RTT = the maximum amount of data "in flight" needed to fully utilize the link. The TCP receive window must be at least as large as the BDP for optimal throughput. Default TCP windows (64KB) limit throughput to ~5 Mbps on a 100ms RTT link regardless of bandwidth. TCP window scaling (enabled by default on modern OSes) allows windows up to 1 GB, enabling full utilization of high-BDP paths.

Q5. How does HTTP/2 header compression (HPACK) work?

HPACK maintains a static table of common header name-value pairs (:method GET, :status 200, etc.) and a dynamic table of previously sent headers. Instead of sending full header strings, HPACK sends table indices. Headers not in the table are Huffman-encoded. This eliminates redundant headers (User-Agent, Authorization are the same on every request) — typically 85%+ header size reduction, critical for mobile where headers can exceed body size.

Q6. What is QUIC 0-RTT resumption and what are its security implications?

0-RTT: on the first connection, the server sends a session ticket. On reconnect, the client sends data alongside the session ticket in the first packet — no handshake RTT. Security concern: 0-RTT data can be replayed by an attacker who captures the initial packet and retransmits it. Mitigation: only use 0-RTT for idempotent requests (GET), not for state-changing operations (POST payments). gRPC over HTTP/3 and browsers enforce this.

Q7. What tools would you use to diagnose poor network performance in production?

ping/mtr for latency and packet loss. traceroute to identify slow hops. curl -w timing for TTFB breakdown (DNS, connect, TLS, first byte). iperf3 for raw bandwidth measurement. ss -s or netstat for connection states (TIME_WAIT accumulation, SYN queues). tcpdump/Wireshark for deep packet inspection. APM tools (Datadog, New Relic) for application-level latency breakdown. dmesg for kernel errors (TCP buffer overruns).

Q8. What is TCP slow start and how does it affect first-request performance?

TCP slow start begins every new connection with a small congestion window (typically 10 MSS = ~14KB initial window in modern Linux). It doubles each RTT until a threshold. For a 1 MB response on a 50ms RTT link, slow start means: RTT 1: 14KB, RTT 2: 28KB, RTT 3: 56KB... taking 4-5 RTTs (200-250ms) to reach full throughput. Mitigation: keep-alive (avoids new connections), smaller responses, CDN edge caching (shorter RTT), increase initial congestion window (ip route change ... initcwnd 100).