Computer Networks — Cheat Sheet

Seven layers: Physical, Data Link, Network, Transport, Session, Presentation, Application. The honest framing is that OSI is a teaching model, not an implementation. The internet runs on the TCP/IP model, which has four layers: Link, Internet, Transport, and Application. OSI's session and presentation layers have no distinct implementation — TLS is sometimes called presentation-layer, but it sits awkwardly between transport and application in practice. What is genuinely useful is the layering principle: each layer provides a service to the one above and does not care how the one below works. TCP does not know whether it is running over Ethernet or Wi-Fi. HTTP does not know whether TCP retransmitted a segment. That is why you can reason about an HTTP problem without thinking about frames, and why encapsulation works — each layer adds its header and treats the payload as opaque. The practical value in an interview is the vocabulary. "Layer 4 load balancer" versus "layer 7" is a meaningful distinction, and it is the main reason the numbers are worth knowing at all.

A layer 4 load balancer routes on transport information — IP addresses and ports. It forwards packets or proxies TCP connections without inspecting the payload. A layer 7 load balancer terminates the connection, parses the application protocol, and routes on its contents: URL path, headers, cookies, HTTP method. The trade is capability against cost. Layer 4 is faster and cheaper — it does not parse anything and can often forward at near line rate. But it cannot do path-based routing, header-based canary releases, sticky sessions by cookie, or per-request retries, because it cannot see requests at all, only a byte stream. Layer 7 gives all of that, plus TLS termination, response caching, compression, and request-level observability. The cost is CPU and latency, and it must terminate TLS to read anything. The practical consequence people miss: a layer 4 balancer distributes connections, not requests. With HTTP/2 or keep-alive, one long-lived connection carries many requests, so a busy client can pin disproportionate load to one backend. That is exactly when you need layer 7.

The Maximum Transmission Unit is the largest payload a link can carry in one frame. Ethernet is conventionally 1500 bytes. A packet larger than the MTU must be fragmented or dropped. IPv4 routers could fragment, but that is inefficient — losing one fragment means retransmitting the whole datagram — and IPv6 removed router fragmentation entirely, requiring the sender to handle it. Path MTU Discovery is how senders find the right size: send with the Don't Fragment bit set, and a router that cannot forward it replies with an ICMP "fragmentation needed" message carrying the correct MTU. The failure mode is the interesting part. Many networks blindly block ICMP, so that message never arrives. The sender keeps retransmitting oversized packets that are silently dropped, and the connection hangs after the handshake succeeds — small packets work, large ones vanish. This is the classic "TLS handshake completes then everything stalls" symptom. It bites hard with tunnels and overlays. VPNs and VXLAN add headers, reducing the effective MTU, which is why container networking so often needs explicit MTU configuration.

A hub is a layer 1 repeater: it broadcasts every incoming signal to every other port. Everyone sees everyone's traffic, collisions are frequent, and bandwidth is shared. Obsolete. A switch operates at layer 2, forwarding frames by MAC address. It learns which MAC lives on which port and forwards only where needed, so each port gets dedicated bandwidth and traffic is not broadcast to everyone. A switch defines a broadcast domain — broadcasts still go everywhere within it. A router operates at layer 3, forwarding packets between different networks by IP address, consulting a routing table. Routers separate broadcast domains, which is what stops a broadcast storm from crossing the whole internet. The distinction that matters for debugging: a switch problem is local and shows as devices on the same subnet not reaching each other; a router problem shows as cross-subnet failures where local traffic is fine. Layer 3 switches blur the line by doing routing in hardware, which is what most data centre fabrics use.

Latency is the time for one unit of data to travel from source to destination — measured in milliseconds. Bandwidth is the theoretical maximum rate a link can carry — measured in bits per second. Throughput is the rate actually achieved. They are independent, which is the point of the question. A satellite link can have enormous bandwidth and 600 ms latency. A short cable can have low latency and modest bandwidth. Throughput is bounded by bandwidth but usually far below it, because of protocol overhead, congestion, packet loss, and — critically — the interaction of latency with window size. TCP can only have one window of unacknowledged data in flight, so maximum throughput is roughly window size divided by round-trip time. This is the bandwidth-delay product, and it is why a high-bandwidth high-latency link performs badly without window scaling. The engineering consequence is that adding bandwidth does not fix a latency problem. A page making thirty sequential requests over a 100 ms link takes three seconds regardless of bandwidth, which is exactly why HTTP/2 multiplexing and request batching matter more than raw pipe size.

The bandwidth-delay product is bandwidth multiplied by round-trip time — the amount of data that can be "in flight" on a link at any moment. It matters because TCP can only send one window of unacknowledged data before it must wait for an acknowledgement. If the window is smaller than the bandwidth-delay product, the sender stalls waiting for ACKs while the link sits idle. Concretely: a 1 Gbps link with 100 ms round trip has a product of about 12.5 MB. The original TCP window field is 16 bits, capping the window at 64 KB — which on that link would achieve roughly 5 Mbps, half a percent of capacity. That is why the window scaling option exists, multiplying the window by a power of two so it can reach hundreds of megabytes. The practical implications: cross-region replication needs large socket buffers, and tuning net.ipv4.tcp_rmem and tcp_wmem is what unlocks throughput on long fat networks. It also explains why a transfer between continents is slow even on fast links — the limit is often window size, not bandwidth.

The classic question, and the value is in naming the layers rather than reciting steps. The browser checks its own cache and HSTS list. It resolves the hostname: browser cache, OS cache, hosts file, then a recursive DNS query walking root, TLD, and authoritative servers unless something along the way has it cached. With an IP, it opens a TCP connection — three-way handshake, one round trip. For HTTPS, a TLS handshake follows: one round trip with TLS 1.3, two with 1.2, negotiating cipher and validating the certificate chain. It sends the HTTP request. The server — usually via a load balancer and reverse proxy — routes it, the application handles it, and a response returns. The browser parses the HTML, and discovers subresources which each may need their own DNS, TCP and TLS work unless connections are reused. It builds the DOM and CSSOM, runs JavaScript, computes layout, and paints. The details worth volunteering: connection reuse and HTTP/2 multiplexing avoid repeating the handshakes, and each of DNS, TCP and TLS costs a round trip, which is why latency compounds.

Network Address Translation rewrites addresses as packets cross a boundary, letting many devices on a private network share one public IP. The router keeps a table mapping internal address and port to external port so replies can be routed back. It exists because IPv4 addresses ran out, and it has extended IPv4's life by decades. The problems are real. It breaks the end-to-end principle: an outside host cannot initiate a connection inward without explicit port forwarding, which is why peer-to-peer applications need STUN, TURN and hole-punching. It requires the router to hold connection state, so it is a stateful bottleneck and a single point of failure. Protocols that embed addresses in their payload, such as classic FTP and SIP, break unless the NAT inspects and rewrites them. And it complicates logging and rate limiting — many users behind one carrier-grade NAT share an IP, so blocking by IP punishes everyone. IPv6 removes the need entirely by giving every device a globally routable address, though NAT's accidental firewall effect is one reason adoption has been slow.

Address Resolution Protocol maps an IP address to a MAC address on a local network. It is needed because IP is layer 3 and Ethernet is layer 2. To actually put a frame on the wire, the sender needs the destination's MAC, and it only knows the IP. The host broadcasts "who has 192.168.1.50?" to the whole segment. The owner replies with its MAC, and the sender caches the mapping for a few minutes. For a destination outside the subnet, the host ARPs for the default gateway instead, because the frame goes to the router even though the IP is remote. That distinction — layer 2 destination is the next hop, layer 3 destination is the final target — is the conceptual point. ARP has no authentication, so ARP spoofing is trivial on a local network: an attacker replies claiming to own an IP and intercepts traffic. That is the basis of many man-in-the-middle attacks on shared Wi-Fi, and part of why TLS matters even on a "trusted" network. IPv6 replaces ARP with Neighbour Discovery over ICMPv6.

The headline is address space: 32 bits giving 4.3 billion addresses versus 128 bits giving an effectively unlimited number. But several design changes matter more day to day. The IPv6 header is simpler and fixed-length, with optional extension headers chained after it, which makes router processing faster. The IPv4 header checksum is gone entirely, since link and transport layers already checksum. Routers no longer fragment — only the sender does, which pushes Path MTU Discovery from optional to required. Address configuration changes: IPv6 has stateless autoconfiguration, so a host can derive an address from the router advertisement without DHCP. Broadcast is replaced by multicast, which reduces unnecessary interrupts on hosts that do not care. And NAT is unnecessary, restoring the end-to-end model. The practical concern in application code is that address parsing, storage and logging must handle both, that literal IPv6 addresses in URLs need brackets, and that dual-stack hosts resolve to both — Happy Eyeballs exists because a broken IPv6 path would otherwise stall connections.

Subnetting splits an IP address into a network portion and a host portion. CIDR notation writes the split as a suffix: /24 means the first 24 bits identify the network and the remaining 8 identify hosts. So 192.168.1.0/24 has 256 addresses, of which 254 are usable — the first is the network address and the last is the broadcast address. The arithmetic worth being fluent in: /24 is 256 addresses, /25 is 128, /26 is 64, and each additional bit halves it. Going the other way, /23 is 512, /22 is 1024, /16 is 65,536. CIDR replaced the old class A/B/C system, which allocated in fixed enormous blocks and wasted addresses catastrophically — an organisation needing 300 addresses got a class B with 65,000. The practical relevance is cloud networking. A VPC is a CIDR block, subnets carve it up, and security group rules are expressed in CIDR. Sizing a subnet too small is painful to fix later, since you cannot resize in place, and overlapping CIDRs between VPCs make peering impossible — both are common and expensive mistakes.

A routing table maps destination network prefixes to next hops. For each packet, the router finds the matching entry and forwards accordingly. The selection rule is longest prefix match: when several entries match, the most specific one wins. A packet for 10.1.2.3 matching both 10.0.0.0/8 and 10.1.0.0/16 takes the /16 route, because it is more specific. That is what makes hierarchical routing work — a general route can cover a large space while specific exceptions override it. The default route 0.0.0.0/0 matches everything and has the shortest prefix, so it is chosen only when nothing else matches. That is why it is called the gateway of last resort. Beyond prefix length, routers use administrative distance to choose between protocols and metrics to choose within one. For debugging, ip route on Linux shows the table, and ip route get ADDRESS asks the kernel which route it would actually use for a given destination — far more reliable than reading the table and reasoning about it yourself.

Border Gateway Protocol is how autonomous systems — large networks such as ISPs and cloud providers — exchange routing information. It is the protocol that holds the internet together. It works by announcement: each AS tells its neighbours which prefixes it can reach and the path to get there. Routes propagate outward, and each network applies policy to decide which to prefer. The fragility is that BGP was designed for a small trusted network of operators, so it largely trusts announcements. If a network announces a prefix it does not own, others may believe it. That is route hijacking, and it has redirected significant traffic both accidentally and maliciously. The accidental version is more common: a misconfiguration leaks internal routes, or withdraws prefixes, and a provider effectively disappears from the internet. Facebook's 2021 outage was exactly this — a BGP withdrawal removed the routes to their DNS servers, so nothing could resolve. RPKI adds cryptographic validation of who may announce what, and adoption has grown, but propagation remains slow and convergence after a change takes minutes.

Private ranges are reserved for internal networks and are not routable on the public internet: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. Public addresses are globally unique and routable, allocated through regional registries. The consequence is that a device with only a private address cannot be reached from outside without NAT or a proxy, and the same private range is reused by millions of networks simultaneously — which is exactly why 192.168.1.1 is your router and also everyone else's. The practical relevance is cloud architecture. Instances in a private subnet have no public address and reach the internet through a NAT gateway; instances in a public subnet have a route to an internet gateway. Getting that wrong is why a service cannot pull dependencies, or conversely why a database is unexpectedly reachable from outside. The range worth remembering beyond those three is 169.254.0.0/16, link-local. An interface holding a 169.254 address means DHCP failed. And 169.254.169.254 specifically is the cloud instance metadata endpoint, which is why SSRF against it is such a serious vulnerability.

It exploits the TTL field. Every IP packet has a Time To Live that each router decrements; when it hits zero the router discards the packet and returns an ICMP "time exceeded" message. Traceroute sends packets with TTL 1, then 2, then 3. The first expires at the first router, which identifies itself in the ICMP reply. The second expires at the second router. Incrementing reveals each hop in turn. Three probes per hop give a sense of latency variance. The interpretation traps are worth knowing. Asterisks do not mean a hop is down — many routers deprioritise or block ICMP responses, so a silent hop can be forwarding traffic perfectly. Latency to an intermediate hop reflects how quickly that router generates ICMP, not the path quality, so a spike in the middle that does not persist to later hops is meaningless. And the return path may differ from the forward path, so the numbers are round trips over possibly asymmetric routes. mtr is the better tool in practice, running continuously so you see loss patterns rather than a single sample.

Internet Control Message Protocol carries control and error messages for IP — destination unreachable, time exceeded, fragmentation needed, echo request and reply. Ping and traceroute are built on it, which is why people think of it as a diagnostic protocol. But it is also load-bearing for normal operation. Blocking it wholesale is a common and damaging mistake. Path MTU Discovery depends on receiving "fragmentation needed" messages; block them and connections hang when packets exceed the path MTU, with the maddening symptom that the handshake succeeds and then large transfers stall. The security rationale — that ping reveals host existence and ICMP has been used for tunnelling and amplification — is real but narrow. The correct response is to rate limit and to block specific types, not all of them. The pragmatic rule is to allow type 3 (destination unreachable, including fragmentation needed) and type 11 (time exceeded) even if you block echo. That preserves the protocol's essential function while removing the trivial host discovery.

Anycast advertises the same IP address from multiple locations, and routing delivers each client to the topologically nearest instance. It is a routing trick rather than a protocol feature: several sites announce the same prefix via BGP, and each network picks whichever announcement looks best from its position. The benefits are latency and resilience. Clients reach a nearby node without any DNS or application logic, and if a site fails its announcement is withdrawn and traffic shifts automatically. It is what makes public DNS resolvers like 8.8.8.8 and 1.1.1.1 fast everywhere, and it underpins CDN edge routing and DDoS absorption — an attack is spread across many sites rather than concentrated. The limitation is that anycast is connectionless-friendly but awkward for long-lived stateful connections. A routing change mid-connection can send packets to a different node with no knowledge of that TCP session, breaking it. That is fine for DNS over UDP, which is a single exchange, and is why anycast plus TCP needs care — usually solved by terminating at the edge and keeping state local.

Unicast is one-to-one: a packet addressed to a single host. Almost all traffic is unicast. Broadcast is one-to-all within a subnet. Every host on the segment receives and processes it, which is why broadcast domains are kept small — a large one wastes CPU on every machine. ARP uses broadcast. Multicast is one-to-many, but only to hosts that have subscribed to a group. Routers and switches build distribution trees so a single sent packet is duplicated only where paths diverge, which is far more efficient than sending N unicast copies. Multicast is used heavily in financial market data feeds and IPTV, where thousands of receivers want the same stream. It is largely absent from the public internet because it requires router support and cooperation that ISPs never broadly deployed. IPv6 drops broadcast entirely in favour of multicast to specific groups, which avoids interrupting hosts that have no interest. The practical relevance for backend work is mostly service discovery on local networks — mDNS and similar — and understanding why cloud VPCs generally do not support multicast.

A Virtual LAN partitions a physical switch into multiple logical broadcast domains. Ports assigned to different VLANs behave as if they were on separate switches, even sharing the same hardware. The motivations are segmentation and flexibility. Segmentation limits broadcast traffic and contains the blast radius of a compromise — a device on the guest VLAN cannot reach the finance VLAN without passing through a router where policy is enforced. Flexibility means grouping by function rather than physical location, so machines in different rooms can share a network. Traffic between VLANs must be routed, which is where access control is applied. VLAN tags are carried in the frame using 802.1Q, adding four bytes — which reduces the effective MTU and is one reason MTU mismatches appear in enterprise networks. The cloud equivalent is the VPC and its subnets, and the same reasoning applies: isolate by trust level, force cross-boundary traffic through a control point. VLAN hopping attacks exist, which is why unused ports should be disabled and native VLANs handled carefully.

A forward proxy sits in front of clients and acts on their behalf. The client is configured to use it, and the destination server sees the proxy's address rather than the client's. Used for corporate egress filtering, caching, and anonymity. A reverse proxy sits in front of servers and acts on their behalf. Clients are unaware of it — they simply connect to what they believe is the service. It routes to backends, terminates TLS, caches, compresses, and load balances. So the difference is which side it represents and which side knows about it. Nginx, HAProxy, Envoy and cloud application load balancers are reverse proxies. A corporate web filter or Squid in the classic configuration is a forward proxy. The practical implications for backend engineers are mostly reverse-proxy concerns: the real client IP arrives in X-Forwarded-For rather than the socket, so logging and rate limiting must read it — and must not trust it blindly, since a client can forge the header unless the proxy overwrites it. Timeouts at the proxy and at the application must be coordinated, or you get confusing 502s where the proxy gives up first.

The client sends SYN with its initial sequence number. The server replies SYN-ACK, acknowledging the client's number and sending its own. The client sends ACK, acknowledging the server's. Three messages are needed because both sides must agree on sequence numbers, and each direction needs independent confirmation. Two would let one side know the other received its number, but not the reverse. Initial sequence numbers are randomised, not zero. That is a security measure: predictable numbers allow an off-path attacker to inject data into an established connection. The cost is one full round trip before any data flows, which is why connection reuse matters so much. On a 100 ms link, opening a connection costs 100 ms before the request is even sent — and TLS adds more. TCP Fast Open lets data ride along with the SYN using a cookie from a previous connection, eliminating that round trip, though middlebox interference has limited adoption. The SYN queue is what a SYN flood attacks: half-open connections consume server state, which SYN cookies mitigate by encoding state in the sequence number instead of storing it.

Because TCP connections are full-duplex and each direction is closed independently. One side sends FIN meaning "I have no more data". The other ACKs it, but may still have data to send, so the connection is now half-closed — one direction shut, the other still flowing. When that side finishes it sends its own FIN, which is ACKed. So four messages: FIN, ACK, FIN, ACK. The middle two are sometimes combined if the second side has nothing more to send, giving three. Half-close is a real feature, not an artefact. A client can signal end-of-request with a FIN while still reading the response, which is how some protocols delimit messages. The side that initiates the close enters TIME_WAIT, holding the socket for twice the maximum segment lifetime — typically 60 seconds on Linux. That is why a busy client can exhaust ephemeral ports while the server is fine: the active closer pays the cost. Designing so the server closes first moves that burden, which is one reason keep-alive and connection pooling matter for high-throughput clients.

After the side that initiated the close sends its final ACK, the socket sits in TIME_WAIT for twice the maximum segment lifetime — 60 seconds on Linux. Two reasons. First, if that final ACK is lost, the peer retransmits its FIN, and the socket must still exist to re-acknowledge it. Without TIME_WAIT the retransmitted FIN would get an RST, and the peer would think the connection failed. Second, it prevents delayed segments from an old connection being delivered to a new connection reusing the same four-tuple. Waiting two segment lifetimes guarantees any straggler has expired. The operational problem is port exhaustion. A client opening thousands of short connections per second accumulates TIME_WAIT sockets and runs out of ephemeral ports, seeing connection failures while the server looks healthy. The correct fixes are connection pooling and keep-alive so connections are reused rather than churned. tcp_tw_reuse allows safe reuse for outgoing connections. tcp_tw_recycle was the dangerous knob that broke clients behind NAT and has been removed from modern kernels — worth knowing so you do not suggest it.

Flow control protects the receiver. Congestion control protects the network. Flow control uses the advertised receive window: the receiver tells the sender how much buffer space it has, and the sender must not exceed it. If the application is slow to read, the window shrinks, eventually to zero, and the sender stops. This is a two-party agreement with an explicit signal. Congestion control protects the shared path between them, and there is no explicit signal — the network does not send "slow down". The sender infers congestion from packet loss or increasing delay, and maintains its own congestion window. The amount actually in flight is the minimum of the two windows. The distinction matters for diagnosis. A zero window in a packet capture means the receiving application is not reading fast enough — an application problem. Repeated retransmissions and a collapsing congestion window mean the path is lossy or saturated — a network problem. They look similar from the outside as "slow transfer" but have completely different fixes.

Slow start is misleadingly named — it ramps up exponentially. The congestion window begins small, typically ten segments, and doubles every round trip as acknowledgements arrive. It is "slow" only compared to immediately blasting at full rate. This continues until the window reaches the slow start threshold or loss occurs, at which point the connection enters congestion avoidance and grows linearly — roughly one segment per round trip — probing carefully for more capacity. On loss, the behaviour depends on how it was detected. Three duplicate ACKs indicate a single lost segment while data still flows, so fast recovery halves the window and continues. A timeout indicates something worse, so the window collapses to the initial value and slow start restarts. The practical consequence is that short connections never reach full speed. A connection carrying a 50 KB response may finish entirely within slow start, so measured throughput is far below link capacity. That is a strong argument for connection reuse: an established warm connection has a large congestion window, while a fresh one starts from scratch. It is also why increasing the initial window to ten segments measurably improved web performance.

Head-of-line blocking is when the first item in a queue prevents everything behind it from progressing, even though those items are ready. It occurs at several layers, and distinguishing them matters. At the HTTP/1.1 layer: one connection handles one request at a time, so a slow response blocks every queued request. Browsers worked around this by opening six connections per host. At the HTTP/2 layer: multiplexing solved the application-level version — many streams share one connection. But TCP still delivers bytes in order, so a single lost packet stalls every stream on that connection until it is retransmitted. HTTP/2 made the problem worse under loss than HTTP/1.1 with parallel connections. That is precisely why HTTP/3 moved to QUIC over UDP: QUIC implements per-stream ordering, so a lost packet blocks only its own stream. It also appears in message queues, where one slow consumer blocks a partition, and in thread pools, where one long task blocks queued work. The general fix is always the same — independent lanes rather than one shared ordered queue.

Nagle's algorithm reduces the number of small packets by buffering: if there is unacknowledged data outstanding, small writes are held until either an ACK arrives or a full segment accumulates. It exists because sending a 1-byte payload with 40 bytes of headers is terribly inefficient, and telnet-style traffic once threatened to congest networks with tiny packets. The problem is its interaction with delayed ACKs, which hold acknowledgements for up to 200 ms hoping to piggyback them on outgoing data. Nagle waits for the ACK; the receiver delays the ACK waiting for data. The result is a 200 ms stall on a request that should take microseconds. That combination is a classic source of mysterious latency in request-response protocols where the request is split across two small writes. Disable it with TCP_NODELAY when your protocol is latency-sensitive and you control write batching yourself — which is why almost every RPC framework, database driver and game server sets it. The better fix where possible is to write the whole message in one call, which avoids the small-write pattern entirely rather than working around it.

The ones worth recognising in ss or netstat output: ESTABLISHED is a healthy open connection. LISTEN is a server socket awaiting connections. SYN_SENT in quantity means outgoing connections are not completing — a firewall dropping packets or an unreachable destination. SYN_RECV in quantity on a server can indicate a SYN flood. TIME_WAIT in large numbers is normal for a busy active-closing side, and only a problem when approaching ephemeral port exhaustion. CLOSE_WAIT is the one that signals an application bug. It means the peer sent FIN and the kernel acknowledged it, but your application has not called close on the socket. They accumulate indefinitely and leak file descriptors, because only the application can move them forward. A growing CLOSE_WAIT count is almost always a missing close in an error path. FIN_WAIT_2 means you closed and are waiting for the peer's FIN; a build-up suggests the peer is not closing. ss -s gives the summary, and ss -tan state close-wait finds the specific sockets.

Connection refused means the packet reached the host and the host actively rejected it with a TCP RST — usually because nothing is listening on that port. It is fast, because you get an explicit answer. Connection timeout means no response at all. The SYN was sent, retransmitted several times, and nothing came back. That takes the full timeout, often tens of seconds. The diagnostic value is in what each implies. Refused means routing works, the host is up, and the port is closed — so check whether the service is running and bound to the right interface. A service bound to 127.0.0.1 rather than 0.0.0.0 gives exactly this from another machine. Timeout means packets are being silently dropped, which almost always means a firewall or security group rule, or a routing problem, or the host is down. The distinction is why firewalls prefer DROP over REJECT — dropping produces a timeout that slows down port scanning, whereas rejecting answers immediately. In cloud environments a timeout is nearly always a security group, and refused is nearly always the application.

They share a name and are unrelated, which is exactly why this gets asked. TCP keep-alive is a transport mechanism that periodically sends an empty probe on an idle connection to check the peer is still there. Linux defaults are conservative — two hours idle before the first probe — so it detects dead peers eventually, not promptly. It exists to reap connections where the peer vanished without sending FIN, such as a crashed machine or a removed NAT mapping. HTTP keep-alive is an application-layer convention meaning the TCP connection is reused for multiple HTTP requests rather than closed after each response. It is the default in HTTP/1.1, and it avoids repeating the TCP and TLS handshakes. So TCP keep-alive checks liveness; HTTP keep-alive avoids reconnection cost. The practical trap is idle timeouts in the middle. Load balancers and NAT devices drop idle connections silently, often after 60 seconds, so a pooled connection can be dead while the client believes it is fine — producing intermittent failures on the first request after idleness. Setting the client idle timeout below the infrastructure timeout is the fix.

Because TCP is designed to keep trying rather than fail fast, so several problems present as a stall. A path MTU black hole: large packets are dropped and the ICMP notification is blocked, so the handshake succeeds and the first large transfer hangs. Classic symptom of small requests working and large ones not. A zero receive window: the peer's application has stopped reading, so the sender is blocked. Visible in a packet capture as window updates of zero. A silently dropped connection mid-path: a firewall or NAT expired the state, so packets go nowhere and there is no RST to signal it. Without TCP keep-alive, both sides wait indefinitely. Application-level deadlock: both sides waiting to read, neither writing, which no amount of network debugging will reveal. The diagnostic sequence is to check ss for the socket state and queue sizes — Send-Q growing means data is not being acknowledged, Recv-Q growing means the application is not reading — then take a packet capture to see whether anything is on the wire at all. Always set socket read timeouts; a hang with no timeout is an outage.

They are two distinct queues in the kernel, and conflating them makes tuning ineffective. The SYN queue holds half-open connections — a SYN has arrived and SYN-ACK been sent, but the final ACK has not. Its size is controlled by net.ipv4.tcp_max_syn_backlog. Overflow here happens under a SYN flood or a very high connection rate, and SYN cookies are the mitigation. The accept queue holds fully established connections waiting for the application to call accept. Its size is the backlog argument to listen(), capped by net.core.somaxconn. Overflow here means the application is not accepting fast enough. The distinction matters because they indicate different problems. SYN queue overflow is a network-facing or attack condition. Accept queue overflow is an application throughput problem — your server is too slow to pick up connections. When the accept queue is full, Linux by default drops the ACK silently, so the client retransmits and eventually times out, which looks like a mysterious intermittent connection failure. ss -lnt shows Recv-Q and Send-Q for listening sockets as current and maximum accept queue depth.

An outgoing connection needs a local port, drawn from the ephemeral range — typically 32768 to 60999 on Linux, about 28,000 ports. A connection is identified by the four-tuple of source IP, source port, destination IP, destination port. So the limit is 28,000 concurrent connections to the same destination IP and port, not 28,000 overall — connections to different destinations can reuse the same local port. Exhaustion happens with high connection churn, because TIME_WAIT holds ports for 60 seconds after close. Two thousand new connections per second to one backend means 120,000 sockets in TIME_WAIT, far exceeding the range. The symptom is intermittent "cannot assign requested address" errors on the client while the server appears healthy. The real fix is connection pooling with keep-alive, so connections are reused instead of churned — this addresses the cause rather than the symptom. Secondary measures: widen the range with net.ipv4.ip_local_port_range, enable net.ipv4.tcp_tw_reuse for outgoing connections, or spread load across more destination IPs. Never use tcp_tw_recycle; it breaks NAT clients and has been removed.

Two mechanisms, with very different costs. Fast retransmit: when a receiver gets an out-of-order segment it re-sends the acknowledgement for the last in-order byte. Three duplicate ACKs tell the sender a segment was lost while later ones arrived, so it retransmits immediately without waiting for a timeout. The congestion window halves and the connection continues — a modest penalty. Retransmission timeout: if no acknowledgement arrives at all, the sender waits for a timeout computed from measured round-trip time, retransmits, and doubles the timeout on each failure. This is expensive: the window collapses to the initial value and slow start restarts, so throughput craters. The difference matters because one lost packet in a stream recovers cheaply, while a burst that loses everything in flight triggers a timeout. Selective acknowledgement improves things by letting the receiver report exactly which ranges arrived, so the sender retransmits only the gaps rather than everything from the loss point. The application sees none of this — only latency. Which is why unexplained tail latency is often loss on the path.

QUIC is a transport protocol built on UDP that provides what TCP provides plus several things TCP structurally cannot. It eliminates head-of-line blocking. TCP guarantees byte ordering for the whole connection, so one lost packet stalls everything. QUIC maintains independent streams, so loss affects only the stream it belongs to. It merges the transport and cryptographic handshakes. TCP plus TLS 1.3 costs two round trips; QUIC costs one, and zero for a resumed connection. It supports connection migration. A QUIC connection is identified by a connection ID rather than the four-tuple, so a phone moving from Wi-Fi to cellular keeps the connection alive where TCP would break it. And because it lives in user space rather than the kernel, congestion control can be updated without an OS upgrade — TCP improvements take a decade to deploy. The cost is CPU: UDP packet processing is less optimised, and encryption is mandatory. Some networks also block or throttle UDP. HTTP/3 is HTTP over QUIC, which is the main reason it exists.

SO_REUSEADDR lets a socket bind to a port still held in TIME_WAIT by a previous instance. Without it, restarting a server fails with "address already in use" for a minute — which is why every server sets it. SO_REUSEPORT allows multiple sockets to bind the same port, with the kernel load balancing incoming connections across them. This lets several worker processes accept independently rather than contending on one socket, and it enables zero-downtime restarts. TCP_NODELAY disables Nagle's algorithm, avoiding the delayed-ACK interaction that adds up to 200 ms to small request-response exchanges. SO_KEEPALIVE enables liveness probing, though the defaults are far too slow to be useful without also tuning the per-socket interval. SO_LINGER controls close behaviour, and setting it to zero sends an RST instead of a graceful FIN — occasionally useful to avoid TIME_WAIT, but it discards unsent data. SO_RCVBUF and SO_SNDBUF size the socket buffers, which matters on high bandwidth-delay paths where the default caps throughput. Most frameworks set the important ones, but knowing which and why is what lets you diagnose when they are wrong.

TCP guarantees that bytes arrive in order, without duplication, and without gaps. What it does not preserve is message boundaries. There is no relationship between how many times the sender called write and how many times the receiver's read returns. Two writes may arrive in one read; one write may be split across several reads. TCP may coalesce or segment freely. That is the single most common source of bugs in hand-rolled protocols. Code that assumes one read gives one complete message works in testing, where messages are small and timing is favourable, and fails in production under load or with larger payloads. So any protocol over TCP must define its own framing. The three standard approaches are a length prefix, which is what most binary protocols use; a delimiter, such as the blank line ending HTTP headers or a newline; or a self-describing format the parser can incrementally consume. HTTP uses both — a delimiter for headers, then Content-Length or chunked encoding for the body. The practical rule is: always read into a buffer and parse from it, never assume a read boundary is a message boundary.

Congestion control decides how fast to send when the network gives no explicit capacity signal. CUBIC, the Linux default for years, is loss-based. It grows the window along a cubic curve, backs off on packet loss, and probes back toward the previous maximum. It assumes loss means congestion. That assumption breaks in two ways. On links with random loss unrelated to congestion — wireless, long-haul — CUBIC backs off unnecessarily and underuses capacity. And because it only backs off at loss, it fills router buffers to overflowing, causing bufferbloat: high latency even though throughput looks fine. BBR takes a different approach, modelling the path's bottleneck bandwidth and minimum round-trip time, and pacing to match. It aims to keep the pipe full without filling buffers, so it achieves high throughput with much lower latency, and it is resilient to random loss. The criticism is fairness — early BBR could take more than its share when competing with CUBIC, which BBRv2 works to address. For a backend engineer the relevance is that switching to BBR is a one-line sysctl that measurably improves throughput on lossy or long-distance links.

When the cost of TCP's guarantees exceeds their value. TCP gives ordering, reliability, and congestion control, paid for with a handshake, retransmission delays, and head-of-line blocking. For some workloads that is exactly wrong. Real-time media is the clearest case. In a voice call, a packet that arrives 500 ms late is useless — retransmitting it is worse than dropping it, because it delays everything behind it. Better to lose a frame and continue. DNS uses UDP because a query and response fit in one packet each; a handshake would triple the cost of a single exchange, and retrying is trivial. High-frequency telemetry and metrics often use UDP, accepting occasional loss to avoid backpressure on the application. Multicast requires UDP, since TCP is inherently point-to-point. And QUIC uses UDP as a substrate specifically so it can implement better transport semantics in user space. The caveat worth adding: choosing UDP means implementing whatever reliability you actually need yourself, and hand-rolled retransmission and congestion control usually end up worse than TCP's.

Very few, deliberately. It provides multiplexing via port numbers, so multiple applications can share a host. It provides an optional checksum detecting corruption — mandatory in IPv6, optional in IPv4. And it preserves message boundaries: one send produces one receive, unlike TCP's byte stream. That last point is genuinely useful and often overlooked. With UDP you never write framing code. What it does not provide: no delivery guarantee, no ordering, no duplicate detection, no flow control, no congestion control. The absence of congestion control is the one with consequences beyond your own application. A UDP sender that ignores network conditions can crowd out TCP traffic, which backs off politely. That is why unrestrained UDP is a bad citizen and why QUIC implements congestion control explicitly rather than relying on the transport. The practical implication is that "UDP is faster" is only true for the right workload. For bulk transfer, a naive UDP implementation is usually far slower than TCP once you account for the loss you must handle yourself.

DNS uses UDP because a typical query and response each fit in a single small packet. A TCP handshake would add a round trip to an exchange that otherwise takes one, tripling the latency of every lookup. Since DNS is idempotent and the client can simply retry, the reliability TCP provides is not worth its cost. DNS falls back to TCP in specific cases. When a response exceeds the UDP size limit — historically 512 bytes — the server sets the truncated flag and the client retries over TCP. Zone transfers between servers always use TCP because they are large and must be reliable. And DNS over TLS and DNS over HTTPS use TCP by necessity. EDNS0 raised the practical UDP limit to around 4096 bytes, reducing TCP fallback, though large responses with DNSSEC signatures still trigger it. The operational trap is firewalls that allow UDP port 53 but block TCP port 53. Most lookups work, and then a large response — often a DNSSEC-signed or many-record answer — fails mysteriously. It is a genuinely common misconfiguration and worth checking when resolution is intermittent by record type.

An attacker sends small UDP requests with a spoofed source address to servers that reply with much larger responses. Those responses flood the victim whose address was spoofed. It works because UDP is connectionless — there is no handshake to verify the source address, so a server cannot tell the request is forged. The amplification factor is what makes it dangerous. DNS can amplify around 50 times; NTP's monlist command amplified over 500 times; memcached exposed to the internet reached factors in the tens of thousands. An attacker with modest bandwidth generates an enormous flood. The mitigations operate at several levels. Network operators should implement BCP 38 source address validation, so spoofed packets never leave their network — the root fix, and poorly deployed. Service operators should not expose amplifiable UDP services publicly, should rate limit responses, and should disable the specific commands with high amplification. For backend engineers the actionable version is simple: never expose memcached, Redis or similar on a public UDP port. That specific mistake caused some of the largest recorded attacks.

You reimplement the parts of TCP you actually need, and only those. Sequence numbers let the receiver detect gaps and duplicates and reorder. Acknowledgements — cumulative or selective — tell the sender what arrived. A retransmission timer, ideally with an adaptive timeout derived from measured round-trip time, resends what was not acknowledged. If you need it, add flow control so a fast sender does not overwhelm a slow receiver, and congestion control so you do not overwhelm the network or crowd out TCP traffic. At that point you have rebuilt TCP, usually worse, which is the honest answer to give. The reason to do it anyway is selective reliability. A game can retransmit critical state updates while dropping stale position updates entirely, because a position from 200 ms ago is worthless. TCP cannot express that — it insists on delivering everything in order. QUIC is the well-engineered version of this idea, and for most purposes the right answer is to use QUIC or an existing library such as ENet or KCP rather than writing your own.

Theoretically a UDP datagram can be 65,507 bytes, but sending anything near that is a mistake. Anything larger than the path MTU — typically 1500 bytes, so roughly 1472 bytes of payload after IP and UDP headers — must be fragmented at the IP layer. And IP fragmentation is fragile: if any single fragment is lost, the entire datagram is discarded, because UDP has no mechanism to retransmit part of it. So an 8 KB datagram spread over six fragments has a much higher effective loss rate than a 1400-byte one. Many firewalls and NAT devices also drop fragments outright, since only the first carries the port numbers needed for stateful inspection. The practical guidance is to keep datagrams under about 1400 bytes to stay within the MTU with room for tunnelling overhead, and to implement application-level fragmentation if you need more, so you control retransmission granularity. This is exactly why DNS truncates and falls back to TCP rather than sending large fragmented responses, and why QUIC keeps its packets small and does its own splitting.

The client checks caches first: the browser cache, the OS resolver cache, and the hosts file. A hit ends it immediately. Otherwise it asks its configured recursive resolver — usually the ISP's, or a public one like 8.8.8.8. That resolver does the work. If the resolver has nothing cached, it queries a root server, which does not know the answer but returns a referral to the .com TLD servers. It queries those, which refer it to the authoritative nameservers for the domain. It queries those, which return the actual record. The resolver caches the answer for the TTL and returns it to the client. The distinction worth being precise about: the client makes a recursive query — "give me the answer" — while the resolver makes iterative queries, following referrals itself. Root and TLD servers only ever give referrals; they never resolve on your behalf. In practice most lookups are served from cache at some level, which is why the full walk is rare. And each step is a round trip, which is why a cold DNS lookup can add 100 ms or more.

A maps a name to an IPv4 address; AAAA maps to IPv6. CNAME aliases one name to another, so the resolver restarts resolution at the target. Crucially, a CNAME cannot coexist with other records at the same name, which is why you cannot put one at the zone apex — the apex must have SOA and NS records. Providers work around this with ALIAS or ANAME records that resolve server-side. MX designates mail servers with priorities. TXT holds arbitrary text and is used for SPF, DKIM and domain ownership verification. NS delegates a zone to authoritative nameservers. SOA holds zone metadata including the default TTL and serial number. SRV specifies a service's host and port, used by SIP and by some service discovery systems. PTR does reverse lookup, mapping an address back to a name — mail servers check this, which is why misconfigured reverse DNS causes deliverability problems. The apex CNAME restriction is the one that most often surprises people setting up a domain behind a load balancer or CDN.

TTL tells resolvers how long they may cache a record. It directly bounds how quickly a change propagates. A 24-hour TTL means some clients will use the old address for a day after you change it. A 60-second TTL means changes take effect in about a minute, at the cost of far more query traffic to your authoritative servers. The standard practice before a planned migration is to lower the TTL well in advance — at least one old TTL period ahead, so every cached copy has expired and picked up the shorter value — then make the change, then raise it again. The uncomfortable reality is that TTL is advisory. Some resolvers, and notably some client libraries, ignore it. The JVM historically cached DNS forever by default for successful lookups when a security manager was present, which caused long outages after failover — networkaddress.cache.ttl exists precisely for this. So DNS-based failover is best-effort and slow. For fast failover, use a load balancer with a stable address, or anycast, and let DNS point at something that itself does not change.

An authoritative server holds the actual records for a zone and answers definitively for names it owns. It never asks anyone else. A recursive resolver holds no records of its own. It performs the resolution on a client's behalf, walking from root to TLD to authoritative, and caches what it learns. So when you query 8.8.8.8 you are using a recursive resolver. When that resolver queries your domain's nameservers, it is querying authoritative servers. The operational distinction matters for two reasons. Caching happens at the recursive layer, so your TTL controls how long recursive resolvers hold your data — you have no control over clients beyond that. And running an open recursive resolver on the public internet is dangerous: it can be used for amplification attacks, since a small query produces a large response and the source can be spoofed. Recursive service should be restricted to your own network; authoritative service is necessarily public. Confusing the two is how open resolvers get accidentally exposed.

DNS round-robin returns multiple A records for one name and rotates their order, so different clients connect to different servers. It distributes load crudely and costs nothing, which is its only real virtue. The problems are substantial. There is no health checking — DNS does not know a server is down, so it keeps handing out a dead address until someone manually removes the record, and even then caching delays it. Distribution is uneven, because clients cache and reuse an answer for the TTL and each client may serve very different traffic volumes. Clients may not honour the rotation at all, some preferring the first record or reordering by their own criteria. And failover is bounded by TTL plus whatever the client caches beyond it. So DNS round-robin is acceptable for spreading load across a few equivalent endpoints where slow failover is tolerable — often used to distribute across regions or across load balancers, not across application servers. For real balancing you want a load balancer that health checks, tracks connections, and can shift traffic immediately.

Caching happens at four levels: the application or runtime, the operating system resolver, the recursive resolver, and sometimes an intermediate forwarder. Each honours the TTL to varying degrees, and each is a place a stale answer can hide. The classic production bug is the JVM. Its InetAddress caching is controlled by networkaddress.cache.ttl, and historically defaulted to caching successful lookups indefinitely under a security manager. An application would resolve a database endpoint at startup and keep using that address after a failover moved it — reconnecting endlessly to a dead host while every other tool on the machine resolved correctly. Browsers cache independently of the OS, which is why a page still resolves wrongly after flushing the system cache. The diagnostic move is to compare layers. dig queries the resolver directly and bypasses OS and application caches; nslookup similarly. If dig returns the new address and your application does not, the cache is inside your process. Setting an explicit short DNS cache TTL in long-running services is the durable fix.

DoH sends DNS queries inside HTTPS requests to a resolver, encrypting them and making them indistinguishable from ordinary web traffic. DNS over TLS does the same over a dedicated port with TLS. The motivation is privacy and integrity. Plain DNS is unencrypted, so anyone on the path sees every domain you look up, and can tamper with responses. That has been used for surveillance, for injecting ads, and for censorship. What it changes operationally is significant. Network-level DNS filtering stops working, because the queries no longer look like DNS and may not even go to the network's resolver — browsers can be configured with their own DoH provider, bypassing the local one entirely. That breaks corporate split-horizon DNS, internal-only names, and DNS-based content filtering. It also centralises: if most clients use a handful of DoH providers, those providers see an enormous share of global lookups, which is a different privacy problem. DoT is easier for network operators, since it uses a distinct port that can be permitted or blocked deliberately, whereas DoH is deliberately hard to distinguish.

dig queries a DNS server directly, bypassing your application and OS caches, which is what makes it the right diagnostic tool. The output has sections. The header shows the status — NOERROR, NXDOMAIN for a name that does not exist, SERVFAIL for a resolver failure — and flags. The flag worth noticing is aa, meaning the answer came from an authoritative server rather than a cache. The QUESTION section echoes what you asked. ANSWER holds the records, with their remaining TTL, which tells you how long the cache will hold them. AUTHORITY shows the nameservers, and ADDITIONAL often includes their addresses. The query time and the SERVER line tell you which resolver answered and how long it took. Useful variants: dig +trace walks the delegation from root yourself, which is how you find where a broken delegation is. dig @8.8.8.8 name queries a specific resolver, so you can compare a public resolver against your local one. dig +short strips everything to the answer. Comparing an authoritative answer against a cached one is how you confirm a propagation problem rather than a configuration problem.

Split-horizon DNS returns different answers for the same name depending on who is asking — typically internal clients get a private address and external clients get a public one. It is implemented by running two views on the authoritative server, or by separate internal and external resolvers. The motivation is that internal traffic should not leave the network and come back. If an internal service resolves api.company.com to the public load balancer, traffic hairpins out and back, adding latency, cost, and an unnecessary dependency on external connectivity. The common problems it causes are worth knowing. A developer's VPN state changes which answer they get, so something works at the office and fails at home. Certificate validation can be surprising if the internal name differs. And DNS over HTTPS in browsers bypasses the internal resolver entirely, so internal names stop resolving — which is a real and increasingly common support issue. In cloud environments the equivalent is private hosted zones, and the same class of confusion arises when a resource is in a VPC without the private zone associated.

Because the failure is in name resolution rather than connectivity, and separating those two is the point of the question. The possibilities: the record does not exist or has a typo. The resolver is unreachable or misconfigured — check /etc/resolv.conf. The record exists but has not propagated, and a cache is holding the old value or a negative answer. Negative caching is the one people forget: a NXDOMAIN response is cached too, per the SOA minimum TTL, so a name queried before it existed stays unresolvable for a while after you create it. Search domain configuration can also interfere — a short name gets suffixes appended in order, and a wrong search path resolves to something unexpected. In containers there is an extra layer: the container has its own resolv.conf pointing at the cluster DNS, so a name that resolves on the host may not resolve inside the pod. Kubernetes ndots defaulting to 5 also causes every external lookup to try several cluster suffixes first, which adds latency and occasionally resolves to the wrong thing. The test is always: does dig work, and does dig against a different resolver work?

HTTP/1.1 is text-based, one request at a time per connection. Keep-alive reuses the connection but requests still queue, so a slow response blocks those behind it. Browsers opened six connections per host to work around it. HTTP/2 is binary and multiplexed: many concurrent streams share one connection, so the application-level head-of-line blocking is gone. It adds header compression with HPACK, which matters because headers are repetitive and often larger than small responses, and server push, which was later deprecated because it was hard to use well. But HTTP/2 still runs over TCP, which guarantees byte ordering for the whole connection. A single lost packet stalls every stream — transport-level head-of-line blocking, which under loss can make HTTP/2 worse than HTTP/1.1 with parallel connections. HTTP/3 fixes that by running over QUIC on UDP, where each stream is independently ordered. It also cuts handshake latency by merging transport and TLS setup, and supports connection migration across network changes. The practical upgrade advice: HTTP/2 helps most with many small resources; HTTP/3 helps most on lossy or mobile networks.

Safe means the request does not change server state — GET, HEAD, OPTIONS. Crawlers and prefetchers rely on this, which is why a GET that deletes something is a genuine bug rather than a style issue. Idempotent means repeating the request has the same effect as making it once. GET, PUT and DELETE are idempotent; POST is not. This is what makes automatic retries safe: a client or proxy can retry an idempotent request after a timeout without risking duplication. That matters enormously in practice. A timeout does not tell you whether the request was processed, so retrying a POST can double-charge a customer. The standard mitigation is an idempotency key that the server records, so a repeated request returns the original result. Cacheable means the response may be stored and reused. GET and HEAD are cacheable by default; POST responses can be, but almost nothing does it. PUT versus PATCH follows the same logic: PUT replaces the whole resource and is idempotent, PATCH applies a partial change and may not be, depending on how the patch is expressed.

1xx informational, 2xx success, 3xx redirection, 4xx client error, 5xx server error. The distinction that matters operationally is 4xx versus 5xx: 4xx means the client sent something wrong and retrying unchanged will not help, 5xx means the server failed and a retry may succeed. Getting this wrong pollutes error budgets — returning 500 for a validation failure makes your service look broken and triggers alerts for a client bug. The ones worth knowing precisely: 201 Created with a Location header for resource creation. 204 No Content for a successful request with no body. 301 permanent versus 302 temporary redirect, where 301 is cached aggressively and hard to undo. 304 Not Modified for conditional requests. 400 for malformed syntax, 401 for missing or invalid authentication, 403 for authenticated but not permitted, 404 not found, 409 conflict, 422 for semantically invalid content, 429 too many requests with a Retry-After header. 500 generic failure, 502 bad gateway meaning an upstream returned garbage, 503 unavailable meaning overloaded or in maintenance, 504 gateway timeout meaning an upstream did not respond in time.

All three come from a gateway or proxy, and each points at a different failure. 502 Bad Gateway means the proxy reached the upstream but got an invalid response — a malformed reply, or the connection was closed unexpectedly. In practice this usually means the backend crashed mid-request, or was restarted while handling one. 503 Service Unavailable means the server itself is refusing to handle the request — overloaded, in maintenance, or with no healthy backends available. A load balancer with every backend failing health checks returns 503. 504 Gateway Timeout means the upstream was reached and accepted the request but did not respond within the proxy's timeout. The backend is alive but slow. That mapping is what makes them useful in an incident: 502 means look for crashes and check logs for stack traces at the moment of failure; 503 means check health checks and capacity; 504 means check backend latency and whether the proxy timeout is shorter than the application's own. The last is a common misconfiguration — a proxy timing out at 30 seconds while the application is happy to take 60 produces 504s for requests that eventually succeed.

Two mechanisms: freshness and validation. Freshness lets a cache serve a response without contacting the server. Cache-Control: max-age=3600 means it may be reused for an hour. During that window there is no request at all — the fastest possible outcome. Validation handles what happens after expiry. The response carries an ETag or Last-Modified, and the client sends If-None-Match or If-Modified-Since. If unchanged, the server returns 304 Not Modified with no body, saving bandwidth but still costing a round trip. The directives worth knowing: public and private control whether shared caches may store it; no-cache means store but always revalidate, which is widely misunderstood as "do not cache"; no-store genuinely forbids storage and is what you want for sensitive data; must-revalidate forbids serving stale content on error. stale-while-revalidate is the useful modern addition — serve the stale copy immediately and refresh in the background, which removes the latency cost of revalidation. The standard strategy for assets is content-hashed filenames with a very long max-age plus immutable, so the URL changes when the content does.

Cross-Origin Resource Sharing is a browser mechanism that relaxes the same-origin policy in a controlled way. The same-origin policy stops a script on one origin reading responses from another. Without it, a malicious page could make requests to your bank with your cookies attached and read the results. CORS lets a server opt in to allowing specific origins. The key point people miss is that CORS is enforced entirely by the browser and protects the user, not the server. A CORS error does not mean the request was blocked — for a simple request the server may have processed it fully; the browser merely refused to let the script read the response. curl and server-to-server calls are unaffected. The flow: for anything beyond a simple request, the browser sends an OPTIONS preflight asking whether the method and headers are permitted. The server answers with Access-Control-Allow-Origin, -Methods and -Headers. Only then is the real request sent. The common trap is credentials: with credentials included, the allowed origin cannot be the wildcard, and Access-Control-Allow-Credentials must be true. That combination is where most CORS debugging time goes.

Cookies are sent automatically with every matching request, which is what makes them work for session authentication — and also what makes them vulnerable to CSRF, since the browser attaches them regardless of who initiated the request. They are small, around 4 KB, and can be marked HttpOnly so JavaScript cannot read them, Secure so they only travel over HTTPS, and SameSite to control cross-site sending. localStorage and sessionStorage are larger, around 5–10 MB, and never sent automatically — the application must read and attach them explicitly. sessionStorage is scoped to a tab and cleared when it closes; localStorage persists indefinitely. The security consequence is the crux. A token in localStorage is readable by any JavaScript on the page, so a single XSS vulnerability exfiltrates it. A token in an HttpOnly cookie is not readable by script at all. So the usual guidance is HttpOnly, Secure, SameSite cookies for session tokens, and web storage for non-sensitive UI state. The counter-argument is that cookies need CSRF protection, but SameSite=Lax largely handles that and is now the browser default.

Chunked encoding lets a server send a response body without knowing its total length upfront. The body is sent as a series of chunks, each prefixed with its size in hex, terminated by a zero-length chunk. It exists because HTTP/1.1 needs to know where a response ends. Normally Content-Length says so. If the length is unknown — the response is generated as it goes, or streamed from another source — chunked encoding provides the framing instead. The benefit is that the server can start sending immediately rather than buffering the whole response to compute its length. That improves time to first byte and bounds memory on large responses. The costs: you cannot show a progress bar, since the total is unknown, and there is slight framing overhead. It is also a security-relevant area. Request smuggling attacks exploit disagreements between a front-end proxy and a back-end server about whether Content-Length or Transfer-Encoding takes precedence, letting an attacker prepend data to another user's request. That is why proxies must reject requests carrying both headers. HTTP/2 and HTTP/3 have framing built in, so chunked encoding does not exist there.

Long polling holds an HTTP request open until data is available, then returns and the client immediately reconnects. It works everywhere but is inefficient — a full request cycle per message, and reconnection overhead. Server-Sent Events keeps one HTTP response open and streams events down it. Simple, uses ordinary HTTP, and gives automatic reconnection with event IDs for resumption. But it is one-directional, server to client only, and limited to text. WebSockets upgrade the HTTP connection to a persistent bidirectional binary channel. After the handshake it is no longer HTTP — it is a framed message protocol in both directions with minimal per-message overhead. The choice follows the requirement. If the client only needs to receive, SSE is simpler and plays better with proxies and HTTP infrastructure. If both directions need low latency — chat, collaborative editing, games — WebSockets are right. The operational costs of WebSockets are worth naming: they hold a connection per client, so scaling means many long-lived connections; load balancers need to support the upgrade and long idle timeouts; and there is no built-in reconnection or message ordering guarantee, so you build those yourself.

Content-Type declares the media type of the body so the recipient knows how to parse it — application/json, text/html, multipart/form-data. The charset parameter declares the character encoding. Without it, the recipient guesses, and guessing wrong produces mojibake — the classic garbled accented characters. For JSON the charset is effectively always UTF-8 and the parameter is redundant by specification. For text/html it matters, and the modern practice is to declare UTF-8 both in the header and in a meta tag, because the header wins but the tag helps when the file is opened locally. Content-Type also has security significance. A response served as text/html when it contains user-supplied content allows stored XSS; serving it as text/plain or application/json does not. That is why X-Content-Type-Options: nosniff matters — it stops browsers second-guessing the declared type, which they historically did and which was exploitable. On the request side, the server should validate that Content-Type matches what it expects rather than parsing whatever arrives, since a mismatch is often the first sign of a malformed or hostile client.

Strict-Transport-Security tells the browser to use HTTPS for this domain for a given period, preventing downgrade and stopping the initial plaintext request that a man-in-the-middle could intercept. Content-Security-Policy restricts where scripts, styles and other resources may load from. It is the strongest defence against XSS, because even if an attacker injects a script tag, the browser refuses to execute it unless the source is allowed. Getting it right takes effort, which is why report-only mode exists. X-Content-Type-Options: nosniff stops the browser guessing content types, which was historically exploitable. X-Frame-Options or CSP frame-ancestors prevents your page being embedded in an iframe, which defeats clickjacking. Referrer-Policy limits how much URL information leaks to other sites, which matters when URLs contain tokens or identifiers. Permissions-Policy restricts access to camera, microphone, geolocation and similar. The practical ordering: HSTS and CSP give the most benefit. X-XSS-Protection is obsolete and should be omitted or set to 0, since its filter caused vulnerabilities of its own.

Authentication establishes who the caller is. Authorization decides what they may do. 401 Unauthorized is the confusingly named one — it actually means unauthenticated. The credentials are missing, invalid, or expired, and the response should include a WWW-Authenticate header indicating how to authenticate. Retrying with valid credentials may succeed. 403 Forbidden means the caller is authenticated and the server knows who they are, but they are not permitted. Retrying with the same credentials will never succeed. Getting these right matters for clients: a 401 should trigger a token refresh and retry, a 403 should not. A subtlety worth raising is information disclosure. Returning 403 for a resource that exists but is not yours confirms its existence. Returning 404 instead hides it. Which is correct depends on whether existence is sensitive — GitHub returns 404 for private repositories you cannot see, deliberately. The transport concern is that credentials must never travel in a URL, since URLs appear in logs, browser history, and Referer headers. Authorization headers or cookies only.

In HTTP/1.1, connections are persistent by default — after a response the connection stays open for the next request, avoiding a fresh TCP handshake and TLS negotiation. The saving is large. A new HTTPS connection costs a TCP handshake plus a TLS handshake: two round trips minimum with TLS 1.3, three with 1.2. On a 100 ms path that is 200–300 ms before a byte of the request is sent. Reuse makes it zero. There is a second, less obvious benefit: an established connection has a warmed TCP congestion window, so it transfers at full speed immediately, while a new connection starts in slow start. For server-to-server calls this is why connection pooling is essential. A client that creates a new connection per request pays those handshakes every time and accumulates TIME_WAIT sockets, eventually exhausting ephemeral ports. The pitfalls: idle connections can be closed by intermediaries without notice, so pools need validation or an idle timeout below the infrastructure timeout. And with HTTP/2, one connection carries everything, so a layer 4 load balancer will pin all that traffic to a single backend.

An attack exploiting disagreement between two servers in a chain — typically a front-end proxy and a back-end server — about where one request ends and the next begins. It arises when a request carries both Content-Length and Transfer-Encoding, and the two servers prioritise them differently. The front-end sees one request; the back-end sees one and a half, treating the remainder as the start of the next request. The attacker has effectively prepended data to whoever's request comes next on that connection. The impact is severe: capturing another user's request including their credentials, bypassing front-end access controls, or poisoning a cache so other users receive attacker-controlled content. It only works because connections are reused between the proxy and back-end, which is exactly what you want for performance. The defences: reject any request containing both headers rather than trying to reconcile them, normalise requests at the front-end before forwarding, use HTTP/2 end to end since its framing is explicit and length-prefixed, and keep proxy and back-end HTTP parsing behaviour consistent. It is a good example of a vulnerability that lives in the gap between two correct-looking implementations.

Content-Length declares the body size in bytes upfront. Transfer-Encoding: chunked frames the body as sized chunks so the total need not be known in advance. They are alternative framing mechanisms and are mutually exclusive. The specification says that if both are present, Transfer-Encoding wins and Content-Length must be ignored. But the correct handling is to reject the request entirely, not to pick one. A request carrying both is either broken or malicious, and it is precisely the condition that enables request smuggling — the attack depends on two servers resolving the ambiguity differently. So modern proxies and servers respond with 400 rather than attempting to interpret it. On the response side, a server should send exactly one. Sending Content-Length when the body is generated dynamically requires buffering the whole thing; chunked avoids that at the cost of losing progress indication. HTTP/2 and HTTP/3 sidestep this entirely — framing is part of the binary protocol, so neither header is used for framing, and Transfer-Encoding: chunked is actually forbidden.

Only retry what is safe to retry, and back off. Idempotency is the first gate. GET, PUT and DELETE can be retried freely. POST cannot, unless the server supports an idempotency key — the client generates a unique key per logical operation, the server records it, and a repeat returns the original result rather than acting twice. The second gate is which failures to retry. Connection errors and timeouts before a response, 502, 503 and 504, and 429 with Retry-After. Not 4xx generally, since the request is wrong and will stay wrong. Backoff must be exponential with jitter. Without jitter, all clients that failed together retry together, producing a synchronised thundering herd that keeps the service down — this is a genuinely common cause of failure amplification during recovery. Cap the total attempts and the total elapsed time, so a retry storm cannot outlive the caller's own deadline. And pair retries with a circuit breaker. Retrying into a service that is comprehensively down multiplies load exactly when it can least handle it; the breaker stops sending entirely until a probe succeeds.

301 is a permanent redirect: the resource has moved for good. 302 is temporary: it is elsewhere for now, but keep using the original URL. The practical difference is caching and its consequences. Browsers cache 301s aggressively, often indefinitely, and search engines transfer ranking to the new URL. A 302 is not cached by default and search engines keep the original indexed. The danger is that a mistaken 301 is very hard to undo. Once a browser has cached it, that user goes to the wrong place until they clear their cache, and you have no way to reach them. Deploying a 301 to the wrong target is a genuinely painful incident. So the rule is: use 302 unless you are certain the move is permanent, and use 301 deliberately for domain migrations and canonical URL consolidation where the SEO transfer is the point. There is also a method subtlety. Historically browsers changed POST to GET when following 301 and 302, contrary to the spec. 307 and 308 were introduced to preserve the method explicitly, so use those when redirecting a non-GET request.

Both support conditional requests, letting a client ask "has this changed?" and receive 304 Not Modified if not. Last-Modified carries a timestamp, and the client sends If-Modified-Since. It is simple but has one-second granularity, so changes within the same second are missed. It also assumes the server has a meaningful modification time, which dynamically generated content may not. ETag carries an opaque validator — typically a hash or version of the content — and the client sends If-None-Match. It detects any change regardless of timing, and works for generated content where no file timestamp exists. ETags come in strong and weak forms. A strong ETag means byte-for-byte identical; a weak one, prefixed W/, means semantically equivalent, which permits differences like whitespace or compression. ETag is generally preferable for correctness, and servers often send both so clients can use either. The operational trap is multiple servers behind a load balancer generating different ETags for identical content — usually because the ETag includes an inode or file timestamp that differs per machine. That defeats caching entirely, and is why content-hash-based ETags are the safe choice.

In TLS 1.3, the client sends ClientHello with its supported cipher suites and a key share — a public key for key exchange — guessing the server's preferred group. The server replies with ServerHello containing its own key share, then immediately encrypts the rest: its certificate, a signature proving it holds the private key, and Finished. Both sides now derive the same shared secret. The client validates the certificate chain, sends Finished, and application data flows. One round trip total. TLS 1.2 needed two, because the cipher suite was negotiated first and only then was key exchange performed. The essential point is what each part achieves. The key exchange establishes a shared secret without transmitting it — with ephemeral Diffie-Hellman, neither side's long-term key can decrypt a recorded session later, which is forward secrecy. The certificate and signature prove identity. The Finished messages prove neither side's messages were tampered with. Session resumption with a pre-shared key allows 0-RTT, sending data with the first flight — at the cost that 0-RTT data is replayable, so it must only carry idempotent requests.

A certificate binds a public key to an identity, signed by a certificate authority. Validation checks that signature, then the signature on the CA's own certificate, and so on up to a root the client already trusts. Root certificates are pre-installed in the OS or browser trust store. That is the anchor — trust is not derived from anything, it is configured. Intermediate certificates sit between, so the root's private key can stay offline and a compromised intermediate can be revoked without invalidating everything. Beyond the signatures, the client checks the validity dates, that the hostname matches a Subject Alternative Name, that the certificate is not revoked, and that key usage permits server authentication. The most common production failure is an incomplete chain: the server sends its leaf certificate but omits the intermediate. Browsers often paper over this by fetching the missing intermediate, but many programmatic clients — Java, curl, Go — do not, producing the classic "works in the browser, fails from the service" problem. openssl s_client -showcerts is how you check what the server actually sends.

Server Name Indication is a TLS extension where the client sends the hostname it wants during the handshake, before encryption is established. It exists because TLS begins before HTTP. Without SNI, a server hosting many domains on one IP cannot know which certificate to present — the Host header that would tell it arrives only after the handshake completes. That is a chicken-and-egg problem, and SNI resolves it. It is what makes virtual hosting over HTTPS possible, which is essentially all shared hosting and every CDN. The privacy weakness is that SNI is sent in plaintext, so anyone on the path sees which site you are visiting even though the traffic is encrypted. That has been used for censorship and surveillance. Encrypted Client Hello is the fix, encrypting the whole ClientHello including SNI, though deployment depends on DNS-delivered keys and is still rolling out. The practical failure is old clients without SNI support receiving the server's default certificate rather than the right one, producing a hostname mismatch. Rare now, but it still appears with very old Java or Android versions.

In ordinary TLS only the server presents a certificate; the client verifies it and authenticates separately, usually with a token or password. In mutual TLS both sides present certificates and both verify. The client's identity is therefore established at the transport layer, cryptographically, before any application data flows. It is used where you control both ends and want strong identity without shared secrets: service-to-service communication inside a mesh, machine-to-machine APIs, and high-security integrations such as banking. The advantage over bearer tokens is that a certificate cannot be replayed by an interceptor — proving possession requires the private key, which never leaves the client. The cost is certificate management: issuing, distributing, rotating and revoking certificates for every client. That operational burden is why mTLS was rare until service meshes automated it — Istio and Linkerd issue and rotate short-lived certificates transparently, which is what made it practical. Revocation remains the weak point, which is why short lifetimes are preferred over relying on CRLs or OCSP.

Forward secrecy means that compromising a server's long-term private key does not allow decryption of past recorded sessions. Without it — as in the old RSA key exchange — the client encrypted a premaster secret with the server's public key. Anyone who recorded that traffic and later obtained the private key could decrypt everything retroactively, potentially years of sessions. With ephemeral Diffie-Hellman, each session generates a fresh key pair used only for that session and discarded afterwards. The long-term key only signs the exchange to prove identity; it never encrypts the session key. So a later compromise reveals nothing about past traffic. This is why TLS 1.3 removed static RSA key exchange entirely — every cipher suite provides forward secrecy by construction, rather than leaving it as a configuration choice. The threat model it addresses is real: adversaries record encrypted traffic now expecting to obtain keys or break the cryptography later. The caveat is session resumption. Tickets encrypted with a long-lived server key can undermine forward secrecy, which is why ticket keys must be rotated frequently.

Encoding transforms data into another representation for transport or compatibility. Base64 and URL encoding are examples. It is fully reversible by anyone and provides no security whatsoever. Treating Base64 as obfuscation is a recurring mistake. Hashing is a one-way function producing a fixed-length digest. You cannot recover the input. Used for integrity checking and for password storage. For passwords you need a deliberately slow hash with a salt — bcrypt, scrypt or Argon2 — because fast hashes like SHA-256 can be brute-forced at billions of guesses per second on a GPU. Encryption is reversible with a key. Symmetric uses one key for both directions and is fast, so it carries the bulk data. Asymmetric uses a key pair, is slow, and is used for key exchange and signatures. TLS combines them: asymmetric to establish a shared secret and prove identity, symmetric for the session, and hashing for integrity. The interview tell is whether someone says "we encrypt passwords". You hash passwords; encrypting them means they can be decrypted, which is exactly what you are trying to prevent.

Through the combination of encryption and authenticated identity — and the identity half is what actually stops the attack. Encryption alone would not help. An attacker who intercepts the connection could negotiate their own encrypted session with you and another with the server, reading everything in between. Both connections would be encrypted and completely compromised. What prevents that is certificate validation. The attacker must present a certificate for the domain, signed by a CA the client trusts. They cannot forge that without a CA issuing it wrongly or being compromised. The remaining gaps are worth naming. If the user clicks through a certificate warning, the protection is gone — which is why browsers made that increasingly difficult. If an attacker controls a trusted CA, or installs their own root certificate on the device, they can issue valid-looking certificates; this is exactly how corporate TLS inspection works. HSTS closes the downgrade gap by preventing the initial plaintext request. Certificate Transparency logs make wrongly-issued certificates detectable. Certificate pinning is the strongest defence but is brittle and has caused outages.

TLS termination is where the encrypted connection is decrypted. Traffic beyond that point is plaintext unless re-encrypted. Terminating at the load balancer or CDN edge is common. It centralises certificate management, offloads the cryptographic cost from application servers, and — crucially — lets a layer 7 balancer read the request to route, cache and rewrite. It cannot do path-based routing on an encrypted stream. The consequence is that traffic between the balancer and the backend is unencrypted. Inside a trusted VPC that is often accepted; under stricter compliance it is not. The alternatives: re-encrypt after termination, so the balancer decrypts to inspect then opens its own TLS connection to the backend — the usual compromise. Or pass through, forwarding the encrypted stream untouched, which gives end-to-end encryption but reduces the balancer to layer 4. Service meshes take a different approach: terminate at the edge for routing, then use mutual TLS between sidecars for every internal hop, giving both inspection and encryption. The detail to remember is that after termination the backend sees the proxy's IP, so the client address must come from X-Forwarded-For.

Several reasons, and they are common enough to be worth knowing as a checklist. Incomplete chain: the server omits the intermediate certificate. Browsers often fetch it automatically via the Authority Information Access extension; Java, curl and Go generally do not, so they fail where the browser succeeds. This is the most frequent cause. Trust store differences: the JVM has its own cacerts file, separate from the OS. A CA trusted by the system may be absent from the JVM's store, particularly for internal CAs. Protocol or cipher mismatch: an old client offering only TLS 1.0, or a server requiring a cipher the client lacks. Java 8 without recent updates lacks some modern suites. SNI: a client not sending SNI receives the default certificate, which may be for a different hostname. Hostname verification differences: some clients are stricter about wildcards or about IP addresses in SANs. Clock skew: a certificate is valid only within a date range, and a badly wrong system clock makes everything expired or not-yet-valid. openssl s_client is the tool that shows exactly what the server presents.

Pinning means the client accepts only a specific certificate or public key for a host, rather than anything signed by a trusted CA. It defends against a compromised or coerced CA issuing a fraudulent certificate — the residual risk that ordinary validation cannot address, since any of hundreds of trusted CAs can issue for any domain. The reason it is discouraged is operational fragility. If you pin a certificate and must rotate it — expiry, compromise, a CA change — every client with the old pin fails, and you often cannot update them quickly. Mobile applications with pinned certificates have bricked themselves this way, requiring an app store update to recover. HTTP Public Key Pinning made this worse by letting a server pin itself in browsers for a period, so a mistake caused a self-inflicted outage with no remedy. It was deprecated and removed. The modern alternative is Certificate Transparency: all issued certificates are logged publicly, so wrongly-issued ones are detectable, and Expect-CT let sites require it. Combined with CAA records restricting which CAs may issue for your domain, that covers most of the threat with far less risk.

Round robin cycles through backends. Simple and fine when servers are identical and requests are uniform. Weighted round robin accounts for differing capacity — useful during a migration where new instances are larger. Least connections routes to the backend with fewest active connections. Better when request durations vary widely, because round robin would keep sending work to a server already stuck on slow requests. Least response time combines connection count with observed latency, adapting to backends that are degraded rather than merely busy. IP hash or consistent hashing routes deterministically by client, which gives session affinity without shared state and is essential for cache locality — you want the same key to reach the same cache node. Random with two choices is the underrated one: pick two backends at random and send to the less loaded. It gets almost all the benefit of least-connections without the coordination cost, and it avoids the herd problem where every balancer picks the same "least loaded" server simultaneously. The practical default is least connections for application traffic and consistent hashing for caches.

Consistent hashing maps both keys and nodes onto a ring, and a key is served by the first node clockwise from it. The problem it solves is redistribution on membership change. With naive modulo hashing — hash(key) % N — changing N remaps almost every key. Removing one node from a ten-node cache invalidates roughly 90% of entries, and the resulting stampede on the origin can take the system down. With consistent hashing, adding or removing a node only affects the keys in that node's arc — roughly 1/N of them. The rest are undisturbed. The refinement is virtual nodes: each physical node is placed at many points on the ring. Without that, distribution is uneven because random placement leaves arcs of very different sizes, and removing a node dumps its entire load onto one neighbour rather than spreading it. It underpins distributed caches like memcached clients, Cassandra and DynamoDB partitioning, and CDN request routing. Bounded-load consistent hashing extends it with a capacity cap per node, so a hot key cannot overwhelm a single server — which plain consistent hashing does not prevent.

Distinguish liveness from readiness. Liveness asks "is this process broken and in need of a restart?" Readiness asks "should this instance receive traffic right now?" Conflating them is a classic failure. If liveness checks a database dependency and the database has a hiccup, every instance fails liveness and the orchestrator restarts the entire fleet — turning a brief dependency blip into a full outage. Liveness should check only the process itself. Readiness may check dependencies, because removing an instance from rotation is reversible and harmless. The check should be shallow enough to be cheap and fast — it runs constantly across every instance — but deep enough to be meaningful. A handler that returns 200 unconditionally tells you nothing; one that runs a full query per check adds real load. Other details that matter: set thresholds so a single failure does not eject an instance, but recovery is quick. Ensure the check fails during shutdown before the process stops accepting connections, so traffic drains first. And exclude health check requests from latency metrics, or they skew everything.

A retry handles transient failure by trying again. A circuit breaker handles sustained failure by refusing to try at all. The breaker has three states. Closed: requests flow normally while failures are counted. Open: after crossing a failure threshold, requests fail immediately without contacting the service. Half-open: after a cooldown, a limited number of probes are allowed; success closes the breaker, failure reopens it. The reason it matters is that retries alone amplify an outage. When a dependency is down, retrying multiplies load on a service that is already failing, delaying its recovery. Meanwhile callers hold threads and connections waiting on timeouts, so the failure propagates upward — a slow dependency exhausts your thread pool and takes you down too. The breaker converts slow failures into fast ones, which frees resources and lets you degrade gracefully — serve cached data, return a partial response, or fail cleanly. They are complementary: retry with backoff for transient errors, breaker for sustained ones. Combined with a bulkhead limiting concurrent calls per dependency, that is the standard resilience triad.

When a request passes through a proxy, the backend sees the proxy's IP as the source. X-Forwarded-For carries the original client address, and each proxy appends the address it saw. So the header is a comma-separated list, oldest first: the leftmost is the client as reported by the first proxy. The security problem is that a client can send the header themselves. If you blindly take the leftmost value, an attacker sets it to anything they like — defeating IP-based rate limiting, allowlists, and audit logging. The correct handling is to count from the right. You know how many proxies you control; the address that many positions from the end is the one your outermost trusted proxy actually observed. Everything to the left of that is client-supplied and untrusted. Better still, have the outermost proxy overwrite rather than append, so no forged values survive. The modern standard is the Forwarded header, which carries proto and host too, but X-Forwarded-For remains ubiquitous. Frameworks usually have a trusted-proxy setting — configuring it is what makes this safe, and leaving it at the default is a common oversight.

Session affinity, or sticky sessions, routes a given client consistently to the same backend — by hashing the client IP, or by a cookie the load balancer sets and reads. It exists because some applications hold session state in memory, so a request routed elsewhere loses the session. The problems are substantial. Load becomes uneven, since sessions have different lifetimes and weights. Scaling out does not help existing sessions, because they stay pinned. Losing an instance loses every session on it. And rolling deployments become disruptive rather than transparent. IP-based affinity is worse still: many users share an IP behind NAT, so they all pin to one backend, and mobile clients changing networks lose their session. The better answer is to make the application stateless — keep session data in Redis or a database, or in a signed token the client carries. Then any instance can serve any request, scaling and deployment are trivial, and instance loss is invisible. Affinity remains legitimate for genuinely stateful protocols such as WebSockets, and as a transitional measure while migrating a stateful application.

A service mesh moves cross-cutting network concerns out of application code into a sidecar proxy deployed alongside every service instance. All traffic in and out flows through that proxy, so the mesh can implement retries, timeouts, circuit breaking, load balancing, mutual TLS, traffic splitting for canaries, and detailed telemetry — uniformly, without each service implementing them separately in whichever language it happens to use. That language-independence is the main argument. Otherwise every service needs its own resilience library, and they drift. The control plane distributes configuration; the data plane is the proxies doing the work. Envoy is the usual data plane; Istio and Linkerd are common control planes. The costs are real and often underestimated. Every hop adds a proxy in each direction, so latency increases and resource usage roughly doubles in pod count. Operational complexity rises substantially, and debugging now involves an extra layer that can itself misbehave. So it earns its place with many services in multiple languages needing consistent mTLS and traffic control. For a handful of services in one language, a good library is simpler.

Start with the algorithm. A fixed window counts requests per interval — simple but allows a burst of double the limit across a window boundary. A sliding window log stores timestamps and is exact but memory-heavy. A sliding window counter interpolates between two fixed windows and is a good compromise. Token bucket is usually the right default: tokens accrue at a steady rate up to a cap, and each request consumes one. It permits bursts up to the bucket size while enforcing a long-run average, which matches how people actually want to limit. Leaky bucket enforces a strictly smooth output rate, which suits protecting a downstream with fixed capacity. Then the distributed question. Per-instance limits are simple but the effective limit multiplies by instance count and shifts as you scale. A shared store — Redis with an atomic script — gives a global limit at the cost of a network round trip on every request. A common compromise is local limiting with periodic reconciliation. Finally the semantics: return 429 with Retry-After, and decide what to key on — API key, user, IP — remembering that IP is shared behind NAT.

Separate the layers, and measure rather than assume. First, establish where time is going. If you have distributed tracing, that answers it directly — you see whether the time is in the network hop or inside the callee. Without it, compare the caller's observed latency against the callee's own reported processing time. A large gap is the network, a proxy, or queueing. Check for packet loss and latency variance with mtr or ping over a sustained period, not a single sample — intermittent problems do not show up in a five-packet ping. Check connection-level symptoms: ss for socket states and queue depths, retransmission counts from netstat -s, which tell you whether TCP is struggling. Rule out DNS, which is a surprisingly common cause of intermittent latency — a slow or failing resolver adds seconds occasionally. And check whether the pattern correlates with connection churn rather than the network being slow: a client without connection pooling pays handshake cost on every request, which looks like network latency but is a client bug. The usual finding is that it is not the network.

tcpdump captures packets on an interface so you can see what actually went on the wire, rather than what the application believes happened. A typical invocation: tcpdump -i any -n port 443 and host 10.0.0.5 -w capture.pcap. The -n avoids DNS lookups that slow the capture and pollute the output, and -w writes a file for later analysis in Wireshark. What you look for depends on the symptom. For a connection failure: is the SYN going out at all, and is anything coming back? A SYN with no reply means it is being dropped, usually by a firewall. A SYN followed by RST means actively refused. For slowness: retransmissions indicate loss, duplicate ACKs indicate out-of-order delivery, zero-window advertisements indicate the receiver is not reading. For protocol problems: the actual bytes show whether the request was malformed. The practical cautions are that capturing on a busy interface is expensive and can affect the system, so always filter narrowly; and that with TLS you see the handshake but not the payload, so failures after the handshake need application-level logging instead.

Each tests a different layer, and choosing the right one narrows the problem quickly. ping uses ICMP and tests layer 3 reachability only. A successful ping says the host is up and routing works; it says nothing about whether your service is listening. A failed ping may just mean ICMP is blocked, which is common — so a failed ping is weak evidence. telnet host port and nc -zv host port test TCP connectivity to a specific port. This is the right tool for "can I reach the service at all?" It distinguishes refused (nothing listening) from timeout (packets dropped, usually a firewall). curl tests the full application layer: DNS resolution, TCP, TLS, and the HTTP exchange. curl -v shows each stage, so you can see exactly where it fails. curl -w with a timing format breaks down DNS, connect, TLS and transfer time, which is excellent for finding which phase is slow. openssl s_client sits between nc and curl, testing TLS specifically and showing the certificate chain. The method is to work up the stack until something fails.

The difference is the whole clue, so enumerate what differs. Network position: different subnet, VPC, security group, or firewall rules. In cloud environments this is the most common cause — check security groups and network ACLs in both directions, remembering that NACLs are stateless so return traffic needs an explicit rule. DNS: the two machines may use different resolvers and get different answers. Compare dig output from both, and check /etc/resolv.conf. Routing: ip route get DESTINATION on each shows which interface and gateway would be used. A VPN or a stale route can send traffic somewhere unexpected. TLS trust: different CA bundles, or a different JVM with its own cacerts. Source address: if the destination has an IP allowlist, the working machine may be on it. Check what address the destination actually sees — curl to a service that echoes it. Proxy configuration: environment variables like HTTP_PROXY set on one machine and not the other silently change everything, and they are easy to overlook because they are invisible in the command.

Bufferbloat is excessive latency caused by oversized buffers in network equipment. The logic that created it seemed sound: memory is cheap, and a bigger buffer means fewer dropped packets. But TCP uses loss as its congestion signal. If a router buffers hundreds of milliseconds of traffic instead of dropping, TCP never learns to slow down and keeps filling the buffer. The result is that a single bulk transfer — an upload, a backup — fills the buffer and every other packet on that link queues behind it. Throughput is fine; latency goes from 20 ms to several seconds. Video calls stutter and interactive sessions become unusable while a download runs. It is particularly bad on home routers and mobile networks. The fixes are active queue management algorithms — CoDel and FQ-CoDel — which drop or mark packets based on how long they have been queued rather than on queue length, signalling congestion before the buffer fills. Fair queueing additionally isolates flows so one bulk transfer cannot monopolise the queue. BBR helps from the sender side by pacing to the estimated bottleneck rather than filling buffers until loss.

Because a missing timeout turns a slow dependency into an outage. A thread blocked indefinitely holds a connection and a pool slot, and once every thread is blocked the service is down even though nothing crashed. The layers each need one: connection timeout for establishing TCP, TLS handshake timeout, socket read timeout for waiting on data, and an overall request timeout bounding the whole operation. Choosing values follows from the dependency's actual latency distribution — set the timeout somewhat above the p99, not the mean, or you will abandon requests that would have succeeded. The critical structural rule is that timeouts must decrease as you go deeper. If a caller times out at 2 seconds and its callee waits 5 seconds on a database, the callee is still working on a request nobody is waiting for — wasting capacity. Propagating a deadline down the chain, as gRPC does, solves this properly. And the proxy timeout must exceed the application timeout, or you get 504s for requests that would have completed, with no error in the application logs to explain them.

A thundering herd is many clients acting simultaneously, overwhelming a resource that could handle them if they were spread out. The common triggers: a cache entry expires and every request that needed it hits the database at once. A service restarts and every client reconnects simultaneously. A dependency recovers and all the queued retries fire together. Or a scheduled job runs at exactly midnight across every instance. The preventions are specific to each. For cache expiry, use request coalescing so only one caller recomputes while the others wait for that result, and add jitter to TTLs so entries do not expire together. Serving stale content while refreshing in the background removes the stampede entirely. For retries and reconnections, exponential backoff with jitter is essential — backoff alone keeps clients synchronised, and jitter is what actually spreads them. For scheduled work, randomise the start within a window. The general principle is that synchronisation is the enemy. Anything that makes many independent clients act at the same instant will eventually produce a herd, and randomisation is usually the cheapest fix.

A CDN caches content at edge locations near users. A request is routed — usually by anycast or DNS — to the nearest edge, which serves from cache or fetches from the origin and caches the result. The benefits are latency, since content travels a shorter distance; origin offload, since most requests never reach you; and DDoS absorption, since the edge network has far more capacity than your origin. The things to be careful about start with cache keys. By default the key is the URL, so if your response varies by header — language, authentication, device — you must include those in the key with Vary, or users will receive each other's content. Serving a personalised page from a shared cache is a real and serious incident. Cache invalidation is slow and sometimes expensive, which is why content-hashed asset filenames are preferable to purging. Also: the origin sees the edge's IP, so client addresses come from headers. Long TTLs on HTML make deployments confusing. And an edge that cannot reach the origin may serve stale content or errors, so configure the stale-if-error behaviour deliberately rather than discovering it during an incident.

A stateless firewall evaluates each packet independently against rules. It has no memory, so to permit a TCP connection you must explicitly allow both the outbound request and the inbound response. A stateful firewall tracks connections. Allowing an outbound connection automatically permits the return traffic, because it recognises those packets as belonging to an established session. Stateful is easier to configure correctly and can enforce protocol sanity — rejecting a packet that claims to be part of a connection that does not exist. The cost is memory per connection and a table that can be exhausted, which is itself an attack vector. The distinction matters concretely in AWS. Security groups are stateful, so allowing inbound on port 443 automatically permits the response. Network ACLs are stateless, so you must also allow outbound on the ephemeral port range — and forgetting that is one of the most common causes of "the security group looks right but it still does not work". Knowing which layer is which, and that ephemeral ports need explicit handling in the stateless one, is the practical takeaway.

The honest answer is that the deep protocol internals rarely matter, but a specific subset matters constantly. Connection lifecycle and pooling: understanding handshake cost, keep-alive, and TIME_WAIT explains a large share of client-side performance problems and port exhaustion. Timeouts and retries at every layer, with backoff and jitter, and why a missing timeout is an availability risk. DNS caching behaviour, especially in long-running processes, because stale resolution causes outages that look inexplicable. HTTP semantics — idempotency, status code meaning, caching headers — because these determine whether retries and caches are safe. TLS enough to debug a chain problem, since certificate issues are routine. And the proxy layer: what a load balancer can and cannot see, where the client IP goes, and how timeouts compose across hops. What matters less day to day is subnetting arithmetic, routing protocols, and the OSI layer numbering beyond the layer 4 versus layer 7 distinction. The framing worth giving in an interview is that networking knowledge earns its keep during incidents — when the application logs look clean and the problem is a layer down.