Find a concept

Search Notes and Discovery

Enter a term to search published Notes and Discovery.

    In this topic

    Structured note

    Networking and Communications from First Principles

    Trace a request through IP, DNS, transports, proxies, load balancers, CDNs, and API gateways, then choose communication patterns from the problem and its limits.

    Classical System DesignFoundation
    Published
    Reviewed
    Reading time
    48 min

    Prerequisites

    No prerequisite note—start here.

    On this page

    A customer opens an online store, sees a product image immediately, and presses Place order. The image loads, but checkout spins and reports a timeout. Did the order never reach the application, or did the application create it and lose the response? Those are different situations. Retrying blindly can turn a slow connection into a duplicate purchase.

    Networking is the machinery that moves messages between processes. System design asks what the messages mean, where they travel, how much work they create, and how the application recovers when a step fails. Addresses, names, connections, caches, and gateways contribute different guarantees. None of them individually proves that an order was created correctly.

    Follow the request from browser to application through worked examples. Use the tables and outline for interview revision.

    Trace one request before drawing the architecture

    The customer visits https://shop.example/products/42. That URL contains a scheme, hostname, and path. The scheme asks for HTTPS, the hostname identifies the intended service, and the path identifies the application target. Before the browser can send the HTTP request, it needs a usable destination and connection.

    A simplified cold path is:

    1. Discover: find an address for shop.example using DNS if a usable answer is not already cached.
    2. Reach: send packets through the client’s network toward that address. An IPv4 router may translate the private source address and port.
    3. Connect: establish a transport and a protected connection to the public endpoint.
    4. Reuse or forward: an edge cache may answer an eligible request; otherwise an intermediary forwards toward the origin.
    5. Process: the application reads or changes state and returns a result through the relevant connections.

    This is a logical path, not a shopping list of required products. One service can implement several roles. A small application might need one reverse proxy and one backend. An internal call might skip the public edge. The deployment determines where each connection ends.

    Explore the mechanism

    Follow one request

    DNS discovers an endpoint separately from the HTTP data path. This is one possible architecture, not a mandatory chain.

    Name lookup — separate exchange

    Browsershop.exampleQuery
    DNS resolverReturns an addressAddress found

    HTTP request and response

    BrowserStarts connectionNot connected
    EdgeCache / forwardingWaiting
    GatewayEntry policyWaiting
    ServiceBusiness logicWaiting

    Step 1 of 5

    Resolve the name

    A lookup returns the endpoint address. DNS does not carry the HTTP request.

    Read every step without motion
    1. Resolve the name. A lookup returns the endpoint address. DNS does not carry the HTTP request.
    2. Connect to the edge. The browser establishes a transport connection and, for HTTPS, validates the certificate.
    3. Cross the policy boundary. For this cache miss or dynamic route, the edge forwards over a separate upstream connection.
    4. Run the operation. The service authorizes the resource and runs the operation. A database call is needed only if this operation requires one.
    5. Return the response. The response returns across the connections. A cache hit would have returned from the edge without reaching the gateway or service.
    A browser performs a DNS lookup, then sends a request through a CDN edge, API gateway, application service, and database; an eligible cache hit returns from the edge.

    DNS supplies discovery information rather than forwarding HTTP. A public asset can finish at the edge, while an order mutation may need application and database work.

    Keep three questions separate throughout the journey:

    • Reachability: can packets and connections reach the intended endpoint?
    • Delivery: can the chosen protocol carry a complete request and response?
    • Correctness: did the application perform the authorized operation with the intended effect?

    A successful TCP connection establishes only part of the second answer. A fast cached image establishes little about checkout. This separation will make the timeout easier to diagnose.

    Choose a technique from the problem

    Read the request trace first if the terms are new; return to this table when choosing a design. The techniques solve different problems and can be combined. Follow each link for the mechanism and a worked example. On a narrow screen, scroll tables sideways; keyboard users can focus a table and use the arrow keys.

    ProblemTechniqueExample use caseLimitation
    Organize network destinations.

    CIDR and private IPs

    Separate application and database networks.Address ranges need explicit access rules.
    Reach the Internet from private IPv4 hosts.

    NAT / port translation

    Office clients share a public address.Mappings and ports are finite.
    Find an endpoint that may move.

    DNS and caching

    Keep one API hostname during migration.Caches and existing connections delay changes.
    Exchange ordinary requests securely.

    HTTP over a protected transport

    Read products and submit orders.Delivery is not a committed business effect.
    Choose one-way, two-way, or deferred updates.

    Polling, SSE, WebSocket, or a queue

    Order status, chat, or receipt generation.Match replay, waiting, and persistence needs.
    Control outbound or inbound access.

    Forward or reverse proxy

    Web filtering or service ingress.Role does not guarantee privacy or encryption.
    Spread work across workers.

    Load balancing

    Survive one application-instance failure.Survivors need measured headroom.
    Reuse eligible content near readers.CDNPublic versioned images and video segments.Keys, freshness, and permissions determine safe reuse.
    Apply shared API entry policies.API gatewayValidate credentials and tenant quotas.Services still check resource permissions.
    Bound slow calls and uncertain outcomes.

    Deadlines and application idempotency

    Recover an order whose response was lost.A timeout does not prove the order failed.
    Prevent queues from consuming the service.

    Backpressure and admission limits

    Isolate expensive reports from checkout.Rejection and degradation need explicit policy.

    IP addresses, subnets, ports, and routing

    An Internet Protocol, or IP, address identifies a network destination for packet delivery. IPv4 has 32-bit addresses, conventionally written as four decimal octets, such as 192.0.2.17. IPv6 has 128-bit addresses, written as hexadecimal groups, such as 2001:db8::17. The examples use IPv4 documentation ranges and IPv6 documentation space, rather than real service endpoints.

    Addresses are not permanent identities. An interface can have multiple addresses, assignments can change, and many clients can share an externally visible address. Do not use “same IP” as proof of the same user or device.

    A packet carries network-layer addressing and a payload. A route tells a router how to forward packets for a destination prefix. IP does not by itself promise delivery or arrival order. A valid route can still lead to a failed link, an access-rule rejection, or a destination without a listening service.

    CIDR: calculate an address range

    A subnet groups addresses for network organization and routing. Classless Inter-Domain Routing, or CIDR, writes a prefix as an address plus a length. For IPv4, /26 means 26 of the 32 bits are fixed by the prefix, leaving six variable bits. This avoids treating old address classes as the modern allocation rule. CIDR specification.

    Consider 192.0.2.64/26:

    IPv4 bits:       32
    Prefix bits:     26
    Variable bits:   32 - 26 = 6
    Total addresses: 2^6 = 64
    Address range:   192.0.2.64 through 192.0.2.127
    Subnet mask:     255.255.255.192

    A conventional IPv4 host subnet reserves its network and broadcast addresses. In this example, the traditional host range is 192.0.2.65 through 192.0.2.126, leaving 62 addresses. “Subtract two” is not a universal allocation formula: point-to-point /31 links, /32 routes, IPv6, and cloud-reserved addresses have different rules.

    Now suppose a router has 192.0.2.0/24 → gateway A and 192.0.2.64/26 → gateway B. Destination 192.0.2.90 matches both, but the longer /26 prefix is more specific. Under ordinary longest-prefix routing, B wins. Destination 192.0.2.140 matches only the /24. A default route catches destinations without a more specific match; it does not prove its next hop can deliver them.

    Public, private, and translated addresses

    The standard private IPv4 ranges are 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. Organizations reuse them in internal networks instead of allocating a unique public IPv4 address to every host. They are not intended for global public routing, but internal routes, VPNs, and compromised peers can still reach private systems where policy permits.

    Private addressing provides structure; firewall rules and application permissions provide explicit controls. Plan ranges to avoid overlap when networks must connect. Two companies using the same internal range cannot simply join their routes and expect every address to remain unambiguous.

    Static and dynamic describe assignment, not public versus private. A database can have a static private address; a home connection can receive a changing public one. DHCP, the Dynamic Host Configuration Protocol, or another provisioning mechanism can assign local network settings. DNS can keep a service name stable over changing addresses.

    Network Address Translation, or NAT, rewrites addressing at a boundary. Port translation can map many private flows onto one public IPv4 address. For one illustrative TCP flow:

    Before translation:
    10.0.0.25:53000 -> 203.0.113.80:443
    
    After translation:
    198.51.100.9:62001 -> 203.0.113.80:443
    
    The recorded mapping sends corresponding replies
    back to 10.0.0.25:53000.

    The translator maintains state to associate replies with internal flows. Different source ports can distinguish multiple connections sharing a public address. Mapping resources and ports are finite, idle mappings expire, and unsolicited inbound access needs an explicit reachability design. NAT is neither encryption nor a substitute for authorization. Traditional NAT mechanism.

    Explore the mechanism

    Two clients share one translated address

    Illustrative TCP flow: two private hosts use source port 53000. PAT assigns distinct translated source ports so replies can be mapped back.

    Client A

    Private socket10.0.0.25:53000Ready
    PAT mapping198.51.100.9:62001No mapping yet
    Server203.0.113.80:443Waiting

    Client B

    Private socket10.0.0.26:53000Ready
    PAT mapping198.51.100.9:62002No mapping yet
    Server203.0.113.80:443Waiting

    Step 1 of 5

    Same source port, different hosts

    A and B are different sockets because their private addresses differ. External addresses in this example use documentation blocks.

    Read every step without motion
    1. Same source port, different hosts. A and B are different sockets because their private addresses differ. External addresses in this example use documentation blocks.
    2. Create two mappings. The router rewrites source IP and port. Both flows share the translated address but receive different translated ports.
    3. Forward the packets. The server sees source sockets 198.51.100.9:62001 and :62002. PAT is neither encryption nor a complete security policy.
    4. Use the reverse mapping. Replies to port 62001 map to A, and replies to 62002 map to B. Mappings have a lifetime; filtering behavior depends on the implementation.
    5. Deliver each reply. The two responses reach the correct private sockets. The encrypted application payload was not decrypted by this address translation.
    Two private client connections share one public IPv4 address through distinct port mappings, and replies follow the mapping table back to the correct client.

    Port translation distinguishes flows sharing an external address. The mapping must remain available for the return path; its existence does not authenticate the user.

    IPv6’s larger address space reduces the conservation reason for IPv4 NAT. It does not automatically encrypt traffic or remove firewall and permission requirements. On local networks, IPv4 commonly uses ARP to find a link-layer neighbor address; IPv6 uses Neighbor Discovery and multicast rather than IPv4-style broadcast. IPv6 routers do not fragment packets as IPv4 routers may, so path size and endpoint handling matter for oversized packets. IPv6 specification, Neighbor Discovery.

    Ports and layers: name the boundary

    A port is a transport endpoint number, not a physical connector. A server may listen on TCP port 443 while a client selects a temporary source port. TCP and UDP have separate port spaces: TCP 443 and UDP 443 are not one socket. A common flow description includes source address, source port, destination address, destination port, and transport protocol.

    A local link carries frames between neighbors; IP carries packets between networks; a transport carries bytes or datagrams between endpoints; an application protocol gives the messages meaning. In common OSI vocabulary, Layer 3 is network, Layer 4 is transport, and Layer 7 is application. The vocabulary is useful even when one appliance implements several roles.

    A documentation-address subnet routes a client through a router to a server; socket labels distinguish IP addresses from TCP ports.

    Routing selects a network path; ports identify transport endpoints. This diagram shows socket addressing, while the separate NAT example adds translation state.

    When a connection fails, inspect the bind address, route, return path, and access rules separately. A process listening only on 127.0.0.1 normally serves that host, even if the machine has a public address. Opening a port does not grant a safe application contract.

    DNS: find an address without hard-coding it

    The Domain Name System, or DNS, is a distributed naming system. Resolving a hostname lets a client discover addresses without hard-coding them. Other record types support mail, aliases, delegation, and protocol-specific metadata. Resolution and request delivery remain separate: DNS does not carry the store’s HTTP request.

    A recursive resolver finds answers for its clients. An authoritative nameserver supplies data for a zone, the managed part of the naming system for which it has responsibility. In a simplified cold lookup, the resolver follows referrals from root to top-level-domain servers to the domain’s authoritative servers, then obtains the requested data. Cached answers or delegations can skip earlier work. The browser does not usually walk that entire chain itself. DNS concepts and resolution.

    The root service’s named authorities are backed by distributed instances, not a small list of thirteen individual physical machines. This distinction matters when discussing capacity and failure domains. IANA root-server information.

    Records: the query type matters

    The store’s illustrative records might include:

    shop.example.     300 IN A     203.0.113.80
    shop.example.     300 IN AAAA  2001:db8::80
    assets.example.   300 IN CNAME images.example.
    example.         3600 IN MX    10 mail.example.

    An A record contains IPv4; AAAA contains IPv6. CNAME aliases a name to another name, rather than redirecting the browser’s URL. MX identifies mail exchangers, NS identifies authoritative nameservers, and TXT carries text interpreted by a particular application or protocol. A name can exist without a record of the requested type. DNS records, IPv6 DNS extensions.

    Explore the mechanism

    Watch a cached DNS answer age

    Illustrative successful lookup with TTL 300 seconds. Existing cache entries retain their original expiry when the authoritative record changes.

    Authority and one resolver cache

    Authoritative recordshop.example A192.0.2.20
    Resolver cacheAcquired at 12:00TTL 300 s
    Client answerUses cached address192.0.2.20

    Step 1 of 4

    12:00 — cache the answer

    The resolver obtains 192.0.2.20 with TTL 300 s, so this cache entry expires at 12:05.

    Read every step without motion
    1. 12:00 — cache the answer. The resolver obtains 192.0.2.20 with TTL 300 s, so this cache entry expires at 12:05.
    2. 12:02 — change the authority. The authoritative record changes to 192.0.2.30. This resolver still has 180 s remaining on its old cached answer.
    3. 12:04 — use the remaining TTL. A client using this resolver can still receive the old address; other caches may have different acquisition times.
    4. 12:05 — expire and query again. In this simplified case, expiry causes a new successful lookup and the resolver obtains the new address. Serving stale and prefetch policies are separate considerations.
    A recursive resolver checks its cache and follows root and top-level-domain referrals to an authoritative server on a miss, then caches an illustrative A-record answer.

    The A-record illustration uses an IPv4 documentation address. DNS returns data or an error; the subsequent application connection uses a separate path.

    TTL, negative caching, and traffic changes

    The time to live, or TTL, limits how long record data may normally be reused from a cache before refreshing. Suppose a resolver obtains TTL 300 seconds at 12:00. At 12:02 it has about 180 seconds remaining. If the authoritative address changes just then, the resolver may still serve the earlier answer until 12:05 under this illustrated caching policy.

    Changing the authoritative TTL at 12:02 does not retroactively shorten the cached 300 seconds. A migration lowers TTL in advance if faster refresh is necessary, keeps the old endpoint usable through transition, and accounts for pooled connections that can remain on the old endpoint after DNS refresh. Some explicit serve-stale policies can preserve answers beyond ordinary freshness during failures; actual client and resolver behavior needs testing.

    A DNS timeline shows an answer cached for 300 seconds, an authoritative change while the cache remains fresh, and a later refresh after the existing cached lifetime.

    A record change and a cache refresh are different events. Lowering a new answer’s TTL does not rewrite TTLs already stored elsewhere.

    Failures can be cached too. NXDOMAIN means the queried name does not exist; an empty answer for one record type is a different result. Fixing a previously absent name may therefore not cause immediate retries from every resolver. DNS negative caching.

    DNS-based balancing advertises destinations for new connections. Configured policies may use weights, geography, observed latency, or health. Anycast advertises an address from several locations and lets network routing select a path. These approaches help broad destination selection, but cached answer sets, resolver grouping, and client choices limit precise per-request control. Removing an unhealthy address from future answers does not move an existing connection.

    For dual-stack services, test both published address families. An AAAA record for a broken IPv6 path can cause client-dependent failures. Do not diagnose such behavior only by checking the healthy IPv4 endpoint.

    DNS security: match the control to the threat

    DNSSEC lets validating resolvers check signed data’s origin and integrity; it does not encrypt queries or guarantee availability. Authenticated DNS-over-HTTPS and DNS-over-TLS protect the connection to the chosen resolver; that resolver still needs to be trusted appropriately. Opportunistic DNS-over-TLS profiles can offer weaker protection. None makes an attacker-controlled website benign or replaces HTTPS to the application. DNSSEC, DNS-over-HTTPS, DNS-over-TLS, strict and opportunistic DNS privacy.

    ThreatWhat goes wrongUseful controlLimit
    Poisoning / spoofingA false answer reaches a cache or client.Validate signed data where DNSSEC applies.Not every domain is signed; confidentiality is separate.
    DNS floodingQueries overwhelm lookup capacity.Capacity, admission controls, and monitoring.Distributed serving does not make resources unlimited.
    Reflection / amplificationSpoofed requests direct larger replies at a victim.Restrict recursion and use appropriate response controls.Requires network-level measures too.
    Connection interceptionLookup traffic is observed or modified in transit.Protected transport to a trusted resolver.Resolver trust and application TLS still matter.
    Random-name exhaustionMany new names force expensive misses.Bound query work and monitor miss/error patterns.Negative caching cannot collapse arbitrary unique names.
    TunnelingQueries or answers carry unwanted data.Controlled DNS egress and anomaly monitoring.Unusual names are signals, not proof by themselves.
    HijackingAn account or configuration changes DNS control.Protect registrar accounts and audit changes.Valid signatures do not undo authorized malicious changes.
    RebindingA name later resolves to a prohibited address.Validate resolved destinations at the connection boundary.A hostname-only check can be insufficient.

    Transports move data; application protocols give it meaning

    Addressing identifies where packets go. A transport defines the communication service at the endpoints; an application protocol defines how to interpret its messages.

    Latency is elapsed time for a particular operation. Bandwidth is an available transfer rate. Throughput is the useful rate actually achieved at the boundary you measure. Round-trip time, or RTT, is time to travel to a peer and return. A fast link does not eliminate propagation distance, connection setup, server work, or queueing.

    For a 1 MB response using decimal units, the payload contains 8 Mb. At an ideal 20 Mb/s useful rate, payload transfer alone takes 8 / 20 = 0.4 seconds. Add any setup, first-byte wait, protocol overhead, loss recovery, and competing traffic to reason about completion. This arithmetic is a lower-level component of latency, not a forecast of page load time.

    TCP and UDP: choose the actual delivery contract

    Transmission Control Protocol, or TCP, provides a reliable, ordered byte stream with flow control and congestion control. Establishment ordinarily uses a SYN, SYN-ACK, ACK exchange. The receiver acknowledges bytes; retransmission repairs missing data while the connection remains viable. TCP preserves byte order, not the application’s message boundaries: one application write need not equal one receiver read. A framing rule must identify complete application messages. TCP specification.

    If later bytes arrive before an earlier lost portion, TCP cannot deliver that later portion of the stream as though nothing were missing. This is useful for an ordered contract but can delay independent application work sharing the stream. TCP also cannot guarantee that a remote application committed a database transaction. Acknowledged bytes and committed orders are different facts.

    User Datagram Protocol, or UDP, sends datagrams without supplying TCP’s ordered reliable stream contract. A receiver may see loss, duplication, or reordering, and a datagram can be too large for a path. The application or a protocol built above UDP must add the recovery, congestion handling, authentication, and sequencing it needs. UDP specification.

    For voice, a packet containing an old audio interval might be less useful than moving on. For an order, quietly discarding a mutation is unacceptable. Yet “UDP is for unreliable applications” is too broad: QUIC builds secure connections and reliable streams over UDP. Choose the complete protocol, not a transport slogan. DNS also supports TCP, so “DNS always uses UDP” is incorrect. DNS over TCP requirements.

    TLS protects a hop

    Transport Layer Security supplies confidentiality and integrity for a connection and normally authenticates the server through certificate validation. Mutual TLS also authenticates the client with a certificate. TLS 1.3 typically needs one handshake round trip before ordinary application data on a new connection; resumption changes the path. Its optional early data can be replayed and requires suitable application handling. TLS 1.3 specification.

    For checkout, choose a protected connection and validate the intended hostname and trust chain. Do not turn off certificate checks to make an error disappear. The TLS endpoint might be an edge proxy rather than the application server. If it terminates TLS, plaintext is available there, and edge-to-origin encryption needs its own configuration. Encryption to a gateway does not magically encrypt a later unprotected hop.

    TLS authentication answers who the peer is according to the configured credentials. It does not answer whether that peer may refund order 42. IPv6 does not automatically turn this protection on either; address format and connection security are independent decisions.

    HTTP versions change transport behavior

    HTTP expresses operations on a target resource using methods, headers, and response status. An illustrative resource read is:

    GET /products/42 HTTP/1.1
    Host: shop.example
    Accept: application/json
    
    HTTP/1.1 200 OK
    Content-Type: application/json
    Cache-Control: public, max-age=60
    
    {"id":"42","name":"Desk lamp"}

    This is a human-readable contract sketch; a complete wire response needs appropriate message framing. HTTP is stateless at the protocol-semantic level, but applications can store users, sessions, carts, and orders. Persistent connections mean HTTP does not require a new connection for every request. HTTP semantics, HTTP/1.1 message framing and connection management.

    HTTP/2 multiplexes streams on a connection and compresses headers. It can interleave independent HTTP exchanges, avoiding the requirement to finish one response before starting every other one. Its usual TCP connection still has ordered delivery, so a transport loss can delay all streams sharing that connection. HTTP/2 specification.

    HTTP/3 carries HTTP over QUIC. QUIC provides secure, multiplexed streams over UDP, and packet loss affecting one stream’s data need not stop delivery of available data on another stream. This removes TCP’s cross-stream delivery dependency, not congestion, CPU limits, or every application dependency. UDP reachability, supported implementations, and fallback behavior affect adoption. HTTP/3 specification, QUIC specification.

    For a small loss example, let responses A and B share one TCP connection. Bytes 1,000–1,499 are missing, but later bytes 1,500–1,999 carrying B arrive. TCP cannot deliver those later bytes past the gap. With QUIC, if the missing data belongs to stream A and all required data for stream B has arrived, B can be delivered independently. One packet can carry data for several streams, so losing a packet may still affect several responses; the difference is their delivery dependency.

    Explore the mechanism

    What does one lost packet delay?

    Compare HTTP/2 over TCP with HTTP/3 over QUIC. The lanes represent application stream data; delivery ordering differs from shared congestion control.

    One TCP connectionBoth streams share connection-wide congestion control.
    One shared ordered-delivery gateNo missing byte range

    Stream A

    SendA dataReady
    Buffered stream AInside this shared connectionIn transit
    ApplicationA deliveryWaiting

    Stream B

    SendB dataReady
    Buffered stream BInside this shared connectionIn transit
    ApplicationB deliveryWaiting

    Step 1 of 4

    Send multiplexed data

    Both application streams share the connection. This is a simplified loss example, not a packet trace.

    Read every step without motion

    HTTP/2 / TCP

    1. Send multiplexed data. Both application streams share the connection. This is a simplified loss example, not a packet trace.
    2. Lose A's packet. TCP exposes one ordered byte stream. Later bytes cannot be delivered past a missing range, even when they carry another HTTP/2 stream.
    3. Deliver what is available. B's received data remains behind the TCP delivery gap until retransmission fills it.
    4. Recover the missing data. After retransmission, A can continue. QUIC still provides reliable delivery and congestion control over UDP.

    HTTP/3 / QUIC

    1. Send multiplexed data. Both application streams share the connection. This is a simplified loss example, not a packet trace.
    2. Lose A's packet. QUIC orders delivery within each stream. Missing A data does not inherently block delivery of received B data.
    3. Deliver what is available. B reaches the application while A waits. Congestion control and some application dependencies can still affect both streams.
    4. Recover the missing data. After retransmission, A can continue. QUIC still provides reliable delivery and congestion control over UDP.
    Three panels show serial HTTP/1.1 over TLS and TCP, multiplexed HTTP/2 streams sharing TCP loss delays, and HTTP/3 streams over QUIC with TLS 1.3 and UDP.

    The HTTP/1.1 panel illustrates serial requests, not a ban on persistent connections or pipelining. Multiplexing reduces some serialization; QUIC avoids TCP’s cross-stream delivery blocking while congestion and application dependencies still influence completion.

    A cold-versus-warm latency calculation

    Assume a small response, a 40 ms client-to-endpoint RTT, a 20 ms DNS lookup, and 30 ms server processing. Assume sequential DNS, a new conventional TCP connection, a full TLS 1.3 handshake without early data, and one request-response round trip. Ignore response serialization, retransmission, and queueing for this simplified first-byte budget:

    Cold path = DNS + TCP setup + TLS setup + request/response + server work
              = 20 + 40 + 40 + 40 + 30
              = 170 ms
    
    Warm path = request/response + server work
              = 40 + 30
              = 70 ms

    The example explains why connection reuse can help. It does not imply every request saves 100 ms: DNS may be cached, connections may already exist, negotiations can overlap, the public peer may be nearby, and a large response adds transfer time. Measure DNS, connect, handshake, first byte, and completion separately before optimizing.

    API and realtime communication patterns

    The client initiates a particular interaction; the server accepts or serves it. These are roles, not permanent machine categories. An Application Programming Interface (API) is the contract through which software asks another system for data or an operation. The store API is a server to the browser and a client when it calls inventory. A backend can handle a request asynchronously in its implementation while the external protocol still uses request-response.

    Separate three meanings of asynchronous: a caller’s thread may continue without blocking; a connection may carry ongoing events; or a workflow may acknowledge now and finish later. Calling all three “async” hides their different failure and recovery contracts.

    Request-response APIs: REST-style and RPC

    Representational State Transfer, or REST, is an architectural style. A REST-style HTTP API emphasizes resources and standard HTTP operations. A product can be read with GET /products/42; an order intent can be submitted with POST /orders. JSON is a representation choice, not the definition of REST. A resource-oriented URL alone does not establish the full REST architectural style, and REST does not imply a blocking client thread.

    A Remote Procedure Call, or RPC, interface exposes operations, such as ReserveInventory or CalculateShipping. gRPC commonly uses service definitions and generated client/server interfaces, and supports unary, client-streaming, server-streaming, and bidirectional-streaming calls. This can make internal contracts explicit, while still requiring compatible schema evolution and error handling. gRPC core concepts.

    Choose based on consumers and contracts. A public product API may value HTTP tooling and stable resource representations. Internal inventory calls may benefit from typed operation schemas. Neither choice is inherently faster, safer, or easier in every system. Payload size, serialization, connection reuse, implementation, and deployment affect performance.

    Here is an illustrative mutation contract:

    POST /orders HTTP/1.1
    Host: shop.example
    Content-Type: application/json
    Authorization: Bearer <validated-user-token>
    Idempotency-Key: <one-key-for-this-order-intent>
    
    {"product_id":"42","quantity":1}

    The token authenticates a caller only after validation. The idempotency key has meaning only if the server implements deduplication. The server must still validate price, stock, ownership, and payment rules. A browser-supplied total cannot become authoritative merely because the request is syntactically valid.

    Stateless application workers do not require all state to vanish. They can load cart and session information from a shared store and persist orders in a database. The useful scaling property is that a subsequent request does not depend on one worker’s private memory. A self-contained token also does not remove every shared-state need, such as revocation or deduplication.

    Polling, SSE, WebSocket, and asynchronous messages

    PatternUse whenExampleImportant limitation
    Request–responseThe caller needs this operation’s result.Product read or inventory reservation.Nonblocking client code still awaits a result.
    PollingUpdates are infrequent and simplicity matters.Check an export’s status.Repeated requests trade cost for freshness.
    Server-sent eventsServer-to-client updates dominate.Order-status feed.Reconnect needs retained events to replay.
    WebSocketBoth peers send frequent messages.Chat or collaborative editing.Persistent connections need limits and recovery.
    Durable queueWork may finish after acceptance.Receipt generation.Acceptance is not completion; duplicates can occur.

    Explore the mechanism

    Separate waiting from durable work

    Nonblocking HTTP changes how a caller waits. A durable job needs persisted acceptance, an independent worker, and a way to obtain its eventual result.

    Blocking HTTP

    CallerThread waitsSend
    ServiceRuns operationWaiting
    Response at callerHTTP responseNo response
    →request

    Nonblocking HTTP

    CallerOther work can runSend
    ServiceSame request semanticsWaiting
    Response at callerAwait HTTP responseNo response
    →request

    Durable job

    SubmitRequest acceptanceSend
    Job storePersist + acknowledgeNot accepted yet
    WorkerRuns independentlyWaiting
    ResultStatus / notificationPending

    Job store → submitter: acknowledgment only after durable acceptance.

    Step 1 of 4

    Submit the operation

    Blocking and nonblocking callers both initiate an HTTP exchange. The job flow submits an intent for independent processing.

    Read every step without motion
    1. Submit the operation. Blocking and nonblocking callers both initiate an HTTP exchange. The job flow submits an intent for independent processing.
    2. Wait, yield, or accept. The blocking thread waits; a nonblocking caller can yield. The job acknowledgment is durable only after the configured persistent acceptance boundary.
    3. Complete the work. HTTP responses complete their exchanges. A job can continue after the submitter disconnects, subject to persistence and worker recovery guarantees.
    4. Obtain the job result. The caller fetches status or receives an update. SSE is a server-to-client stream; WebSocket is bidirectional. Neither creates durable job storage by itself.
    Three lanes distinguish blocking HTTP, nonblocking HTTP, and durable queued jobs; smaller panels show server-to-client SSE and bidirectional WebSocket messaging.

    Choose who sends, when the caller continues, and how missed or duplicate work is recovered. A persistent connection is not a durable job queue.

    Polling asks repeatedly: “Has the status changed?” It is simple and works well for infrequent updates. With 10,000 active clients polling every 5 seconds, evenly spread, the average request rate is 10,000 / 5 = 2,000 requests/s even when no status changes. If all clients poll together, instantaneous bursts are worse; stagger or jitter the schedule.

    Server-sent events, or SSE, provide an HTTP event stream from server to client, typically with text/event-stream. Browser EventSource supports reconnect behavior and event IDs, but replay requires the application to retain and retrieve missed events. The feed is one-way; ordinary requests can carry client actions. Server-sent event standard.

    WebSocket supports bidirectional messages on an established connection. Its classic HTTP/1.1 handshake uses Upgrade; HTTP/2 and HTTP/3 can use negotiated extended CONNECT. Once connected, peers can send without opening a new HTTP request for each message. The application still needs message schemas, authentication, reconnects, and limits. WebSocket specification, WebSocket over HTTP/2, WebSocket over HTTP/3.

    For order status, SSE or occasional polling may suffice. For chat, a WebSocket can fit frequent two-way updates. Every persistent connection consumes some memory, socket capacity, and routing state. During reconnect storms, add backoff and bounded admission; define how clients resume without dropping or duplicating important events. Heartbeats help detect dead connections but add traffic and do not prove application health.

    An asynchronous queue separates acceptance from completion. After recording an order and its receipt task durably, the API can return a job identifier while a worker generates the receipt. Consumers need duplicate handling, bounded retries, backlog monitoring, and a way to inspect terminal failure. For a business effect plus an event, a transactional outbox or equivalent durable coordination avoids the gap between “database commit succeeded” and “message publication failed.” This changes the workflow, not the laws of networking.

    Forward and reverse proxies place the boundary

    A proxy is an intermediary acting on another party’s behalf. Forward and reverse describe the role from the deployment’s perspective, not opposite packet directions. Both requests and replies flow through either kind.

    A forward proxy serves clients’ outbound access:

    Managed client -> organization's forward proxy -> external service

    It can enforce destination policy, provide audit visibility, and sometimes cache eligible responses. For HTTPS, an HTTP CONNECT tunnel can carry encrypted traffic without revealing its HTTP content to the proxy. Inspecting that content requires a deliberate TLS interception trust model, which changes the security boundary. A VPN generally tunnels network traffic; it is not simply another name for an HTTP forward proxy.

    A reverse proxy serves a backend’s incoming access:

    External client -> store reverse proxy -> application backend

    It can terminate TLS, route hostnames or paths, impose request-size limits, and forward to backend pools. It may also implement caching, load balancing, or gateway policies. Those are configurable capabilities, not mandatory properties of every proxy. One reverse proxy can fill several architectural roles.

    A forward proxy represents managed clients reaching external services, while a reverse proxy represents application servers receiving clients; arrows show requests and responses, with a TLS-interception trust warning.

    Ask whom the intermediary represents and which connections terminate there. TLS on one hop and address masking do not establish end-to-end encryption or anonymity.

    Three TLS arrangements and certificate ownership

    Choose the connection boundary before placing certificates:

    • Termination with plaintext upstream: ingress ends the public TLS connection and forwards an unprotected application connection. Make a deliberate trust decision about that upstream network.
    • Termination with re-encryption: ingress ends the client session and starts a separately validated TLS session to the backend. Both links are protected; the intermediary can read the message.
    • Passthrough: encrypted traffic reaches the backend, where the client’s TLS connection ends. The intermediary cannot also inspect the encrypted HTTP path.
    Three TLS arrangements show termination with a plaintext upstream hop, termination with a second TLS session upstream, and passthrough carrying one encrypted session to the backend.

    These are valid design alternatives with different plaintext and routing boundaries. Re-encryption protects both links; it does not hide the message from the terminating proxy.

    A server certificate identifies an endpoint to clients. A commonly named full chain file contains the leaf certificate and needed intermediate certificates, not a private key. Keep the corresponding private key secret at the endpoint that uses it. In TLS 1.3 certificate-based authentication, that key authenticates the handshake; negotiated traffic keys protect application data. The phrase “the certificate decrypts all traffic” obscures these different roles.

    If an ingress forwards to a private service listening on port 8000, that service does not automatically need its own public hostname. It does need whatever identity, connection protection, and access rules the chosen upstream design requires. DNS discovery, certificate hostname matching, and application path routing are three separate checks.

    Explore the mechanism

    Where can the proxy read plaintext?

    TLS protects a session between its endpoints. Select a trust-boundary design; encryption on two hops does not hide plaintext from the proxy between them.

    Client hop and upstream hop

    ClientValidate expected endpointHandshake
    ProxyTLS endpointWaiting
    BackendPlain HTTP upstreamWaiting

    Client ↔ proxy: TLS session A. Proxy ↔ backend: plaintext HTTP.

    Step 1 of 4

    Establish the session

    The client establishes TLS with the proxy and validates its certificate for the requested hostname.

    Read every step without motion

    Termination

    1. Establish the session. The client establishes TLS with the proxy and validates its certificate for the requested hostname.
    2. Reach the proxy. The proxy terminates the client TLS session and sees the HTTP request in plaintext.
    3. Cross the upstream hop. The upstream hop uses plaintext HTTP here. Its trust boundary must be deliberately assessed; remote hops need appropriate protection.
    4. Keep authorization separate. TLS endpoint validation does not decide whether this caller may access a particular object. The application must enforce that permission.

    Re-encryption

    1. Establish the session. The client establishes TLS with the proxy and validates its certificate for the requested hostname.
    2. Reach the proxy. The proxy terminates the client TLS session and sees the HTTP request in plaintext.
    3. Cross the upstream hop. The proxy creates and validates a separate TLS session to the backend. Both hops are encrypted; the proxy still sees plaintext.
    4. Keep authorization separate. TLS endpoint validation does not decide whether this caller may access a particular object. The application must enforce that permission.

    Passthrough

    1. Establish the session. The client's TLS session ends at the backend. The proxy forwards encrypted bytes and cannot route by decrypted HTTP paths.
    2. Reach the proxy. The proxy can use available transport or handshake metadata, but does not decrypt this TLS session.
    3. Cross the upstream hop. The backend decrypts and processes the request. This is the same TLS session initiated by the client.
    4. Keep authorization separate. TLS endpoint validation does not decide whether this caller may access a particular object. The application must enforce that permission.

    Trust forwarded identity only from trusted paths

    An origin may see a proxy’s network address instead of the client’s. A Forwarded or similar header can convey earlier information, but a client can also send a forged header. Strip or overwrite untrusted values at ingress and accept a defined chain only from trusted intermediaries. Never authorize a user solely from a header that anyone can supply. Forwarded HTTP extension and its trust cautions.

    Likewise, hiding an origin IP is useful only if direct paths are controlled. Restrict origin ingress to intended intermediaries when appropriate, authenticate upstream connections, and keep the backend’s own checks. A proxy does not make cookies, identifiers, payloads, or logs anonymous. TLS termination centralizes some certificate work but also makes that proxy a sensitive plaintext boundary.

    The cost is another hop with connection pools, buffers, timeouts, and configuration. Match upstream and downstream protocols deliberately, avoid buffering a streaming response unintentionally, and make tracing cross the boundary. A proxy that returns 502 tells you an intermediary could not obtain a usable upstream response; it does not identify the precise cause without logs and timing evidence.

    Load balancing: distribute work without inventing capacity

    A server has finite CPU, memory, network bandwidth, connections, and dependency capacity. Vertical scaling gives one server more resources. Horizontal scaling adds servers. A load balancer selects targets so clients need not know every backend address. Adding application workers helps only when the limiting work can spread; a saturated shared database may remain saturated.

    Suppose one application instance sustains 400 requests/s at the required latency in a representative load test. Three equally capable instances offer an ideal aggregate 1,200 requests/s for that workload if distribution and shared dependencies permit it. For a 900 requests/s peak, all three would average 300 requests/s. After one fails, each survivor would receive 450 requests/s, exceeding the demonstrated per-instance rate. The system is redundant, but it lacks the assumed failure capacity.

    Four such instances would leave three after one failure, averaging 300 requests/s. That is a better starting budget, not a guarantee: expensive requests, bursts, cold caches, uneven routing, and correlated failures can break the equal-distribution assumption. Size from the required workload and failure condition, then test it. Autoscaling is not instantaneous; it needs spare capacity or controlled admission while new instances become ready.

    Layer 4 and Layer 7 answer different questions

    Layer 4 is transport. Balancing at this layer selects using transport information such as addresses, ports, and connection state. It can pass encrypted application traffic to a backend without reading HTTP paths. Its unit of selection is often a connection or flow. If a client sends many requests on one long-lived connection, those requests may stay on one selected target.

    Layer 7 is application. Balancing at this layer interprets an application protocol. For HTTP, routing can depend on hostname, path, headers, or a controlled cookie. If HTTP is encrypted, the component needs an appropriate termination or visibility arrangement to read those fields. It can choose different backend pools for /products and /orders, but must preserve intended request semantics and trust boundaries.

    A reverse proxy, an L7 load balancer, and an API gateway can share features. The useful distinction is their main responsibility: represent backends, distribute work, or enforce an API contract and policies. Do not infer a product’s exact capabilities from a label.

    Choose an algorithm from the workload

    Round robin cycles through targets. For equal short requests and similar servers, it is a simple baseline. Six requests across A, B, C might select A, B, C, A, B, C. It balances request counts, not necessarily CPU time: two expensive requests on A can outweigh several cheap ones on B.

    Weighted round robin assigns proportions. Weights A=2, B=1, C=1 imply shares of 50%, 25%, 25% over a suitable sample, not necessarily a particular four-request sequence. If offered load is 800 requests/s, the intended rates are 400, 200, and 200 requests/s. Use weights justified by measured capacity and update them when the workload changes.

    Least connections chooses a target with fewer active connections. It can help when connection durations vary, but one busy multiplexed connection can carry more work than many idle ones. Least outstanding requests, response-time-aware selection, or explicit load signals can fit other workloads. Each depends on timely measurements and careful handling of feedback so the fleet does not oscillate between targets.

    Hash-based affinity sends a key repeatedly to a target. Source-IP hashing is coarse: a whole office behind NAT may share one source IP and overload one backend. Cookie affinity can keep a session near local state, but failures still break that local state and sticky assignments can skew traffic. Prefer shared durable state when any worker should serve a request; retain stickiness only for a justified benefit.

    Consistent hashing reduces reassignment when membership changes compared with naïvely taking a hash modulo the server count. It helps cache locality or partition routing, but does not replicate data, eliminate hot keys, or make a failed target’s data available. “Same key goes to the same server” is a routing property, not a durability property.

    Explore the mechanism

    Balancer and parallel target pool

    Illustrative evenly distributed load: 180 requests/s, three targets with a target rate of 100 requests/s each. These are teaching values, not measured capacity.

    Target pool

    Balancer180 requests/s3 eligible
    A100 requests/s target60 requests/s
    B100 requests/s target60 requests/s
    C100 requests/s target60 requests/s

    R1, R2, R3 represent different requests, each routed to one target. Movement is illustrative, not a 180 requests/s simulation.

    Step 1 of 4

    All three targets healthy

    Even routing gives each target 60 requests/s. Real algorithms and connection reuse need not produce an exactly equal short-term split.

    Read every step without motion
    1. All three targets healthy. Even routing gives each target 60 requests/s. Real algorithms and connection reuse need not produce an exactly equal short-term split.
    2. C fails before detection. There is a detection window. Traffic may still be sent to C until health checks or passive failures mark it ineligible.
    3. Remove C from the eligible pool. After detection, new eligible routing excludes C. Existing or in-flight requests need their own recovery policy.
    4. The remaining targets take more load. A and B now receive about 90 requests/s each. Shared dependencies and changed latency can still make the system fail despite this rate being below the illustrative target.
    A balancer spreads 180 requests per second across three 100-request-per-second servers, then directs 90 requests per second to each survivor after removing a failed target.

    This separate numerical illustration assumes 180 requests/s and 100 requests/s capacity per server: 60 each with three targets, then 90 each with two. It does not measure a real fleet; detection delay and in-flight failures still matter.

    Health checks, draining, and the balancer itself

    A TCP-connect probe establishes that something accepts connections. An HTTP readiness probe can test whether an instance should receive new requests. These are active checks: the balancer sends probes. Passive checks use observations from real traffic, such as repeated upstream errors or timeouts. Neither proves that every real request will succeed. Readiness checks should represent useful serving ability without turning a shared dependency incident into an uncontrolled removal of the entire fleet.

    Detection has delay. With checks every 5 seconds and removal after 3 consecutive failures, the observation window is roughly 10–15 seconds plus probe duration, depending on when failure begins. Requests can fail before detection, and in-flight work on the failed server may be lost. Recovery needs thresholds too, to avoid rapid add/remove flapping.

    For maintenance, drain a target: stop assigning new work, allow eligible in-flight work to finish within a deadline, then terminate. Persistent WebSockets require an explicit reconnect or handover policy. Removing a server from a routing list does not move its already-established connections automatically.

    Finally, a single manually operated balancer can itself become a bottleneck or single point of failure. Use the platform’s redundancy model or design a redundant ingress, and understand how failover affects existing connections. Multiple balancers still depend on routing, configuration distribution, and certificates; count those dependencies instead of assuming replicas make them independent.

    CDN: cache reuse-safe responses near the reader

    A Content Delivery Network, or CDN, places delivery capacity at distributed points of presence, or PoPs. An origin holds or generates the authoritative content; an edge can answer eligible requests from a local cache. Routing may use DNS decisions, anycast, latency, load, and network topology. The geographically nearest building is not always the best reachable endpoint, and failover can interrupt existing connections.

    A cache hit finds an eligible reusable representation. A miss needs an upstream fetch, often followed by storage if policy allows it. A stale entry might instead be validated or served under a deliberate stale-serving policy. A hit still costs network and processing time; a miss does not always mean “fetch from the database.” There may be an origin shield or another cache in between.

    Cache keys are part of correctness

    A cache key identifies which requests can share a stored response. Hostname, path, query parameters, relevant headers, and sometimes application-defined dimensions can matter. Ignoring a parameter that changes the representation returns the wrong content. Including meaningless tracking parameters can fragment the cache and lower reuse. Normalize only with knowledge of the application contract.

    For product images, a versioned URL may identify one public object. For GET /account, the response depends on the authenticated user. Caching every account response under only /account could disclose one customer’s data to another. Default to an explicit private-response policy; do not assume GET means public or safe to share.

    Vary identifies request-header dimensions used to select a representation. It does not automatically cover every application-specific identity or tenant dimension. Shared-cache handling of authorization, cookies, and operator overrides needs deliberate review. Test requests as different users and tenants, not just two repeated requests from one account.

    Freshness, validation, and invalidation

    HTTP cache directives distinguish storage and reuse. max-age sets freshness; s-maxage can set shared-cache freshness. no-cache permits storage but requires validation before reuse. no-store directs caches not to store the response. A freshness expiry does not require immediate deletion or a background fetch at that exact moment; a later request may trigger validation or replacement. HTTP caching rules.

    Illustrative public and private response policies:

    # Public versioned asset: representation never changes at this URL.
    Cache-Control: public, max-age=31536000, immutable
    
    # Personalized checkout result: do not store it in HTTP caches.
    Cache-Control: no-store
    
    # Reusable product description, with an explicit freshness budget.
    Cache-Control: public, max-age=60
    ETag: "product-42-v7"

    These are separate response examples, not headers to combine. An entity tag can let a cache ask whether its representation is still current; a 304 Not Modified response avoids transferring a new body when validation succeeds. immutable suits content that will not change at its current URL. Immutable cache-control extension.

    For a new asset, publish /assets/app.a17c.js and update the HTML reference. A later version uses a different URL. This cache busting selects a new cache entry; it does not purge old URLs or update cached HTML automatically. A manual purge removes eligible cached entries according to the provider’s operation, but propagation and scope need verification. Protect purge credentials because invalidation affects availability and origin load.

    Stale-while-revalidate can serve a permitted stale response while refreshing in the background. That improves continuity for tolerant content, but inventory, authorization, and payment decisions have different freshness requirements. A stale product description may be acceptable; an old permission grant may not be. Stale-serving HTTP extensions.

    Explore the mechanism

    A hit is useful only when reuse is safe

    This example compares a cacheable public versioned asset with an uncached private response. Eligibility, cache key, and freshness are separate checks.

    Request / response path

    BrowserVersioned public assetRequest
    Edge cacheEligibility + key + freshnessCheck eligibility
    OriginGenerate responseWaiting

    Check eligibility before choosing an edge hit or an origin fetch.

    Step 1 of 4

    Identify the requested representation

    A public asset is eligible for shared caching under this policy, using the correct representation key.

    Read every step without motion

    Public cache hit

    1. Identify the requested representation. A public asset is eligible for shared caching under this policy, using the correct representation key.
    2. Check the cached entry. The cache key matches a fresh eligible entry, so the edge can answer without contacting the origin.
    3. Serve or fetch. The edge sends the cached public asset to the browser. No origin work was needed for this hit.
    4. Finish without unsafe reuse. The browser receives the cached public asset. The edge retains the eligible entry under its existing freshness policy; the origin was not contacted.

    Public cache miss

    1. Identify the requested representation. A public asset is eligible for shared caching under this policy, using the correct representation key.
    2. Check the cached entry. There is no matching fresh reusable entry. The edge must fetch from the origin.
    3. Serve or fetch. The origin returns a response. A miss does not mean every response is safe to store.
    4. Finish without unsafe reuse. A fetched public response is stored only if its policy permits reuse. Freshness expiry, revalidation, and invalidation need explicit rules.

    Private response

    1. Identify the requested representation. A personal response must not be reused across users. This route is configured to bypass shared reuse.
    2. Check the cached entry. Private content bypasses shared cache reuse. Authentication data and cache policy must not leak representations across callers.
    3. Serve or fetch. The origin returns a response. A miss does not mean every response is safe to store.
    4. Finish without unsafe reuse. The browser receives its private response; the shared edge does not store it for other users in this example.
    A CDN edge checks a cache key and freshness, serves an eligible hit, and fetches from origin on a miss; public asset caching and personalized response exclusion are contrasted.

    Cacheability is a correctness decision before it is a performance decision. Freshness, key selection, validation, and private-response rules determine whether a stored answer may be reused.

    Calculate origin load and average latency

    Assume 10,000 requests/s for cache-eligible images and a 90% request hit ratio, with each miss causing one origin request. Ignore revalidation, collapsed simultaneous misses, prefetching, and other traffic:

    Hit requests/s    = 10,000 × 0.90 = 9,000
    Origin requests/s = 10,000 × 0.10 = 1,000

    Now assume an end-to-end hit completes in 20 ms and a complete miss path in 120 ms:

    Mean latency = 0.90 × 20 + 0.10 × 120
                 = 18 + 12
                 = 30 ms

    This weighted mean does not establish p95 or p99 latency. A minority of misses can dominate the tail. A request hit ratio also differs from a byte hit ratio: caching many tiny icons does not necessarily save most bytes if large video responses miss.

    A sudden global purge or many distinct cache keys can create an origin surge. Request coalescing, origin shielding, controlled warming, and stale-serving policies can reduce pressure when supported and appropriate. Provision and test the origin’s miss-path behavior; a warm-cache load test alone hides this failure case.

    Compression, video, and multiple CDNs

    Compression and optimized image formats reduce bytes; they do not shorten propagation distance. Keep content negotiation and cache variants consistent so a client receives an encoding it supports. Dynamic compression costs CPU, and already-compressed media may benefit little. Prefer measured byte savings and useful rendering performance to automatically transforming every response.

    Video delivery often caches segments at several bitrate levels. Adaptive bitrate players choose representations based on available throughput and playback buffer; the CDN delivers requested segments. Live manifests and segments have different freshness needs. Cache delivery cannot substitute for correct media packaging or entitlement checks.

    A multi-CDN design can reduce provider dependence or improve regional performance. It also duplicates configuration, purge operations, certificates, observability, and cache warming. Define steering and failure detection, and test whether moving traffic overwhelms the destination’s cold cache or origin. Two providers sharing the same fragile origin are not independent end-to-end systems.

    API gateway: a contract and policy boundary

    An API gateway is an application-facing entry point that routes API calls and applies selected cross-cutting policies. Common responsibilities include credential validation, request-size limits, schema checks, tenant quotas, version routing, and telemetry. It may aggregate responses or translate protocols. Each additional transformation needs a clear contract, because it can alter latency, error handling, and compatibility.

    A gateway is useful when several APIs need consistent admission policy. For one small service, a normal reverse proxy plus well-defined application middleware may be sufficient. Add a separate gateway when ownership, client contracts, or policy coordination justify it. Feature overlap with L7 balancing is normal.

    A load balancer distributes traffic among eligible backends while an API gateway applies a shared entry contract; a Venn diagram shows overlapping capabilities and services retaining object permission checks.

    Responsibilities overlap across products. Select the required behavior and ownership rather than assuming a label defines an exclusive feature set. The branches show alternative destinations: an ordinary request uses one selected backend or matching service route in this illustration.

    Explore the mechanism

    Identity, quota, and permission are different checks

    Illustrative token bucket: capacity 3, refill 1 token/s. Admission at the gateway still leaves object-level authorization to the service.

    Following R1: credential verification; quota admission is next.

    Admission and authorization

    CallerIdentity credentialRequest
    GatewayVerify + bucket + routeIdentity verified
    ServiceObject permissionWaiting
    ResultAllowed or rejectedPending

    Step 1 of 4

    Verify the caller

    The gateway verifies credentials under its configured policy. A valid caller is not automatically permitted to access every object.

    Read every step without motion
    1. Verify the caller. The gateway verifies credentials under its configured policy. A valid caller is not automatically permitted to access every object.
    2. Admit a burst within the bucket. Three immediate requests consume the initial 3 tokens. This is a burst budget, not evidence of a sustainable throughput capacity.
    3. Reject the next immediate request. A fourth request before refill is denied under this example quota. Waiting one second would refill one token; policy determines the rejection response.
    4. Return to admitted request R1. This is R1, admitted earlier, not rejected R4. The service checks R1's permission on the requested object. R4 never reached the service; authentication alone does not grant R1 access.

    Authentication does not finish authorization

    Authentication establishes a claimed identity using validated credentials. Authorization decides whether that identity may perform an operation on a particular resource. A gateway might reject an invalid token and check a broad orders:read scope. The orders service must still check that this caller may read order 42 in the correct tenant.

    For a JSON Web Token, or JWT, decoding JSON is not validation. Validate the allowed algorithm, signature with trusted keys, intended issuer and audience, applicable time claims, and the token’s expected use. Key rotation and cache refresh must be planned without accepting arbitrary keys supplied by an attacker. JWT best current practices.

    Remove client-forged internal identity headers and propagate identity through a controlled, authenticated path. Restrict direct backend access when gateway checks are required. API keys can identify an application or quota owner but do not inherently establish an end user’s identity. Mutual TLS authenticates peers; it still needs authorization policy.

    An API gateway applies authentication and admission policy, then forwards a request to its selected catalog or order service route; resource permission checks remain in that service.

    Centralize appropriate entry policies while keeping domain decisions with the service that owns the resource. A validated identity is an input to authorization, not its conclusion.

    Rate limits, concurrency limits, and quotas

    A rate limit bounds requests over time; a concurrency limit bounds work currently in flight. They solve different problems. One hundred fast reads/s and one hundred slow reports/s can have the same rate but very different resource use. Apply limits at meaningful boundaries such as tenant, credential, route, or cost class, while considering unauthenticated traffic too. IP-only quotas can unfairly combine users behind NAT.

    The algorithm determines what the limit means, especially around bursts. Choose from the actual admission behavior:

    AlgorithmMechanismExample use caseLimitation
    Fixed windowCount within a fixed interval.Simple per-minute API allowance.Boundary bursts can straddle two windows.
    Sliding windowCount over a moving interval.Smoother recent-request limits.Requires state or an approximation.
    Token bucketRefill credits; each admitted request spends them.Allow a bounded burst with a sustained rate.Independent buckets can multiply a shared quota.
    Leaky-bucket-style shaperDrain a bounded queue at a chosen rate.Smooth work admitted to a dependency.Queue size and waiting time must be limited.

    For an initially full bucket of 200 tokens, refill rate 50 tokens/s, and cost one token per request, a simultaneous 250-request burst can immediately admit 200 and reject or queue the remaining 50 according to policy. If no tokens are consumed during the next second, the bucket gains 50 tokens, up to capacity. This bounds the burst; it is not the same rule as “exactly 50 in every second.”

    Throttling terminology varies across systems. It can mean rejection, delayed admission, or reduced service rate. HTTP 429 Too Many Requests can communicate a rate-limit rejection, sometimes with Retry-After; a client should respect applicable guidance within its own deadline. Do not build an unbounded waiting room merely to avoid returning errors. HTTP 429 definition.

    Distributed quota enforcement needs an explicit consistency model. Independent instances enforcing a full per-tenant limit each can accidentally multiply the allowance. Shared counters, partitioned budgets, or deliberate approximate limits offer different latency and availability trade-offs. Gateway replicas and their policy/key distribution also need capacity and failure planning.

    Rate limiting at a gateway does not stop a volumetric attack from saturating a link before traffic reaches it. Layered network protection, controlled origin access, and application admission address different points. Logs and traces should reveal decisions while redacting tokens, private payloads, and unnecessary identifiers.

    Operating a gateway: self-hosted or managed

    A self-hosted deployment gives the team more direct control over configuration, rollout, and runtime, along with responsibility for upgrades, capacity, and recovery. A managed service moves some operations to a provider, while the team still owns route contracts, permissions, quotas, monitoring, and dependency behavior.

    Kong, Nginx, and Traefik are examples of software that teams may operate. AWS API Gateway, Apigee, and Azure API Management illustrate managed offerings. These are vocabulary examples, not rankings; editions, deployment modes, limits, and costs need checking against the actual requirements.

    Deadlines, retries, and idempotency close the failure loop

    Communication fails in several ways: no route, connection refusal, handshake failure, overload, partial response, or response loss after successful work. The application needs a policy for each relevant outcome. Most importantly, “I did not receive success” does not imply “the operation did not happen.”

    A timeout bounds how long a caller waits for a stage. An end-to-end deadline bounds the whole operation. A connection timeout, idle timeout, per-attempt timeout, and total deadline are different controls; name which one you mean. Propagate the remaining budget rather than giving every downstream call a fresh full budget.

    Suppose checkout has a 1,000 ms total budget. Gateway validation and queueing consume 100 ms, leaving 900 ms. Inventory takes 250 ms, leaving 650 ms. Reserve 100 ms for returning a response and bookkeeping, so a subsequent payment call receives at most 550 ms under this simplified serial allocation. Independent operations may run in parallel, but a payment depending on validated inventory cannot simply ignore that dependency.

    Deadline propagation must account for elapsed time and the protocol’s clock-handling rules. Cancellation can stop unnecessary work when supported, but a caller timing out does not prove a remote effect was undone. Record the operation’s state durably and provide a way to query its outcome. Dependency-specific pools and concurrency limits, often called bulkheads, can prevent one slow dependency from consuming every request slot. A circuit breaker temporarily stops eligible attempts to a failing dependency and probes recovery; it needs a defined failure signal and does not repair that dependency.

    Retry only when the operation and budget allow it

    Retry a transient failure only when another attempt is useful, the operation is safe to repeat under its contract, and the remaining time and load budget allow it. Invalid input, forbidden access, and permanent absence ordinarily require a corrected request or a final answer, not repeated identical attempts. An overload response can mean “send less work,” not “retry immediately.”

    Exponential backoff increases delay between attempts; jitter varies delays so many clients do not wake together. Bound attempts, elapsed time, and total retry traffic. Assign retry ownership deliberately. If a client, gateway, and service each permit 3 total attempts and each failed upper-level attempt triggers a full lower-level retry sequence, one logical operation can cause 3 × 3 × 3 = 27 terminal dependency attempts in the worst case. This is an assumption-specific upper bound, not the average rate of every system.

    Measure original operations and attempts separately. A superficially healthy request success rate can hide a retry storm consuming the spare capacity needed for recovery. Preserve request identity across retries so traces show one intent and its several attempts.

    Explore the mechanism

    A timeout leaves the outcome unknown

    Illustrative order operation. A stable idempotency key protects one intent only when the service durably and atomically records its outcome and rejects conflicting payloads.

    Request and uncertain response

    ClientIntent KPOST order
    ServiceCreate orderCheck key K
    Order storeCommit + durable outcomeNot committed
    ResponseMay be lostPending
    Separate nested attempt budget
    Client owns 3attempts to gatewayGateway owns 3per call to serviceService owns 3per call to dependency

    Up to 27 terminal dependency attempts; total attempts, not 27 created orders. Concurrency depends on timing.

    Step 1 of 5

    Send the operation

    The client sends a request. The safe example associates key K with this caller and intent, not with every future order.

    Read every step without motion

    Same intent / same key

    1. Send the operation. The client sends a request. The safe example associates key K with this caller and intent, not with every future order.
    2. Commit, then lose the response. The service commits the order, but the response is lost. A timeout does not undo that commit or prove that the order failed.
    3. Retry the uncertain operation. The same intent reuses K and the same payload. The durable deduplication record lets the service look up the existing result instead of creating another order.
    4. Return a result or reconcile. Return the recorded order result within the retention contract. Payload conflicts, in-flight concurrency, and expired keys require explicit handling.
    5. Bound the whole retry tree. Three client-owned attempts to the gateway, up to three gateway-owned attempts per call to the service, and up to three service-owned attempts per call to its dependency can produce 27 terminal dependency attempts. These are total attempts; concurrency depends on timing. Coordinate a deadline and budget; jitter does not make a mutation idempotent.

    Uncoordinated retry

    1. Send the operation. The client sends a request. The safe example associates key K with this caller and intent, not with every future order.
    2. Commit, then lose the response. The service commits the order, but the response is lost. A timeout does not undo that commit or prove that the order failed.
    3. Retry the uncertain operation. Without a coordinated durable deduplication boundary, the retry can execute a second effect even though the first response was never received.
    4. Return a result or reconcile. A duplicate effect needs reconciliation. Three total attempts at each of the client, gateway, and service layers can create 3 × 3 × 3 = 27 leaf attempts in the worst case.
    5. Bound the whole retry tree. Three client-owned attempts to the gateway, up to three gateway-owned attempts per call to the service, and up to three service-owned attempts per call to its dependency can produce 27 terminal dependency attempts. These are total attempts; concurrency depends on timing. Coordinate a deadline and budget; jitter does not make a mutation idempotent.
    Nested client, gateway, and service retry budgets can create up to 3, 9, and 27 outgoing attempts to the next layer; a lost order response can be recovered using the same scoped durable idempotency key.

    Retry budgets protect capacity; durable intent deduplication protects business effects. The nested counts are totals under the stated retry budgets, not concurrent request counts. Timed-out work may still be running. Neither control replaces the other.

    Idempotency: handle the unknown outcome

    An operation is idempotent when repeating it has the same intended effect as performing it once. It need not return identical bytes or status every time. Reading a product is normally repeatable; adding another order is not automatically so. HTTP method semantics guide clients, but an application must implement its advertised behavior correctly.

    For one order intent, an application-defined idempotency key can connect a retry to the first attempt:

    Key: <one stable key for this caller's order intent>
    Scope: authenticated caller / tenant + create-order operation
    Request fingerprint: the canonical order inputs
    Stored result: state and response of the accepted intent
    
    Same key + same inputs -> return or resume the recorded intent
    Same key + different inputs -> reject as a conflicting reuse
    New intentional order -> new key

    This is a contract sketch, not an executable deduplication algorithm. The server needs a durable record, uniqueness for concurrent duplicates, a retention policy covering expected retries, and coordination with the business effect. Writing the key after creating the order leaves a crash gap; checking the key without atomic uniqueness lets simultaneous duplicates through. Payment providers and other external effects may need their own stable operation identifiers and reconciliation.

    If the response is lost after order creation, the client retries with the same intent key or queries the recorded outcome. It must not invent a new key just because it received a timeout. Such a design controls one application effect within its defined scope; it does not grant universal “exactly once delivery” across every network and storage boundary.

    Backpressure: keep waiting work within capacity

    A service can be reachable and healthy while having more admitted work than it can finish on time. A queue hides that mismatch briefly; it does not remove it. If useful arrivals exceed sustainable completions, waiting work grows until memory, deadlines, or customers’ patience run out.

    Backpressure communicates limited capacity to upstream producers so they slow down, pause, or reduce work. Load shedding rejects some work when admitting it would harm the service’s useful throughput or latency. They are related controls with different actions. A rejection with a clear policy can be better than accepting work into a queue whose completion time already exceeds the caller’s deadline. Google’s overload discussion describes why balancing alone cannot prevent every overload.

    Explore the mechanism

    Longer residence time means more concurrent work

    Illustrative stable averages: Little's Law L = λ × W. This relation is not a maximum-capacity formula and should not be applied as a stable average to a growing unbounded queue.

    Arrival rate × residence time = in-system work

    λ180 requests/s180/s
    ×
    WSeconds in the system50 ms = 0.05 s
    =
    LMean concurrent requests180 × 0.05 = 9
    9 mean requests in the system

    Representative mean occupancy, not individual tracked requests or worker capacity.

    Step 1 of 4

    Start with a stable short residence time

    At λ = 180/s and W = 0.05 s, L = 9 requests on average. W includes the time spent inside the chosen system boundary.

    Read every step without motion
    1. Start with a stable short residence time. At λ = 180/s and W = 0.05 s, L = 9 requests on average. W includes the time spent inside the chosen system boundary.
    2. Increase the residence time. At the same arrival rate and a new stable W = 0.2 s, the mean work in the system becomes 36 requests.
    3. Observe the fourfold concurrency. Four times the residence time means four times the average in-system work at the same stable arrival rate. This does not measure available workers or throughput capacity.
    4. Check whether the system is stable. If arrival work persistently exceeds completion capacity, backlog grows. Use bounded queues, admission control, and measured service limits; there is no stationary mean to substitute casually.

    The explorer’s smaller scenario uses 180 requests/s: 50 ms of mean residence gives nine requests in flight, while 200 ms gives 36. The following larger calculation uses different illustrative values at the same kind of boundary.

    Little’s Law connects rate, time, and in-flight work

    For the stable averages used here, Little’s Law gives:

    L = λ × W
    
    L: average number of requests inside the chosen boundary
    λ: average admitted rate across that boundary, in requests per second
    W: average time spent inside that boundary, in seconds

    Assume the boundary is one API worker pool including its waiting queue. At 600 requests/s and 0.2 seconds average residence time:

    L = 600 × 0.2 = 120 requests

    If the pool remains stable at that rate but residence time rises to 1 second, average in-flight work becomes 600 requests. The arrival rate has not changed, yet five times as much work now occupies the boundary. This explains why a slower dependency can exhaust connections or memory without a new traffic spike.

    The assumptions matter. In this stable system, admitted and completed rates balance over time; count admitted requests rather than offered traffic rejected before the boundary. Use the same boundary for rate and time, convert milliseconds to seconds, and include waiting time when the boundary includes a queue. The equation relates long-run averages; it is not a safe maximum concurrency setting, a tail-latency estimate, or a description of an indefinitely growing unstable queue.

    A queue microscope

    Suppose a pool can complete 500 of these requests/s, but it admits 650/s for 10 seconds. Ignore changing service time and failures:

    Backlog growth per second = 650 - 500 = 150 requests
    Added backlog after 10 s = 150 × 10 = 1,500 requests
    Time to drain with no new work = 1,500 / 500 = 3 seconds

    If arrivals then fall only to 450/s, the pool has 50 requests/s of spare drain capacity. The same backlog takes roughly 1,500 / 50 = 30 seconds to clear. This is why a brief surge can leave a long recovery tail. The calculation assumes a uniform request cost and constant completion capacity; real workloads require measurement.

    Bound both how much may wait and how long it may wait. A queue of 1,000 requests is not inherently safe if their costs differ or their callers have 100 ms left. Reserve capacity for essential operations, isolate expensive routes, and stop work that no longer has a viable deadline where cancellation is supported.

    A practical admission policy has several parts:

    • Rate budget: limits new arrivals over time.
    • Concurrency budget: limits work currently using scarce resources.
    • Queue budget: limits waiting work and its age.
    • Retry budget: prevents recovery attempts from consuming all spare capacity.

    A transport’s flow control can slow a sender when buffers fill, but it cannot decide which business operation is valuable or make a database transaction cheaper. Application admission needs the domain’s priorities. Prefer an explicit overload response or a clearly authorized degraded read to silently performing a partial mutation.

    At checkout, it may be reasonable to omit an optional recommendation widget. It is not reasonable to “degrade” an order by skipping permission or payment validation. Capacity policy must preserve the operation’s correctness contract.

    Put the design together and debug its boundaries

    Now choose from the actual requirements. Suppose the store has globally distributed readers, public product images, authenticated checkout, occasional order-status events, and a requirement to survive one application-instance failure. A reasonable first design is:

    1. Stable DNS names resolve to the intended public ingress; both advertised IP families have working paths.
    2. TLS protects client-to-edge and required upstream hops with valid identity checks.
    3. A CDN caches versioned public assets using reviewed keys and freshness rules. Personalized responses bypass shared caching under explicit policy.
    4. Ingress applies controlled identity propagation and gateway admission. The application checks resource and tenant permissions.
    5. A load balancer routes to ready application workers. Durable state is shared appropriately, and tested headroom covers the chosen failure condition.
    6. Order creation records a caller-scoped idempotent intent and coordinates its durable business effects. A lost response can be reconciled.
    7. Receipt generation can run through a durable asynchronous workflow; SSE or polling exposes status according to the required freshness.
    8. Deadlines, bounded retries, concurrency limits, traces, and recovery procedures cross the relevant boundaries.

    This design is a starting hypothesis. Measure workload, payload sizes, geographic latency, peak rate, tail latency, acceptable staleness, and failure requirements before sizing it. Multi-region writes, multi-CDN steering, or a separate gateway are additional decisions, not mandatory badges of a serious system.

    Debug from the observed boundary

    When the image loads but checkout times out, start with evidence. A successful image might have come from browser cache or an edge, so it does not prove the origin is reachable. Follow the failing request’s own trace and distinguish name resolution, connection, protected handshake, intermediary response, and application outcome.

    SymptomFirst boundary to inspectUseful evidenceAvoid assuming
    Name cannot be resolved.DNS answer and resolver path.

    Requested type, answer/error, TTL, authoritative data, selected resolver.

    Every failure is a missing A record; negative caching and resolver failure differ.

    Connection is refused or times out.Listener, routes, and network policy.

    Target address/port, bind interface, reachability from the actual client network.

    A route proves a reachable application or a timeout proves its exact cause.

    TLS fails.The peer reached at that hop.

    Hostname, certificate chain, time validity, protocol negotiation, termination point.

    Disabling validation is a correct fix.
    Proxy returns an upstream error.Proxy-to-upstream connection and response.

    Selected backend, error reason, connect/read timing, matching upstream logs.

    All 502 errors mean the application crashed.
    Only some users see stale or wrong content.Cache selection and freshness.

    Key dimensions, cache status, age, variants, user/tenant separation.

    A cache hit is necessarily the correct representation.
    Latency spikes during a server failure.Health convergence and surviving capacity.

    Removal timing, active work, queue depth, attempt amplification, warmup.

    Remaining healthy targets have enough spare capacity.
    Checkout timed out after submission.Application intent and durable outcome.

    Idempotency key, request identity, committed order/payment state, reconciliation.

    No response means no order was created.

    Record timings at a named boundary. Browser total time, edge origin-fetch time, gateway queue time, and backend processing time measure different things. Correlate them with a request or trace identifier, and distinguish connection attempts from business operations. Protect logs from credentials and private payloads.

    In an interview or design review, use the same discipline: state the requirement, identify the bottleneck or failure boundary, explain the mechanism, verify a small example, and name the limitation. “Use a CDN” becomes “cache versioned public images near readers, keep authenticated checkout responses out of shared caches, and provision for origin misses.” “Add replicas” becomes “distribute eligible work and show that the survivors meet the required load.” The concrete claim can then be tested.

    The central question for every box is: what responsibility does it own, what evidence does it produce, and what happens when its message or response is lost? That question connects addressing, delivery, performance, and application correctness into one design.

    Interview revision: turn a component into a defended decision

    A useful design answer follows assumption → mechanism → numbers → failure → decision. Saying “add a load balancer” names a component. Explaining which workload it spreads, how the survivors meet the peak, and how failed requests recover defends the design.

    Use these short prompts to practise the reasoning, then return to the linked mechanisms when an answer feels vague.

    Explain why an image can load while checkout times out

    • Separate the paths: an image may finish in browser or edge cache; checkout needs an origin call.
    • Locate the failure: inspect destination selection, connection timing, intermediary results, and dependency work.
    • Keep the outcome open: a timeout can occur before delivery or after the order commits.
    • Recover the same intent: query or retry under the idempotency contract. Check origin admission and timing before changing a healthy image cache.

    Defend a scaling decision with a failure budget

    • State the target: request mix, peak rate, latency, and tolerated failures.
    • Show the capacity assumption: three instances at 100 requests/s each receive 180 requests/s in this small illustration.
    • Calculate survivors: after one failure, even distribution gives 90 requests/s per survivor.
    • Challenge the model: expensive requests or a shared database can invalidate the calculation.
    • Defend the decision: keep the pool only if representative failure testing meets the target; otherwise add headroom or admission limits.

    Decide whether a response belongs in a shared cache

    • Define eligible readers: a public image and a private account response have different reuse contracts.
    • Choose the key: a versioned image URL identifies content; account data requires controlled identity and tenant isolation.
    • Define the lifecycle: permitted storage, freshness, validation, and update behavior.
    • Test correctness: use two different users and a changed object, then assess performance. Review cache keys and freshness policy.

    Place the TLS and authorization boundaries

    • Place the TLS endpoint: name the component that terminates the connection and needs HTTP visibility.
    • Explain the trade-off: passthrough preserves encrypted forwarding; re-encryption protects two separate hops while the proxy sees plaintext.
    • Verify identity: identify which component authenticates the caller and how trusted identity reaches the service.
    • Check resource permission: token acceptance does not prove order ownership. The service still enforces business access rules.

    Show how a small failure can become an overload

    • Name the budgets: original operation rate, total attempts at each layer, remaining deadline, and downstream capacity.
    • Show amplification: three total attempts at each of three nested layers can create 27 terminal attempts in the stated worst case.
    • Explain occupancy: longer residence time holds more slots; admitted work above completion capacity grows a backlog.
    • Bound recovery work: choose retry ownership, safe operation semantics, queue limits, and admission controls. A larger queue alone does not restore capacity.

    If you can explain those five cases with stated assumptions and one checkable example, you have a working model of networking in system design. The next design review can change the workload or failure condition without forcing you to memorize a new list of boxes.

    Sources

    1. RFC 4632: Classless Inter-domain Routing
    2. RFC 1918: Address Allocation for Private Internets
    3. RFC 8200: Internet Protocol Version 6
    4. RFC 3022: Traditional IP Network Address Translator
    5. RFC 1034: Domain Names, Concepts and Facilities
    6. RFC 1035: Domain Names, Implementation and Specification
    7. RFC 2308: Negative Caching of DNS Queries
    8. RFC 9293: Transmission Control Protocol
    9. RFC 8446: Transport Layer Security Version 1.3
    10. RFC 9110: HTTP Semantics
    11. RFC 9111: HTTP Caching
    12. RFC 9113: HTTP/2
    13. RFC 9114: HTTP/3
    14. RFC 9000: QUIC
    15. RFC 6455: The WebSocket Protocol
    16. HTML Standard: Server-sent Events
    17. RFC 7239: Forwarded HTTP Extension
    18. gRPC deadlines
    19. Google SRE: Handling Overload
    20. Little’s Law as Viewed on Its 50th Anniversary