Latency, Throughput & Bandwidth
Basics
Visual Representation
🛣️ The Highway Analogy
Think of it like a highway connecting two cities.
Bandwidth = how many lanes the highway has. A 6-lane highway CAN carry more cars than a 2-lane road. It's the MAXIMUM capacity.
Throughput = how many cars ACTUALLY pass through per hour right now. Even a 6-lane highway might only see 100 cars/hour if there's construction ahead. It's the REAL performance.
Latency = how long it takes ONE car to drive from City A to City B. Even on an empty highway, driving 200 km takes time. It's the DELAY.
In tech language:
💡 Simple Summary: Bandwidth = size of the pipe (max capacity). Throughput = water actually flowing through right now. Latency = how long one drop takes to travel from one end to the other.
⏱️ Latency: The Delay You Can Feel
When you ping google.com and see "20ms" — that means it took 20 milliseconds (0.02 seconds) for a tiny packet to travel from your computer to Google's server and back.
That's latency. The raw delay of one round trip.
Why does latency exist? Two reasons:
Real numbers you should know:
💡 Simple Summary: Latency is the time ONE request takes to travel there and back. It's limited by the speed of light and physical distance. You can't make it zero — you can only reduce it by moving the server closer.
🌐 Why Latency Matters: The CDN Story
Imagine you're in Mumbai using an app whose server is in the US (Virginia). Every single request takes 200ms just to travel there and back — before the server even DOES anything.
Click a button → wait 200ms for the signal to reach the US → server processes for 50ms → wait 200ms for the response to come back. Total: 450ms for something that feels like it should be instant.
Now imagine the company puts a copy of their server in Mumbai. Same click → 5ms travel → 50ms processing → 5ms back. Total: 60ms. It feels INSTANT!
That's what a CDN (Content Delivery Network) does. Companies like Netflix and Instagram place servers in cities all over the world, so your data doesn't need to cross oceans. The server comes to YOU.
❓ But wait — can't I just buy faster internet to reduce latency?
No! This is the biggest misconception. Upgrading from 100 Mbps to 1 Gbps internet increases your BANDWIDTH (bigger pipe) but does NOTHING for latency. The speed of light doesn't care about your internet plan. A 1 KB request on a 100 Mbps connection transfers in 0.08ms — bandwidth isn't the bottleneck. The 200ms to reach the US server is pure distance-based latency.
💡 Simple Summary: Latency = distance problem. Fix it by moving the server closer (CDN), NOT by buying faster internet.
📊 Throughput: How Much Actually Flows
Your internet plan says 100 Mbps. But when you run a speed test, you get 60 Mbps. What happened to the other 40?
Welcome to the difference between bandwidth and throughput!
Bandwidth = the theoretical maximum (what you pay for)
Throughput = what you actually get (reality)
The gap happens because of:
Think of it like a water pipe: bandwidth is how wide the pipe is, throughput is how much water actually flows. If the pipe is clogged or the water pressure is low, you won't fill the pipe to capacity.
For system design, throughput means: how many requests per second (RPS) can your system handle?
💡 Simple Summary: Bandwidth = what the pipe CAN carry. Throughput = what it ACTUALLY carries right now. They're not the same! Plan for throughput, not bandwidth.
🔬 Bandwidth vs Throughput: The Pipe Analogy
Let's make this crystal clear with one example:
Scenario: You have a 100 Mbps internet connection (bandwidth). You're downloading a file from a server.
This is called the bandwidth-delay product. A bigger pipe doesn't help if the "water" takes too long to arrive. It's like having a 6-lane highway between Mumbai and New York — the lanes are mostly empty because cars take hours to arrive.
💡 Simple Summary: Bandwidth is the width of the pipe. Throughput is the water actually flowing. High latency can make a fat pipe practically useless. You need BOTH low latency AND high bandwidth for great throughput.
📈 P99 Latency: The Number That Actually Matters
Here's something interviewers love asking about.
Your API's AVERAGE latency is 100ms. Sounds fine, right? But let's look deeper:
That 1% sounds small. But at 1 million requests per day, that's 10,000 users having a TERRIBLE experience daily.
It gets worse. Imagine a single page makes 50 API calls to load. The chance that AT LEAST ONE of those 50 calls hits the slow P99 path: 1 - (0.99)^50 = 39%! Nearly 4 out of 10 page loads will feel slow.
This is called tail latency amplification — the more services you call, the more likely you hit a slow one.
❓ But wait — why don't we just use averages?
Averages hide the pain! If 99 requests take 10ms and 1 request takes 10,000ms, the average is 109ms — sounds fine! But that one user waited 10 SECONDS. Percentiles (P50, P95, P99) show you the real distribution.
💡 Simple Summary: P99 latency = the worst experience 1% of users have. At scale, that 1% is thousands of unhappy people. Always measure P99, not just averages.
🎯 Key Points for Interviews
❌ Common Mistakes in Interviews
Mistake 1: Confusing bandwidth with throughput — "Our bandwidth is 10,000 requests/sec." No! Bandwidth is maximum capacity. Throughput is actual performance. They're different.
Mistake 2: Using average latency — "Average response is 100ms." But if P99 is 5 seconds, 1% of users are suffering. Always discuss P50, P95, and P99.
Mistake 3: Thinking more bandwidth fixes latency — Upgrading from 100 Mbps to 1 Gbps won't make a 200ms round-trip faster. Latency is limited by speed of light, not bandwidth.
Mistake 4: Ignoring tail latency amplification — A page with 50 API calls: even if each call has P99 = 100ms, the PAGE-level P99 is much worse because one of those 50 calls is likely to be slow.
Mistake 5: Forgetting latency is additive in microservices — Each service call adds network + processing latency. A chain of 5 services at 30ms each = 150ms minimum. Use parallel calls when possible!
🎯 Interview One-Liner
"Latency is how long one request takes (reduce with CDNs, caching, and fewer hops), throughput is how many requests per second the system handles (increase with horizontal scaling), and I always use P99 latency because tail latencies determine real user experience at scale."
Interview Q&A
Q: How would you reduce latency in a system?
Multiple strategies layered together: (1) CDNs to move data physically closer to users. (2) Redis/Memcached to cache hot data in memory instead of slow database reads. (3) Parallel calls instead of sequential chains — call 5 services simultaneously instead of one after another. (4) Connection pooling to skip repeated TCP/TLS handshakes. (5) Pre-compute and pre-fetch data the user will likely need next. (6) Faster serialization (Protobuf over JSON).
Q: System handles 10K requests/sec but users say it's slow. What's happening?
High throughput but high latency. The system processes lots of requests but each one takes too long. I'd check P99 latency — likely some requests hit slow paths (cache miss, heavy database query, garbage collection pause). Solutions: add caching for hot data, optimize slow queries, add circuit breakers for slow downstream services, and add read replicas to reduce DB contention.
Q: Explain tail latency and why it matters at scale.
If P99 is 500ms, 1 in 100 requests is slow. At 100K requests/second, that's 1000 unhappy users PER SECOND. Worse: if a page calls 20 services in parallel, the chance at least ONE is slow = 1-(0.99^20) = 18%. So 18% of page loads feel slow to users. Google's approach: focus on P99.9, not averages. Tail latency amplification is why microservice architectures feel slow without careful optimization.
Q: Batching — when would you use it and what's the trade-off?
Batching = collecting multiple operations and executing them together. Instead of 100 individual DB inserts (100 round trips × 5ms = 500ms total), batch all 100 into one call (1 round trip × 20ms = 20ms). Trade-off: individual items wait in the batch buffer (slightly higher per-item latency) for much higher overall throughput. Use it when throughput matters more than instant response — like processing analytics events or bulk notifications.
Q: How do you calculate if your system can handle expected load?
Back-of-envelope math: 1M daily users, peak = 10x average. Average = 1M/86400 ≈ 12 req/sec. Peak = 120 req/sec. If one server handles 500 req/sec at acceptable latency, 1 server is enough (with 2-3 for redundancy). For 100M users? 12,000 req/sec peak → need 24+ servers. Add latency budget: page must load in 200ms → server processing must be under 100ms (leaving room for network travel).
Q: CDN reduces latency — explain how.
Instead of every request traveling from India to a US server (200ms round trip), the CDN stores a copy of the content on servers in India (Mumbai, Delhi). Now the same content is served from 5ms away! The first request goes to the origin (US), but all subsequent requests for the same content are served locally. For static content (images, JS, videos), this eliminates 95% of latency. Netflix places CDN servers literally inside ISP buildings — so the video barely travels at all.
Q: What's the bandwidth-delay product and why does it matter?
It's bandwidth × latency = how much data is "in flight" (in the network pipe) at any time. High latency means the pipe is mostly empty even if it's wide. Example: a 1 Gbps connection with 200ms latency has 25 MB of data in flight. TCP's window size must be at least this large to fill the pipe. If the window is too small, you'll never use your full bandwidth — the sender keeps waiting for acknowledgments. This is why high-latency links (satellites) struggle even with massive bandwidth.
Q: Why do companies measure P99.9 instead of P99?
At massive scale, even the 0.1% matters. Google handles 8.5 billion searches daily. P99.9 = 8.5 million slow searches per day! Users who consistently get P99.9 responses might have a systematically bad experience (bad cache locality, distant server, complex queries). Measuring P99.9 helps find and fix these edge cases that P99 misses.
Quick Quiz
1/5Your game shows "ping: 150ms." What does this mean?