Three metrics that define system performance. Understanding these is essential because every design decision trades off one for another. You'll learn what each really means, why they matter, and how they interact.
Latency is the time it takes for a request to travel from client to server and back. It's the delay users experience. Every millisecond counts—especially in competitive markets.
Network Latency
Time for data to travel through the internet. Physics limits this— speed of light is ~300,000 km/s. San Francisco to London: ~80ms minimum.
Processing Latency
Time the server takes to compute a response. Depends on database queries, algorithm complexity, I/O operations.
Queueing Latency
Time requests wait when the server is overloaded. The most unpredictable and often largest component under load.
Average latency can lie. If 99% of requests complete in 10ms but 1% take 2 seconds, your average might be 30ms—useless for understanding user experience.
p99 latency = the time at which 99% of requests complete. This is what users in the slowest 1% experience.
Throughput is the number of requests (or data units) a system can handle per unit time. Usually measured in requests-per-second (RPS) or transactions-per-second (TPS).
Your server can process 1 request every 10ms (0.01 seconds). With 4 cores, you can handle ~400 requests per second.
Throughput = Cores × (1 / Processing Time)
= 4 × (1 / 0.01s)
= 400 RPSUnder Load
Incoming requests: 100 RPS, Server capacity: 150 RPS → System keeps up, requests complete quickly
Overloaded
Incoming requests: 200 RPS, Server capacity: 150 RPS → Queue grows, latency spikes, requests timeout
Throughput is limited by bottlenecks. If your database can handle 1000 queries/sec but your server logic calls it 2x per request, you're capped at 500 RPS—no matter how many servers you add.
Availability is the percentage of time your system is operational and responding to requests. It's often expressed as "nines": 99% (two nines), 99.9% (three nines), etc.
Redundancy
Multiple servers, replicated databases. If one fails, others take over.
Health Checks
Detect failures fast. Route around broken servers before users notice.
Graceful Degradation
When something fails, still serve users (reduced features, cached data) rather than returning errors.
Monitoring & Alerts
Know instantly when problems occur. Act fast.
You can't maximize all three. Every architecture decision prioritizes some at the cost of others.
Low Latency + High Availability
Requires replication and caching everywhere. Expensive. Limits throughput.
High Throughput + Low Latency
Requires massive infrastructure. Hurts availability (complex = more failure points).
High Availability + High Throughput
Distributed systems introduce latency (coordination, consensus, replication lag).
Smart systems don't try to be perfect at all three. Instead, they prioritize based on their use case:
Adjust the controls below to see how latency, throughput, and availability interact in real time. Watch what happens when you overload the system, introduce failures, or scale horizontally.
System Design Rudiments — Interactive Simulation
Scenario
Controls
Live pipeline
Throughput vs Latency over time
Queue depth
0
Error rate
0.0%
Utilization
100%
Event log