Building a reliable system at scale requires achieving High Availability (HA) and Fault Tolerance. This ensures that even when individual components fail, the system as a whole remains operational and accessible to users.
Adapted from Mastering System Design from Basics to Cracking Interviews (Udemy) — https://www.udemy.com/course/mastering-system-design-from-basics-to-cracking-interviews/
Redundancy Strategies
Redundancy is the practice of including extra components that are not strictly necessary for functionality but are crucial for reliability.
1. N+1 Redundancy
Providing one extra instance beyond what is required for the baseline capacity. If you need 2 nodes to handle traffic, you deploy 3.
- Benefit: Ensures availability during a single node failure.
2. Active-Active vs. Active-Passive
- Active-Active: Multiple nodes work together simultaneously, each handling a portion of the traffic.
- Best for: Load distribution and maximum throughput.
- Active-Passive: One node is active while others stay on standby (hot/warm/cold). The standby node only takes over if the primary fails.
- Best for: Systems where state synchronization is complex.
Active-active spreads traffic across two synced services, while active-passive sends traffic to a primary and keeps a heartbeat-linked standby in reserve.
Graceful Degradation
If a system cannot operate at 100% capacity due to a partial failure, it should attempt to stay online by disabling non-essential features.
- Example: During a database outage for comments, an e-commerce site might still allow users to browse products and place orders, while simply hiding the "Reviews" section.
- Goal: Maintain user experience even when full service is not possible.
HA Patterns in Real-World Systems
- Load Balancers: Distribute traffic evenly across healthy nodes. If a node fails health checks, the balancer stops sending traffic to it.
- Replication: Copying data across multiple nodes or even geographical regions.
- Automated Failover: The process of automatically switching to a backup node or service in case of failure without manual intervention.
Health Monitoring & Self-Healing
A resilient system must be able to detect its own failures and take corrective action.
- Health Monitoring:
- Tracking the status of system components (CPU, Memory, Disk, Service Uptime).
- Setting up Alerts for performance degradation or total failure.
- Self-Healing:
- Automatically repairing or replacing failed components.
- Example: In Kubernetes, if a container crashes, the system automatically restarts it or provisions a new one on a healthy node.
Self-healing runs as a loop: monitoring metrics detects a failure, the failed component is repaired or replaced, and monitoring resumes.
Designing for Redundancy
To protect against catastrophic failures (like an entire data center going offline), we use:
- Redundant Components: Multiple servers, databases, and network paths.
- Geographical Redundancy: Deploying across different cloud regions or continents.
- Automated Failover: Ensuring the switch to a Different region happens in seconds, not hours.
Interview Questions - High Availability & Fault Tolerance 💡
1. What is High Availability (HA) and why is it important in system design?
Answer: High Availability (HA) refers to the ability of a system to be continuously operational with minimal downtime. It is crucial because modern services must be reliable 24/7; downtime leads to revenue loss and reputational damage. Designing for HA means the system can withstand failures gracefully.
2. Describe the difference between active-active and active-passive redundancy.
Answer:
- Active-Active: Multiple instances run in parallel, sharing the workload. If one fails, others handle the traffic without interruption. Better for high-traffic and maximum fault tolerance.
- Active-Passive: One node handles all traffic while another stays idle. In case of failure, the passive node takes over. It's simpler to set up but has a brief recovery window during the switch.
3. What is fault tolerance, and how does it differ from high availability?
Answer:
- High Availability focuses on minimizing downtime (e.g., using failover).
- Fault Tolerance goes further, ensuring the system continues to operate during a failure with zero downtime and no service interruption (e.g., redundant hardware power supplies).
4. What is graceful degradation, and how does it improve UX?
Answer: It is the design principle of reducing functionality rather than failing completely.
- Example: A store might disable "Reviews" during a DB outage but still allow "Cart" and "Checkout," maintaining trust and allowing users to complete their core goals.
5. How do load balancers contribute to high availability?
Answer:
- Fault Tolerance: They automatically redirect traffic away from unhealthy servers.
- Scalability: They spread load evenly to prevent any single component from being a bottleneck or failing due to overload.
6. What is failover, and how does it help?
Answer: Failover is the automatic shift to a backup component when the primary fails. It minimizes downtime and improves resilience by ensuring continuity without manual intervention.
7. How do health monitoring and self-healing systems improve reliability?
Answer:
- Health Monitoring provides proactive detection of issues (CPU spikes, crashes).
- Self-Healing automatically takes action (restarting a container, provisioning a new node), reducing MTTR and maintaining availability without human intervention.
Summary: High Availability is about redundancy and automation. By implementing LB-driven failover, self-healing loops, and graceful degradation, we ensure the system survives both minor glitches and major outages.
Next up? Distributed reliability protocols — Paxos, Raft & Quorum