System Design Foundations: Architecture, Scaling & Fault Tolerance

Architectural Paradigms: Monoliths, SOA & Microservices

The architectural decomposition of an enterprise backend dictates how engineering teams build, deploy, scale, and maintain software over decades.

1. Monolithic Architecture

In a Monolith, all application components—user authentication, billing, product catalog, search indexing, and notification dispatch—reside within a single unified codebase, compile into a single binary or deployment archive, and interact via in-memory function calls and shared local heap objects over a single relational database.

• Advantages: Unmatched developer velocity for greenfield projects, simple end-to-end debugging, zero network serialization latency, and atomic ACID database transactions spanning business domains.
• Failure Modes: Tight coupling, ballooning blast radius (a memory leak in PDF generation crashes the entire store), deployment bottlenecks (a 1-line CSS fix requires re-deploying the 10GB monolith), and scaling inefficiency (scaling the CPU-bound search requires duplicating the entire application memory footprint).

2. Service-Oriented Architecture (SOA)

Popularized in the early 2000s, SOA decomposes enterprise applications into coarse-grained, shared business services communicating over a centralized communication bus termed an Enterprise Service Bus (ESB) using heavyweight protocols such as SOAP, WSDL, and XML schemas. While it achieved cross-department reusability, the ESB frequently became a massive single point of failure and architectural bottleneck.

3. Microservices Architecture

Microservices decouple an application into a collection of small, independently deployable, loosely coupled services organized around specific business domains (bounded contexts according to Domain-Driven Design). Each microservice owns its private datastore (polyglot persistence) and communicates exclusively via lightweight network protocols: HTTP/REST, gRPC over HTTP/2, or asynchronous message brokers.

Scaling Strategies: Scale-Up vs Scale-Out & Stateless Compute

System growth necessitates scaling hardware resources to handle exponential increases in concurrent transactions and data volumes:

• Vertical Scaling (Scale-Up): Adding more computational capacity to an individual machine (e.g., migrating an AWS EC2 instance from t4g.medium with 2 vCPUs to u-12tb1.metal with 448 vCPUs and 12 TB of RAM). While trivial to implement with zero code modifications, it suffers from a hard physical ceiling, exponential cost curves, and creates a massive single point of failure (hardware restart takes the whole service down).
• Horizontal Scaling (Scale-Out): Adding more commodity server nodes into an autoscaling pool (e.g., expanding from 3 pods to 50 pods behind a load balancer). Scale-out provides theoretical infinite scaling and extreme resilience, but requires the application layer to be completely stateless.

To achieve pure statelessness, application containers must treat their local disks and memory as strictly ephemeral:

1. Session Persistence: Offloaded to Redis clusters or encoded cryptographically into stateless JSON Web Tokens (JWTs) held by the client.
2. File Uploads: Dispatched directly to distributed object storage (AWS S3, Google Cloud Storage) via presigned URLs, never persisted to local container file systems.
3. Background Processing: Offloaded to distributed message brokers (RabbitMQ, SQS, BullMQ) rather than in-memory timer objects.

Traffic Distribution: Layer 4 vs Layer 7 Load Balancing

A load balancer acts as the single public entry point (Reverse Proxy) for distributed backend clusters, distributing incoming client traffic across healthy application instances to maximize throughput and eliminate downtime.

Layer 4 vs Layer 7 Load Balancing

• Layer 4 (Transport Layer - TCP/UDP): Operates strictly at the transport packet layer without decrypting or inspecting application contents (e.g., AWS Network Load Balancer, HAProxy TCP mode). The balancer reads the source IP, destination IP, and TCP port to direct packet streams. Capable of routing millions of packets per second with ultra-low latency, but lacks HTTP awareness.
• Layer 7 (Application Layer - HTTP/HTTPS): Terminates the TLS/SSL handshake and fully inspects the HTTP request payload: HTTP headers, cookies, URL paths, and query parameters (e.g., NGINX, AWS Application Load Balancer, Envoy). Enables intelligent path-based routing (e.g., /api/v1/auth → Auth Pods, /static/* → CDN), at the expense of higher CPU overhead for TLS decryption and HTTP parsing.

The API Gateway Pattern & Cross-Cutting Concerns

In a microservices ecosystem, clients (web applications, mobile apps, third-party partners) should never communicate with dozens of internal microservices directly. Exposing internal endpoints creates tight coupling, leaks internal network topology, forces clients to perform multiple HTTP roundtrips, and duplicates authentication logic across every service.

An API Gateway serves as the centralized, high-performance facade situated between external clients and internal microservices. It encapsulates all cross-cutting infrastructure concerns:

• SSL/TLS Termination: Decrypts HTTPS traffic at the perimeter; internal inter-service communication runs over optimized VPC networks or mTLS.
• Identity & Authentication: Validates JWTs, API keys, or OAuth tokens once at the gateway; forwards validated user context downstream via internal headers (e.g., X-User-Id: 4082).
• Rate Limiting & DDoS Defense: Drops malicious traffic spikes at the edge before internal services suffer CPU starvation.
• Request Routing & Aggregation (BFF Pattern): Accepts a single client request and fans out to three backend microservices in parallel, aggregating their responses into a single JSON response envelope.

import express, { Request, Response, NextFunction } from 'express';
import httpProxy from 'http-proxy';
import jwt from 'jsonwebtoken';

const app = express();
const proxy = httpProxy.createProxyServer({});

// Microservice Internal Service Discovery Map
const SERVICES: Record<string, string> = {
  auth: process.env.AUTH_SVC_URL || 'http://auth-service.internal:4001',
  catalog: process.env.CATALOG_SVC_URL || 'http://catalog-service.internal:4002',
  orders: process.env.ORDERS_SVC_URL || 'http://orders-service.internal:4003'
};

// Gateway Edge Middleware: Centralized JWT Authentication
function authenticateGateway(req: Request, res: Response, next: NextFunction) {
  if (req.path.startsWith('/api/v1/auth') || req.path === '/health') {
    return next(); // Public bypass
  }

  const authHeader = req.headers.authorization;
  if (!authHeader?.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'Missing or malformed Authorization bearer token' });
  }

  try {
    const token = authHeader.split(' ')[1];
    const decoded = jwt.verify(token, process.env.JWT_PUBLIC_KEY!, { algorithms: ['RS256'] }) as any;
    
    // Inject verified claims into downstream headers
    req.headers['x-user-id'] = decoded.sub;
    req.headers['x-user-roles'] = decoded.roles.join(',');
    next();
  } catch (err) {
    return res.status(401).json({ error: 'Invalid or expired token' });
  }
}

app.use(authenticateGateway);

// Route: Catalog Microservice
app.use('/api/v1/catalog', (req: Request, res: Response) => {
  proxy.web(req, res, { target: SERVICES.catalog, changeOrigin: true });
});

// Route: Orders Microservice
app.use('/api/v1/orders', (req: Request, res: Response) => {
  proxy.web(req, res, { target: SERVICES.orders, changeOrigin: true });
});

// Global Proxy Error Handling (Circuit Breaker / Service Unavailable)
proxy.on('error', (err, req, res) => {
  const response = res as Response;
  response.status(503).json({ error: 'Upstream Microservice Unavailable' });
});

app.listen(8080, () => console.log('[API Gateway] Running on port 8080'));

Database Scaling: Primary-Replica Topology & Read Replicas

In production web applications, database read workloads typically dwarf write workloads by a ratio of 10:1 to 100:1. Scaling database capacity begins by offloading read queries to an array of Read Replicas.

Architectural mechanisms to resolve Primary-Replica hazards:

• Asynchronous vs Synchronous Replication: Synchronous replication forces the Primary to wait until at least one replica writes the log to disk before confirming a commit. This guarantees zero replication lag, but elevates write latency to the slowest replica's network roundtrip. Production backends almost universally run asynchronous replication for maximum write throughput.
• Read-After-Write Consistency (Sticky Routing): When a user mutates data, the backend flags the user session in Redis with a timestamp (e.g., last_write:timestamp). For the next 2 seconds, all read queries from that specific user bypass replicas and are routed directly to the Primary database, ensuring the user immediately observes their own edits.
• Connection Pooling (PgBouncer): Relational databases like PostgreSQL allocate a dedicated operating system process per TCP connection (~10MB RAM each). Placing an external connection pooler between 200 autoscaled microservice pods and the database multiplexes thousands of incoming client requests across a pool of ~50 physical server connections.

Database Sharding & The Consistent Hashing Ring

When total data volume exceeds the storage capacity of a single physical disk, or write throughput exceeds what a single Primary node can absorb, the database must be sharded. Sharding (horizontal partitioning) splits a massive table across multiple independent physical database instances (shards).

Sharding Architectures

• Range-Based Sharding: Partitioning based on numeric or alphabetical boundaries (e.g., Shard 1: User IDs 1 - 1,000,000; Shard 2: User IDs 1,000,001 - 2,000,000). Highly vulnerable to write hotspots if IDs increment monotonically.
• Hash-Based Sharding (Modulo N): Routing via Shard = Hash(User_ID) % N. Distributes data uniformly, but causes catastrophic re-sharding overhead: adding an (N+1)th shard relocates almost 100% of all keys!
• Consistent Hashing: Maps both shard nodes and partition keys onto a continuous circular hash ring ($0$ to $2^{32}-1$). Adding or removing a node relocates only a minimal fraction ($K/N$) of keys to neighboring nodes.

The code below demonstrates a production TypeScript Consistent Hashing Ring utilizing MD5 cryptographic hashing and binary search (O(log N)) for key-to-node routing.

import crypto from 'crypto';

export class ConsistentHashRing {
  private readonly virtualNodesCount: number;
  private readonly ring: Map<number, string> = new Map();
  private sortedKeys: number[] = [];

  constructor(virtualNodesCount: number = 150) {
    this.virtualNodesCount = virtualNodesCount;
  }

  private hashKey(key: string): number {
    const hash = crypto.createHash('md5').update(key).digest();
    // Read first 4 bytes as unsigned 32-bit integer
    return hash.readUInt32BE(0);
  }

  public addNode(nodeAddress: string): void {
    for (let i = 0; i < this.virtualNodesCount; i++) {
      const vNodeKey = `${nodeAddress}#vnode_${i}`;
      const hashValue = this.hashKey(vNodeKey);
      this.ring.set(hashValue, nodeAddress);
      this.sortedKeys.push(hashValue);
    }
    this.sortedKeys.sort((a, b) => a - b);
  }

  public getNode(partitionKey: string): string {
    if (this.sortedKeys.length === 0) {
      throw new Error('Consistent hash ring is empty');
    }

    const keyHash = this.hashKey(partitionKey);

    // Binary Search: Find the first node hash >= keyHash (Clockwise walk)
    let low = 0;
    let high = this.sortedKeys.length - 1;

    if (keyHash > this.sortedKeys[high]) {
      // Wrap around ring to first node
      return this.ring.get(this.sortedKeys[0])!;
    }

    while (low <= high) {
      const mid = Math.floor((low + high) / 2);
      if (this.sortedKeys[mid] >= keyHash) {
        high = mid - 1;
      } else {
        low = mid + 1;
      }
    }

    return this.ring.get(this.sortedKeys[low])!;
  }
}

High Availability, Fault Tolerance & SLA Mathematics

High Availability (HA) is the characteristic of a system that aims to ensure an agreed level of operational performance (uptime) for a higher than normal period.

1. MTBF and MTTR Availability Formula

System availability is defined mathematically by Mean Time Between Failures (MTBF) and Mean Time To Recover (MTTR):

2. The "Nines" of Availability

3. Fault Tolerance & Circuit Breakers

Fault tolerance defines the ability of a system to continue operating without interruption despite the physical failure of one or more sub-components. Systems achieve fault tolerance via the Circuit Breaker Pattern:

• Closed State: Normal operations; requests pass through to the downstream dependency.
• Open State: Triggered when consecutive failures cross a threshold (e.g., 50% errors over 10 seconds). The circuit breaker trips immediately, failing fast and returning a cached fallback response without hitting the failing downstream service.
• Half-Open State: After a timeout window (e.g., 30 seconds), the circuit allows a canary probe request through. If it succeeds, the breaker resets to Closed; if it fails, it trips back to Open.

The CAP & PACELC Theorems in Distributed Systems

Formulated by Eric Brewer in 2000, the CAP Theorem proves that a distributed data store can guarantee at most two of the following three guarantees simultaneously:

1. Consistency (Linearizability): Every read receives the most recent write or an error. All nodes across the cluster return identical data at the identical instant.
2. Availability: Every non-failing node returns a successful response for every request—without guarantee that it contains the most recent write.
3. Partition Tolerance: The system continues to operate despite an arbitrary number of messages being dropped or delayed by the physical network between nodes.

The PACELC Theorem

Daniel Abadi extended CAP to address normal operations when no network partition exists:

Designing Scalable, Production-Grade APIs

Building APIs that sustain enterprise scale requires defensive engineering protocols that guard downstream services against cascade failures:

• Idempotency Guarantees: Enforce unique Idempotency-Key HTTP headers on all mutating POST requests. If a client network timeout triggers an automatic retry, the server identifies the stored key in Redis and returns the cached original result without double-charging the user.
• Exponential Backoff with Full Jitter: When clients retry failed network requests, simple exponential backoff (t = 2^attempt) causes synchronized request waves that swamp recovering servers. Full jitter randomizes the sleep interval: sleep = random_between(0, min(cap, base * 2^attempt)), breaking up request herd dynamics.
• Load Shedding & Priority Tiers: When CPU or memory saturation breaches 85%, servers prioritize critical transactional requests (e.g., checkout payments) and shed low-priority background traffic (e.g., profile analytics, telemetry) with an immediate HTTP 503 Service Unavailable.

University & Industrial Examination Review