Load Balancing From One Server to a Hundred: Scaling Up, Scaling Out, and Keeping Microservices Alive

Every backend starts the same way. One server, one process, one database. It works beautifully right up until the day it doesn't — usually the day something you built gets shared somewhere popular and 40,000 people show up at once.

Then you learn the hard truth: your app didn't crash because your code was bad. It crashed because nothing in front of it was deciding who goes where.

That decision-maker is a load balancer. And once you have more than one server, or more than one service, the load balancer stops being "a thing you set up once" and becomes the most important piece of infrastructure you own.

This guide walks through the whole thing from scratch: how scaling actually works (up and out), how to build a load balancer for both cases with real configs, and then the harder half — how to stop one slow microservice from taking down every other service in your system.

No hand-waving. Real configs, real code, real failure modes.

---

Part 1: Scaling, Explained Without the Jargon

Before load balancers make sense, you need to be clear on what problem you're actually solving.

Vertical scaling (scaling up)

You take the server you already have and make it stronger. More CPU cores, more RAM, faster disks.

It's the equivalent of your delivery guy trading his scooter for a truck. Same guy, same route, just more capacity per trip.

Why people love it: you change nothing in your code. No distributed state, no session problems, no new networking. You resize the box and go back to sleep.

Where it stops working:

• There is a hardware ceiling. You cannot buy an infinitely large machine.
• Price scales worse than linearly. The jump from 32 GB to 64 GB costs more than the jump from 16 to 32.
• Resizing usually means a reboot, which means downtime.
• And the big one: one machine is one failure. That server dies, your product is offline. There is no "the others picked up the slack" because there are no others.

Horizontal scaling (scaling out)

You keep the same size servers and add more of them. Three medium boxes instead of one huge one.

Now you're not upgrading the scooter, you're hiring four more delivery guys.

Why it wins in production:

• Capacity is basically unbounded. Need more? Add another node.
• One node dying is an inconvenience, not an outage.
• You can add capacity during a spike and remove it after, so you're not paying for a monster machine at 3 AM when nobody's awake.
• Deploys can be rolling — update node 1, then node 2, then node 3, with zero downtime.

What it costs you:

• You now need a load balancer, which is itself a thing that can break.
• Your app must be stateless (huge — we'll cover this properly).
• Network hops between nodes add latency.
• Debugging goes from "check the log" to "check which of the nine logs".

The answer in real life: both

Almost nobody picks one. The pattern that actually ships is called diagonal scaling: scale the box up until you hit the sensible price/performance ceiling, then start cloning that box.

A practical rule I'd stand behind:

• Stateless things — your API servers, web servers, workers, anything that doesn't remember anything between requests — scale horizontally from day one. Running two small instances instead of one is barely more expensive and buys you real availability.
• Stateful things — your primary database, message brokers with ordering guarantees, caches that need strong consistency — scale vertically first. Coordinating state across many nodes gets expensive fast, and the complexity curve is brutal. Give Postgres a bigger machine before you give it friends.

That single distinction saves teams months of pain.

---

Part 2: What a Load Balancer Actually Does

Most people think a load balancer just splits traffic. That's maybe 30% of the job.

A load balancer sits between the internet and your servers, and here's the full list of what it's really doing:

1. Distributing requests across your healthy servers using some algorithm.
2. Health checking — constantly probing every backend and quietly removing the broken ones from rotation.
3. Terminating TLS — handling HTTPS once, at the edge, so your app servers can speak plain HTTP internally and not burn CPU on encryption.
4. Connection management — keeping warm connections open to backends so every request doesn't pay a new TCP handshake.
5. Draining — when you're deploying, it stops sending new requests to a server while letting its in-flight requests finish.
6. Protecting — rate limiting, request size limits, timeouts, and shedding traffic when everything is on fire.

Think of it as the receptionist at a busy clinic. She doesn't treat patients. She knows which doctors are in, which ones are free, which one is on a break, and she makes sure nobody walks into an empty room and waits forever.

---

Part 3: Layer 4 vs Layer 7 (and why you care)

You'll see load balancers described as L4 or L7. This just refers to how deep into the request they look.

Layer 4 — the transport layer

An L4 balancer sees IP addresses and ports. That's it. It doesn't open the envelope; it just reads the address and forwards the packets.

• Extremely fast, because there's nothing to parse. Kernel-level L4 balancers push enormous throughput with almost no overhead.
• Works for any TCP or UDP protocol — databases, Redis, game servers, custom binary protocols, gRPC streams.
• Can't do anything smart. It has no idea whether the request was /api/checkout or /health.

Layer 7 — the application layer

An L7 balancer parses the actual HTTP request. It sees the path, the headers, the cookies, the method.

That unlocks the things people actually want:

• Route /api/* to the API service and /images/* to the media service.
• Route based on headers — send users with x-beta: true to the new build.
• Retry a failed request on a different server, because it understands what a request is.
• Compress responses, cache them, rewrite paths, inject headers.
• Rate limit per user rather than per IP.

The tradeoff is CPU. Parsing HTTP costs more than forwarding packets. In practice, for 99% of web apps, that cost is irrelevant and L7 is what you want.

Simple rule: if it speaks HTTP, use L7. If it speaks anything else — Postgres, Redis, raw TCP — use L4.

---

Part 4: The Algorithms (this is where weighted routing ties everything together)

The algorithm decides which server gets the next request. Picking the wrong one is a classic source of "one server is at 95% CPU and the other two are asleep".

Round Robin

Requests go 1, 2, 3, 1, 2, 3. Dead simple, and completely fine when all your servers are identical and all your requests cost roughly the same.

Where it breaks: if request A takes 5 ms and request B takes 4 seconds, round robin will happily keep feeding a server that's already drowning in slow requests.

Weighted Round Robin

Same idea, but each server gets a weight. A server with weight 3 gets three times the traffic of a server with weight 1.

This is the algorithm that connects vertical and horizontal scaling. In the real world your fleet is rarely uniform — you upgraded two boxes last quarter and left one alone. Weights let a mixed fleet behave sanely: the 8-core machine takes proportionally more work than the 2-core one, instead of both getting an equal share and the small one falling over.

Least Connections

Send the request to whichever server currently has the fewest open connections.

This is the safest default for most real applications, because it self-corrects. A server stuck on slow requests naturally accumulates open connections, so it automatically stops receiving new ones until it catches up. If your response times vary at all — and they do — prefer this over round robin.

Least Response Time

Combines active connections with measured latency, and favours the server that's actually answering fastest. More accurate, slightly more overhead, and available in HAProxy, Envoy, and most cloud balancers.

IP Hash / Consistent Hashing

Hash something about the request (client IP, a header, a user ID) and always route the same hash to the same server.

This is how you get sticky behaviour without cookies. It genuinely matters for caches: if you're fronting a cache layer, you want user 12345 hitting the same cache node every time, otherwise your hit rate collapses.

Consistent hashing is the important variant. With naive hashing (hash % server_count), adding one server reshuffles nearly every key. Consistent hashing only remaps the small slice of keys near the new node, so a scale-up doesn't nuke your cache.

Power of Two Choices

Pick two servers at random, send the request to whichever has fewer connections.

Sounds lazy. Works remarkably well, and it's what a lot of modern proxies and meshes use by default, because tracking global state across many balancer instances is expensive and this approximation gets you most of the benefit for almost nothing.

---

Part 5: Building a Load Balancer for a Vertically Scaled Server

Here's the part most tutorials skip, and it's the one that matters most for people running a single VPS.

You need a load balancer even with one server.

Why? Because Node.js runs your JavaScript on a single thread. Python's default WSGI setup is similar. If you buy a 16-core machine and run one Node process, you're using one core. You paid for sixteen. Fifteen of them are watching.

So on a vertically scaled box, you run one app process per core and load balance across them locally. Same principle as horizontal scaling, just inside one machine.

Step 1: Run multiple app instances

Using PM2 in cluster mode:

Start it with pm2 start ecosystem.config.js.

What this does, line by line:

• instances: "max" tells PM2 to spawn one process per available CPU core. On a 16-core box, that's 16 workers.
• exec_mode: "cluster" is the important one. It uses Node's cluster module so all 16 workers share a single listening port, and the OS kernel distributes incoming connections between them. You get free load balancing at the kernel level.
• max_memory_restart is a safety net. If a worker leaks memory past 500 MB, PM2 kills and restarts it. Users never notice, because the other 15 keep serving.

That alone typically gives you a 5–10x throughput improvement on a multi-core machine, with zero code changes.

Step 2: Put Nginx in front

Kernel-level distribution works, but you still want a proper reverse proxy in front for TLS, compression, static files, timeouts, and rate limiting. If you want explicit control over distribution, run your app instances on separate ports (or Unix sockets) and let Nginx balance them.

Reading this config properly:

• upstream api_backend is the pool. Everything in that block describes your servers and how to pick between them.
• least_conn picks the instance with the fewest active connections — the self-correcting default we talked about.
• Unix sockets instead of TCP ports. Since all processes are on the same machine, going over localhost:3001 means a full TCP round trip through the network stack for no reason. Unix domain sockets skip that entirely and are measurably faster for local traffic.
• max_fails=3 fail_timeout=10s is passive health checking. If a worker fails three requests within 10 seconds, Nginx stops sending it traffic for 10 seconds, then tries again.
• keepalive 64 keeps 64 idle connections open to the backend pool. Without this, every single request opens and closes a connection, which is pure waste. This one line is often a double-digit percentage latency win.
• proxy_http_version 1.1 plus proxy_set_header Connection "" is mandatory for keepalive to actually work. Nginx defaults to HTTP/1.0 upstream, which doesn't support persistent connections. If you set keepalive without these two lines, it silently does nothing. Very common mistake.
• The X-Forwarded-* headers preserve the real client IP and protocol. Without them your app thinks every request came from 127.0.0.1, which breaks rate limiting, logging, and geolocation.
• proxy_connect_timeout 3s — if a worker won't even accept the connection in 3 seconds, it's dead. Don't wait 60.
• proxy_next_upstream tells Nginx to silently retry on a different worker if one returns a gateway error. proxy_next_upstream_tries 2 caps that at two attempts so a broken backend can't cause an infinite retry storm.

Step 3: Tune the OS

The default Linux limits assume you're running a desktop, not a load balancer.

worker_processes auto matches Nginx workers to CPU cores. worker_connections is how many simultaneous connections each worker handles. Multiply the two and that's your theoretical ceiling — but it means nothing if the OS file descriptor limit is still 1024, which is why worker_rlimit_nofile is there. If you've ever seen "too many open files" in a log at exactly the worst moment, this is why.

---

Part 6: Building a Load Balancer for Horizontally Scaled Servers

Now the servers are on different machines. The config is similar, but the failure modes are completely different — because now the network is involved, and the network lies.

Nginx across multiple machines

Note the weights. Nodes .10 and .11 are 8-core machines, .12 is an older 2-core box, so it takes a third of the traffic. That's the mixed fleet problem solved in one line. The backup server receives nothing until every primary is down — a cheap way to keep a warm spare.

HAProxy, when you want active health checks

Open-source Nginx only does passive health checks — it learns a server is broken by failing real user requests on it. HAProxy does active checks: it probes backends on a schedule, independently of user traffic, so a dead server is removed before any user hits it.

What each piece is doing:

• option httpchk GET /healthz with http-check expect status 200 is the active check. HAProxy hits that endpoint on every backend on a schedule.
• check inter 3s fall 3 rise 2 — probe every 3 seconds; mark the server down after 3 consecutive failures; mark it healthy again only after 2 consecutive successes. Requiring multiple successes stops a flapping server from being repeatedly added and removed.
• slowstart 30s is underrated. When a server comes back, HAProxy ramps its traffic up gradually over 30 seconds instead of instantly hitting it with a full share. A freshly booted JVM or a cold cache will fall over if you slam it at full load the millisecond it says "I'm ready".
• option redispatch with retries 2 means if a connection to a backend fails, HAProxy retries on a different backend rather than the same broken one.
• The stick-table block is a distributed rate limiter. It tracks request rate per source IP over a 10-second window and returns 429 to anyone exceeding 200 requests. This runs at the edge, so abusive traffic never reaches your app at all.

The load balancer is now your single point of failure

You just carefully removed the single point of failure from your app tier and accidentally created a new one at the front door. If that one Nginx box dies, everything is down, no matter how many healthy app servers you have.

Two standard fixes:

1. Use your cloud's managed balancer. AWS ALB, GCP Cloud Load Balancing, Azure Load Balancer. They're distributed and multi-zone by default, and you don't operate them. For most teams this is simply the correct answer — stop building infrastructure you don't need to own.

2. Run two balancers with a floating IP. If you're on bare metal or plain VPSes, run two Nginx/HAProxy nodes and use keepalived to share a virtual IP that lives on whichever node is alive.

Your DNS points at 10.0.1.100, not at either machine. The two nodes gossip over VRRP; if the master stops responding — or if check_haproxy finds the process gone — the backup grabs the IP within a couple of seconds. Failover is nearly invisible to users. The second node has state BACKUP and a lower priority, everything else identical.

The Kubernetes version

If you're on Kubernetes, most of this is built in and you describe intent instead of writing proxy configs.

The parts that actually matter:

• topologySpreadConstraints spreads your three replicas across availability zones. Three replicas on one physical node that dies is the same as one replica.
• resources.requests is what the scheduler uses to place pods, and what the autoscaler measures against. Without requests, autoscaling based on CPU is meaningless because there's no baseline.
• Note there's a memory limit but no CPU limit. CPU limits cause throttling — your container gets artificially paused even when the node has spare CPU, which shows up as mysterious latency spikes. Memory limits are different: memory isn't compressible, so a leak without a limit takes down the whole node.
• The preStop sleep of 10 seconds is the single most valuable line in this file. We'll get to why in the graceful shutdown section.

---

Part 7: The Prerequisite Nobody Warns You About — Statelessness

Here's the failure that catches everyone the first time they add a second server:

You add server 2. You test the site. You get logged out at random. You log in again. Fine for a minute. Logged out again.

What happened: your session was stored in the memory of server 1. The load balancer sent your next request to server 2, which has never heard of you.

Rule: any state stored in a server's local memory or local disk is a bug the moment you add a second server.

Fix each category:

Sessions — move them to Redis, or use signed stateless tokens.

Every server now reads sessions from the same Redis. Any server can serve any user. That's the whole point.

Uploaded files — never write to local disk. Push to S3, R2, or any object store. A file saved on node 2's disk simply doesn't exist for node 3, and it definitely doesn't survive a container restart.

In-memory caches — a local cache per node isn't wrong, but understand you now have N copies that can disagree. For anything that must be consistent, use shared Redis. For read-heavy immutable data, local is fine and faster.

Background jobs and cron — if you run a cron inside your app and scale to 5 replicas, that job now runs 5 times. Use a proper queue (BullMQ, SQS, RabbitMQ) with a single scheduler, or a distributed lock.

WebSockets — connections are inherently sticky to one node. Use a Redis pub/sub adapter so a message published on node 1 reaches sockets connected to node 3.

A word on sticky sessions

Most load balancers offer sticky sessions: pin a user to one server via a cookie, and the memory problem disappears.

It's tempting. It's also a trap.

• When that server dies, every user pinned to it loses their session anyway. You've just made the failure rarer and more confusing.
• Load becomes uneven — the server that happened to catch the heavy users stays hot.
• Autoscaling gets awkward, because you can't remove a node without dropping its users.
• You've built a distributed system that pretends not to be one.

Use stickiness when the protocol genuinely requires it (WebSockets, some legacy apps). Don't use it to avoid fixing session storage.

---

Part 8: Health Checks — Getting These Wrong Causes Outages

Your load balancer only knows a server is broken if the server tells it. Health checks are that conversation, and there are three distinct kinds that people constantly conflate.

Liveness: "are you alive?"

Is the process running and able to respond at all? If this fails, the right action is to restart the container.

Readiness: "are you ready for traffic?"

The process is up, but can it actually serve? Maybe it's still warming a cache, still connecting to the database, or currently overloaded. If this fails, the right action is to remove it from the load balancer — but not restart it. It might recover on its own.

Startup: "are you done booting?"

Some apps take 60 seconds to boot. Without a startup probe, the liveness check fails during boot, Kubernetes restarts the container, and you get an infinite crash loop of an app that was working fine.

Here's the implementation:

Why it's written this way:

• /healthz is deliberately dumb. It checks nothing except "am I running". This is critical — read the next section.
• /readyz checks the dependencies this instance genuinely needs to serve a request: can it reach its database, and is it keeping up?
• The database check is wrapped in a 1-second race. A health check that hangs is worse than one that fails, because the probe times out, then everything times out, and now you're diagnosing a health check instead of the actual problem.
• The event loop lag check is the self-defence bit. If the Node event loop is more than 200 ms behind, this instance is overloaded. Reporting "not ready" makes the load balancer route traffic elsewhere until it catches up. It sheds its own load instead of degrading for everyone.
• isShuttingDown connects to graceful shutdown, coming up next.

The mistake that turns a small problem into a total outage

Never check shared dependencies in a liveness probe.

Picture this: your liveness endpoint checks the database. The database has a 20-second hiccup. Every one of your 30 instances fails its liveness probe simultaneously. Kubernetes restarts all 30. Now the database is fine, but you have 30 cold instances all reconnecting at once, and you have zero capacity for two minutes.

The database blipped. You caused the outage.

Liveness = "is this process wedged?" Readiness = "should I get traffic right now?" Shared dependencies belong in readiness only, and even there, be careful — if every instance marks itself unready at once, you've taken the whole service out of rotation for a problem that only affected some requests.

---

Part 9: Graceful Shutdown — Where Your Mystery 502s Come From

Deploys shouldn't drop requests. They usually do, and here's the exact sequence of why.

When a server is being removed — a deploy, a scale-down, a node replacement — two things happen at nearly the same time:

1. The orchestrator sends SIGTERM to your process.
2. The orchestrator tells the load balancer to stop routing to it.

These are not instant, and they are not ordered. Load balancer config propagation takes a few seconds. Your process, meanwhile, exits in milliseconds. So there's a window where the balancer is still confidently sending requests to a process that already died. Every one of those becomes a 502 for a real user.

The fix has two parts.

Part one: delay before dying.

That preStop: sleep 10 in the Kubernetes spec earlier isn't padding. It tells the container: after being marked for termination, keep serving normally for 10 more seconds. That's the window the load balancer needs to notice and stop routing. On AWS, the equivalent is the target group deregistration delay.

Part two: drain properly.

Walking through it:

• isShuttingDown = true immediately flips the health endpoints to 503, which is the fastest way to tell the load balancer to back off. This is why that flag exists in the health check code.
• The 10-second wait is the drain window. During it, the app is still fully serving traffic — it just told the balancer it's leaving. Requests in flight complete normally.
• server.close() stops accepting new connections but lets existing ones finish. It's not a kill; it's "no new customers, we're closing".
• Only after the HTTP server is fully closed do we tear down database pools, Redis connections, and queue workers. Closing the DB pool first would break the requests you're currently trying to finish.
• The final setTimeout is the escape hatch. If something hangs — a stuck query, a socket that won't close — force exit after 30 seconds. .unref() means this timer alone won't keep the process alive.

Get these two pieces right and deploys become genuinely invisible to users. This one section fixes more real-world 502s than anything else in this guide.

---

Part 10: Autoscaling Without Making Things Worse

Autoscaling adds capacity when load rises. It also has a talent for turning a small incident into a large one.

The reasoning behind each number:

• minReplicas: 3, never 1. Two gives you redundancy; three lets you lose one and still have redundancy while the replacement boots.
• Target 65% CPU, not 90%. The gap is your headroom. Scaling isn't instant — pods take tens of seconds, nodes take minutes. If you only start scaling at 90%, you're already too late.
• Scale up fast, scale down slow. Look at the asymmetry: up allows doubling every 30 seconds, down allows removing 25% per minute after a 5-minute stabilisation window. Scaling up too slowly costs you an outage. Scaling down too quickly causes flapping — remove pods, load rises, add them back, repeat — and each cycle means cold starts and dropped connections.
• maxReplicas is a blast radius limit, not an ambition. Without a ceiling, a runaway retry loop can scale you to 400 pods and a five-figure bill before anyone notices.

Pick the right metric. CPU is the default and it's often wrong. If your service is I/O-bound — waiting on a database or a third-party API — CPU stays low while latency climbs and users suffer. Scale on requests-per-second, queue depth, or p95 latency instead. Queue depth in particular is the perfect signal for worker fleets: it's a direct measure of "how far behind are we".

And always pair the HPA with a Pod Disruption Budget so that voluntary operations like node upgrades can't take all your replicas at once:

---

Part 11: Making Microservices Stable

Everything so far keeps a single service healthy. This section is about stopping one sick service from killing the rest.

The defining failure of microservices is the cascading failure. Service D gets slow. Service C calls D and its threads pile up waiting. Service B calls C and starts timing out. Service A calls B, users retry, retries multiply, and within ninety seconds the entire platform is down because one non-critical service got slow.

Notice: D didn't crash. It got slow. Slowness spreads faster than crashes, because a crash gives you an instant error while slowness quietly consumes every resource you have.

Here are the patterns that stop it, roughly in the order you should adopt them.

1. Timeouts — the non-negotiable

Every network call gets a timeout. Every single one. A call without a timeout is a promise to wait forever, and forever is exactly how long a hung TCP connection takes.

The subtle part is the timeout budget. If your API gateway times out at 30 seconds and calls a service that times out at 60, the inner timeout is meaningless — the gateway gave up long ago and the inner service is still burning resources on a request nobody's waiting for.

Timeouts must shrink as you go deeper: gateway 10s → service A 5s → service B 2s → database 1s. Each layer needs enough time for its own work plus a retry, and less time than its caller allows.

2. Retries — useful, and the most common cause of self-inflicted outages

Retries fix transient failures. They also multiply load at the exact moment your system can least afford it. If three layers each retry three times, one user request becomes 27 requests to the bottom service. That's not resilience, that's a self-inflicted DDoS.

Three rules make retries safe:

Only retry idempotent operations. Reading is safe. Charging a card is not — unless you use idempotency keys, where the client sends a unique key and the server guarantees the same key never executes twice. This is exactly how Stripe's API handles it, and it's the right pattern for any write you might retry.

Always add jitter. Without it, every client that failed at the same moment retries at the same moment, producing a synchronised thundering herd that knocks the service back down the instant it recovers.

Cap the total, not just the count. Use a retry budget: allow retries only while they're a small fraction of total traffic (say 10%). If everything is failing, retries stop entirely — the system is down, and hammering it won't help.

What's going on:

• The delay grows exponentially — roughly 200 ms, 400, 800, 1600 — giving the downstream service progressively more room to recover.
• Math.random() * exponential is full jitter: instead of waiting exactly 800 ms, wait a random amount between 0 and 800. This spreads a thousand simultaneous clients across a window instead of stacking them on one instant. Full jitter consistently outperforms fixed backoff in practice, and it's one line.
• isRetryable is the guard rail. A 400 means your request was wrong; retrying it will be wrong again. A 500, a connection reset, or a 429 might succeed later. Retrying non-retryable errors is pure waste.

3. Circuit Breakers — stop calling the dead

If a service has failed its last twenty calls, the twenty-first will probably fail too. A circuit breaker notices this and fails instantly instead of waiting for another timeout.

It works exactly like the breaker in your house, with three states:

• Closed — normal. Requests flow, failures are counted.
• Open — the failure threshold was crossed. All calls fail immediately without touching the network. Your service stops wasting threads on a dependency that isn't answering, and the struggling service gets breathing room to recover instead of being hammered.
• Half-open — after a cooldown, let a small number of trial requests through. Succeed, and the breaker closes. Fail, and it opens again.

That half-open state is the clever bit: it's automatic recovery detection with a strictly limited blast radius.

Why these settings:

• volumeThreshold: 20 prevents a hair trigger. Without it, two failures out of three requests at 4 AM reads as 66% failure and opens the circuit on a sample size that means nothing. Require real volume before making a judgement.
• errorThresholdPercentage: 50 — open when half of a meaningful sample is failing.
• resetTimeout: 15000 — wait 15 seconds before probing recovery. Too short and you keep hammering a service that's trying to restart; too long and you stay degraded after it's healthy.
• breaker.fallback() is the part that turns an error into a non-event. Recommendations are dead, so serve cached popular items. The page still renders. The user sees slightly worse recommendations instead of a 500. Nobody files a ticket.

If you're on a service mesh you get this without touching application code:

outlierDetection is the circuit breaker: after 5 consecutive 5xx responses, that specific pod is ejected from the pool for 30 seconds. maxEjectionPercent: 50 is the safety valve — never eject more than half the pool, because ejecting everything means guaranteed total failure. The connectionPool section is a bulkhead, which is next.

4. Bulkheads — contain the damage

Named after ships. A ship's hull is divided into sealed compartments so a hole in one doesn't sink the whole vessel.

Applied to services: give each dependency its own isolated pool of resources — connections, threads, or concurrency slots. If the recommendations service hangs, it can only ever consume its own 20 slots. Your checkout path still has its slots free, and checkout keeps working.

Without bulkheads, one slow dependency drains the shared pool and every unrelated feature dies with it. This is the difference between "recommendations are down" and "the site is down".

The most common place this bites people is the database connection pool. If your pool has 20 connections and one slow report query holds 18 of them, your login endpoint can't get a connection and users can't sign in. Separate pools — or at minimum separate read replicas for heavy analytical queries — fix it.

Sizing rule of thumb, from Little's Law: pool size ≈ expected concurrent requests × average call duration. Measure, don't guess, and leave headroom.

5. Load Shedding — say no early

When you're beyond capacity, the kindest thing you can do is reject requests immediately.

It sounds backwards. It isn't. A request you accept but can't serve consumes memory, a connection, and a thread, then times out anyway — so you paid the full cost and delivered nothing. Worse, the client retries, so you pay again.

Rejecting fast with a 429 and a Retry-After header costs almost nothing and keeps the requests you did accept fast.

Reading it:

• The event loop delay histogram is the most honest overload signal Node gives you. High lag means the process cannot keep up, full stop — more reliable than CPU percentage.
• Above 400 ms lag, non-critical requests get a fast 503 with Retry-After. Critical paths — checkout, auth — are exempt, so you shed the traffic you can afford to lose and protect the traffic that makes money. That prioritisation is the whole point.
• The rate limiter keys on user ID where possible, falling back to IP. Keying only on IP punishes everyone behind a corporate NAT or mobile carrier gateway while doing nothing to stop a distributed abuser.

6. Asynchronous Boundaries — stop calling, start queueing

The strongest form of decoupling: don't make the call at all.

If service A needs service B to do something but doesn't need the answer right now, put a message on a queue. Now B being down doesn't affect A — messages just accumulate and get processed when B returns.

Order placement is the classic example. The synchronous path should be: validate, take payment, write the order, return success. Everything else — confirmation email, warehouse notification, analytics event, loyalty points, recommendation model update — goes on a queue.

If the email service is down, orders still complete. Emails go out late. Nobody notices. Compare that with a synchronous chain where a dead email service means customers can't buy anything.

The queue also acts as a shock absorber. A traffic spike fills the queue rather than crushing your workers, and the queue drains at whatever rate your workers can sustain.

7. Observability — you can't stabilise what you can't see

Three things, minimum:

RED metrics per service — Rate (requests/sec), Errors (failure %), Duration (latency distribution). Track p50, p95, and p99 latency. Averages lie: a service with a 100 ms average might have a p99 of 8 seconds, meaning 1 in 100 users is having a terrible time and your dashboard looks fine.

Distributed tracing — a request ID that flows through every service so you can see the full path of one request. Without it, "checkout is slow" means opening nine dashboards and guessing. With it, you see the 4-second gap and exactly which hop owns it.

SLOs with error budgets — decide the target ("99.9% of requests under 300 ms"), measure it, and treat the remaining 0.1% as a budget. Budget intact? Ship features. Budget burning? Stop and fix reliability. It turns "should we prioritise stability this sprint" from an argument into a number.

8. Test the failures on purpose

Every pattern above is a theory until you've watched it work.

Kill a pod during business hours. Add 500 ms of latency to a dependency. Return 500s from a service on purpose. Fill a disk. Do it in staging first, then, when you trust your tooling, in production during working hours with everyone watching.

You will find things. Timeouts that were never configured. A circuit breaker with a fallback that itself calls the broken service. A "non-critical" dependency that turns out to be on the login path. Far better to find them at 11 AM on a Tuesday with the team online than at 2 AM on a Saturday.

---

Part 12: Do You Need a Service Mesh?

A service mesh puts a proxy next to every service and handles retries, timeouts, circuit breaking, mTLS, and tracing at the network layer instead of in your code.

Genuinely great when: you have dozens of services across several languages, you need encryption between all services for compliance, and you want consistent policy without asking every team to implement it correctly in their own framework.

Not worth it when: you have five services. The operational overhead of the mesh will exceed the problem it solves, and you'll spend your week debugging sidecar configuration instead of shipping.

The honest rule: adopt a mesh when implementing resilience in each service separately has become more painful than running the mesh. For most teams that's somewhere north of 20 services. Before that, libraries and good defaults in a shared internal package do the job fine.

---

Part 13: Real-World Walkthrough 1 — A Node API That Survives a Traffic Spike

The setup: a Node.js API on a single 8-core VPS, one process, Nginx already in front. It handles about 200 requests/second. Then a launch post goes viral and traffic hits 3,000 requests/second. The site dies.

What actually broke, in order:

1. One Node process = one core busy, seven idle.
2. That process's event loop backed up, so every response got slower.
3. Slow responses meant connections stayed open longer, so Nginx hit its worker connection limit.
4. Users got timeouts, hit refresh, and doubled the incoming traffic.
5. The database connection pool (default: 10) was fully saturated by the slow requests still in flight.

The fix, in the order it was applied:

Step 1 — use all the cores. Switch to PM2 cluster mode with 8 instances. Throughput goes from ~200 to ~1,400 requests/second immediately. Zero code changes. This is vertical scaling done correctly: the hardware was already paid for and mostly unused.

Step 2 — fix the connection reuse. Add keepalive 128, proxy_http_version 1.1, and proxy_set_header Connection "" to the Nginx config. p95 latency drops meaningfully because requests stop paying for a fresh TCP handshake each time.

Step 3 — size the database pool properly. Raise the pool from 10 to 40 (8 processes × 5 connections), and confirm Postgres max_connections can actually take it. Add a 2-second statement timeout so one bad query can't hold a connection hostage.

Step 4 — add rate limiting at the edge. Nginx limit_req at 30 requests/second per IP with a burst allowance. Bots and refresh-spammers get 429s at the front door and never reach the app.

Step 5 — go horizontal. One box still can't do 3,000 rps sustainably, and it's still a single point of failure. Clone the VPS twice, move sessions to Redis, put a managed load balancer in front with least_conn, health checks on /readyz, and a 15-second deregistration delay.

Result: three 8-core boxes, 24 processes, comfortably handling 3,500 rps with headroom. Any single box can die without users noticing. Deploys are rolling and invisible.

The lesson: they nearly bought a 64-core server on day one. The actual problem was that they were using one core out of eight. Fix utilisation before you buy hardware, and fix statelessness before you scale out.

---

Part 14: Real-World Walkthrough 2 — Microservices in a Flash Sale

The setup: an e-commerce platform on Kubernetes. Services: gateway, auth, catalog, cart, pricing, inventory, payments, recommendations, notifications. A flash sale starts at 12:00 PM.

What happens at 12:00:01:

Traffic goes up 40x in seconds. The pricing service — which calls a third-party currency API — starts taking 8 seconds per call because that third party is now also getting hammered.

Without the patterns, here's the cascade:

• Catalog calls pricing and waits 8 seconds per product.
• Catalog's threads all end up parked on pricing calls.
• The gateway calls catalog and times out.
• Users see errors and refresh, tripling traffic.
• Auth shares a database connection pool with catalog and starts starving.
• Users can't log in. The entire site is down.

One slow third-party currency API took down authentication. That's a cascading failure.

With the patterns applied, here's what happens instead:

Timeout budget: gateway allows 5s, catalog allows 2s for pricing, pricing allows 800 ms for the currency API. When the third party takes 8 seconds, pricing gives up at 800 ms — long before anyone upstream is even worried.

Circuit breaker on the currency API: after 5 consecutive failures the circuit opens. Pricing stops calling the third party entirely and serves exchange rates from a Redis cache that's refreshed every 5 minutes. Rates are slightly stale. In a 90-minute sale, nobody cares.

Bulkhead: the pricing client in the catalog service has its own 20-slot concurrency pool. Even at its worst, pricing can only ever consume those 20 slots. Catalog's product-listing path is untouched.

Separate connection pools: auth has its own database pool. Whatever happens to catalog, login keeps working. This alone converts a total outage into a partial degradation.

Graceful degradation: recommendations time out at 300 ms and fall back to a cached "trending now" list. Product pages render fully. The recommendations are generic. No user notices, and if they do, they don't care — they're here for the sale price.

Load shedding: the gateway sheds non-critical traffic first. /api/recommendations and /api/reviews start returning 503 with Retry-After. /api/checkout and /api/auth are never shed. Revenue paths stay fast while nice-to-haves take the hit.

Async writes: order confirmation emails, loyalty points, and analytics events all go to a queue. The queue backs up to 200,000 messages during the peak and drains over the next 20 minutes. Customers get their confirmation emails a few minutes late. Orders complete instantly.

Autoscaling: HPA scales catalog and cart from 5 to 25 pods over about 90 seconds. Because the target was 65% CPU rather than 90%, scaling began before anything actually saturated.

The outcome: the sale runs. Checkout stays under 400 ms p95. Recommendations are degraded for 12 minutes. Emails are 4 minutes late. Zero orders lost.

The lesson: stability isn't about preventing failure. Something will always be slow. Stability is deciding in advance which parts are allowed to fail and making sure they fail in a small, contained, boring way.

---

The Mistakes I See Most Often

• No timeouts on outbound calls. The single most common cause of cascading failure. Nothing else on this list matters if you skip this.
• Retries without jitter. Turns a brief blip into a synchronised stampede that keeps re-killing a recovering service.
• Liveness probes that check the database. One database hiccup restarts your entire fleet.
• No preStop delay or drain window. Every deploy quietly drops requests, and you blame the network.
• Sticky sessions used as a substitute for shared session storage. Works until the server dies, then fails in a confusing way.
• CPU limits on containers. Causes throttling and phantom latency spikes. Set memory limits, set CPU requests, skip CPU limits.
• Autoscaling on CPU for I/O-bound services. CPU stays at 20% while latency triples and nothing scales.
• Missing keepalive on the proxy. Silently adds a TCP handshake to every request.
• One shared database connection pool for everything. One slow report query starves your login endpoint.
• Never testing failure. Every pattern above is a hypothesis until you've broken something on purpose and watched it hold.

---

The Checklist

Before you call an architecture production-ready:

Scaling

• App servers are stateless — no local sessions, no local file writes
• Sessions in Redis or stateless tokens
• Uploads go to object storage
• Cron and scheduled jobs run once, not once per replica
• Minimum 2 replicas, ideally 3, spread across zones

Load balancer

• L7 for HTTP, L4 for everything else
• least_conn or least-response-time unless you have a reason otherwise
• Weights reflect actual machine sizes in a mixed fleet
• Active health checks against a real readiness endpoint
• Keepalive to backends configured and the HTTP/1.1 headers set
• Balancer itself is redundant — managed service or two nodes with a floating IP
• TLS terminated at the edge, X-Forwarded-* headers passed through

Health and lifecycle

• Separate liveness, readiness, and startup endpoints
• Liveness checks nothing shared
• Readiness checks real dependencies with short internal timeouts
• SIGTERM handled, drain window before exit
• preStop delay or deregistration delay configured

Stability

• Every outbound call has a timeout, and timeouts shrink with depth
• Retries only on idempotent calls, with full jitter and a cap
• Circuit breakers on every external and cross-service dependency
• Fallbacks defined for anything non-critical
• Bulkheads separating dependency pools, including database pools
• Rate limiting at the edge, keyed on user where possible
• Load shedding that protects revenue paths first
• Non-urgent work moved to queues

Visibility

• RED metrics per service, p95 and p99 tracked, not averages
• Distributed tracing with a request ID across all hops
• Alerts on symptoms users feel, not on CPU
• Failure drills actually run

---

Wrapping Up

Load balancing sounds like a networking topic. It isn't. It's the decision layer that makes everything above it survivable.

Scale up first because it's easy and your hardware is probably underused. Scale out when you need availability, and accept the price — statelessness, health checks, drain windows. And once you're running more than one service, understand that your job shifts from preventing failure to containing it.

Failure is not the exception in distributed systems. It's the steady state. Something is always slow, restarting, or briefly unreachable. The systems that feel reliable aren't the ones where nothing breaks — they're the ones where things break constantly and nobody outside the engineering team ever finds out.

Start with timeouts. Add graceful shutdown. Then circuit breakers and fallbacks. You'll get 80% of the stability from those three, and they take an afternoon each.

Then go break something on purpose and see if it holds.