Building Resilient Systems: The Circuit Breaker Pattern
In a distributed microservices environment, failure is not an exception; it is the rule. Network glitches happen, databases overload, and dependencies go down.
If Service A calls Service B, and Service B hangs, Service A will eventually run out of threads waiting for B, crashing itself. This causes a Cascading Failure, taking down your entire platform.
Enter the Circuit Breaker
The Circuit Breaker pattern prevents an application from performing an operation that is likely to fail. It wraps a protected function call and monitors for failures.
The 3 States
- CLOSED (Normal): Requests pass through. We count errors. If the error rate exceeds a threshold (e.g., 50%), the breaker TRIPS.
- OPEN (Failing): The breaker prevents requests from executing and immediately returns a fallback error. This gives the failing service time to recover.
- HALF-OPEN (Testing): After a timeout, we let a *few* requests through. If they succeed, we reset to CLOSED. If they fail, we go back to OPEN.
Implementation in Node.js
We can use libraries like opossum.
const CircuitBreaker = require('opossum');async function riskyServiceCall() { // Call unreliable API return axios.get('http://unreliable-api.com'); }
const options = { timeout: 3000, // If request takes > 3s, fail errorThresholdPercentage: 50, // Trip if 50% fail resetTimeout: 10000 // Wait 10s before trying again };
const breaker = new CircuitBreaker(riskyServiceCall, options);
breaker.fallback(() => ({ body: "Service unavailable, try later." }));
// Use it
const response = await breaker.fire();
`
Use Cases
- External APIs: Payment gateways, Maps API, etc.
- Database Queries: Protecting against DB overload.
- Microservice Inter-communication.
Best Practices
- Fail Fast: Don't wait 30 seconds for a timeout. If the breaker is open, fail instantly.
- Fallbacks: Always provide a degraded experience. If the "Recommendations" service is down, show "Popular Items" from cache instead of a 500 error.
- Monitoring: Log every state change. Knowing when breakers trip gives you early warning of system instability.
Adding circuit breakers makes your system strictly more robust and is a hallmark of senior engineering architecture.