Circuit breaker design pattern

Circuit breaker is a design pattern used in modern software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties.

Common uses

Assume that an application connects to a database 100 times per second and the database fails. The application designer does not want to have the same error reoccur constantly. They also want to handle the error quickly and gracefully without waiting for TCP connection timeout.

Generally Circuit Breaker can be used to check the availability of an external service. An external service can be a database server or a web service used by the application.

Circuit breaker detects failures and prevents the application from trying to perform the action that is doomed to fail (until it's safe to retry).

Implementation

Implementations of the Circuit Breaker Design Pattern need to retain the state of the connection over a series of requests. It must offload the logic to detect failures from the actual requests. Therefore, the state machine within the circuit breaker needs to operate in some sense concurrently with the requests passing through it. One way this can be achieved is asynchronously.

In a multi-node (clustered) server, the state of the upstream service will need to be reflected across all the nodes in the cluster. Therefore, implementations may need to use a persistent storage layer, e.g. a network cache such as Memcached or Redis, or local cache (disk or memory based) to record the availability of what is, to the application, an external service.

Circuit Breaker records the state of the external service on a given interval.

Before the external service is used from the application, the storage layer is queried to retrieve the current state.

Performance implication

While it's safe to say that the benefits outweigh the consequences, implementing Circuit Breaker will negatively affect the performance.

By how much depends on the storage layer used and generally available resources. The largest factors in this regard are the type of cache, for example, disk-based vs. memory-based and local vs. network.

Example implementation

PHP

The following is a sample implementation in PHP. The proof of concept stores the status of a MySQL server into a shared memory cache (APC).

Check

The following script could be run on a set interval through crontab.

$mysqli = new mysqli('localhost', 'user', 'pass');

if ($mysqli->connect_error) {
    apc_store('dbStatus', 'down');
} else {
    apc_store('dbStatus', 'up');
    $mysqli->close();
}

Usage in an application

if (apc_fetch('dbStatus') === 'down') {
    echo 'The database server is currently not available. Please try again in a minute.';
    exit;
}

$mysqli = new mysqli('localhost', 'user', 'pass', 'database');
$result = $mysqli->query('SELECT * FROM table');

External links

• Example of PHP implementation with diagrams
• Example of Retry Pattern with Polly using C#
• Example of C# implementation from Anders Lybeckers using Polly
• Polly NuGet package
• Example of C# implementation from Alexandr Nikitin
• Implementation in Python
• Implementation in PHP
• Stability patterns applied in a RESTful architecture