How-To create exponential backoff timeouts

In distributed systems, it can be expected that some nodes either do not respond, or a response is delayed – so we try again. But trying again should not happen at regular intervals, or you could risk a “retry storm” in which all or a large subset of nodes decide to retry at the same time.

A popular tactic for avoiding this is first to wait at ever increasing intervals, and second to randomize the wait time. The function liberate::timeout::backoff() combines both.

Randomized exponential backoff

// Always have a minimum timeout of 300
uint16_t min_delay = 300;

for (size_t retries = 1 ; retries < MAX_RETRIES ; ++retries) {
  auto delay = liberate::timeout::exponential_backoff(1000, retries,
     min_delay);
  std::cout << "Delay for " << delay << " before trying again." << std::endl;
}

You can see that the outputs are both randomized, and increasing exponentially and all are above the minimum delay.