A dependency does not have to disappear to break your service. It can accept connections and answer slowly enough to occupy every worker you have. It can perform a write and lose the response. It can recover just as thousands of callers decide to try again.
“Retry on failure” is a remarkably incomplete design for those situations. Before an outage, decide what your application owes the caller when a dependency becomes slow, unavailable, or uncertain. That decision belongs beside the integration code, with an owner and an executable test.
The useful deliverable is a small dependency policy: how long you will wait, which operations can be retried, what the caller sees when attempts stop, and how normal traffic resumes. Different operations against the same service may need different answers.
Start with the consequence of losing the dependency
Take an imaginary storefront. Product recommendations, inventory reservations, and payment authorization are all remote calls. Losing each one creates a different product problem.
Recommendations might be optional. Omitting them can preserve a useful page. Inventory and payment affect whether the business can honestly say that an order exists. Returning an optimistic success there can create a reconciliation problem larger than the original outage.
For every operation, write down:
- The customer-visible capability that depends on it.
- Whether the operation reads state, changes state, or does both.
- Whether old data is acceptable, and for how long.
- What outcome is safe when the remote result is unknown.
- Who owns the policy and the dependency relationship.
Avoid labeling a whole service “noncritical” because one endpoint is optional. A recommendation endpoint and an entitlement endpoint can share infrastructure without sharing acceptable failure behavior. In particular, an authorization failure should not quietly become permission to proceed.
Give the entire request a budget
A per-call timeout is useful only in the context of the parent request. Three individually reasonable timeouts can add up to a response the caller has already abandoned.
The AWS Builders’ Library discussion of timeouts and retries recommends bounding remote waits and warns that timeout implementations can exclude work such as name resolution or TLS setup. It also explains why retries need restraint: they can add load to an already struggling dependency, and a lost response does not prove that a side effect never occurred.
Here is an illustrative budget for a recommendation lookup inside a page request. These numbers demonstrate accounting; they are not suggested defaults for your service.
| Work | Maximum planned time |
|---|---|
| Local admission, parsing, and preparation | 100 ms |
| Recommendation attempt one | 250 ms |
| Randomized delay before an eligible retry | 0–100 ms |
| Recommendation attempt two | 250 ms |
| Fallback selection and response construction | 100 ms |
| Scheduling and other overhead allowance | 200 ms |
| Overall request deadline | 1,000 ms |
The upper bounds total 1,000 ms. The dependency gets at most 600 ms including the delay, and no more than two attempts total. The parent deadline still wins if earlier work consumes more than its allowance.
Before starting another attempt, calculate the remaining time using the runtime's appropriate elapsed-time clock. Preserve enough time for the terminal response. Do not begin a 250 ms attempt with 80 ms left and hope the outer timeout will tidy everything up.
A connection timeout is a limit within an attempt, not an extra allowance after it. Similarly, a read timeout may mean an interval between received chunks rather than a total response deadline. Check the actual client contract, including pool acquisition, DNS, TLS, response-body consumption, and cancellation. A slow stream can defeat an inactivity timer while making almost no useful progress. For a concrete example, HTTPX documents separate connection, read, write, and pool timeouts; its read timeout bounds the wait for a chunk, not the duration of the entire response.
Assign retries to one responsible layer
Inspect the complete call path before adding a retry loop. Your application, SDK, proxy, and service mesh may already retry. Two attempts at each of three nested layers can produce up to eight downstream attempts for one original call. That arithmetic is a design review worth having before an incident.
Choose the layer with enough context to distinguish a safe repeat from a dangerous one. Configure the other layers consistently. Record attempts as total attempts rather than an ambiguous “retry count,” and test what actually reaches the dependency.
A useful eligibility check asks whether all of these conditions hold:
- The operation can safely be repeated under its documented contract.
- The failure is one for which another attempt could help.
- The parent deadline leaves room for the delay, attempt, and terminal response.
- The caller's retry allowance and dependency load controls permit it.
Randomize a bounded backoff interval so simultaneous failures do not schedule simultaneous repeats. For a longer-lived background operation, exponential backoff may make sense; a short request might have room for only one randomized delay. Neither should continue indefinitely. Treat any server retry instruction as an input constrained by the remaining deadline, not permission to sleep beyond it.
Do not classify every HTTP error as transient. An invalid request needs correction. A permission error needs a permission decision. Retrying either unchanged is usually just a more expensive way to obtain the same answer.
Treat a timed-out write as an unknown outcome
Suppose an inventory reservation request times out. The inventory service may have committed the reservation before the response was lost. Repeating the operation with a new identifier can reserve twice; pretending it failed can leave an orphaned reservation.
If the service offers an idempotency contract, understand its scope, retention period, and behavior when the same key arrives with different input. Reuse the identity of the same logical operation across its attempts. A locally generated key alone provides no protection unless the receiver enforces the contract.
Define how you resolve uncertainty: query operation status, reconcile an existing reservation, or expose a pending state with a supported follow-up path. If none exists, the integration may not support automatic retry safely. That is a capability gap to address, not a reason to hide the exception.
This is different from retrying a failed CI job. Runtime retries can repeat business effects while the customer is still waiting. Verification jobs have their own authority, evidence, and cost boundaries.
Make fallback semantics explicit
A fallback must preserve an acceptable meaning, not merely return a successful status code. Consider the policies below as discussion examples for the imaginary storefront.
| Operation | Failure policy |
|---|---|
| Recommendation read | At most two eligible attempts within budget; omit the optional panel when attempts stop. Avoid simultaneous cache refreshes on recovery. |
| Product description read | One bounded attempt; use an approved, age-limited cached version or report unavailable. Invalidate incompatible or withdrawn content. |
| Inventory reservation | Repeat only under the receiver's idempotency contract. Return confirmed, pending, or unavailable according to known state; reconcile uncertain outcomes. |
| Permission check | One bounded attempt under the access policy. Deny or report unavailable on uncertainty; restore checks without bypassing authorization. |
For a cached fallback, specify the maximum acceptable age, tenant or user boundaries, and invalidation rules. An old description is different from an old price or entitlement. “Serve stale” is not a universal product requirement.
A secondary provider also needs capacity, compatible semantics, and a tested switch. If it depends on the same failing network or identity service, the fallback may share the original failure. Draw that dependency before congratulating yourself on redundancy.
Recover without reopening the floodgates
Write down what signals allow normal traffic to return. If you use a circuit breaker, test its open, probe, and closed behavior, including how many probes multiple application instances can collectively send. A healthy probe says little about whether the dependency can handle the whole fleet.
A recovery policy might gradually increase admitted traffic while watching successful completions, latency, and saturation. That is an engineering choice to validate with the dependency owner, not a magic percentage to copy. Keep backlogs bounded and decide what happens to expired work instead of replaying everything accumulated during the outage.
Observe original requests separately from attempts. Record retry volume, exhausted budgets, fallback use, ambiguous writes, and recovery transitions. A flat error rate can conceal a rising number of customers receiving degraded results. A retry-success metric can look excellent while additional attempts consume all remaining capacity.
Prove the policy before relying on it
Use a controlled test environment with representative concurrency and explicit stop conditions. Include more than a connection refusal:
- A connection that succeeds but never returns a complete response.
- Slow DNS or connection setup, and exhaustion of the client pool.
- A response body that arrives in tiny intermittent chunks.
- A successful remote write whose response is deliberately dropped.
- An overload response during a burst of original requests and retries.
- A stale or unavailable fallback, followed by gradual dependency recovery.
Assert customer outcomes and resource bounds: attempts per logical request, maximum elapsed time, released connections and workers, correct duplicate handling, and bounded recovery traffic. Verify that cancellation stops local work where supported; it does not automatically undo a remote write.
A discussion-only incident rehearsal can exercise ownership and communications. These integration tests establish what the program actually does. Use both where useful, with a named owner for any gap discovered.
Start with one important dependency and one operation. Put the policy next to its configuration, verify it under failure, and make the customer-visible tradeoff explicit. An outage is a poor time to discover that your fallback is a lie or that every layer has been helpfully retrying.
Find more practical engineering and operations guidance at Slaptijack.