Every network call in a distributed system can fail. A dependency may slow down, a connection may disappear, or the client may time out after the server has already completed the operation. A resilient API treats these outcomes as normal operating conditions and defines how to degrade, recover, and communicate failure.

Start with a time budget

Requests that wait forever exhaust threads and connection pools. Each dependency call needs a timeout, while the complete request needs a slightly larger but still finite budget. Retries belong only on transient failures and operations that are safe to repeat. A circuit breaker temporarily stops traffic to a failing dependency, and bulkhead isolation prevents one dependency from consuming all available resources.

An idempotent order request

Blindly retrying a POST for an order or payment can create duplicate business records. A client-generated idempotency key ensures that one logical request is processed once.

app.MapPost("/orders", async (
    HttpRequest request,
    CreateOrder command,
    OrderService service,
    CancellationToken ct) =>
{
    var key = request.Headers["Idempotency-Key"].ToString();
    if (string.IsNullOrWhiteSpace(key))
        return Results.BadRequest("Idempotency-Key is required");

    var result = await service.CreateOnceAsync(key, command, ct);
    return Results.Ok(result);
});

CreateOnceAsync stores the key and result behind a unique database constraint. A repeated key returns the previous result instead of performing the operation again. Retry policies can then recover from uncertain network outcomes without duplicating business data.

Implementation checklist

1. Define measurable connection and request timeouts for every external dependency.
2. Retry only transient outcomes such as 408, 429, and carefully selected 5xx responses.
3. Use exponential backoff with jitter to avoid synchronized retry storms.
4. Protect critical POST operations with idempotency keys and database uniqueness.
5. Expose circuit-breaker state through metrics, structured logs, and alerts.

What to avoid

• Retrying every exception three times and amplifying permanent failures.
• Setting only an HttpClient timeout without defining an end-to-end request budget.
• Returning internal exception details to clients and leaking implementation information.

Conclusion

Resilience is not a NuGet package checkbox. Business semantics, time budgets, and observability must be designed together. Start with explicit timeouts and narrow retry rules, then add circuit breaking, idempotency, and isolation based on real measurements. The system will not hide failures, but it will contain their impact and recover predictably.