Skip to content
All articles

Distributed systems · Failure

A timeout is not a failure

A local call returns or throws. A network call has a third ending: no reply, and no way to know whether the work was done. Two of the three outcomes look identical to your code.

· 4 min read

Three outcomes of a payment call. Request arrives and the reply comes back: the money moved, and the client sees 200 OK. Request never arrives: the money did not move, and the client sees a timeout. Request arrives but there is no reply: the money moved, and the client sees a timeout. Rows two and three are byte-identical to the client's code. Every repair has to work without knowing which one happened; a 5xx can be the third outcome too, so retry with the same key.

Call a method in the same process and it returns a value or it throws. Two outcomes, and both are information: either way you know what happened, so you can write the next line of code accordingly.

Put a network between the caller and the method and most engineers' mental model quietly stays the same: it worked, or it failed. That model is wrong, and the gap between those two outcomes is where the money goes missing.

Three outcomes, two of them identical

A client calls a payment service. Three things can happen:

What happenedDid the money move?What the client sees
The request arrives and the reply comes backYes, charged200 OK
The request never arrivesNotimeout
The request arrives and no reply comes backYes, chargedtimeout

Look only at the right-hand column, because that is all your code can see. Two different states of the world, the customer's money untouched and the customer's money gone, arrive at your service as byte-identical timeouts. Nothing available to the caller separates them.

Every repair works blind

Because your code cannot tell those two cases apart, every repair has to work without knowing which one happened. The course builds the rest of its material on that column.

You have probably met this already. You wrote if (orderExists) return existing; before an insert. You added a UNIQUE constraint on a reference number and caught the violation. You built a nightly reconciliation job that fixes up rows nobody meant to create twice. Each of those was a way of handling the third outcome without a name for it.

What about an HTTP 500?

Common Mistake

Reading a 5xx as proof that nothing was committed. An exception thrown after the commit returns 500, a gateway that gives up waiting returns 504, and one whose backend connection broke mid-reply returns 502, in every case with the money possibly moved.

Unless the service guarantees that a 5xx means nothing was committed, treat it like a timeout and retry with the same key, so that a second attempt cannot charge twice.

Get one diagram a week

A short article built around one engineering diagram, from the same library as these courses.

One diagram-led article a week on AI and systems engineering. We email you once to confirm, and every newsletter has an unsubscribe link. Privacy policy