Things I learned about distributed systems

The biggest shift in how I think about systems is accepting that partial failure is normal. A request can succeed on one machine while its caller never receives a response.

Start with the failure case Before adding a queue, I now ask what happens if a worker stops after doing the work but before acknowledging the message. That question usually leads straight to idempotency.

Make retries safe Retries are useful only when repeating an operation is safe. A stable operation identifier and a record of completed work can make a repeated request harmless. Backoff and a retry limit prevent a struggling dependency from receiving even more traffic.

Observe the whole journey A dashboard of averages can hide a long tail of slow requests. Queue age, latency percentiles, and correlated logs make the actual user experience easier to understand. The lesson is simple: build a system you can explain when something goes wrong.