how clock drift causes production bugs
The Most Useful Debugging Tool Is Often a Clock
A surprising number of production bugs are not mysterious. They are merely events recorded in different time zones, with clocks that disagree by several minutes.
I learned this from an old payment system where the application log said a request arrived before the database log showed the transaction that created it. The team spent hours discussing retries and duplicate messages. The real culprit was one server with a drifting hardware clock.
A Small Habit That Helps Whenever I investigate a distributed failure, I first check how timestamps are produced, stored, and displayed. UTC is useful, but synchronized clocks are more useful. A timestamp without its source and accuracy is only a confident-looking guess.
Before adding another dashboard or tracing product, ask whether the existing clocks can agree. Boring infrastructure still gets the vote.