← Home

An Incident Timeline Needs Two Clocks

Jul 5, 2026

At 10:03 a payment provider emits a failure event. At 10:11 your webhook worker receives it after a network partition. At 10:14 an alert fires. Which time belongs in the incident timeline?

All three, if they answer different questions.

Most event pipelines preserve one timestamp and call it “the time.” That makes postmortems look orderly until a retry, queue delay, or a machine with a bad clock reverses cause and effect. An event timestamp says when the producer claims something happened. An observation timestamp says when this system first received or recorded it. Alert and action timestamps tell yet other parts of the story.

{
  "provider_event_at": "2026-07-05T10:03:12Z",
  "received_at": "2026-07-05T10:11:47Z",
  "persisted_at": "2026-07-05T10:11:48Z"
}

Do not overwrite provenance

The cheap model replaces provider_event_at with received_at during ingestion. It makes sorting easy, but destroys the evidence needed to distinguish a late event from a late processor. Keep both. When the producer time is untrusted or absent, label that fact rather than inventing precision.

The same applies to logs. A service's timestamp is useful for local sequencing, but a collector timestamp is often the better basis for deciding whether the telemetry pipeline was delayed. Clock synchronization reduces error; it does not eliminate the conceptual difference.

OpenTelemetry makes this explicit by allowing a span's start and end time to be supplied separately from export timing. That separation is a design clue: telemetry is an observation of work, not the work itself.

Practical review question

When an incident report says “X happened before Y,” ask which clock establishes that ordering. If the answer is merely the order in which a dashboard displayed rows, the conclusion may be a queue artifact.

References

Final take

One clock tells you when something claims to have occurred. The other tells you when you became responsible for it. Incident analysis needs both.

Changelog