An Incident Timeline Needs Two Clocks
At 10:03 a payment provider emits a failure event. At 10:11 your webhook worker receives it after a network partition. At 10:14 an alert fires. Which time belongs in the incident timeline?
All three, if they answer different questions.
Most event pipelines preserve one timestamp and call it “the time.” That makes postmortems look orderly until a retry, queue delay, or a machine with a bad clock reverses cause and effect. An event timestamp says when the producer claims something happened. An observation timestamp says when this system first received or recorded it. Alert and action timestamps tell yet other parts of the story.
{
"provider_event_at": "2026-07-05T10:03:12Z",
"received_at": "2026-07-05T10:11:47Z",
"persisted_at": "2026-07-05T10:11:48Z"
}
Do not overwrite provenance
The cheap model replaces provider_event_at with received_at during ingestion. It makes sorting easy, but destroys the evidence needed to distinguish a late event from a late processor. Keep both. When the producer time is untrusted or absent, label that fact rather than inventing precision.
The same applies to logs. A service's timestamp is useful for local sequencing, but a collector timestamp is often the better basis for deciding whether the telemetry pipeline was delayed. Clock synchronization reduces error; it does not eliminate the conceptual difference.
OpenTelemetry makes this explicit by allowing a span's start and end time to be supplied separately from export timing. That separation is a design clue: telemetry is an observation of work, not the work itself.
Practical review question
When an incident report says “X happened before Y,” ask which clock establishes that ordering. If the answer is merely the order in which a dashboard displayed rows, the conclusion may be a queue artifact.
References
- OpenTelemetry trace API specification documents explicit span timestamps.
- RFC 3339 defines an interoperable timestamp format; it does not make independently observed times identical.
Final take
One clock tells you when something claims to have occurred. The other tells you when you became responsible for it. Incident analysis needs both.
Changelog
- 2026-10-03T00:08:49-04:00: Recovered the 2026-07-05 scheduled slot; topic is distinct from the archive’s Git/publishing lineage articles.