Skip to content

Commit

Permalink
Update website/docs/reference/resource-configs/event-time.md
Browse files Browse the repository at this point in the history
Co-authored-by: Joel Labes <[email protected]>
  • Loading branch information
mirnawong1 and joellabes authored Jan 13, 2025
1 parent 34fdef2 commit 9764ae0
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion website/docs/reference/resource-configs/event-time.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ Set the `event_time` to the name of the field that represents the actual timesta
However, if an ingestion date (like `loaded_at`, `ingested_at`, or `last_updated_at`) are the only timestamps you use, you can set `event_time` to these fields. Here are some considerations to keep in mind if you do this:

- Using `last_updated_at` or `loaded_at` &mdash; May result in duplicate entries in the resulting table in the data warehouse over multiple runs. Setting an appropriate [lookback](/reference/resource-configs/lookback) value can reduce duplicates but it can't fully eliminate them since some updates outside the lookback window won't be processed.
- Using `ingested_at` &mdash; Represents when data was ingested, (not when the event occurred). This is less semantically meaningful for the underlying data and may require more filtering during query time.
- Using `ingested_at` &mdash; Because this column is created by your ingestion/EL tool instead of coming from the original source, it will change if/when you need to resync your connector for some reason. This means that data will be reprocessed and loaded into your warehouse for a second time against a second date. As long as this never happens (or you run a full refresh when it does), microbatches will be processed correctly when using `ingested_at`.

Here are some examples of recommended and not recommended `event_time` columns:

Expand Down

0 comments on commit 9764ae0

Please sign in to comment.