Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

postgres cdc: update mirror lsn_offset when wal processing raises consumedXLogPos #823

Merged
merged 13 commits into from
Dec 20, 2023

Conversation

serprex
Copy link
Contributor

@serprex serprex commented Dec 14, 2023

Not doing so could cause failures if consumption restarted, since retrieval of mirror lsn_offset would start before updated lsn sent to pg, making possible that pg had gotten rid of that wal in meantime

Also always use GREATEST to make sure lsn_offset is monotonic

@serprex serprex force-pushed the start-lsn-investigation branch 2 times, most recently from ed66fc7 to fceefb1 Compare December 14, 2023 15:00
…sumedXLogPos

Not doing so could cause failures if consumption restarted,
since retrieval of mirror lsn_offset would start before updated lsn sent to pg,
making possible that pg had gotten rid of that wal in meantime
@serprex serprex force-pushed the start-lsn-investigation branch from fceefb1 to 313c708 Compare December 14, 2023 15:18
@serprex serprex marked this pull request as draft December 14, 2023 15:31
@serprex serprex force-pushed the start-lsn-investigation branch from a4a4d61 to 9c73c0f Compare December 14, 2023 17:00
@serprex serprex force-pushed the start-lsn-investigation branch from 9c73c0f to 68bf3ce Compare December 14, 2023 17:03
flow/connectors/core.go Outdated Show resolved Hide resolved
flow/connectors/core.go Show resolved Hide resolved
flow/connectors/postgres/cdc.go Show resolved Hide resolved
flow/activities/flowable.go Show resolved Hide resolved
flow/connectors/postgres/client.go Show resolved Hide resolved
flow/connectors/postgres/client.go Outdated Show resolved Hide resolved
flow/connectors/snowflake/snowflake.go Outdated Show resolved Hide resolved
@serprex serprex marked this pull request as ready for review December 14, 2023 17:10
@serprex serprex force-pushed the start-lsn-investigation branch from 9641287 to 065e205 Compare December 15, 2023 06:15
@serprex serprex enabled auto-merge (squash) December 16, 2023 16:50
@serprex serprex merged commit 9521994 into main Dec 20, 2023
7 checks passed
@serprex serprex deleted the start-lsn-investigation branch December 25, 2023 21:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants