-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WAL-Archiving + PITR #231
Merged
Merged
WAL-Archiving + PITR #231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davissp14
force-pushed
the
object-storage-integration
branch
2 times, most recently
from
June 25, 2024 19:01
fefe347
to
3e1a877
Compare
davissp14
force-pushed
the
object-storage-integration
branch
from
June 25, 2024 19:08
9957c56
to
c7e764a
Compare
can we change the env vars to match the default |
benwaffle
reviewed
Jun 27, 2024
benwaffle
reviewed
Jun 27, 2024
davissp14
commented
Jul 2, 2024
davissp14
commented
Jul 3, 2024
benwaffle
reviewed
Jul 3, 2024
I'm seeing an error listing backups:
here's the json: postgres@6833e41a77d908:~$ barman-cloud-backup-list --cloud-provider aws-s3 --endpoint-url https://fly.storage.tigris.dev --profile restore --format json s3://dbdbdbbucket dbdbdb | python3 -m json.tool
{
"backups_list": [
{
"backup_label": null,
"begin_offset": 216,
"begin_time": "Tue Jul 2 21:05:44 2024",
"begin_wal": "00000005000000000000002A",
"begin_xlog": "0/2A0000D8",
"compression": null,
"config_file": "/data/postgresql/postgresql.conf",
"copy_stats": null,
"deduplicated_size": null,
"end_offset": null,
"end_time": null,
"end_wal": null,
"end_xlog": null,
"error": "failure uploading data (connection already closed)",
"hba_file": "/data/postgresql/pg_hba.conf",
"ident_file": "/data/postgresql/pg_ident.conf",
"included_files": [
"/data/postgresql/postgresql.auto.conf",
"/data/postgresql/postgresql.internal.conf"
],
"mode": null,
"pgdata": "/data/postgresql",
"server_name": "cloud",
"size": null,
"status": "FAILED",
"systemid": "7332222271544570189",
"tablespaces": null,
"timeline": 5,
"version": 150006,
"xlog_segment_size": 16777216,
"backup_id": "20240702T210544"
},
{
"backup_label": "'START WAL LOCATION: 0/2E000028 (file 00000005000000000000002E)\\nCHECKPOINT LOCATION: 0/2E000060\\nBACKUP METHOD: streamed\\nBACKUP FROM: primary\\nSTART TIME: 2024-07-02 21:07:03 UTC\\nLABEL: Barman backup cloud 20240702T210703\\nSTART TIMELINE: 5\\n'",
"begin_offset": 40,
"begin_time": "Tue Jul 2 21:07:03 2024",
"begin_wal": "00000005000000000000002E",
"begin_xlog": "0/2E000028",
"compression": null,
"config_file": "/data/postgresql/postgresql.conf",
"copy_stats": {
"total_time": 5.192484,
"number_of_workers": 2,
"analysis_time": 0,
"analysis_time_per_item": {
"data": 0
},
"copy_time_per_item": {
"data": 4.423057
},
"serialized_copy_time_per_item": {
"data": 2.484593
},
"copy_time": 4.423057,
"serialized_copy_time": 2.484593
},
"deduplicated_size": null,
"end_offset": 80,
"end_time": "Tue Jul 2 21:07:05 2024",
"end_wal": "00000005000000000000002F",
"end_xlog": "0/2F000050",
"error": null,
"hba_file": "/data/postgresql/pg_hba.conf",
"ident_file": "/data/postgresql/pg_ident.conf",
"included_files": [
"/data/postgresql/postgresql.auto.conf",
"/data/postgresql/postgresql.internal.conf"
],
"mode": null,
"pgdata": "/data/postgresql",
"server_name": "cloud",
"size": null,
"status": "DONE",
"systemid": "7332222271544570189",
"tablespaces": null,
"timeline": 5,
"version": 150006,
"xlog_segment_size": 16777216,
"backup_id": "20240702T210703"
}
]
}
|
What do the query params look like for the EDIT UPDATE |
…ith the restore target
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a WAL Archiving solution that integrates with any S3 compatible object storage. It also provides various remote recovery options for restoring into a new app. While in-place recovery options are possible with the provided functionality, the commands must be issued manually. The current automated recovery process will fail if there's an existing
postgresql
directory.Docs for in-place restores will be coming soon.
Enabling WAL Archiving
To enable, set the following secret:
This most straight forward way to get started would be to create a new Tigris bucket via:
Then set the secret with
fly secrets set S3_ARCHIVE_CONFIG=... --app <pg-app-name>
Viewing/Updating WAL Archiving Configuration
Definitions
recovery_window (default: 7d)
Used as a retention policy. Backups older than the specified value will be pruned.
The shortest allowed recovery window is
1d
.Units available:
minimum_redundancy (default 3)
The minimum number of backups that should always be available. Must be
>= 0
full_backup_frequency (default: 24h)
The frequency in which full backups are taken. Must be
>= 1h
archive_timeout (default: 60s)
This is a limit on how old unarchived data can be. Archiving typically happens once the WAL segment is full, however, this can cause issues for less active databases. So if the WAL segment doesn't fill by the specified timeout, a wal-switch will occur and force an archive event.
WARNING - Having too short of a archive timeout can have performance implications and bloat object storage.
View your current configuration
First, you can view the existing settings by running the following curl command:
Update your configuration
To update a configuration option, you can run:
Then when we re-run the view command we can see the recovery window has been updated:
Note - This is a manual process, but the idea is that it make it straight forward to surface within a client and/or UI. These settings currently require a cluster restart, but in the future it should be possible to configure at runtime.
Remote Restore
Provision a new Postgres with the following secret:
S3_ARCHIVE_REMOTE_RESTORE_CONFIG=https://<access_key_id>:<access_secret>@<s3-endpoint>/<source-bucket-name>/<source-bucket-directory>
Note: If you don't specify a target it will default to restoring to the most recent point-in-time.
Target options
target
Specifies that recovery should end as soon as a consistent state is reached. The only allowed value here is "immediate".
targetTime
This parameter specifies the time stamp up to which recovery will proceed. The timestamp should follow RFC3339 formatting.
targetName
This parameter specifies the named restore point.
targetLSN (Not implemented)
This parameter specifies the LSN of the write-ahead log location up to which recovery will proceed.
TargetXID (Not implemented)
This parameter specifies the transaction ID up to which recovery will proceed.
Additional target options
targetInclusive (default: true)
Specifies whether to stop just after the specified recovery target (true), or just before the recovery target (false).
targetAction (default: promote)
Specifies what action the server should take once the recovery target is reached
targetTimeline (default: current-timeline)
Specifies recovering into a particular timeline. The default is to recover along the same timeline that was current when the base backup was taken.
Example usage
S3_ARCHIVE_REMOTE_RESTORE_CONFIG=https://<access_key_id>:<access_secret>@<s3-endpoint>/<source-bucket-name>/<source-bucket-directory>?targetTime=2024-07-03T20:23:46Z
For more information on these settings, see: https://www.postgresql.org/docs/current/runtime-config-wal.html#RUNTIME-CONFIG-WAL-RECOVERY-TARGET