Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created by
brew bump
Created with
brew bump-formula-pr
.release notes
PostgreSQL 17 support - revised ported library to support PostgreSQL 17
Database Subset - a new feature that allows you to define a subset of the database, allowing you to scale down the dump size (#110). This is robust for multipurpose and especially useful for testing and development environments. It supports:
circular dependencies in the subset by generating a recursive query. The query is generated with integrity checks of the subset ensuring that the data gathered from circular dependencies is consistent.
polymorphic_exprs
attribute and use greenmask to generate a subset for such tables.pgzip support for faster compression and [decompression] https://greenmask.io/v0.2.0/commands//restore#pgzip-decompression) — setting
--pgzip
can speed up the dump and restoration processes through parallel compression. In some tests, it shows up to 5x faster dump and restore operations.Restoration in topological order - This flag ensures that dependent tables are not restored until the tables they depend on have been restored. This is useful when you want to be notified of errors as immediately as possible without waiting for the entire table to be restored.
Insert format restoration - For a flexible restoration process, Greenmask now supports data restoration in the
INSERT
format. It generates the insert statements based onCOPY
records from the dump. You do not need to re-dump your data to use this feature; it can be defined in therestore
command. The list of new features related to theINSERT
format:INSERT
statements with theON CONFLICT DO NOTHING
clause if the flag--on-conflict-do-nothing
is set.Restore data batching (#173) - By default, the COPY protocol returns the error only on transaction commit. To override this behavior, use the
--batch-size
flag to specify the number of rows to insert in a single batch during the COPY command. This is useful when you want to control the transaction size and commit.Introduced
keep_null
parameter forRandomPerson
transformer.Introduced dynamic parameters in the transformers
cast_to
. These functions cover frequent operations such asUnixTimestampToDate
andIntToBool
.The transformation logic has been significantly refactored, making transformers more customizable and flexible than before.
Introduced transformation engines
random
- generates transformer values based on pseudo-random algorithms.hash
- generates transformer values using hash functions. Currently, it utilizessha3
hash functions, which are secure but perform slowly. In the stable release, there will be an option to choose betweensha3
andSipHash
.Introduced static parameters value template
Dumps retention management - Introduced retention parameters (#201) for the delete command. Introduced two new statuses: failed and in progress. A dump is considered failed if it lacks a "done" heartbeat or if the last heartbeat timestamp exceeds 30 minutes. The delete command now supports the following retention parameters:
--dry-run
: Runs the deletion operation in test mode with verbose output, without actually deleting anything.--before-date 2024-08-27T23:50:54+00:00
: Deletes dumps older than the specified date. The date must be providedin RFC3339Nano format, for example:
2021-01-01T00:00:00Z
.--retain-recent 10
: Retains the N most recent dumps, where N is specified by the user.--retain-for 1w2d3h4m5s6ms7us8ns
: Retains dumps for the specified duration. The format supports weeks (w), days (d), hours (h), minutes (m), seconds (s), milliseconds (ms), microseconds (us), and nanoseconds (ns).--prune-failed
: Prunes (removes) all dumps that have failed.--prune-unsafe
: Prunes dumps with "unknown-or-failed" statuses. This option only works in conjunction with--prune-failed
.Docker image mirroring into the GitHub Container Registry
Core
Parametrizer
interface, now implemented for both dynamic and static parameters.Driver
initialization logic.Driver
.Parametrizer
interface.TransformationContext
, as the first step towards enabling new feature transformation conditions (#34).Transformers
RandomEmail - Introduces a new transformer that supports both random and deterministic engines. It allows for flexible email value generation; you can use column values in the template and choose to keep the original domain or select any from the
domains
parameter.NoiseDate, NoiseFloat, NoiseInt - These transformers support both random and deterministic engines, offering dynamic mode parameters that control the noise thresholds within the
min
andmax
range. Unlike previous implementations which used a singleratio
parameter, the new release featuresmin_ratio
andmax_ratio
parameters to define noise values more precisely. Utilizing thehash
engine in these transformers enhances security by complicating statistical analysis for attackers, especially when the same salt is used consistently over long periods.NoiseNumeric - A newly implemented transformer, sharing features with
NoiseInt
andNoiseFloat
, but specifically designed for numeric values (large integers or floats). It provides adecimal
parameter to handle values with fractions.RandomChoice - Now supports the
hash
engineRandomDate, RandomFloat, RandomInt - Now enhanced with hash engine support. Threshold parameters
min
andmax
have been updated to support dynamic mode, allowing for more flexible configurations.RandomNumeric - A new transformer specifically designed for numeric types (large integers or floats), sharing similar features with
RandomInt
andRandomFloat
, but tailored for handling huge numeric values.RandomString - Now supports hash engine mode
RandomUnixTimestamp - This new transformer generates Unix timestamps with selectable units (
second
,millisecond
,microsecond
,nanosecond
). Similar in function toRandomDate
, it supports the hash engine and dynamic parameters formin
andmax
thresholds, with the ability to override these units usingmin_unit
andmax_unit
parameters.RandomUuid - Added hash engine support
RandomPerson - Implemented a new transformer that replaces
RandomName
,RandomLastName
,RandomFirstName
,RandomFirstNameMale
,RandomFirstNameFemale
,RandomTitleMale
, andRandomTitleFemale
. This new transformer offers enhanced customizability while providing similar functionalities as the previous versions. It generates personal data such asFirstName
,LastName
, andTitle
, based on the providedgender
parameter, which now supports dynamic mode. Future minor versions will allow for overriding the default names database.Added tsModify - a new template function for time.Time objects modification
Introduced a new RandomIp transformer capable of generating a random IP address based on the specified netmask.
Added a new RandomMac transformer for generating random Mac addresses.
Deleted transformers include
RandomMacAddress
,RandomIPv4
,RandomIPv6
,RandomUnixTime
,RandomTitleMale
,RandomTitleFemale
,RandomFirstName
,RandomFirstNameMale
,RandomFirstNameFemale
,RandomLastName
, andRandomName
due to the introduction of more flexible and unified options.Fixes and improvements
validate
command with the--table
flag, which had the wrong order of the table name representation{{ table_name }}.{{ schema }}
instead of{{ schema }}.{{ table_name }}
.Row.SetColumn
out of range validation.restoreWorker
panic caused when the worker received an error from pgx.restore
command.--exit-on-error
works incorrectly in therestore
command. Now, the--exit-on-error
flag works correctly with thedata
section.validate
command.latest
to exclude specific keywords.RandomPerson
transformer.--exclude-table
,--table
, etc.Email
transformer.columns_type_override
did not work.min
andmax
parameter values were ignored in transformersNoiseDate
,NoiseNumeric
,NoiseFloat
,NoiseInt
,RandomNumeric
,RandomFloat
, andRandomInt
.pg_restore 1724504511561 --file 1724504511561.sql
is backward compatible and works as expected.Full Changelog: v0.1.14...v0.2.0
Contributors
Special thanks
Links
Feel free to reach out to us if you have any questions or need assistance: