diff --git a/docs/built_in_transformers/apply_for_references.md b/docs/built_in_transformers/apply_for_references.md new file mode 100644 index 00000000..3299bad8 --- /dev/null +++ b/docs/built_in_transformers/apply_for_references.md @@ -0,0 +1,171 @@ +# Apply for References + +## Description + +Using `apply_for_references`, you can apply transformations to columns involved in a primary key or in tables with a +foreign key that references that column. This simplifies the transformation process by requiring you to define the +transformation only on the primary key column, which will then be applied to all tables referencing that column. + +The transformer must support `hash` engine and the `hash` engin must be set in the configuration file. + +## End-to-End Identifiers + +End-to-end identifiers in databases are unique identifiers that are consistently used across multiple tables in a +relational database schema, allowing for a seamless chain of references from one table to another. These identifiers +typically serve as primary keys in one table and are propagated as foreign keys in other tables, creating a direct, +traceable link from one end of a data relationship to the other. + +Greenmask can detect end-to-end identifiers and apply transformations across the entire sequence of tables. These +identifiers are detected when the following condition is met: the foreign key serves as both a primary key and a foreign +key in the referenced table. + +## Limitations + +- The transformation must be deterministic. +- The transformation condition will not be applied to the referenced column. +- Not all transformers support `apply_for_references` + +!!! warning + + We do not recommend using `apply_for_references` with transformation conditions, as these conditions are not + inherited by transformers on the referenced columns. This may lead to inconsistencies in the data. + +List of transformers that supports `apply_for_references`: + +* Hash +* NoiseDate +* NoiseFloat +* NoiseInt +* NoiseNumeric +* RandomBool +* RandomDate +* RandomEmail +* RandomFloat +* RandomInt +* RandomIp +* RandomMac +* RandomNumeric +* RandomString +* RandomUuid +* RandomUnixTimestamp + +## Example 1. Simple table references + +This is ordinary table references where the primary key of the `users` table is referenced in the `orders` table. + +```sql +-- Enable the extension for UUID generation (if not enabled) +CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; + +CREATE TABLE users +( + user_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), + username VARCHAR(50) NOT NULL +); + +CREATE TABLE orders +( + order_id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), + user_id UUID REFERENCES users (user_id), + order_date DATE NOT NULL +); + +INSERT INTO users (username) +VALUES ('john_doe'); +INSERT INTO users (username) +VALUES ('jane_smith'); + +INSERT INTO orders (user_id, order_date) +VALUES ((SELECT user_id FROM users WHERE username = 'john_doe'), '2024-10-31'), + ((SELECT user_id FROM users WHERE username = 'jane_smith'), '2024-10-30'); +``` + +To transform the `username` column in the `users` table, you can use the following configuration: + +```yaml +- schema: public + name: users + apply_for_inherited: true + transformers: + - name: RandomUuid + apply_for_references: true + params: + column: "user_id" + engine: "hash" +``` + +This will apply the `RandomUuid` transformation to the `user_id` column in the `orders` table automatically. + +## Example 2. Tables with end-to-end identifiers + +In this example, we have three tables: `tablea`, `tableb`, and `tablec`. All tables have a composite primary key. +In the tables `tableb` and `tablec`, the primary key is also a foreign key that references the primary key of `tablea`. +This means that all PKs are end-to-end identifiers. + +```sql +CREATE TABLE tablea +( + id1 INT, + id2 INT, + data VARCHAR(50), + PRIMARY KEY (id1, id2) +); + +CREATE TABLE tableb +( + id1 INT, + id2 INT, + detail VARCHAR(50), + PRIMARY KEY (id1, id2), + FOREIGN KEY (id1, id2) REFERENCES tablea (id1, id2) ON DELETE CASCADE +); + +CREATE TABLE tablec +( + id1 INT, + id2 INT, + description VARCHAR(50), + PRIMARY KEY (id1, id2), + FOREIGN KEY (id1, id2) REFERENCES tableb (id1, id2) ON DELETE CASCADE +); + +INSERT INTO tablea (id1, id2, data) +VALUES (1, 1, 'Data A1'), + (2, 1, 'Data A2'), + (3, 1, 'Data A3'); + +INSERT INTO tableb (id1, id2, detail) +VALUES (1, 1, 'Detail B1'), + (2, 1, 'Detail B2'), + (3, 1, 'Detail B3'); + +INSERT INTO tablec (id1, id2, description) +VALUES (1, 1, 'Description C1'), + (2, 1, 'Description C2'), + (3, 1, 'Description C3'); +``` + +To transform the `data` column in `tablea`, you can use the following configuration: + +```yaml +- schema: public + name: "tablea" + apply_for_inherited: true + transformers: + - name: RandomInt + apply_for_references: true + params: + min: 0 + max: 100 + column: "id1" + engine: "hash" + - name: RandomInt + apply_for_references: true + params: + min: 0 + max: 100 + column: "id2" + engine: "hash" +``` + +This will apply the `RandomInt` transformation to the `id1` and `id2` columns in `tableb` and `tablec` automatically. diff --git a/docs/built_in_transformers/index.md b/docs/built_in_transformers/index.md index 79e5e8fa..a237183e 100644 --- a/docs/built_in_transformers/index.md +++ b/docs/built_in_transformers/index.md @@ -3,10 +3,15 @@ Transformers in Greenmask are methods which are applied to anonymize sensitive data. All Greenmask transformers are split into the following groups: -- [Transformation engines](transformation_engines.md) — the type of generator used in transformers. Hash (deterministic) - and random (randomization) - [Dynamic parameters](dynamic_parameters.md) — transformers that require an input of parameters and generate random data based on them. +- [Transformation engines](transformation_engines.md) — the type of generator used in transformers. Hash (deterministic) + and random (randomization) +- [Parameters templating](parameters_templating.md) — generate static parameters values from templates. +- [Transformation conditions](transformation_condition.md) — conditions that can be applied to transformers. If the + condition is not met, the transformer will not be applied. +- [Apply for references](apply_for_references.md) — apply transformation on a column that is involved in a primary key + and tables with a foreign key that references that column. - [Standard transformers](standard_transformers/index.md) — transformers that require only an input of parameters. - [Advanced transformers](advanced_transformers/index.md) — transformers that can be modified according to user's needs with the help of [custom functions](advanced_transformers/custom_functions/index.md). diff --git a/docs/built_in_transformers/transformation_condition.md b/docs/built_in_transformers/transformation_condition.md index 6b0b1cd4..276fd8b7 100644 --- a/docs/built_in_transformers/transformation_condition.md +++ b/docs/built_in_transformers/transformation_condition.md @@ -9,7 +9,7 @@ The condition must be defined as a boolean expression that evaluates to `true` o `expr` library. You can use the same functions that are described in -the [built-in transformers](/docs/built_in_transformers/advanced_transformers/custom_functions/index.md) +the [built-in transformers](./advanced_transformers/custom_functions/index.md) The transformers are executed one by one - this helps you create complex transformation pipelines. For instance depending on value chosen in the previous transformer, you can decide to execute the next transformer or not. diff --git a/internal/db/postgres/context/config_builder.go b/internal/db/postgres/context/config_builder.go index f534d7bb..b1ae4dc8 100644 --- a/internal/db/postgres/context/config_builder.go +++ b/internal/db/postgres/context/config_builder.go @@ -11,6 +11,7 @@ import ( "github.com/greenmaskio/greenmask/internal/db/postgres/entries" "github.com/greenmaskio/greenmask/internal/db/postgres/subset" + "github.com/greenmaskio/greenmask/internal/db/postgres/transformers" transformersUtils "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" "github.com/greenmaskio/greenmask/internal/domains" "github.com/greenmaskio/greenmask/pkg/toolkit" @@ -51,7 +52,7 @@ func (tcm *tableConfigMapping) hasTransformerWithApplyForReferences() bool { // may contain the schema affection warnings that would be useful for considering consistency func validateAndBuildEntriesConfig( ctx context.Context, tx pgx.Tx, entries []*entries.Table, typeMap *pgtype.Map, - cfg *domains.Dump, registry *transformersUtils.TransformerRegistry, + cfg *domains.Dump, r *transformersUtils.TransformerRegistry, version int, types []*toolkit.Type, graph *subset.Graph, ) (toolkit.ValidationWarnings, error) { var warnings toolkit.ValidationWarnings @@ -66,14 +67,11 @@ func validateAndBuildEntriesConfig( } // Assign settings to the Tables using config received - //entriesWithTransformers := findTablesWithTransformers(cfg.Transformation, Tables) - entriesWithTransformers, err := setConfigToEntries(ctx, tx, cfg.Transformation, entries, graph) + entriesWithTransformers, setConfigWarns, err := setConfigToEntries(ctx, tx, cfg.Transformation, entries, graph, r) if err != nil { return nil, fmt.Errorf("cannot get Tables entries config: %w", err) } - // TODO: - // Check if any has relkind = p - // If yes, then find all children and remove them from entriesWithTransformers + warnings = append(warnings, setConfigWarns...) for _, cfgMapping := range entriesWithTransformers { // set subset conditions setSubsetConds(cfgMapping.entry, cfgMapping.config) @@ -122,7 +120,7 @@ func validateAndBuildEntriesConfig( setColumnTypeOverrides(cfgMapping.entry, cfgMapping.config, typeMap) // Set transformers for the table - transformersInitWarns, err := initAndSetupTransformers(ctx, cfgMapping.entry, cfgMapping.config, registry) + transformersInitWarns, err := initAndSetupTransformers(ctx, cfgMapping.entry, cfgMapping.config, r) enrichWarningsWithTableName(transformersInitWarns, cfgMapping.entry) warnings = append(warnings, transformersInitWarns...) if err != nil { @@ -176,9 +174,8 @@ func validateTableExists( return warnings, nil } -// findTablesWithTransformers - assigns settings from the config to the table entries. This function -// iterates through the Tables and do the following: -// 1. Compile when condition and set to the table entry +// findTablesWithTransformers - finds Tables with transformers in the config and returns them as a slice of +// tableConfigMapping func findTablesWithTransformers( cfg []*domains.Table, tables []*entries.Table, ) []*tableConfigMapping { @@ -200,12 +197,19 @@ func findTablesWithTransformers( func setConfigToEntries( ctx context.Context, tx pgx.Tx, cfg []*domains.Table, tables []*entries.Table, g *subset.Graph, -) ([]*tableConfigMapping, error) { + r *transformersUtils.TransformerRegistry, +) ([]*tableConfigMapping, toolkit.ValidationWarnings, error) { var res []*tableConfigMapping + var warnings toolkit.ValidationWarnings for _, tcm := range findTablesWithTransformers(cfg, tables) { if tcm.hasTransformerWithApplyForReferences() { // If table has transformer with apply_for_references, then we need to find all reference tables // and add them to the list + ok, checkWarns := checkApplyForReferenceMetRequirements(tcm, r) + if !ok { + warnings = append(warnings, checkWarns...) + continue + } refTables := getRefTables(tcm.entry, tcm.config, g) res = append(res, refTables...) } @@ -216,15 +220,18 @@ func setConfigToEntries( } // If the table is partitioned, then we need to find all children and remove parent from the list if !tcm.config.ApplyForInherited { - return nil, fmt.Errorf( - "the table \"%s\".\"%s\" is partitioned use apply_for_inherited", - tcm.entry.Schema, tcm.entry.Name, + warnings = append(warnings, toolkit.NewValidationWarning(). + SetMsg("the table is partitioned use apply_for_inherited"). + AddMeta("SchemaName", tcm.entry.Schema). + AddMeta("TableName", tcm.entry.Name). + SetSeverity(toolkit.ErrorValidationSeverity), ) + continue } parts, err := findPartitionsOfPartitionedTable(ctx, tx, tcm.entry) if err != nil { - return nil, fmt.Errorf( + return nil, nil, fmt.Errorf( "cannot find partitions of the table %s.%s: %w", tcm.entry.Schema, tcm.entry.Name, err, ) @@ -248,7 +255,7 @@ func setConfigToEntries( }) } } - return res, nil + return res, warnings, nil } func getRefTables(rootTable *entries.Table, rootTableCfg *domains.Table, graph *subset.Graph) []*tableConfigMapping { @@ -532,3 +539,59 @@ func initAndSetupTransformers(ctx context.Context, t *entries.Table, cfg *domain } return warnings, nil } + +func checkApplyForReferenceMetRequirements( + tcm *tableConfigMapping, r *transformersUtils.TransformerRegistry, +) (bool, toolkit.ValidationWarnings) { + warnings := toolkit.ValidationWarnings{} + for _, tr := range tcm.config.Transformers { + allowed, w := isTransformerAllowedToApplyForReferences(tr, r) + if !allowed { + warnings = append(warnings, w...) + } + } + return !warnings.IsFatal(), warnings +} + +// isTransformerAllowedToApplyForReferences - checks if the transformer is allowed to apply for references +// and if the engine parameter is hash and required +func isTransformerAllowedToApplyForReferences( + cfg *domains.TransformerConfig, r *transformersUtils.TransformerRegistry, +) (bool, toolkit.ValidationWarnings) { + td, ok := r.Get(cfg.Name) + if !ok { + return false, toolkit.ValidationWarnings{ + toolkit.NewValidationWarning(). + SetMsg("transformer not found"). + AddMeta("TransformerName", cfg.Name). + SetSeverity(toolkit.ErrorValidationSeverity), + } + } + allowApplyForReferenced, ok := td.Properties.GetMeta(transformers.AllowApplyForReferenced) + if !ok || !allowApplyForReferenced.(bool) { + return false, toolkit.ValidationWarnings{ + toolkit.NewValidationWarning(). + SetMsg( + "cannot apply transformer for references: transformer does not support apply for references", + ). + AddMeta("TransformerName", cfg.Name). + SetSeverity(toolkit.ErrorValidationSeverity), + } + } + requireHashEngineParameter, ok := td.Properties.GetMeta(transformers.RequireHashEngineParameter) + if !ok { + return false, nil + } + if !requireHashEngineParameter.(bool) { + return true, nil + } + if string(cfg.Params[engineParameterName]) != transformers.HashEngineParameterName { + return false, toolkit.ValidationWarnings{ + toolkit.NewValidationWarning(). + SetMsg("cannot apply transformer for references: engine parameter is not hash"). + AddMeta("TransformerName", cfg.Name). + SetSeverity(toolkit.ErrorValidationSeverity), + } + } + return true, nil +} diff --git a/internal/db/postgres/context/config_builder_test.go b/internal/db/postgres/context/config_builder_test.go new file mode 100644 index 00000000..75fe5138 --- /dev/null +++ b/internal/db/postgres/context/config_builder_test.go @@ -0,0 +1,70 @@ +package context + +import ( + "testing" + + "github.com/stretchr/testify/require" + + "github.com/greenmaskio/greenmask/internal/db/postgres/transformers" + "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" + "github.com/greenmaskio/greenmask/internal/domains" + "github.com/greenmaskio/greenmask/pkg/toolkit" +) + +func Test_isTransformerAllowedToApplyForReferences(t *testing.T) { + r := utils.DefaultTransformerRegistry + + t.Run("RandomInt and hash engine", func(t *testing.T) { + cfg := &domains.TransformerConfig{ + Name: transformers.RandomIntTransformerName, + ApplyForReferences: true, + Params: toolkit.StaticParameters{ + "column": toolkit.ParamsValue("id"), + "engine": toolkit.ParamsValue("hash"), + }, + } + ok, w := isTransformerAllowedToApplyForReferences(cfg, r) + require.Empty(t, w) + require.True(t, ok) + }) + + t.Run("RandomInt and without hash engine", func(t *testing.T) { + cfg := &domains.TransformerConfig{ + Name: transformers.RandomIntTransformerName, + ApplyForReferences: true, + Params: toolkit.StaticParameters{ + "column": toolkit.ParamsValue("id"), + "engine": toolkit.ParamsValue("random"), + }, + } + ok, w := isTransformerAllowedToApplyForReferences(cfg, r) + require.NotEmpty(t, w) + require.False(t, ok) + }) + + t.Run("Template", func(t *testing.T) { + cfg := &domains.TransformerConfig{ + Name: transformers.TemplateTransformerName, + ApplyForReferences: true, + Params: toolkit.StaticParameters{ + "column": toolkit.ParamsValue("id"), + }, + } + ok, w := isTransformerAllowedToApplyForReferences(cfg, r) + require.NotEmpty(t, w) + require.False(t, ok) + }) + + t.Run("Unknown name", func(t *testing.T) { + cfg := &domains.TransformerConfig{ + Name: "unknown", + ApplyForReferences: true, + Params: toolkit.StaticParameters{ + "column": toolkit.ParamsValue("id"), + }, + } + ok, w := isTransformerAllowedToApplyForReferences(cfg, r) + require.NotEmpty(t, w) + require.False(t, ok) + }) +} diff --git a/internal/db/postgres/transformers/cmd.go b/internal/db/postgres/transformers/cmd.go index 196d412e..35a8ebb2 100644 --- a/internal/db/postgres/transformers/cmd.go +++ b/internal/db/postgres/transformers/cmd.go @@ -47,11 +47,11 @@ var defaultRowDriverParams = toolkit.DriverParams{ CsvAttributesFormat: toolkit.CsvAttributesConfigNumeratingFormatName, } -var cmdTransformerName = "Cmd" +var CmdTransformerName = "Cmd" var CmdTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - cmdTransformerName, + CmdTransformerName, "Transform data via external program using stdin and stdout interaction", ), @@ -145,7 +145,7 @@ func NewCmd( ctx context.Context, driver *toolkit.Driver, parameters map[string]toolkit.Parameterizer, ) (utils.Transformer, toolkit.ValidationWarnings, error) { - name := cmdTransformerName + name := CmdTransformerName var columns []*Column var executable string var args []string diff --git a/internal/db/postgres/transformers/default_params.go b/internal/db/postgres/transformers/default_params.go index 35f464ef..a7806460 100644 --- a/internal/db/postgres/transformers/default_params.go +++ b/internal/db/postgres/transformers/default_params.go @@ -8,8 +8,8 @@ import ( ) const ( - randomEngineName = "random" - hashEngineName = "hash" + RandomEngineParameterName = "random" + HashEngineParameterName = "hash" ) var ( @@ -42,7 +42,7 @@ var ( func engineValidator(p *toolkit.ParameterDefinition, v toolkit.ParamsValue) (toolkit.ValidationWarnings, error) { value := string(v) - if value != randomEngineName && value != hashEngineName { + if value != RandomEngineParameterName && value != HashEngineParameterName { return toolkit.ValidationWarnings{ toolkit.NewValidationWarning(). SetMsg("Invalid engine value"). diff --git a/internal/db/postgres/transformers/dict.go b/internal/db/postgres/transformers/dict.go index 190223ab..876d77fc 100644 --- a/internal/db/postgres/transformers/dict.go +++ b/internal/db/postgres/transformers/dict.go @@ -24,9 +24,11 @@ import ( const defaultNullSeq = `\N` +const DictTransformerName = "Dict" + var DictTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Dict", + DictTransformerName, "Replace values matched by dictionary keys", ), diff --git a/internal/db/postgres/transformers/email.go b/internal/db/postgres/transformers/email.go index 6fade11c..887f9716 100644 --- a/internal/db/postgres/transformers/email.go +++ b/internal/db/postgres/transformers/email.go @@ -19,15 +19,18 @@ import ( const emailTransformerGeneratorSize = 64 +const RandomEmailTransformerName = "RandomEmail" + var emailTransformerRegexp = regexp.MustCompile(`^([a-zA-Z0-9_.+-]+)@([a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)$`) //var emailTransformerAllowedChars = []rune("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!#$%&'*+-/=?^_`{|}~.") var EmailTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomEmail", + RandomEmailTransformerName, "Generate random email", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewEmailTransformer, diff --git a/internal/db/postgres/transformers/hash.go b/internal/db/postgres/transformers/hash.go index d9326b2d..4363dfab 100644 --- a/internal/db/postgres/transformers/hash.go +++ b/internal/db/postgres/transformers/hash.go @@ -32,11 +32,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const HashTransformerName = "Hash" + var HashTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Hash", + HashTransformerName, "Generate hash of the text value using Scrypt hash function under the hood", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, false), NewHashTransformer, diff --git a/internal/db/postgres/transformers/json.go b/internal/db/postgres/transformers/json.go index fb41ec49..94d612b5 100644 --- a/internal/db/postgres/transformers/json.go +++ b/internal/db/postgres/transformers/json.go @@ -40,10 +40,12 @@ const ( JsonSetOpName = "set" ) +const JsonTransformerName = "Json" + var JsonTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Json", + JsonTransformerName, "Update json document", ), diff --git a/internal/db/postgres/transformers/masking.go b/internal/db/postgres/transformers/masking.go index ba3640e7..a3eb2b76 100644 --- a/internal/db/postgres/transformers/masking.go +++ b/internal/db/postgres/transformers/masking.go @@ -21,6 +21,7 @@ import ( "strings" "github.com/ggwhite/go-masker" + "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" "github.com/greenmaskio/greenmask/pkg/toolkit" ) @@ -38,9 +39,11 @@ const ( MDefault string = "default" ) +const MaskingTransformerName = "Masking" + var MaskingTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Masking", + MaskingTransformerName, "Mask a value using one of masking type", ), diff --git a/internal/db/postgres/transformers/noise_date.go b/internal/db/postgres/transformers/noise_date.go index 48a69b79..f49de6fd 100644 --- a/internal/db/postgres/transformers/noise_date.go +++ b/internal/db/postgres/transformers/noise_date.go @@ -21,19 +21,21 @@ import ( "github.com/jackc/pgx/v5/pgtype" - "github.com/greenmaskio/greenmask/internal/generators/transformers" - "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" + "github.com/greenmaskio/greenmask/internal/generators/transformers" "github.com/greenmaskio/greenmask/pkg/toolkit" ) // TODO: Ensure pqinterval.Duration returns duration in int64 for date and time +const NoiseDateTransformerName = "NoiseDate" + var NoiseDateTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "NoiseDate", + NoiseDateTransformerName, "Add some random value (shift value) in the provided interval", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewNoiseDateTransformer, diff --git a/internal/db/postgres/transformers/noise_float.go b/internal/db/postgres/transformers/noise_float.go index 09b88c6c..4a11b627 100644 --- a/internal/db/postgres/transformers/noise_float.go +++ b/internal/db/postgres/transformers/noise_float.go @@ -23,11 +23,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const NoiseFloatTransformerName = "NoiseFloat" + var NoiseFloatTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "NoiseFloat", + NoiseFloatTransformerName, "Add noise to float value in min and max thresholds", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewNoiseFloatTransformer, toolkit.MustNewParameterDefinition( diff --git a/internal/db/postgres/transformers/noise_int.go b/internal/db/postgres/transformers/noise_int.go index 929bc0b3..e8a1a7b0 100644 --- a/internal/db/postgres/transformers/noise_int.go +++ b/internal/db/postgres/transformers/noise_int.go @@ -23,11 +23,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const NoiseIntTransformerName = "NoiseInt" + var NoiseIntTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "NoiseInt", + NoiseIntTransformerName, "Make noise value for int", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewNoiseIntTransformer, diff --git a/internal/db/postgres/transformers/noise_numeric.go b/internal/db/postgres/transformers/noise_numeric.go index b976056d..f73db84f 100644 --- a/internal/db/postgres/transformers/noise_numeric.go +++ b/internal/db/postgres/transformers/noise_numeric.go @@ -25,11 +25,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const NoiseNumericTransformerName = "NoiseNumeric" + var NoiseNumericTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "NoiseNumeric", + NoiseNumericTransformerName, "Add noise to numeric value in min and max thresholds", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewNumericFloatTransformer, toolkit.MustNewParameterDefinition( diff --git a/internal/db/postgres/transformers/random_bool.go b/internal/db/postgres/transformers/random_bool.go index fe01c6be..72e74385 100644 --- a/internal/db/postgres/transformers/random_bool.go +++ b/internal/db/postgres/transformers/random_bool.go @@ -23,12 +23,15 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomBoolTransformerName = "RandomBool" + var boolTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomBool", + RandomBoolTransformerName, "Generate random bool", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewBooleanTransformer, diff --git a/internal/db/postgres/transformers/random_choice.go b/internal/db/postgres/transformers/random_choice.go index 7c406933..71e67d19 100644 --- a/internal/db/postgres/transformers/random_choice.go +++ b/internal/db/postgres/transformers/random_choice.go @@ -18,18 +18,21 @@ import ( "context" "fmt" - "github.com/greenmaskio/greenmask/internal/generators/transformers" "github.com/tidwall/gjson" "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" + "github.com/greenmaskio/greenmask/internal/generators/transformers" "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomChoiceTransformerName = "RandomChoice" + var ChoiceTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomChoice", + RandomChoiceTransformerName, "Replace values chosen randomly from list", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewRandomChoiceTransformer, diff --git a/internal/db/postgres/transformers/random_date.go b/internal/db/postgres/transformers/random_date.go index e3941004..9329bf84 100644 --- a/internal/db/postgres/transformers/random_date.go +++ b/internal/db/postgres/transformers/random_date.go @@ -25,6 +25,8 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomDateTransformerName = "RandomDate" + var truncateParts = []string{ transformers.YearTruncateName, transformers.MonthTruncateName, transformers.DayTruncateName, transformers.HourTruncateName, transformers.SecondTruncateName, transformers.MillisecondTruncateName, @@ -33,9 +35,10 @@ var truncateParts = []string{ var timestampTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomDate", + RandomDateTransformerName, "Generate date in the provided interval", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewTimestampTransformer, diff --git a/internal/db/postgres/transformers/random_faker.go b/internal/db/postgres/transformers/random_faker.go index 88868bb3..14fb004b 100644 --- a/internal/db/postgres/transformers/random_faker.go +++ b/internal/db/postgres/transformers/random_faker.go @@ -25,6 +25,31 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const ( + RandomLatitudeTransformerName = "RandomLatitude" + RandomLongitudeTransformerName = "RandomLongitude" + RandomMonthNameTransformerName = "RandomMonthName" + RandomYearStringTransformerName = "RandomYearString" + RandomDayOfWeekTransformerName = "RandomDayOfWeek" + RandomDayOfMonthTransformerName = "RandomDayOfMonth" + RandomCenturyTransformerName = "RandomCentury" + RandomTimezoneTransformerName = "RandomTimezone" + RandomDomainNameTransformerName = "RandomDomainName" + RandomURLTransformerName = "RandomURL" + RandomUsernameTransformerName = "RandomUsername" + RandomPasswordTransformerName = "RandomPassword" + RandomWordTransformerName = "RandomWord" + RandomSentenceTransformerName = "RandomSentence" + RandomParagraphTransformerName = "RandomParagraph" + RandomCCTypeTransformerName = "RandomCCType" + RandomCCNumberTransformerName = "RandomCCNumber" + RandomCurrencyTransformerName = "RandomCurrency" + RandomAmountWithCurrencyTransformerName = "RandomAmountWithCurrency" + RandomPhoneNumberTransformerName = "RandomPhoneNumber" + RandomTollFreePhoneNumberTransformerName = "RandomTollFreePhoneNumber" + RandomE164PhoneNumberTransformerName = "RandomE164PhoneNumber" +) + type FakerFunc func(opts ...options.OptionFunc) string type FakerTransformerDef struct { @@ -35,14 +60,14 @@ type FakerTransformerDef struct { var FakerTransformersDes = map[string]*FakerTransformerDef{ // Faker geo - "RandomLatitude": { + RandomLatitudeTransformerName: { Generator: func(opts ...options.OptionFunc) string { return fmt.Sprintf("%f", faker.Latitude()) }, SupportedTypes: []string{"float4", "float8", "numeric"}, Description: "Generates a random latitude value.", }, - "RandomLongitude": { + RandomLongitudeTransformerName: { Generator: func(opts ...options.OptionFunc) string { return fmt.Sprintf("%f", faker.Longitude()) }, @@ -50,110 +75,110 @@ var FakerTransformersDes = map[string]*FakerTransformerDef{ Description: "Generates a random longitude value.", }, - "RandomMonthName": { + RandomMonthNameTransformerName: { Generator: faker.MonthName, SupportedTypes: []string{"text", "varchar"}, Description: "Generates the name of a random month.", }, - "RandomYearString": { + RandomYearStringTransformerName: { Generator: faker.YearString, SupportedTypes: []string{"text", "varchar", "int2", "int4", "int8", "numeric"}, Description: "Generates a random year as a string.", }, - "RandomDayOfWeek": { + RandomDayOfWeekTransformerName: { Generator: faker.DayOfWeek, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random day of the week.", }, - "RandomDayOfMonth": { + RandomDayOfMonthTransformerName: { Generator: faker.DayOfMonth, SupportedTypes: []string{"text", "varchar", "int2", "int4", "int8", "numeric"}, Description: "Generates a random day of the month.", }, - "RandomCentury": { + RandomCenturyTransformerName: { Generator: faker.Century, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random century.", }, - "RandomTimezone": { + RandomTimezoneTransformerName: { Generator: faker.Timezone, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random timezone.", }, // Faker Internet - "RandomDomainName": { + RandomDomainNameTransformerName: { Generator: faker.DomainName, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random domain name.", }, - "RandomURL": { + RandomURLTransformerName: { Generator: faker.URL, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random URL.", }, - "RandomUsername": { + RandomUsernameTransformerName: { Generator: faker.Username, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random username.", }, - "RandomPassword": { + RandomPasswordTransformerName: { Generator: faker.Password, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random password.", }, // Faker words and Sentences - "RandomWord": { + RandomWordTransformerName: { Generator: faker.Word, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random word.", }, - "RandomSentence": { + RandomSentenceTransformerName: { Generator: faker.Sentence, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random sentence.", }, - "RandomParagraph": { + RandomParagraphTransformerName: { Generator: faker.Paragraph, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random paragraph.", }, // Faker Payment - "RandomCCType": { + RandomCCTypeTransformerName: { Generator: faker.CCType, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random credit card type.", }, - "RandomCCNumber": { + RandomCCNumberTransformerName: { Generator: faker.CCNumber, SupportedTypes: []string{"text", "varchar", "int4", "int8", "numeric"}, Description: "Generates a random credit card number.", }, - "RandomCurrency": { + RandomCurrencyTransformerName: { Generator: faker.Currency, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random currency code.", }, - "RandomAmountWithCurrency": { + RandomAmountWithCurrencyTransformerName: { Generator: faker.AmountWithCurrency, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random monetary amount with currency.", }, // Faker Phone - "RandomPhoneNumber": { + RandomPhoneNumberTransformerName: { Generator: faker.Phonenumber, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random phone number.", }, - "RandomTollFreePhoneNumber": { + RandomTollFreePhoneNumberTransformerName: { Generator: faker.TollFreePhoneNumber, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random toll-free phone number.", }, - "RandomE164PhoneNumber": { + RandomE164PhoneNumberTransformerName: { Generator: faker.E164PhoneNumber, SupportedTypes: []string{"text", "varchar"}, Description: "Generates a random phone number in E.164 format.", diff --git a/internal/db/postgres/transformers/random_float.go b/internal/db/postgres/transformers/random_float.go index 3bded542..b823ef0f 100644 --- a/internal/db/postgres/transformers/random_float.go +++ b/internal/db/postgres/transformers/random_float.go @@ -29,11 +29,14 @@ const ( float8Length = 8 ) +const RandomFloatTransformerName = "RandomFloat" + var floatTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomFloat", + RandomFloatTransformerName, "Generate float value in min and max thresholds and round up to provided digits", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewFloatTransformer, diff --git a/internal/db/postgres/transformers/random_int.go b/internal/db/postgres/transformers/random_int.go index e780fc48..75a339a9 100644 --- a/internal/db/postgres/transformers/random_int.go +++ b/internal/db/postgres/transformers/random_int.go @@ -30,11 +30,14 @@ const ( Int8Length = 8 ) +const RandomIntTransformerName = "RandomInt" + var integerTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomInt", + RandomIntTransformerName, "Generate integer value in min and max thresholds", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewIntegerTransformer, diff --git a/internal/db/postgres/transformers/random_ip.go b/internal/db/postgres/transformers/random_ip.go index f5daddef..84e33cba 100644 --- a/internal/db/postgres/transformers/random_ip.go +++ b/internal/db/postgres/transformers/random_ip.go @@ -24,8 +24,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomIpTransformerName = "RandomIp" + var RandomIpDefinition = utils.NewTransformerDefinition( - utils.NewTransformerProperties("RandomIp", "Generate V4 or V6 IP in the provided subnet"), + utils.NewTransformerProperties( + RandomIpTransformerName, + "Generate V4 or V6 IP in the provided subnet", + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewIpTransformer, diff --git a/internal/db/postgres/transformers/random_mac.go b/internal/db/postgres/transformers/random_mac.go index abbf046e..f2a4346d 100644 --- a/internal/db/postgres/transformers/random_mac.go +++ b/internal/db/postgres/transformers/random_mac.go @@ -38,8 +38,14 @@ const ( managementTypeNameAny = "any" ) +const RandomMacTransformerName = "RandomMac" + var RandomMacAddressDefinition = utils.NewTransformerDefinition( - utils.NewTransformerProperties("RandomMac", "Generate random mac address"), + utils.NewTransformerProperties( + RandomMacTransformerName, + "Generate random mac address", + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewMacAddressTransformer, diff --git a/internal/db/postgres/transformers/random_numeric.go b/internal/db/postgres/transformers/random_numeric.go index c7f1e650..26d6d458 100644 --- a/internal/db/postgres/transformers/random_numeric.go +++ b/internal/db/postgres/transformers/random_numeric.go @@ -13,11 +13,14 @@ import ( const bigIntegerTransformerGenByteLength = 20 +const RandomNumericTransformerName = "RandomNumeric" + var numericTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomNumeric", + RandomNumericTransformerName, "Generate numeric value in min and max thresholds", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewRandomNumericTransformer, diff --git a/internal/db/postgres/transformers/random_person.go b/internal/db/postgres/transformers/random_person.go index bb7365a6..bf92d88f 100644 --- a/internal/db/postgres/transformers/random_person.go +++ b/internal/db/postgres/transformers/random_person.go @@ -19,9 +19,11 @@ const ( const randomPersonAnyGender = "Any" +const RandomPersonTransformerName = "RandomPerson" + var randomPersonTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomPerson", + RandomPersonTransformerName, "Generate random person data (Title, FirstName, LastName, Gender)", ), @@ -103,9 +105,9 @@ func NewRandomNameTransformer(ctx context.Context, driver *toolkit.Driver, param var engineMode int switch engine { - case randomEngineName: + case RandomEngineParameterName: engineMode = randomEngineMode - case hashEngineName: + case HashEngineParameterName: engineMode = hashEngineMode } diff --git a/internal/db/postgres/transformers/random_string.go b/internal/db/postgres/transformers/random_string.go index 8ab94cf5..0d9866c9 100644 --- a/internal/db/postgres/transformers/random_string.go +++ b/internal/db/postgres/transformers/random_string.go @@ -23,11 +23,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomStringTransformerName = "RandomString" + var stringTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomString", + RandomStringTransformerName, "Generate a string withing the specified length with provided char set", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewRandomStringTransformer, diff --git a/internal/db/postgres/transformers/random_unix_timestamp.go b/internal/db/postgres/transformers/random_unix_timestamp.go index 74db8fca..f626ed13 100644 --- a/internal/db/postgres/transformers/random_unix_timestamp.go +++ b/internal/db/postgres/transformers/random_unix_timestamp.go @@ -19,15 +19,18 @@ const ( nanoUnit = "nanosecond" ) +const RandomUnixTimestampTransformerName = "RandomUnixTimestamp" + var timestampUnitValues = []string{ secondsUnit, milliUnit, microUnit, nanoUnit, } var unixTimestampTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomUnixTimestamp", + RandomUnixTimestampTransformerName, "Generate UnixTimestamp in the provided interval with unit", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewUnixTimestampTransformer, diff --git a/internal/db/postgres/transformers/random_uuid.go b/internal/db/postgres/transformers/random_uuid.go index 817ce831..c930aaf9 100644 --- a/internal/db/postgres/transformers/random_uuid.go +++ b/internal/db/postgres/transformers/random_uuid.go @@ -23,11 +23,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RandomUuidTransformerName = "RandomUuid" + var uuidTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RandomUuid", + RandomUuidTransformerName, "Generate UUID", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewRandomUuidTransformer, diff --git a/internal/db/postgres/transformers/real_address.go b/internal/db/postgres/transformers/real_address.go index 98f2f204..dde4a4f5 100644 --- a/internal/db/postgres/transformers/real_address.go +++ b/internal/db/postgres/transformers/real_address.go @@ -27,9 +27,11 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RealAddressTransformerName = "RealAddress" + var RealAddressTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RealAddress", + RealAddressTransformerName, "Generate a real address", ), diff --git a/internal/db/postgres/transformers/regexp_replace.go b/internal/db/postgres/transformers/regexp_replace.go index 61761a60..b6fedfa4 100644 --- a/internal/db/postgres/transformers/regexp_replace.go +++ b/internal/db/postgres/transformers/regexp_replace.go @@ -23,11 +23,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const RegexpReplaceTransformerName = "RegexpReplace" + var RegexpReplaceTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "RegexpReplace", + RegexpReplaceTransformerName, "Replace string using regular expression", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewRegexpReplaceTransformer, diff --git a/internal/db/postgres/transformers/replace.go b/internal/db/postgres/transformers/replace.go index aaf7b462..4d03add7 100644 --- a/internal/db/postgres/transformers/replace.go +++ b/internal/db/postgres/transformers/replace.go @@ -22,11 +22,14 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const ReplaceTransformerName = "Replace" + var ReplaceTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Replace", + ReplaceTransformerName, "Replace column value to the provided", - ), + ).AddMeta(AllowApplyForReferenced, true). + AddMeta(RequireHashEngineParameter, true), NewReplaceTransformer, diff --git a/internal/db/postgres/transformers/set_null.go b/internal/db/postgres/transformers/set_null.go index ea581af7..5b884167 100644 --- a/internal/db/postgres/transformers/set_null.go +++ b/internal/db/postgres/transformers/set_null.go @@ -22,9 +22,11 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const SetNullTransformerName = "SetNull" + var SetNullTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "SetNull", + SetNullTransformerName, "Set NULL value", ), NewSetNullTransformer, diff --git a/internal/db/postgres/transformers/template.go b/internal/db/postgres/transformers/template.go index 8685ce80..d1c57e17 100644 --- a/internal/db/postgres/transformers/template.go +++ b/internal/db/postgres/transformers/template.go @@ -25,9 +25,11 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const TemplateTransformerName = "Template" + var TemplateTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "Template", + TemplateTransformerName, "Modify the value using gotemplate", ), NewTemplateTransformer, diff --git a/internal/db/postgres/transformers/template_record.go b/internal/db/postgres/transformers/template_record.go index 1a5a9542..b6fa2554 100644 --- a/internal/db/postgres/transformers/template_record.go +++ b/internal/db/postgres/transformers/template_record.go @@ -25,9 +25,11 @@ import ( "github.com/greenmaskio/greenmask/pkg/toolkit" ) +const TemplateRecordTransformerName = "TemplateRecord" + var TemplateRecordTransformerDefinition = utils.NewTransformerDefinition( utils.NewTransformerProperties( - "TemplateRecord", + TemplateRecordTransformerName, "Modify the record using gotemplate", ), NewTemplateRecordTransformer, diff --git a/internal/db/postgres/transformers/utils.go b/internal/db/postgres/transformers/utils.go index 4584088c..7e9e3db4 100644 --- a/internal/db/postgres/transformers/utils.go +++ b/internal/db/postgres/transformers/utils.go @@ -6,18 +6,20 @@ import ( "encoding/binary" "fmt" + "github.com/greenmaskio/greenmask/internal/db/postgres/transformers/utils" "github.com/greenmaskio/greenmask/internal/generators" ) const ( - Sha1HashFunction = "sha1" + AllowApplyForReferenced utils.MetaKey = "AllowApplyForReferenced" + RequireHashEngineParameter utils.MetaKey = "RequireHashEngineParameter" ) func getGenerateEngine(ctx context.Context, engineName string, size int) (generators.Generator, error) { switch engineName { - case randomEngineName: + case RandomEngineParameterName: return getRandomBytesGen(size) - case hashEngineName: + case HashEngineParameterName: salt, err := getSaltFromCtx(ctx) if err != nil { return nil, fmt.Errorf("error getting salt from context: %w", err) diff --git a/internal/db/postgres/transformers/utils/properties.go b/internal/db/postgres/transformers/utils/properties.go index 82c16839..fa2b87a3 100644 --- a/internal/db/postgres/transformers/utils/properties.go +++ b/internal/db/postgres/transformers/utils/properties.go @@ -14,10 +14,13 @@ package utils +type MetaKey string + type TransformerProperties struct { - Name string `json:"name"` - Description string `json:"description"` - IsCustom bool `json:"is_custom"` + Name string `json:"name"` + Description string `json:"description"` + IsCustom bool `json:"is_custom"` + Meta map[MetaKey]any `json:"meta"` } func NewTransformerProperties( @@ -26,5 +29,16 @@ func NewTransformerProperties( return &TransformerProperties{ Name: name, Description: description, + Meta: make(map[MetaKey]any), } } + +func (tp *TransformerProperties) AddMeta(key MetaKey, value any) *TransformerProperties { + tp.Meta[key] = value + return tp +} + +func (tp *TransformerProperties) GetMeta(key MetaKey) (any, bool) { + value, ok := tp.Meta[key] + return value, ok +} diff --git a/mkdocs.yml b/mkdocs.yml index d05ae518..c5bc83c1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,6 +65,7 @@ nav: - Transformation engines: built_in_transformers/transformation_engines.md - Parameters templating: built_in_transformers/parameters_templating.md - Transformation conditions: built_in_transformers/transformation_condition.md + - Applying for references: built_in_transformers/apply_for_references.md - Standard transformers: - built_in_transformers/standard_transformers/index.md - Cmd: built_in_transformers/standard_transformers/cmd.md