Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release: v0.2 #97

Merged
merged 76 commits into from
May 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
f9eaaf7
Parametrizer abstraction
wwoytenko Jan 10, 2024
89642f7
Fixed restore section for playground config
wwoytenko Jan 10, 2024
c7c97a3
Merge branch 'main' into v0.2
wwoytenko Jan 10, 2024
520d70f
Refactored code:
wwoytenko Jan 11, 2024
8d98cdb
Refactored code:
wwoytenko Jan 12, 2024
7b14c5b
Refactored code:
wwoytenko Jan 12, 2024
46a1e4d
Fixes
wwoytenko Jan 12, 2024
849f848
Implemented dynamic parameters basic usage
wwoytenko Jan 14, 2024
8a69fa5
New rails
wwoytenko Jan 14, 2024
5159645
New rails
wwoytenko Jan 15, 2024
b57b84a
Adapted tests for new rails
wwoytenko Jan 17, 2024
c35d4bc
Minor changes:
wwoytenko Jan 17, 2024
56a09a6
Get rid of the deprecated:
wwoytenko Jan 18, 2024
26f2170
Get rid of the deprecated:
wwoytenko Jan 18, 2024
ea5e9e2
Introduced dynamic parameters compatible types
wwoytenko Jan 20, 2024
ad1c6d8
Dynamic parameters
wwoytenko Jan 20, 2024
1cfe53b
Refactored:
wwoytenko Jan 21, 2024
6aa5d98
Fixed unit tests
wwoytenko Jan 22, 2024
760daf4
Merge branch 'main' into v0.2
wwoytenko Jan 26, 2024
1066f9d
Dynamic parameters with cast_template execution
wwoytenko Jan 27, 2024
19a44f0
Merged with main
wwoytenko Jan 31, 2024
e3af948
cast_to implementation and auto-cast
wwoytenko Jan 31, 2024
bf02864
cast_to implementation
wwoytenko Feb 4, 2024
8d129cc
Fixed test errors
wwoytenko Feb 4, 2024
828827f
Fixes
wwoytenko Feb 4, 2024
7c45cc3
Fixes
wwoytenko Feb 4, 2024
b2719c9
Merged with main
wwoytenko Feb 23, 2024
4c848c4
Fixed errors after merge with actual v0.1.6 version
wwoytenko Feb 23, 2024
86a11f9
Added DynamicUnmarshaler logic and fixed type in method name
wwoytenko Feb 23, 2024
f1b276f
Init deterministic lib
wwoytenko Mar 1, 2024
d7750fa
Saved changes
wwoytenko Mar 1, 2024
ab7035f
Implemented random generator for int values
wwoytenko Mar 2, 2024
9fdf89a
Changed layout
wwoytenko Mar 2, 2024
007862b
Changed layout
wwoytenko Mar 2, 2024
5981859
New transformation framework
wwoytenko Mar 3, 2024
e0949e0
Saved changes
wwoytenko Mar 4, 2024
c2544ca
Pulled from deterministic lib
wwoytenko Mar 4, 2024
b3a712d
First deterministic transformer implementation
wwoytenko Mar 4, 2024
cb6c57b
Fixed integer transformer initialization and salt value parsing. Min …
wwoytenko Mar 5, 2024
c1f08d3
Implemented Decimal transformer support
wwoytenko Mar 5, 2024
583b85e
BigInt transformer implementation
wwoytenko Mar 6, 2024
9e1a26d
Fixed dynamic params for big int
wwoytenko Mar 6, 2024
b3d6d50
Re-designed unified transformers added tests from big_integer
wwoytenko Mar 9, 2024
3144369
Implemented common random date transformer
wwoytenko Mar 9, 2024
9ca0561
Fixed errors
wwoytenko Mar 9, 2024
854a4c0
Merged with the latest release v0.1.7
wwoytenko Mar 16, 2024
2c44954
Implemented Timestamp transformer
wwoytenko Mar 16, 2024
0074b48
Refactored engine usage logic
wwoytenko Mar 17, 2024
ff8fe01
Fixed Integer transformer size
wwoytenko Mar 17, 2024
d7101a8
Fixed unit tests
wwoytenko Mar 17, 2024
ce77a23
Implemented Float transformer and fixed hash reducer
wwoytenko Mar 18, 2024
2f7a3a6
Merge remote-tracking branch 'origin/main' into deterministic_lib_v0.2
wwoytenko Mar 22, 2024
e1480c0
New transformers and removed artifacts
wwoytenko Mar 22, 2024
5ea1b46
Bunch of changes
wwoytenko Apr 8, 2024
e8c848b
NoiseDate transformer implementation
wwoytenko Apr 10, 2024
95f71ad
Added template support for static parameter
wwoytenko Apr 17, 2024
582e412
Implemented NoiseFloat and NoiseNumeric transformers
wwoytenko Apr 22, 2024
4feaa05
Implemented Hybrid generator (random + hash) and fixed unit test
wwoytenko Apr 22, 2024
4f252aa
feat: Email transformer is implemented
wwoytenko May 1, 2024
2f1dd67
fix: Fixed email transformer UX
wwoytenko May 2, 2024
3515ecf
fix: Added lost RoRecordContext
wwoytenko May 2, 2024
883f786
merge: Pulled from main
wwoytenko May 2, 2024
31840ad
merge: Pulled from main
wwoytenko May 2, 2024
055f13e
fix: fixed unit test for unixToTime func
wwoytenko May 2, 2024
8427631
feat: RandomFullName transformer implementation
wwoytenko May 3, 2024
b54a298
fix: fixed unit test for TestFuncMap_unixToTime
wwoytenko May 6, 2024
f6faf78
Merge branch 'release/v0.2rc' into feat/name_transformer
wwoytenko May 6, 2024
4d2bdfc
fix: fixed TableColumnsQuery scanning for pg 11
wwoytenko May 6, 2024
c11e8eb
Merge remote-tracking branch 'origin_public/release/v0.2rc' into feat…
wwoytenko May 6, 2024
348d839
feat: Renamed RandomFullName transformer to RandomPerson transformer
wwoytenko May 8, 2024
a31fd23
Merge remote-tracking branch 'origin_public/main' into release/v0.2rc
wwoytenko May 8, 2024
97294c0
Merge remote-tracking branch 'origin_public/release/v0.2rc' into feat…
wwoytenko May 8, 2024
764c446
feat: Added documentation for RandomPerson transformer. Deleted old t…
wwoytenko May 8, 2024
c608ea2
feat: Fixed doc example and deterministic behaviour
wwoytenko May 8, 2024
8eaea68
Merge pull request #98 from GreenmaskIO/feat/name_transformer
wwoytenko May 8, 2024
ba7b9e8
Merge remote-tracking branch 'origin_public/main' into release/v0.2rc
wwoytenko May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/greenmask/cmd/list_transformers/list_transformers.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ func listTransformersText(registry *utils.TransformerRegistry) error {
return nil
}

func getColumnTypes(p *toolkit.Parameter) []string {
func getColumnTypes(p *toolkit.ParameterDefinition) []string {
if p.ColumnProperties != nil && len(p.ColumnProperties.AllowedTypes) > 0 {
return p.ColumnProperties.AllowedTypes
}
Expand Down
8 changes: 4 additions & 4 deletions cmd/greenmask/cmd/show_transformer/show_transformer.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ func run(name string) error {
}

func showTransformerJson(registry *utils.TransformerRegistry, transformerName string) error {
var transformers []*utils.Definition
var transformers []*utils.TransformerDefinition

def, ok := registry.M[transformerName]
if ok {
Expand Down Expand Up @@ -114,8 +114,8 @@ func showTransformerText(registry *utils.TransformerRegistry, name string) error
if p.DefaultValue != nil {
data = append(data, []string{def.Properties.Name, "parameters", p.Name, "default", string(p.DefaultValue), ""})
}
if p.LinkParameter != "" {
data = append(data, []string{def.Properties.Name, "parameters", p.Name, "linked_parameter", p.LinkParameter, ""})
if p.LinkColumnParameter != "" {
data = append(data, []string{def.Properties.Name, "parameters", p.Name, "linked_parameter", p.LinkColumnParameter, ""})
}
if p.CastDbType != "" {
data = append(data, []string{def.Properties.Name, "parameters", p.Name, "cast_to_db_type", p.CastDbType, ""})
Expand Down Expand Up @@ -146,7 +146,7 @@ func showTransformerText(registry *utils.TransformerRegistry, name string) error
return nil
}

func getTransformerDefinition(registry *utils.TransformerRegistry, name string) (*utils.Definition, error) {
func getTransformerDefinition(registry *utils.TransformerRegistry, name string) (*utils.TransformerDefinition, error) {
def, ok := registry.M[name]
if ok {
return def, nil
Expand Down
8 changes: 1 addition & 7 deletions docs/built_in_transformers/standard_transformers/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,7 @@ Standard transformers are ready-to-use methods that require no customization and
1. [RandomCCNumber](random_cc_number.md) — generates a random credit card number.
1. [RandomCurrency](random_currency.md) — generates a random currency code.
1. [RandomAmountWithCurrency](random_amount_with_currency.md) — generates a random monetary amount with currency.
1. [RandomName](random_name.md) — generates a full random name.
1. [RandomLastName](random_last_name.md) — generates a random last name.
1. [RandomFirstName](random_first_name.md) — generates a random first name.
1. [RandomFirstNameMale](random_first_name_male.md) — generates a random male first name.
1. [RandomFirstNameFemale](random_first_name_female.md) — generates a random female first name.
1. [RandomTitleMale](random_title_male.md) — generates a random male title.
1. [RandomTitleFemale](random_title_female.md) — generates a random female title.
1. [RandomPerson](random_person.md) — generates a random person data (first name, last name, etc.)
1. [RandomPhoneNumber](random_phone_number.md) — generates a random phone number.
1. [RandomTollFreePhoneNumber](random_toll_free_phone_number.md) — generates a random toll-free phone number.
1. [RandomE164PhoneNumber](random_e164_phone_number.md) — generates a random phone number in E.164 format.
Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

28 changes: 0 additions & 28 deletions docs/built_in_transformers/standard_transformers/random_name.md

This file was deleted.

121 changes: 121 additions & 0 deletions docs/built_in_transformers/standard_transformers/random_person.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
The `RandomPerson` transformer is designed to populate specified database columns with personal attributes such as
first name, last name, title and gender.

## Parameters

| Name | Description | Default | Required | Supported DB types |
|------------|------------------------------------------------------|---------|----------|--------------------|
| columns | The name of the column to be affected | | Yes | text, varchar |
| gender | set specific gender (possible values: Male, Female, Any) | `Any` | No | - |
| gender_mapping | Specify gender name to possible values when using dynamic mode in "gender" parameter | `Any` |No |- |
| fallback_gender | Specify fallback gender if not mapped when using dynamic mode in "gender" parameter | `Any` |No |- |
| engine | The engine used for generating the values [random, hash]. Use hash for deterministic generation | `random` |No |- |


## Description

The `RandomPerson` transformer utilizes a comprehensive list of first names to inject random first names into the
designated database column. This feature allows for the creation of diverse and realistic user profiles by
simulating a variety of first names without using real user data.


### *column* object attributes

* `name` — the name of the column where the personal attributes will be stored. This value is required.
* `template` - the template for the column value.
You can use the next attributes: `.FirstName`, `.LastName` or `.Title`. For example, if you want to generate a full name, you can use the next template:
`"{{ .FirstName }} {{ .LastName }}"`

* `hashing` - the bool value. Indicates whether the column value must be passed through the hashing function.
The default value is `false`. If all column has `hashing` set to `false` (by default), then all columns will be hashed.


### *gender_mapping* object attributes

`gender_mapping` - a dictionary that maps the gender value when `gender` parameters works in dynamic mode.
The default value is:

```json
{
"Male": [
"male",
"M",
"m",
"man",
"Man"
],
"Female": [
"female",
"F",
"f",
"w",
"woman",
"Woman"
]
}
```

### *fallback_gender*

Gender that will be used if `gender_mapping` was not found. This parameter is optional
and required only for `gender` parameter in dynamic mode. The default value is `Any`.

## Example: Populate random first name and last name for table user_profiles in static mode

This example demonstrates how to use the `RandomPerson` transformer to populate the `name` and `surname` columns in
the `user_profiles` table with random first names, last name, respectively.

```sql title="Create table user_profiles and insert data"

CREATE TABLE personal_data (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
surname VARCHAR(100),
sex CHAR(1) CHECK (sex IN ('M', 'F'))
);

-- Insert sample data into the table
INSERT INTO personal_data (name, surname, sex) VALUES
('John', 'Doe', 'M'),
('Jane', 'Smith', 'F'),
('Alice', 'Johnson', 'F'),
('Bob', 'Lee', 'M');
```

```yaml title="RandomPerson transformer example"
- schema: public
name: personal_data
transformers:
- name: "RandomPerson"
params:
gender: "Any"
columns:
- name: "name"
template: "{{ .FirstName }}"
- name: "surname"
template: "{{ .LastName }}"
engine: "hash"
```

## Example: Populate random first name and last name for table user_profiles in dynamic mode

This example demonstrates how to use the `RandomPerson` transformer to populate the `name`, `surname` using dynamic
gender


```yaml title="RandomPerson transformer example with dynamic mode"
- schema: public
name: personal_data
transformers:
- name: "RandomPerson"
params:
columns:
- name: "name"
template: "{{ .FirstName }}"
- name: "surname"
template: "{{ .LastName }}"
engine: "random"
dynamic_params:
gender:
column: sex
```

This file was deleted.

Loading