S3 upload error: region missing

[joaozanuttoqr]

Hello guys, I'm trying to run the project locally using the latest docker image provided in dockerhub, but I'm getting an error message saying that the region can't be found in my configuration. Here is how my config.yaml file is looking like:

common:
  tmp_dir: /home/temp

log:
  level: debug

s3:
  bucket: BUCKET_NAME
  region: us-east-1
  access_key_id: ACCESS_KEY_ID
  secret_access_key: SECRET_ACCESS_KEY

dump:
  pg_dump_options:
    host: DB_HOST
    dbname: DB_NAME

restore:
  pg_restore_options:
    host: DB_HOST
    dbname: DB_NAME

Here is the log messages I'm getting (db host ommited):

root@173f2c271d32:/home# greenmask dump --config config.yaml
2024-03-27T15:17:24Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:145 > performing snapshot export pid=390
2024-03-27T15:17:26Z DBG ../var/lib/greenmask/internal/db/postgres/pgdump/pgdump.go:44 > pg_dump: pg_dump --file /home/temp/1711552642497597375 --format d --schema-only --snapshot 00000005-00069849-1 --dbname postgres --host DB_HOST --username postgres
 pid=390
2024-03-27T15:17:34Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:197 > reading schema section pid=390
2024-03-27T15:17:34Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:226 > planned 1 workers pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:547 > exited normally WorkerId=1 pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:331 > all the data have been dumped pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:336 > merging toc entries pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/db/postgres/cmd/dump.go:342 > writing built toc file into storage pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/storages/s3/logger.go:33 > s3 storage logging 0="DEBUG: Validate Request s3/PutObject failed, not retrying, error MissingRegion: could not find region configuration" pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/storages/s3/logger.go:33 > s3 storage logging 0="DEBUG: Build Request s3/PutObject failed, not retrying, error MissingRegion: could not find region configuration" pid=390
2024-03-27T15:17:36Z DBG ../var/lib/greenmask/internal/storages/s3/logger.go:33 > s3 storage logging 0="DEBUG: Sign Request s3/PutObject failed, not retrying, error MissingRegion: could not find region configuration" pid=390
2024-03-27T15:17:36Z FTL ../var/lib/greenmask/cmd/greenmask/cmd/dump/dump.go:58 > cannot make a backup error="mergeAndWriteToc stage dumping error: s3 object uploading error: MissingRegion: could not find region configuration" pid=390

I've tried to even export an AWS_REGION environment variable before executing greenmask, but I had no luck. Look forward to hear from you guys, this project is amazing!

Let me know if there is anything I can help with

[wwoytenko]

Hi!
Thank you for reporting I will check your case and come back soon.

[wwoytenko]

I have tried to reproduce your issue in the AWS Free Tier and it runs fine no error occurred. Ensure that the bucket you are using is in the region provided. Try to put something into this bucket using your AWS console client.

If can put the object via the AWS client then I will try to investigate the issue further

[wwoytenko]

Are you using Amazon AWS S3?

[wwoytenko]

However, I've just found a problem with S3 storage when the region is provided via env var AWS_REGION. I was able to access AWS using the region provided in the config. The related merge request is #37

But I don't think that is your case. My suggestion for now is still the same - ensure your bucket in the region provided

[joaozanuttoqr]

@wwoytenko thanks for taking the time! Yes, I'm using S3

I just tested uploading a file to the bucket via aws cli and it worked:

~/db-dump (master*) » aws s3 cp --no-verify-ssl config.yaml s3://<BUCKET_NAME>
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '<BUCKET_NAME>.s3.us-east-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
upload: ./config.yaml to s3://<BUCKET_NMAE>/config.yaml

(I'm getting this SSL error, so I ran the command using the --no-verify-ssl, but it actually helps because the warning message shows the bucket region. I also tried to set disable_ssl: true in greenmask config, but still getting the same error)

[wwoytenko]

Could you try to build the greenmask from the branch s3_storage_region_env_fix and execute it?

I have just added debug information and AWS_REGION env variable usage fix #37.

All changed in the source code you can find here

For doing so

1. Clone greenmask

git clone [email protected]:GreenmaskIO/greenmask.git

2. Go into cloned repo

cd greenmask && git checkout s3_storage_region_env_fix

3. Build greenmask

make build

4. You will find the greenmask binary in the repository root and try to do the same as you did
5. Find the log line s3 storage bucket and check that bucket and region names are correct

[joaozanuttoqr]

@wwoytenko I got this error when trying to build the project

~/greenmask (s3_storage_region_env_fix*) » make build                                     130 ↵ jzanuttoqr@NBVORTX690B18
CGO_ENABLED=0 go build -ldflags="-X github.com/greenmaskio/greenmask/cmd/greenmask/cmd.Version=" -v -o greenmask ./cmd/greenmask/
go: downloading github.com/rs/zerolog v1.31.0
go: downloading github.com/spf13/viper v1.18.2
go: downloading github.com/spf13/cobra v1.8.0
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/aws/aws-sdk-go v1.49.14
go: downloading github.com/jackc/pgx/v5 v5.5.1
go: extracting github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: extracting github.com/spf13/viper v1.18.2
go: downloading github.com/jackc/pgx-shopspring-decimal v0.0.0-20220624020537-1d36b5a1853e
go: extracting github.com/rs/zerolog v1.31.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/locafero v0.4.0
go: extracting github.com/mitchellh/mapstructure v1.5.0
go: extracting github.com/sagikazarmark/locafero v0.4.0
go: downloading github.com/spf13/afero v1.11.0
go: extracting github.com/spf13/cobra v1.8.0
go: extracting github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/mattn/go-colorable v0.1.13
go: extracting github.com/mattn/go-colorable v0.1.13
go: downloading github.com/spf13/pflag v1.0.5
go: extracting github.com/jackc/pgx-shopspring-decimal v0.0.0-20220624020537-1d36b5a1853e
go: downloading github.com/sourcegraph/conc v0.3.0
go: extracting github.com/spf13/afero v1.11.0
go: downloading golang.org/x/text v0.14.0
go: extracting github.com/sourcegraph/conc v0.3.0
go: downloading github.com/mitchellh/go-wordwrap v1.0.1
go: extracting github.com/spf13/pflag v1.0.5
go: downloading github.com/magiconair/properties v1.8.7
go: extracting github.com/mitchellh/go-wordwrap v1.0.1
go: extracting github.com/jackc/pgx/v5 v5.5.1
go: extracting github.com/magiconair/properties v1.8.7
go: downloading github.com/mattn/go-runewidth v0.0.9
go: extracting github.com/mattn/go-runewidth v0.0.9
go: downloading gopkg.in/yaml.v3 v3.0.1
go: extracting gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/spf13/cast v1.6.0
go: extracting github.com/spf13/cast v1.6.0
go: downloading golang.org/x/sync v0.5.0
go: extracting golang.org/x/sync v0.5.0
go: extracting golang.org/x/text v0.14.0
go: downloading github.com/go-faker/faker/v4 v4.2.0
go: extracting github.com/aws/aws-sdk-go v1.49.14
go: extracting github.com/go-faker/faker/v4 v4.2.0
go: downloading golang.org/x/crypto v0.17.0
go: extracting golang.org/x/crypto v0.17.0
go: downloading github.com/tidwall/sjson v1.2.5
go: extracting github.com/tidwall/sjson v1.2.5
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading github.com/shopspring/decimal v1.3.1
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/mattn/go-isatty v0.0.19
go: downloading golang.org/x/sys v0.15.0
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/Masterminds/sprig/v3 v3.2.3
go: downloading github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a
go: extracting github.com/shopspring/decimal v1.3.1
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: extracting github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a
go: downloading go.uber.org/multierr v1.9.0
go: extracting github.com/mattn/go-isatty v0.0.19
go: downloading github.com/ggwhite/go-masker v1.1.0
go: extracting github.com/subosito/gotenv v1.6.0
go: downloading github.com/pkg/errors v0.9.1
go: extracting github.com/hashicorp/hcl v1.0.0
go: extracting github.com/Masterminds/sprig/v3 v3.2.3
go: downloading github.com/google/uuid v1.5.0
go: downloading github.com/imdario/mergo v0.3.11
go: downloading github.com/Masterminds/goutils v1.1.1
go: extracting github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/mitchellh/copystructure v1.0.0
go: extracting go.uber.org/multierr v1.9.0
go: downloading golang.org/x/exp v0.0.0-20230905200255-921286631fa9
go: extracting gopkg.in/ini.v1 v1.67.0
go: extracting github.com/pkg/errors v0.9.1
go: downloading go.uber.org/atomic v1.9.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: extracting github.com/google/uuid v1.5.0
go: downloading github.com/jackc/pgpassfile v1.0.0
go: extracting github.com/Masterminds/goutils v1.1.1
go: downloading github.com/Masterminds/semver/v3 v3.2.0
go: extracting github.com/mitchellh/copystructure v1.0.0
go: downloading github.com/mitchellh/reflectwalk v1.0.0
go: extracting github.com/imdario/mergo v0.3.11
go: downloading github.com/tidwall/gjson v1.17.0
go: extracting github.com/jackc/pgpassfile v1.0.0
go: downloading github.com/huandu/xstrings v1.3.3
go: extracting github.com/mitchellh/reflectwalk v1.0.0
go: extracting github.com/tidwall/gjson v1.17.0
go: downloading github.com/tidwall/pretty v1.2.0
go: downloading github.com/tidwall/match v1.1.1
go: extracting go.uber.org/atomic v1.9.0
go: extracting github.com/ggwhite/go-masker v1.1.0
go: extracting github.com/tidwall/match v1.1.1
go: extracting github.com/tidwall/pretty v1.2.0
go: extracting github.com/jmespath/go-jmespath v0.4.0
go: extracting github.com/Masterminds/semver/v3 v3.2.0
go: extracting github.com/huandu/xstrings v1.3.3
go: extracting golang.org/x/exp v0.0.0-20230905200255-921286631fa9
go: extracting golang.org/x/sys v0.15.0
go: extracting github.com/pelletier/go-toml/v2 v2.1.0
go: finding github.com/rs/zerolog v1.31.0
go: finding github.com/mattn/go-colorable v0.1.13
go: finding github.com/mattn/go-isatty v0.0.19
go: finding golang.org/x/sys v0.15.0
go: finding golang.org/x/sync v0.5.0
go: finding github.com/jackc/pgx/v5 v5.5.1
go: finding github.com/jackc/pgpassfile v1.0.0
go: finding github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a
go: finding golang.org/x/crypto v0.17.0
go: finding golang.org/x/text v0.14.0
go: finding github.com/spf13/cobra v1.8.0
go: finding github.com/spf13/pflag v1.0.5
go: finding github.com/aws/aws-sdk-go v1.49.14
go: finding github.com/jmespath/go-jmespath v0.4.0
go: finding github.com/mitchellh/go-wordwrap v1.0.1
go: finding github.com/olekukonko/tablewriter v0.0.5
go: finding github.com/mattn/go-runewidth v0.0.9
go: finding github.com/ggwhite/go-masker v1.1.0
go: finding github.com/go-faker/faker/v4 v4.2.0
go: finding github.com/google/uuid v1.5.0
go: finding github.com/Masterminds/sprig/v3 v3.2.3
go: finding github.com/Masterminds/goutils v1.1.1
go: finding github.com/Masterminds/semver/v3 v3.2.0
go: finding github.com/huandu/xstrings v1.3.3
go: finding github.com/imdario/mergo v0.3.11
go: finding github.com/mitchellh/copystructure v1.0.0
go: finding github.com/mitchellh/reflectwalk v1.0.0
go: finding github.com/shopspring/decimal v1.3.1
go: finding github.com/spf13/cast v1.6.0
go: finding github.com/tidwall/gjson v1.17.0
go: finding github.com/tidwall/match v1.1.1
go: finding github.com/tidwall/pretty v1.2.0
go: finding github.com/tidwall/sjson v1.2.5
go: finding github.com/pkg/errors v0.9.1
go: finding github.com/jackc/pgx-shopspring-decimal v0.0.0-20220624020537-1d36b5a1853e
go: finding gopkg.in/yaml.v3 v3.0.1
go: finding github.com/spf13/viper v1.18.2
go: finding github.com/fsnotify/fsnotify v1.7.0
go: finding github.com/mitchellh/mapstructure v1.5.0
go: finding github.com/sagikazarmark/locafero v0.4.0
go: finding github.com/sourcegraph/conc v0.3.0
go: finding go.uber.org/multierr v1.9.0
go: finding go.uber.org/atomic v1.9.0
go: finding github.com/spf13/afero v1.11.0
go: finding github.com/sagikazarmark/slog-shim v0.1.0
go: finding golang.org/x/exp v0.0.0-20230905200255-921286631fa9
go: finding github.com/subosito/gotenv v1.6.0
go: finding github.com/hashicorp/hcl v1.0.0
go: finding gopkg.in/ini.v1 v1.67.0
go: finding github.com/magiconair/properties v1.8.7
go: finding github.com/pelletier/go-toml/v2 v2.1.0
build github.com/greenmaskio/greenmask/cmd/greenmask: cannot load embed: malformed module path "embed": missing dot in first path element
make: *** [Makefile:24: build] Error 1

I'm unfamiliar with golang so I wasn't able to fix it. I just installed it via apt and this is the version I'm running locally:

~/greenmask (s3_storage_region_env_fix*) » go version                                       2 ↵ jzanuttoqr@NBVORTX690B18
go version go1.13.8 linux/amd64

If you are able to push a docker image with the branch tag, it would make it easier for me to replicate the scenario on my end.

[joaozanuttoqr]

@wwoytenko an update:

I just tried to replace the s3: configuration with a directory: block, but I got the same error, which is really weird. See below:

root@046b7724d2d2:/home# cat config.yaml
common:
  tmp_dir: /home/temp

directory:
  path: /home/dumps

dump:
  pg_dump_options:
    host: DB_HOST
    dbname: 

restore:
  pg_restore_options:
      host: DB_HOST
    dbname: 
root@046b7724d2d2:/home# greenmask dump --config config.yaml
2024-03-27T19:11:34Z FTL cannot make a backup error="mergeAndWriteToc stage dumping error: s3 object uploading error: MissingRegion: could not find region configuration"
root@046b7724d2d2:/home# 

Am I doing anything wrong? Is there a possibility to configure Greenmask with environment variables that may be overwriting my config.yaml file? (I couldn't find an environment variable configuration section in the docs, but I could contribute with it if it's implemented already)

EDIT: this was done running with the latest dockerhub image version, so doesn't include the modifications in your new branch

[wwoytenko]

Am I doing anything wrong? Is there a possibility to configure Greenmask with environment variables that may be overwriting my config.yaml file?

You've got a problem with config definition. Use the next storage definition:

storage:
  directory:
    path: /home/dumps

It should work fine in this case.

[wwoytenko]

If you are able to push a docker image with the branch tag, it would make it easier for me to replicate the scenario on my end.

Okay, I will push it into public repo

[wwoytenko]

There is docker image with fixes

[wwoytenko]

@joaozanuttoqr please, try to run with the docker image and let me know.
Thank you!

[joaozanuttoqr]

@wwoytenko it seems I was missing the storage: tag for s3 too, I went straight ahead and added it to the s3 config and it worked without updating the docker image (now I got the SSL error which is not working even with disable_ssl: true, but I think this ticket can be closed).

It would help to add this tag to the storage section example in the docs to reduce chance of confusion, as I just copied and pasted mine from there.

Thanks again for your help!

[wwoytenko]

@joaozanuttoqr

I agree with your point about documentation - it is kind of implicit. We will add the explicit storage definition in the docs.

Anyway If you wish, you can contribute it and make a merge request.

I highly appreciate your help! I found a few bugs and added the option no_verify_ssl as well.

[joao-zanutto]

@wwoytenko I've added that contribution with my personal account. I can add the no_verify_ssl flag to the same PR too. Just wanted to confirm with you if the disable_ssl: true option that is described in the docs is implemented

[joao-zanutto]

@wwoytenko I just verified that the disable_ssl option is not implemented in the s3.go file, so I proposed a change into your PR with this one #39, which will lend to less modification on current code/docs. Please let me know your thoughts on it.

[wwoytenko]

You're right. Disable ssl is an artifact without implementation. Let's use NoVerifySsl, I think it's more clear naming in comparison with DisableSsl, because we want to skip certs verification. I will remove this disable_ssl option from code. If you wish you can edit documentation by adding no_verify_ssl option in s3 storage docs with comment like "disable SSL certificates verification"

[joao-zanutto]

Gotcha, I've closed the previous pull request and added a new one changing the disable_ssl to no_verify_ssl in the repo docs. New PR: #40