Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ speed up image sync by replacing s3cmd with awscli #2789

Merged
merged 2 commits into from
Oct 19, 2023
Merged

✨ speed up image sync by replacing s3cmd with awscli #2789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
157fe73
:sparkles: speed up image sync by replacing s3cmd with awscli
Marigold Oct 18, 2023
a516f22
make objects readable
Marigold Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion devTools/docker/sync-s3-images.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ fi
# for local development, it should be owid-image-upload/local-yourname
# at least until we decide to instead host images locally, if ever

s3cmd sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/
aws --endpoint=https://nyc3.digitaloceanspaces.com s3 sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/ --acl public-read
9 changes: 8 additions & 1 deletion site/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,11 @@ This means that any other documents that reference the image will use the update

If you are refreshing your environment's database by importing a database dump from prod, the prod `images` table may make claims about the existence of files in your environment's S3 folder that aren't true, which will lead to 403 errors when trying to bake.

In this project's root Makefile, we have a make command (`make sync-images`) that runs `s3cmd sync` from prod to your environment to solve this problem. Make sure your [s3cmd is configured correctly](https://docs.digitalocean.com/products/spaces/reference/s3cmd/) before running it.
In this project's root Makefile, we have a make command (`make sync-images`) that runs `aws s3 sync` from prod to your environment to solve this problem. Make sure your `~/.aws/config` is configured correctly and contains

```
[owid]
aws_access_key_id = xxx
aws_secret_access_key = xxx
region = nyc3
```
Loading