Skip to content

Commit

Permalink
✨ speed up image sync by replacing s3cmd with awscli (#2789)
Browse files Browse the repository at this point in the history
* ✨ speed up image sync by replacing s3cmd with awscli
  • Loading branch information
Marigold authored Oct 19, 2023
1 parent 8436b31 commit 5f4db6c
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 2 deletions.
2 changes: 1 addition & 1 deletion devTools/docker/sync-s3-images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ fi
# for local development, it should be owid-image-upload/local-yourname
# at least until we decide to instead host images locally, if ever

s3cmd sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/
aws --endpoint=https://nyc3.digitaloceanspaces.com s3 sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/ --acl public-read
9 changes: 8 additions & 1 deletion site/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,11 @@ This means that any other documents that reference the image will use the update

If you are refreshing your environment's database by importing a database dump from prod, the prod `images` table may make claims about the existence of files in your environment's S3 folder that aren't true, which will lead to 403 errors when trying to bake.

In this project's root Makefile, we have a make command (`make sync-images`) that runs `s3cmd sync` from prod to your environment to solve this problem. Make sure your [s3cmd is configured correctly](https://docs.digitalocean.com/products/spaces/reference/s3cmd/) before running it.
In this project's root Makefile, we have a make command (`make sync-images`) that runs `aws s3 sync` from prod to your environment to solve this problem. Make sure your `~/.aws/config` is configured correctly and contains

```
[owid]
aws_access_key_id = xxx
aws_secret_access_key = xxx
region = nyc3
```

0 comments on commit 5f4db6c

Please sign in to comment.