From 5f4db6c08c9847a9bbb0712b1067873e4cd4ebf2 Mon Sep 17 00:00:00 2001 From: Mojmir Vinkler Date: Thu, 19 Oct 2023 13:04:31 +0200 Subject: [PATCH] :sparkles: speed up image sync by replacing s3cmd with awscli (#2789) * :sparkles: speed up image sync by replacing s3cmd with awscli --- devTools/docker/sync-s3-images.sh | 2 +- site/README.md | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/devTools/docker/sync-s3-images.sh b/devTools/docker/sync-s3-images.sh index 98df8f6c39f..b2a8a06987a 100755 --- a/devTools/docker/sync-s3-images.sh +++ b/devTools/docker/sync-s3-images.sh @@ -13,4 +13,4 @@ fi # for local development, it should be owid-image-upload/local-yourname # at least until we decide to instead host images locally, if ever -s3cmd sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/ +aws --endpoint=https://nyc3.digitaloceanspaces.com s3 sync s3://owid-image-upload/production/ s3://$IMAGE_HOSTING_BUCKET_PATH/ --acl public-read \ No newline at end of file diff --git a/site/README.md b/site/README.md index 8dc41a43da6..5a4e94a1a59 100644 --- a/site/README.md +++ b/site/README.md @@ -59,4 +59,11 @@ This means that any other documents that reference the image will use the update If you are refreshing your environment's database by importing a database dump from prod, the prod `images` table may make claims about the existence of files in your environment's S3 folder that aren't true, which will lead to 403 errors when trying to bake. -In this project's root Makefile, we have a make command (`make sync-images`) that runs `s3cmd sync` from prod to your environment to solve this problem. Make sure your [s3cmd is configured correctly](https://docs.digitalocean.com/products/spaces/reference/s3cmd/) before running it. +In this project's root Makefile, we have a make command (`make sync-images`) that runs `aws s3 sync` from prod to your environment to solve this problem. Make sure your `~/.aws/config` is configured correctly and contains + +``` +[owid] +aws_access_key_id = xxx +aws_secret_access_key = xxx +region = nyc3 +```