Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ speed up image sync by replacing s3cmd with awscli #2789

Merged
merged 2 commits into from
Oct 19, 2023

Conversation

Marigold
Copy link
Contributor

s3cmd is really slow. Syncing images takes more than 3 minutes now, which is considerable time for our staging servers. Switching to awscli reduces the time to about 15s. (I'm not sure why is it so much faster, parallel sync perhaps?)

Moreover, using .aws/config is more intuitive than ~/.s3cfg

@Marigold
Copy link
Contributor Author

Eh, damn. aws s3 sync doesn't transfer public/private permissions. We'd have to run it with --acl public-read, but that would make all objects public. @ikesau do we have any private images?

@ikesau
Copy link
Member

ikesau commented Oct 18, 2023

As far as I know, no.

Images from unpublished articles are "private" in one sense, but I doubt there's anything actually sensitive in them. Worst case, they still have typos or something.

Worth checking with our authors, just to make sure.

@Marigold
Copy link
Contributor Author

Thanks! I went over all images in that bucket and none of them was private. Merging this as it impacts staging server performance a lot.

@Marigold Marigold merged commit 5f4db6c into master Oct 19, 2023
13 checks passed
@Marigold Marigold deleted the switch-to-awscli branch October 19, 2023 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants