Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speedup Object Storage Uploads (Async/Multiprocess) #49

Open
franTarkenton opened this issue Mar 24, 2023 · 1 comment
Open

Speedup Object Storage Uploads (Async/Multiprocess) #49

franTarkenton opened this issue Mar 24, 2023 · 1 comment
Assignees
@franTarkenton
Labels
enhancement New feature or request

Comments

@franTarkenton
Copy link
Member

Currently the upload to object storage works on a single thread. Could significantly reduce upload time by making the upload process async.

See example for how to do that here:
download_granules.py which uses the multiprocess module. Similar approach could be used for syncing to object storage.
@franTarkenton franTarkenton added the enhancement New feature or request label Mar 24, 2023
@franTarkenton
Copy link
Member Author

This was completed as part of refactor of code for the s3 backup process

@franTarkenton franTarkenton self-assigned this Jun 9, 2023
@franTarkenton franTarkenton moved this to Done in RFC Backlog Dec 5, 2023
@franTarkenton franTarkenton added the zen_reorg temp label to help with migration to ghp label Dec 5, 2023
@franTarkenton franTarkenton moved this from Done to Review / QA in RFC Backlog Dec 5, 2023
@franTarkenton franTarkenton removed the zen_reorg temp label to help with migration to ghp label Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@franTarkenton franTarkenton

Labels
enhancement New feature or request
Projects
RFC Backlog
Status: Review / QA
Milestone
No milestone
Development

No branches or pull requests
1 participant
@franTarkenton