Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding download & unzipping progress bar & download as stream #49

Closed
wants to merge 1 commit into from
Closed

adding download & unzipping progress bar & download as stream #49

wants to merge 1 commit into from

Conversation

luhangsnn
Copy link

@luhangsnn luhangsnn commented Dec 20, 2021

(#34)
added downloading and unzipping progress bar to the download method that displays the file size, chunk size, and estimated finishing time.

image
minor issue for unzipping: when running on jupyter notebook, the output will first print a separate progress bar that will end at ~1% and then the proper progress bar for unzipping. This issue does not happen when testing in powershell terminal - have not found a solution to this jupyter-specific issue yet

I believe it also splits the download into chunks - 1024 bytes at a time (download as stream?) so that we don't need to hold the whole dataset in memory when downloading.
with requests.get(url, stream=True) as r
then
for chunk in r.iter_content(chunk_size=1024):

minor issue for unzipping: when running on jupyter notebook, the output will first print a separate progress bar that will end at ~1% and then the proper progress bar for unzipping. This issue does not happen when testing in powershell terminal - have not found a solution to this jupyter-specific issue yet
@luhangsnn luhangsnn changed the title adding download & unzipping progress bar adding download & unzipping progress bar & download as stream Dec 20, 2021
@nalmadi
Copy link
Owner

nalmadi commented Dec 21, 2021

Thank you! You made a 2nd pull-request so I am closing this one.

@nalmadi nalmadi closed this Dec 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants