You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I wanted to try using this module instead of request because it would handle ESOCKETTIMEDOUT (and more) errors for me. However, there is a big issue when it comes to streaming.
The bl module uses Node's buffer.js. Node buffers have a limit of 256MB. I get the error
when I use this module, even though only pass in the options object as an argument to requestretry.
Here is the culprit in your code. I may fork and fix the code myself, but I am not sure if I have enough time to think of all the things I might break for other users of this module.
A solution, I think, is to listen to the .on( 'end' ... event instead of passing in the callback to request. Of course, sending the callback is fine IF AND ONLY IF the user passes a callback in as well. The callback destroys the opportunity we have to use the excellent streaming interface on the request module.
When downloading large files (and in my case, multiple large files. Perhaps several over 1GB), this module is impossible to use. Once we pass a callback to request, we are telling request to buffer data in memory. By using the streaming interface, we are able to download several huge files because the data goes in and out of memory (drains) as fast as possible.
---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/37570683-always-uses-an-in-memory-buffer-never-allows-streaming-to-disk?utm_campaign=plugin&utm_content=tracker%2F1861872&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1861872&utm_medium=issues&utm_source=github).
The text was updated successfully, but these errors were encountered:
I wanted to try using this module instead of request because it would handle ESOCKETTIMEDOUT (and more) errors for me. However, there is a big issue when it comes to streaming.
The bl module uses Node's buffer.js. Node buffers have a limit of 256MB. I get the error
when I use this module, even though only pass in the options object as an argument to requestretry.
Here is the culprit in your code. I may fork and fix the code myself, but I am not sure if I have enough time to think of all the things I might break for other users of this module.
A solution, I think, is to listen to the
.on( 'end' ...
event instead of passing in the callback to request. Of course, sending the callback is fine IF AND ONLY IF the user passes a callback in as well. The callback destroys the opportunity we have to use the excellent streaming interface on the request module.When downloading large files (and in my case, multiple large files. Perhaps several over 1GB), this module is impossible to use. Once we pass a callback to request, we are telling request to buffer data in memory. By using the streaming interface, we are able to download several huge files because the data goes in and out of memory (drains) as fast as possible.
--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/37570683-always-uses-an-in-memory-buffer-never-allows-streaming-to-disk?utm_campaign=plugin&utm_content=tracker%2F1861872&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1861872&utm_medium=issues&utm_source=github).The text was updated successfully, but these errors were encountered: