You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
So that we can support retries on part uploads after failures, an entire part must be stored in the circular buffer.
We had a user who attempted to upload a 1.7 PB file. With the limit of 10,000 parts per upload, that means that each buffer must be 170 GB. That is an unreasonable amount of memory especially considering each streaming thread for each upload has its own buffer.
We should do the following:
When calculating the number of parts based on the file size and circular buffer size, check to see if we need more than 10,000 parts. (This should be done no matter what even if all we did was throw an error.)
If we calculate that we would need more than 10,000 parts:
Update the part sizes so that there are 10,000 parts or less.
Drop the requirement that the full part must be in memory.
Drain the circular buffer as we are streaming the bytes from S3.
Do not support retries if the part fails.
The text was updated successfully, but these errors were encountered:
Turns out the user was attempting to upload a 1.7TB file, but that doesn't change the fact we need to add logic for making sure the total number of parts does not exceed 10000.
Also, AWS S3 doesn't support objects exceeding 5TB.
So that we can support retries on part uploads after failures, an entire part must be stored in the circular buffer.
We had a user who attempted to upload a 1.7 PB file. With the limit of 10,000 parts per upload, that means that each buffer must be 170 GB. That is an unreasonable amount of memory especially considering each streaming thread for each upload has its own buffer.
We should do the following:
The text was updated successfully, but these errors were encountered: