Skip to content

Commit

Permalink
uploader: reinit TeeReader on upload retry to reset pointer
Browse files Browse the repository at this point in the history
On an upload fail and retry, teeReader would not be reset and would
point to the end of the buffer being uploaded. We need to reset
teeReader to begin reading from the beginning of fileContents again.
  • Loading branch information
emranemran committed Mar 19, 2024
1 parent 7b1111f commit 5b51b98
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions core/uploader.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,12 @@ func Upload(input io.Reader, outputURI *url.URL, waitBetweenWrites, writeTimeout
return nil, fmt.Errorf("failed to read file")
}

// To count how many bytes we are trying to read then write (upload) to s3 storage
teeReader := io.TeeReader(bytes.NewReader(fileContents), byteCounter)

var out *drivers.SaveDataOutput
err = backoff.Retry(func() error {
// To count how many bytes we are trying to read then write (upload) to s3 storage
teeReader := io.TeeReader(bytes.NewReader(fileContents), byteCounter)
byteCounter.Count = 0

Check warning on line 81 in core/uploader.go

View check run for this annotation

Codecov / codecov/patch

core/uploader.go#L78-L81

Added lines #L78 - L81 were not covered by tests
out, err = session.SaveData(context.Background(), "", teeReader, fields, segmentWriteTimeout)
if err != nil {
glog.Errorf("failed upload attempt for %s (%d bytes): %v", outputURI.Redacted(), byteCounter.Count, err)
Expand Down

0 comments on commit 5b51b98

Please sign in to comment.