Skip to content

Commit

Permalink
uploader: reinit TeeReader on upload retry to reset pointer
Browse files Browse the repository at this point in the history
On an upload fail and retry, teeReader would not be reset and would
point to the end of the buffer being uploaded. We need to reset
teeReader to begin reading from the beginning of fileContents again.
  • Loading branch information
emranemran committed Mar 19, 2024
1 parent 7b1111f commit b794ab7
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions core/uploader.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,12 @@ func Upload(input io.Reader, outputURI *url.URL, waitBetweenWrites, writeTimeout
return nil, fmt.Errorf("failed to read file")
}

// To count how many bytes we are trying to read then write (upload) to s3 storage
teeReader := io.TeeReader(bytes.NewReader(fileContents), byteCounter)

var out *drivers.SaveDataOutput
err = backoff.Retry(func() error {
// To count how many bytes we are trying to read then write (upload) to s3 storage
teeReader := io.TeeReader(bytes.NewReader(fileContents), byteCounter)
byteCounter.Reset()

Check failure on line 80 in core/uploader.go

View workflow job for this annotation

GitHub Actions / Build binaries for linux-arm64

byteCounter.Reset undefined (type *ByteCounter has no field or method Reset)

Check failure on line 80 in core/uploader.go

View workflow job for this annotation

GitHub Actions / Run tests defined for the project

byteCounter.Reset undefined (type *ByteCounter has no field or method Reset)

Check failure on line 80 in core/uploader.go

View workflow job for this annotation

GitHub Actions / Build binaries for darwin-arm64

byteCounter.Reset undefined (type *ByteCounter has no field or method Reset)

Check failure on line 80 in core/uploader.go

View workflow job for this annotation

GitHub Actions / Build binaries for linux-amd64

byteCounter.Reset undefined (type *ByteCounter has no field or method Reset)

Check failure on line 80 in core/uploader.go

View workflow job for this annotation

GitHub Actions / Build binaries for darwin-amd64

byteCounter.Reset undefined (type *ByteCounter has no field or method Reset)

out, err = session.SaveData(context.Background(), "", teeReader, fields, segmentWriteTimeout)
if err != nil {
glog.Errorf("failed upload attempt for %s (%d bytes): %v", outputURI.Redacted(), byteCounter.Count, err)
Expand Down

0 comments on commit b794ab7

Please sign in to comment.