Skip to content

Commit

Permalink
Invert default chunk upload logic
Browse files Browse the repository at this point in the history
  • Loading branch information
radito3 committed Oct 20, 2023
1 parent 8636131 commit f86b520
Show file tree
Hide file tree
Showing 4 changed files with 14 additions and 14 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ The configuration of the MultiApps CF plugin is done via env variables. The foll
* `MULTIAPPS_UPLOAD_CHUNK_SIZE=<POSITIVE_INTEGER>` - By default, large MTARs are not uploaded as a single unit, but are split up into smaller chunks of 45 MBs that are uploaded separately. The goal is to prevent failed uploads due to [gorouter](https://github.com/cloudfoundry/gorouter)'s request timeout. In case the default chunk size is still too large, you can configure it via this environment variable. **The specified values are in megabytes.**
:rotating_light: Note: The total number of chunks in which an MTAR is split cannot exceed 50, since the multiapps-controller could interpret bigger values as a denial-of-service attack. For this reason, the minimum value for this environment variable is computed based on the formula: `MIN = MTAR_SIZE / 50`
For example, with a 100MB MTAR the minimum value for this environment variable would be 2, and for a 400MB MTAR it would be 8. Finally, the minimum value cannot grow over 50, so with a 4GB MTAR, the minimum value would be 50 and not 80.
* `MULTIAPPS_UPLOAD_CHUNKS_IN_PARALLEL=<BOOLEAN>` - By default, MTAR chunks are uploaded sequentially so as not to produce too much network traffic. In case of a good internet connection, the option to upload them in parallel may improve upload performance.
* `MULTIAPPS_UPLOAD_CHUNKS_SEQUENTIALLY=<BOOLEAN>` - By default, MTAR chunks are uploaded in parallel for better performance. In case of a bad internet connection, the option to upload them sequentially will lessen network load.

# How to contribute
* [Did you find a bug?](CONTRIBUTING.md#did-you-find-a-bug)
Expand Down
6 changes: 3 additions & 3 deletions commands/file_uploader.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ type FileUploader struct {
mtaClient mtaclient.MtaClientOperations
namespace string
uploadChunkSizeInMB uint64
isParallel bool
sequentialUpload bool
}

type progressBarReader struct {
Expand Down Expand Up @@ -68,7 +68,7 @@ func NewFileUploader(mtaClient mtaclient.MtaClientOperations, namespace string,
mtaClient: mtaClient,
namespace: namespace,
uploadChunkSizeInMB: uploadChunkSizeInMB,
isParallel: configuration.NewSnapshot().GetUploadChunksInParallel(),
sequentialUpload: configuration.NewSnapshot().GetUploadChunksSequentially(),
}
}

Expand Down Expand Up @@ -170,7 +170,7 @@ func (f *FileUploader) uploadInChunks(fileToUpload *os.File) ([]*models.FileMeta
}
uploadedFilesChannel <- file
}()
if !f.isParallel {
if f.sequentialUpload {
if err := waitForChannelData(uploadedFilesChannel, errorChannel, &uploadedFileParts); err != nil {
return nil, err
}
Expand Down
16 changes: 8 additions & 8 deletions configuration/configuration.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ import (
const unknownError = "An unknown error occurred during the parsing of the environment variable \"%s\". Please report this! Value type: %T"

type Snapshot struct {
backendURL properties.ConfigurableProperty
uploadChunkSizeInMB properties.ConfigurableProperty
uploadChunksInParallel properties.ConfigurableProperty
backendURL properties.ConfigurableProperty
uploadChunkSizeInMB properties.ConfigurableProperty
uploadChunksSequentially properties.ConfigurableProperty
}

func NewSnapshot() Snapshot {
return Snapshot{
backendURL: properties.BackendURL,
uploadChunkSizeInMB: properties.UploadChunkSizeInMB,
uploadChunksInParallel: properties.UploadChunksInParallel,
backendURL: properties.BackendURL,
uploadChunkSizeInMB: properties.UploadChunkSizeInMB,
uploadChunksSequentially: properties.UploadChunksSequentially,
}
}

Expand All @@ -30,8 +30,8 @@ func (c Snapshot) GetUploadChunkSizeInMB() uint64 {
return getUint64Property(c.uploadChunkSizeInMB)
}

func (c Snapshot) GetUploadChunksInParallel() bool {
return getBoolProperty(c.uploadChunksInParallel)
func (c Snapshot) GetUploadChunksSequentially() bool {
return getBoolProperty(c.uploadChunksSequentially)
}

func getStringProperty(property properties.ConfigurableProperty) string {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ package properties

import "strconv"

var UploadChunksInParallel = ConfigurableProperty{
Name: "MULTIAPPS_UPLOAD_CHUNKS_IN_PARALLEL",
var UploadChunksSequentially = ConfigurableProperty{
Name: "MULTIAPPS_UPLOAD_CHUNKS_SEQUENTIALLY",
Parser: booleanParser{},
ParsingSuccessMessage: "Attention: You've specified %v for the environment variable %s.\n",
ParsingFailureMessage: "Invalid boolean value (%s) for environment variable %s. Using default value %v.\n",
Expand Down

0 comments on commit f86b520

Please sign in to comment.