Support a new package type s3

[suzuki-shunsuke]

Feature Overview

Support a new package type s3.

Why is the feature needed?

To install private tools from S3.

• hold other things in registry #1929 (comment)

⚠️ Note that the object URI must conform to semver.

Does the feature include Breaking Changes?

No.

Example Code

registry.yaml

packages:
  - name: mytools/yoo
    type: s3
    bucket: mybucket
    path: mytools/yoo-{{.Version}}-{{.OS}}-{{.Arch}}.{{.Format}}
    format: tar.gz

Authentication

Authenticate with AWS SDK for Go.

https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/config

Reference

• https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html

[gedw99]

will come back to this :) just busy on other stuff. thanks for making a discussion...

[gedw99]

Hey @suzuki-shunsuke

i read on your Blog that a refs toting is planned. Actually o am really happy about this because it’s hard to work on the code base as it is.

regarding s3 it might be useful to use the minio mc cli as it supports all s3 implementions that are common like google , Amazon and minio.

https://github.com/minio/mc#example---amazon-s3-cloud-storage

https://github.com/minio/mc#example---google-cloud-storage

The other thing I was thinking about is a s3 proxy, that works like the aqua proxy. The idea here is that registry files get huge I noticed. The yaml we can’t reduce but he way the Aqua proxy only pulls the actual binary when you call for it is what I mean. We should match that with the s3 provider so you can have huge s3 stores but your local machine only needs to hold the ones you pulled.

[suzuki-shunsuke]

i read on your Blog that a refs toting is planned.

What Blog did you read? What's refs toting?

The other thing I was thinking about is a s3 proxy, that works like the aqua proxy.

Unfortunately, I can't understand your idea well.

The idea here is that registry files get huge I noticed.

Yeah, this is a potential issue of aqua's current architecture.
There is an issue related to this.

• Support extracting required packages from standard registry and generate local registry to improve performance #685

But this potential issue has nothing to do with the new package type s3.