Repo-syncer script

[ferricoxide]

Closes #1524

[lorengordon]

This seems like an awful lot of logic just to sync an s3 repo. Gimme a bit to fix the permissions, and you'll be able to just do this:

aws --no-sign-request --endpoint-url https://watchmaker.cloudarmor.io s3 sync --delete --exact-timestamps s3://releases/ ./wam/

For example, this works currently:

aws --no-sign-request --endpoint-url https://watchmaker.cloudarmor.io s3 cp s3://releases/latest/watchmaker-latest-standalone-linux-x86_64 ./wam/watchmaker-latest-standalone-linux-x86_64

[ferricoxide]

This seems like an awful lot of logic just to sync an s3 repo.

Use-case in question is not an AWS shop (thus, does not have the AWS utilities on their systems or authorized for use) – or any type of public-cloud user.

Gimme a bit to fix the permissions, and you'll be able to just do this:

aws --no-sign-request --endpoint-url https://watchmaker.cloudarmor.io s3 sync --delete --exact-timestamps s3://releases/ ./wam/

For example, this works currently:

aws --no-sign-request --endpoint-url https://watchmaker.cloudarmor.io s3 cp s3://releases/latest/watchmaker-latest-standalone-linux-x86_64 ./wam/watchmaker-latest-standalone-linux-x86_64

Not saying don't do the above: it'd be nice to also be able to do that in suitable environments. 😃

[lorengordon]

Ok, this is working now:

aws --no-sign-request --endpoint-url https://watchmaker.cloudarmor.io s3 ls s3://watchmaker/

Basically any tool that "understands" the s3 api and supports setting a custom endpoint, such as awscli or rclone, will work. ListBucket and GetObject are allowed currently. No profile or credential is necessary.

I'm a bit hesitant on the idea of signing up to maintain a custom script in this project just because some org is so massively backwards that no such tooling can be used. It feels like exactly the kind of thing that org should be maintaining in their own environment... I think, at the very least, to be comfortable keeping it here, we'd need a test that asserts expected functionality/behavior and is executed by CI. And probably I'll end up being super nitpicky about the script structure and var/function names to make it as clean and easy to understand as possible...

[ferricoxide]

The other (AWS-based) environment I thought I might be able to use this in is even more jacked-up than the (non-AWS) environment I originally wrote this script for. So, obviously not a broadly-applicable enough of a work-around to contribute. Withdrawing.