Add '--tamper-proof' flag and Azure Confidential Ledger Logic

[PallabPaul]

Description

The goal of this PR is to introduce a new flag --tamper-proof. The design document to accompany this PR can be found here.

The --tamper-proof flag introduces the ability for AzCopy to store the calculated MD5 Hash to an Azure Confidential Ledger (ACL) instance when using the --put-md5 flag. In addition, the flag will retrieve and compare the stored MD5 Hash with the re-calculated hash when using the '--check-md5' flag.

Implementation

Aside from adding the --tamper-proof flag to the necessary Job Part files and common library, the main logic for the ACL and tamper-proof checks can be found in ste/ledgerHelper.go. This is where the main logic and calls to ACL can be found, specifically in the uploadHash and downloadHash methods.

Testing

More tests will be added soon but so far, the upload/download of the MD5 hash is working as expected.

To prepare for tests you can:

1. [Create a ledger instance].(https://ms.portal.azure.com/#view/Microsoft_Azure_ConfidentialLedger/CreateForm_dx/dxParameters~/%7B%7D)
2. Build an executable go build -o test_azcopy.exe.
3. Set your login type export AZCOPY_AUTO_LOGIN_TYPE=AZCLI.

Upload

Command:
./test_azcopy.exe copy "<local_file>" "<remote_storage_account>" --put-md5 --recursive --tamper-proof "<ledger_url>"

Example:
./test_azcopy.exe copy "tool_clean.sh" "https://blobmanagedapptest.blob.core.windows.net/testaz" --put-md5 --recursive --tamper-proof "https://a79test1ledger20240710T061149652.confidential-ledger.azure.com"

Command Successfully Executed:

MD5 Hash Successfully Stored in ACL:

Download

Command:
./test_azcopy.exe copy "<remote_storage_account>" "<local_file>" --check-md5 FailIfDifferentOrMissing --recursive --tamper-proof "<ledger_url>"

Example:
./test_azcopy.exe copy "https://blobmanagedapptest.blob.core.windows.net/testaz" "testaz" --check-md5 FailIfDifferentOrMissing --recursive --tamper-proof "https://a79test1ledger20240710T061149652.confidential-ledger.azure.com"

Command Successfully Executed and Hash Matching:

Logs of Hash Matching:

Command Successfully Executed and Hash Not Matching:

Logs of Hash Not Matching: