Skip to content

Commit

Permalink
Add instructions for copying benchmarks to webapp (#113)
Browse files Browse the repository at this point in the history
  • Loading branch information
AlexAxthelm authored Nov 19, 2024
1 parent 37c30e4 commit 1299c90
Showing 1 changed file with 64 additions and 0 deletions.
64 changes: 64 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,67 @@ echo $GITHUB_PAT | docker login ghcr.io -u <USERNAME> --password-stdin
```
which will run the script with the defined configuration, and populate files into the paths specified.
## Promotion of Datasets to PROD
### `workflow.transiton.monitor`
Data sets to prepare images from [`workflow.transition.monitor`](https://github.com/RMI-PACTA/workflow.transition.monitor/) are stored in the `pactadatadev` Storage Account (`RMI-SP-PACTA-DEV` Resource Group), in the file share `workflow-prepare-pacta-indices-outputs`.
The dataset used is defined by the direcotry name, which is an MD5 hash of the inputs, including the code used in `workflow.transition.monitor` to generate the results.
### `workflow.pacta.webapp` and `workflow.pacta.dashboard`
For the [workflow.pacta.webapp](https://github.com/RMI-PACTA/workflow.pacta.webapp/) and [workflow.pacta.dashboard](https://github.com/RMI-PACTA/workflow.pacta.dashboard/) images, the benchmark data is expected as a bind mount to the docker image (rather than "baked in" in `/pacta-data`, as with `workflow.transition.monitor`).
For Azure Container Instances running on our tenant, the expected file share to mount is `benchmark-data`, in the `rmipactawebappdata` Storage Account (in the `RMI-SP-PACTA-WEU-PAT-DEV` Resource Group).
The top level directories in that File Share correspond to the directories in the `pactadatadev/workflow-prepare-pacta-indices-outputs` file share, and should be passed as environment variables to the docker image (see workflow repos for more detail).
### Transferring from `pactadatadev` to `rmipactawebappdata`
Prepared datasets can be copied from `pactadatadev` to `rmipactawebappdata` with the following commands:
```sh
# Change as needed.
DIRNAME="65c1a416721b22a98c7925999ae03bc4"
TOKEN_START=$(date -u -j '+%Y-%m-%dT%H:%MZ')
TOKEN_EXPIRY=$(date -u -j -v "+20M" '+%Y-%m-%dT%H:%MZ')
DESTINATION_ACCOUNT_NAME="rmipactawebappdata"
DESTINATION_SHARE="benchmark-data"
DESTINATION_SAS="$(
az storage share generate-sas \
--account-name $DESTINATION_ACCOUNT_NAME \
--expiry $TOKEN_EXPIRY \
--permissions rcw \
--name $DESTINATION_SHARE \
--start $TOKEN_START \
--output tsv
)"
# note permissions are different. rcl allows listing contents, rcw above is to write
SOURCE_ACCOUNT_NAME="pactadatadev"
SOURCE_SHARE="workflow-prepare-pacta-indices-outputs"
SOURCE_SAS="$(
az storage share generate-sas \
--account-name $SOURCE_ACCOUNT_NAME \
--expiry $TOKEN_EXPIRY \
--permissions rcl \
--name $SOURCE_SHARE \
--start $TOKEN_START \
--output tsv
)"
COPY_SOURCE="https://$SOURCE_ACCOUNT_NAME.file.core.windows.net/$SOURCE_SHARE/$DIRNAME"?$SOURCE_SAS
COPY_DESTINATION="https://$DESTINATION_ACCOUNT_NAME.file.core.windows.net/$DESTINATION_SHARE/$DIRNAME?$DESTINATION_SAS"
echo "$COPY_SOURCE"
echo "$COPY_DESTINATION"
azcopy copy \
"$COPY_SOURCE" \
"$COPY_DESTINATION" \
--as-subdir=false \
--recursive
```

0 comments on commit 1299c90

Please sign in to comment.