Finalize issues of S3 proxy features to go live

[chuckbelisle]

Refer to Blob-CSI epic #1001 and move forward with go-live of Blob-CSI and S3 proxy features.

Follow the confirmation of the feature functioning in Prod we will need to action the following

• Update our documentation (See epic #1001) and have it reviewed (I would like this to be lead by Rohan once the knowledge transfer is done)
• Review and plan out deprecating MinIO
• [ ]

[Collinbrown95]

Final Development Items

• feat(central-dashboard): add s3 icon aaw-kubeflow-manifests#309 Add s3 menu button to central dashboard
• https://github.com/StatCan/aaw-argocd-manifests/pull/269 Make sure s3proxy profiles controller is deployed in dev
• https://github.com/StatCan/aaw-argocd-manifests/pull/274

TODO

• Create a pro-b and unclassified version of the application for aws-js-s3-explorer. Reason is purely for UX improvement - the unclassified s3proxy instance that is exposed to the kubeflow UI does not mount the protected-b volume. However, there is only one UI for aws-js-s3-explorer, and the protected-b option is still shown even though there does not exist a protected-b volume mounted to the unclassified s3proxy instance. To improve UX, I propose we just add a line to the init container for each deployment to use sed to remove the irrelevant menu options in each case (see https://github.com/StatCan/aws-js-s3-explorer/blob/main/index.html#L78-L82). E.g. add something like sed -i '/option data-i18n base-value="unclassified"/d' index.html to the init container for copying the static files to the nginx container.

• Verify that the appropriate volumes are mounting correctly to unclassified and protected-b instances of s3proxy.

  • Need to ensure that there is persistence across s3proxy restarts (should work in theory but has not been tested yet).

• Need default virtualservice for the "Storage" button on the kubeflow interface.

  • There needs to be a "404" page for users who have not opted in to s3proxy rather than a blanc "no route found" message from kubeflow.

• Need to figure out how to enable downloads from the s3 aws js explorer UI when the user is accessing the UI through the Kubeflow iFrame

  • Needs to be some kind of virtual service / nginx route to catch these cases.
  • Downloads work correctly when accessed via s3proxy-web service in a novnc notebook

[Collinbrown95]

Testing and Documentation

1. Remove s3proxy from a user namespace that no longer requires it

Case 1: Profile is created from https://github.com/StatCan/aaw-kubeflow-profiles

I created the collin-test profile for this test.

1. In the jsonnet file for the profile, wrap the profile in the addS3 function (see example for my profile here: https://github.com/StatCan/aaw-kubeflow-profiles/commit/a5d4883e678f04c6e245ef9e413c3c5c2952bbcf)

2. User logs into kubeflow (dev) with usual SSO

3. kubeflow central dashboard configmap populates "Storage" option on the left menu of Kubeflow

4. User clicks "Storage" and is presented the aws-js-s3-explorer UI. User must select Unclassified, Unclassified read only, or Protected-b

Note: On the first page load, the service worker may not get loaded into the browser. This can lead to a false "403" forbidden error message. As described in the s3proxy developer docs, we have a service worker that removes an erroneous authorization header that is added automatically by the client-side s3 javascript library used by the aws-js-s3-explorer tool.

Case 1.1: Select "Unclassified" bucket

1. User creates test prefix (folder) in s3 explorer, folder is created successfully.

2. User uploads moderately large single file (122 MB), file is uploaded successfully.

Note: The UI is a bit finicky with file uploads; the progress bar shows 0% progress and turns green when the file upload completes. The file is uploaded correctly, but the UI appears as though no progress has been made.

Note: the aws-js-s3-explorer UI does not currently allow users to download files through the kubeflow interface. The issue is that the request URL in the aws js s3 explorer application does not have the /s3/<user namespace> prefix; this works correctly in a noVNC notebook when requests are made directly against the s3proxy-web service in the cluster. For example, in a noVNC notebook, users can visit s3proxy-web/s3/<user-namespace>/ (or s3proxy-protected-b-web/s3/<user-namespace>/ in a protected b environment) and connect directly to the s3proxy pod within the cluster.

[Jose-Matsuda]

Mathis and Jose work

In relation to Collin's comment here. ✔️ means good to go ❌ means problem (and in this case so does ❗, but was not in Collin's initial comment ). ⭕ means task to be completed.

✔️ Removing Unusable Items from the Menu

The PR just needed slight tweaking as initially it complained about not being able to find the index.html. Fixing it up results in this. As a result when navigating to it on the kubeflow interface you only get unclassified. What we do need to verify is how to get to the protected b version while in novnc and we should only see unclassified-ro and protectedb. --> We accessed this during tech elab using the internal service of s3proxy-protected-b-web and were able to verify seeing those two volumes.

✔️ Verify that the appropriate volumes are mounting correctly to unclassified and protected-b instances of s3proxy.

To test this I started up a few notebooks. The PVC's of aaw-protected-b, aaw-unclassified-ro only got mounted to protected b pods and aaw-unclassified was only mounted to non protected b pods.

❌Need to ensure that there is persistence across s3proxy restarts

However, note that if you delete the s3proxy pod in your namespace (or restart the deployment) any files that were uploaded are gone.

⭕ VS for storage button

• Create default page for non-opted in s3proxy users #1641

❗ IMPORTANT Things to Note

We are able to access kubeflow central dashboard from the protected b remote desktop and as a result, can interact with the unclassified volume. You are also able to upload to the unclassifed-ro volume.

[Collinbrown95]

Notes from debugging session with @Jose-Matsuda and @mathis-marcotte : https://github.com/StatCan/daaas-private/issues/80

Discussion includes some networking details which is why it is outlined in a private repo.

[mathis-marcotte]

Closing this issue. The work will progress through other issues, including

• Create default page for non-opted in s3proxy users #1641
• S3Proxy fix volume mounting  #1648
• Fix s3proxy explorer upload progress bar when in iframe #1650
• Enable downloads from the s3 explorer UI when the user is accessing the UI through the Kubeflow iFrame #1651
• https://github.com/StatCan/daaas-private/issues/80
• Fix s3 proxy New Folder button #1658