Blob CSI: Pro-b Unclassified-ro bucket usage

[Souheil-Yazji]

Describe the bug

Env: prod

What is the usage process for the pro-b unclassified-ro bucket? The bucket is un-writable and the docs don't point out how contents can be stored there. Tried upload/mv/cp/touch and all fail due to permissions (expected). Tried adding contents to the aaw-unclassified bucket on an unclassified notebook in hopes that this content would be available as -ro on a pro-b notebook but that didn't work, bucket was still empty

Expected behaviour

There should be a way to have data in that bucket, otherwise it will always be empty.

Screenshots

Additional context

Resolving PRs:
StatCan/aaw-kubeflow-profiles-controller#119
StatCan/charts#430
https://github.com/StatCan/aaw-argocd-manifests/pull/326
https://github.com/StatCan/aaw-argocd-manifests/pull/327

[mathis-marcotte]

Testing on my end, I encountered 2 different scenarios. These were both in DEV.

In my "mathis-marcotte" namespace, it worked as expected. In my unclassified notebook, I could only see the aaw-unclassified bucket, and I could add files in that folder. Then, when I would go in the protected-b notebook, I could only see the aaw-protected-b and aaw-unclassified-ro buckets, and the contents of my aaw-unclassified bucket were duplicated in the -ro bucket. I also couldn't add files in the -ro bucket, but i could read them.

In my "mathis-test" namespace, it worked as expected for the unclassified bucket. It was the only one that appeared while in an unclassified notebook, and I could read and write to it. But, when going in the protected-b notebook, the files from aaw-unclassified were not copied over to the -ro bucket. Instead, it was the files from the protected-b bucket that were getting copied over to the -ro bucket, which is not suppose to be the case. Although the -ro bucket still had the correct permissions and I could read but not write.

The only real difference that I could notice between how those two namespaces are defined is that at the profile level, my "mathis-marcotte" namespace has the "s3.statcan.gc.ca/enabled=true" label.

One thing to investigate might be to confirm that all the volumes are getting mounted correctly and as expected.
We had worked on the mounting for those volumes in a previous issue as they were not behaving as expected. Maybe the work got overwritten or wasn't fully deployed up to prod properly or something like that. #1648

[chuckbelisle]

pv's are being pointed to the wrong storage account. The controller needs to be modified.

[Souheil-Yazji]

Resolved by StatCan/aaw-kubeflow-profiles-controller#119

[Souheil-Yazji]

PVs must be recreated to reconfigure the storage account. No impact to users as the PV will just point to a different bucket in RO mode, no change in actual data. Should be scheduled maintenance.