Deleting dataset before deleting the pod referring to the dataset leads to orphaned DatasetInternal objects

[srikumar003]

Relates to #341

More description to follow

[srikumar003]

Describing Persistent Volume Claims displays a "Used By" field which can be used to determine if any pods are referencing the PVC (and hence the dataset).

kubectl describe pvc example-dataset

returns

Name:          example-dataset 
Namespace:     xxxx
StorageClass:  csi-s3
Status:        Bound
Volume:       yyyy
Labels:        dataset=example-dataset
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ch.ctrox.csi.s3-driver
               volume.kubernetes.io/storage-provisioner: ch.ctrox.csi.s3-driver
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      9765625000Ki
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       <none>
Events:        <none>

Relevant issue

We can use the same logic used by kubectl describe - i.e. loop through the pods in the namespace and add the name of the pods which have a PVC annotation for the dataset/PVC name to a list. if list is not empty, dataset cannot be detected.

Also, use this logic to populate a similar used-by field for the dataset.

[srikumar003]

The Dataset custom resource creates a DatasetInternal resource which actually "owns" the resources (PVC, ConfigMaps) etc created for the former.

The finaliser of the DatasetInternal checks if the PVC is referenced before deletion and does not exit until this is deleted.

However, Dataset has no finalizer present, so it's deletion goes through, leaving the DatasetInternal object orphaned.

To resolve this issue, we need to catch a DELETE request and check if the corresponding Internal instance has been deleted, or else deny the request