[PROPOSAL] Apply GPU acceleration to ML Node in k8s operator

[YeonghyeonKO]

What are you proposing?

As @stevapple has already suggested #832 how to override CUDA image in nodePools[] in opensearch k8s oprator, this issue is an extended discussion with that concept.

1. Provision GPU-dedicated Worker Node(s) in the existing K8s cluster.
2. Prepare CUDA-based image for OpenSearch
3. Deploy OpenSearch ml-node(s) using nodeSelector in opensearch k8s operator.
   • CR has already offered us nodeSelector property as below:

  nodePools:
    - component: ml
      replicas: 3
      diskSize: "10Gi"
      nodeSelector: # <- assign ml node(ie. Pod) to GPU Worker Node 
      roles:
        - "ml"
      resources:
        requests:
        limits:

4. Test whether ML Node does use GPU resource or not.

But this is all for now because we can't set a GPU-enabled image for opensearch, except opensearchCluster.general.image in yaml.
Is there any further progress in this idea?
For implementing GPU acceleration - OpenSearch Documentation, we need guidelines and an exact standard for it.

[prudhvigodithi]

[Triage]
Hey @YeonghyeonKO coming from my previous comment #832 (comment), today CUDA built-in OpenSearch images are not officially release by the project (related issue opensearch-project/opensearch-build#4743), is there already an open source OpenSearch CUDA supported image ?

Coming from operator side yes its a limitation, custom image at nodePool level spec.nodePools[0].image is not supported in the NodePool struct, is there a way you can PR this change ?

Regarding GPU acceleration - OpenSearch Documentation, Adding @dblock to provide some guidelines.

Thank you
@stevapple @getsaurabh02 @peterzhuamazon @gaiksaya @bshien @swoehrl-mw

[peterzhuamazon]

My thoughts:
opensearch-project/opensearch-build#4743 (comment)

[prudhvigodithi]

Adding @vamshin to provide some thoughts on CUDA OpenSearch development, more than operator I feel this issue should be part of ml-commons repo to continue the discussion related to GPU acceleration - OpenSearch Documentation.
Thank you

[vamshin]

@YeonghyeonKO could you please help us with use cases for GPU with ml nodes?

[YeonghyeonKO]

@prudhvigodithi Thanks for continuing the discussion. I totally agree with you that it's prior to build cuda image than nodePools[] thing.

[YeonghyeonKO]

@vamshin Sure, I would happily help as a tester for deploying cuda image to hosts(k8s worker nodes) where nvidia toolkit has been installed in.

[YeonghyeonKO]

@vamshin
For hosting CUDA-based image for OpenSearch, I've added v100 type GPU node to k8s clusters. (see the below spec)

Capacity:
  cpu:                80
  ephemeral-storage:  204700Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527862092Ki
  nvidia.com/gpu:     8
  pods:               110

ML Nodes will be deployed in this worker node(GPU). I'd already tested the deployment of them via opensearch-k8s-operator using the existing opensearch docker image instead of CUDA-based image. To avoid being disturbed by nodes with other roles, the property of nodeSelector and tolerations is used.

  nodePools:
    - component: ml
      replicas: 3
      nodeSelector:
        gpu: "true"
      tolerations:
      - key: node.kubernetes.io/unschedulable
        operator: "Exists"
        effect: "NoSchedule"