[BUG] 'spark-test.sh' integration tests FAILED on 'ps: command not found" in Rocky Docker environment

[NvTimLiu]

Describe the bug

'spark-test.sh' integration tests FAILED on ps: command not found in Rocky Docker containers, ps is not installed in some nvidia Rocky docker image, e.g. nvidia/cuda:12.0.0-runtime-rockylinux9

ps command is required for starting/stopping spark clusters: e.g. https://github.com/apache/spark/blob/v3.3.2/bin/load-spark-env.sh#L68

As our integration tests support running in Rocky Docker containers, https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/Dockerfile-blossom.integration.rocky. We can install ps tools in Rocky Docker images to fix the issue.

Failure logs

 + stop-worker.sh
 /home/jenkins/agent/workspace/jenkins-rapids_it-3.5.x-SNAPSHOT-dev-github-9/jars/spark-3.5.1-SNAPSHOT-bin-hadoop3.2/bin/load-spark-env.sh: line 68: ps: command not found
 /home/jenkins/agent/workspace/jenkins-rapids_it-3.5.x-SNAPSHOT-dev-github-9/jars/spark-3.5.1-SNAPSHOT-bin-hadoop3.2/bin/load-spark-env.sh: line 68: ps: command not found
 /home/jenkins/agent/workspace/jenkins-rapids_it-3.5.x-SNAPSHOT-dev-github-9/jars/spark-3.5.1-SNAPSHOT-bin-hadoop3.2/sbin/spark-daemon.sh: line 214: ps: command not found
 no org.apache.spark.deploy.worker.Worker to stop
 + stop-master.sh
 /home/jenkins/agent/workspace/jenkins-rapids_it-3.5.x-SNAPSHOT-dev-github-9/jars/spark-3.5.1-SNAPSHOT-bin-hadoop3.2/bin/load-spark-env.sh: line 68: ps: command not found
 /home/jenkins/agent/workspace/jenkins-rapids_it-3.5.x-SNAPSHOT-dev-github-9/jars/spark-3.5.1-SNAPSHOT-bin-hadoop3.2/sbin/spark-daemon.sh: line 214: ps: command not found
 no org.apache.spark.deploy.master.Master to stop

Steps/Code to reproduce bug
1, Run Rocky9 docker container using the docker file : https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/Dockerfile-blossom.integration.rocky
2, Run integration tests https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/spark-test.sh, e.g.

    docker build -f jenkins/Dockerfile-blossom.integration.rocky -t plugin-it:cuda12.0.0-rocky9 --build-arg ARG CUDA_VER=11.8.0 ARG --build-argROCKY_VER=9 ./
    docker run --runtime=nvidia -it plugin-it:cuda12.0.0-rocky9 bash
    bash jenins/spark-test.sh

Expected behavior
Integration tests can PASS in the above provided Rocky8 docker container

Environment details (please complete the following information)

• Environment location: Standalone,
• Rocky 9 docker container : https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/Dockerfile-blossom.integration.rocky
• test script : https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/jenkins/spark-test.sh