capture nsys report

[wjxiz1992]

This is for customer to try capture nsys report. The report is generated before executor exits, so user should be able to upload the report to their persist storage

This requires the change at entry point as well:

...
case "$1" in
  driver)
    shift 1
    CMD=(
      "$SPARK_HOME/bin/spark-submit"
      --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
      --deploy-mode client
      "$@"
    )
    ;;
  executor)
    shift 1
    CMD=(
      nsys launch --cuda-memory-usage=true
      ${JAVA_HOME}/bin/java
      "${SPARK_EXECUTOR_JAVA_OPTS[@]}"
      -Xms$SPARK_EXECUTOR_MEMORY
      -Xmx$SPARK_EXECUTOR_MEMORY
      -cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH"
      org.apache.spark.scheduler.cluster.k8s.KubernetesExecutorBackend
      --driver-url $SPARK_DRIVER_URL
      --executor-id $SPARK_EXECUTOR_ID
      --cores $SPARK_EXECUTOR_CORES
      --app-id $SPARK_APPLICATION_ID
      --hostname $SPARK_EXECUTOR_POD_IP
      --resourceProfileId $SPARK_RESOURCE_PROFILE_ID
      --podName $SPARK_EXECUTOR_POD_NAME
    )
    ;;
...

NOTE --cuda-memory-usage=true will cause GPU perf drop, so if you don't need memory usage analysis, you can remove it from the nsys comamnd.

[firestarman]

Will this make the GPU perf worse ?

[wjxiz1992]

Not really. only enabling cuda memory track(--cuda-memory-usage=true) will make perf worse, but I didn't enable that.
Should we enable that? that is used to see peak memory stuff.

[firestarman]

i mean the "sleep" here, it will take 2 mins.
Customer will also use this branch for benchmarks. I am not sure debug-only code could get in.
@winningsix what's your idea ?

[wjxiz1992]

Oh I got your point. I removed the sleep, as the command itself is blocking, no need to wait.
And as we discussed offline that the report generation time is counted, so for analysis, use the new branch, for perf bench, use the orignal one which doesn't do nsys.