Skip to content

Commit

Permalink
add space
Browse files Browse the repository at this point in the history
  • Loading branch information
rishic3 committed Jan 16, 2025
1 parent 1cc73b2 commit 0e09d33
Show file tree
Hide file tree
Showing 14 changed files with 2,417 additions and 1 deletion.
23 changes: 23 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
*.pyo
*.swp
*~
*.log
*.db
*.pt
*.rst
*.tar.gz
.DS_Store
.cache
.classpath
Expand All @@ -22,3 +27,21 @@
.settings
hs_err*.log
target
/examples/ML+DL-Examples/Spark-DL/dl_inference/pytriton
/examples/ML+DL-Examples/Spark-DL/dl_inference/archive
# /examples/ML+DL-Examples/Spark-Rapids-ML/pca
/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/new_ideas
/examples/ML+DL-Examples/Spark-DL/dl_inference/databricks/torch_notebooks
/examples/ML+DL-Examples/Spark-DL/dl_inference/databricks/tf_notebooks
/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/datasets
/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/spark-dl-datasets
/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/models
**/spark-dl-datasets
**/models
**/predictions
**/dl_inference/databricks/*.ipynb
**/dl_inference/dataproc/*.ipynb
**/dl_inference/huggingface/.ipynb_checkpoints
/examples/ML+DL-Examples/Spark-DL/dl_inference/dataproc/spark-dl-notebooks
/examples/ML+DL-Examples/Spark-DL/dl_inference/test.ipynb
/examples/ML+DL-Examples/Spark-DL/dl_inference/dataproc/setup/copy_notebooks_gcp.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"version":"1.4.0","project":{"name":"spark-rapids-examples_f4dd477a3a-test","directory":"/home/rishic/spark-rapids-examples/.scala-build","workspaceDir":"/home/rishic/spark-rapids-examples","sources":["/home/rishic/spark-rapids-examples/examples/MIG-Support/device-plugins/gpu-mig/src/test/java/com/nvidia/spark/TestNvidiaGPUMigPluginForRuntimeV2.java"],"dependencies":["spark-rapids-examples_f4dd477a3a"],"classpath":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/main","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/com/sourcegraph/semanticdb-javac/0.10.0/semanticdb-javac-0.10.0.jar"],"out":"/home/rishic/spark-rapids-examples/.scala-build/.bloop/spark-rapids-examples_f4dd477a3a-test","classesDir":"/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/test","scala":{"organization":"org.scala-lang","name":"scala-compiler","version":"3.4.2","options":["-Xsemanticdb","-sourceroot","/home/rishic/spark-rapids-examples"],"jars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.4.2/scala3-compiler_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.4.2/scala3-interfaces-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.4.2/tasty-core_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-native/3.25.1/jline-native-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar"],"bridgeJars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.2/scala3-sbt-bridge-3.4.2.jar"]},"java":{"options":["-Xplugin:semanticdb -sourceroot:/home/rishic/spark-rapids-examples -targetroot:javac-classes-directory","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"]},"test":{"frameworks":[{"names":["com.novocode.junit.JUnitFramework"]},{"names":["org.scalatest.tools.Framework","org.scalatest.tools.ScalaTestFramework"]},{"names":["org.scalacheck.ScalaCheckFramework"]},{"names":["org.specs.runner.SpecsFramework","org.specs2.runner.Specs2Framework","org.specs2.runner.SpecsFramework"]},{"names":["utest.runner.Framework"]},{"names":["munit.Framework"]}],"options":{"excludes":[],"arguments":[]}},"platform":{"name":"jvm","config":{"home":"/usr/lib/jvm/java-8-openjdk-amd64","options":[]},"mainClass":[]},"resolution":{"modules":[{"organization":"org.scala-lang","name":"scala3-library_3","version":"3.4.2","artifacts":[{"name":"scala3-library_3","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar"},{"name":"scala3-library_3","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2-sources.jar"}]},{"organization":"org.scala-lang","name":"scala-library","version":"2.13.12","artifacts":[{"name":"scala-library","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar"},{"name":"scala-library","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12-sources.jar"}]}]},"tags":["test"]}}
1 change: 1 addition & 0 deletions .scala-build/.bloop/spark-rapids-examples_f4dd477a3a.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"version":"1.4.0","project":{"name":"spark-rapids-examples_f4dd477a3a","directory":"/home/rishic/spark-rapids-examples/.scala-build","workspaceDir":"/home/rishic/spark-rapids-examples","sources":["/home/rishic/spark-rapids-examples/examples/MIG-Support/device-plugins/gpu-mig/src/main/java/com/nvidia/spark/NvidiaGPUMigPluginForRuntimeV2.java","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/main.scala","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/scala/src/com/nvidia/spark/examples/pca/Main.scala","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/target/classes/com/nvidia/spark/examples/pca/Main.scala","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/DecimalFraction.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/StringWordCount.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/URLDecode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/URLEncode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/CosineSimilarity.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/DecimalFraction.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/NativeUDFExamplesLoader.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/URLDecode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/URLEncode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLDecode.scala","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLEncode.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/agaricus/scala/src/com/nvidia/spark/examples/agaricus/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/agaricus/target/classes/com/nvidia/spark/examples/agaricus/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/Mortgage.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/XGBoostETL.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/Mortgage.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/XGBoostETL.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/Taxi.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/Taxi.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/Benchmark.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/SparkSetup.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/Vectorize.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/XGBoostArgs.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/Benchmark.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/SparkSetup.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/Vectorize.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/XGBoostArgs.scala"],"dependencies":[],"classpath":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/com/sourcegraph/semanticdb-javac/0.10.0/semanticdb-javac-0.10.0.jar"],"out":"/home/rishic/spark-rapids-examples/.scala-build/.bloop/spark-rapids-examples_f4dd477a3a","classesDir":"/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/main","scala":{"organization":"org.scala-lang","name":"scala-compiler","version":"3.4.2","options":["-Xsemanticdb","-sourceroot","/home/rishic/spark-rapids-examples"],"jars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.4.2/scala3-compiler_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.4.2/scala3-interfaces-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.4.2/tasty-core_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-native/3.25.1/jline-native-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar"],"bridgeJars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.2/scala3-sbt-bridge-3.4.2.jar"]},"java":{"options":["-Xplugin:semanticdb -sourceroot:/home/rishic/spark-rapids-examples -targetroot:javac-classes-directory","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"]},"platform":{"name":"jvm","config":{"home":"/usr/lib/jvm/java-8-openjdk-amd64","options":[]},"mainClass":[]},"resolution":{"modules":[{"organization":"org.scala-lang","name":"scala3-library_3","version":"3.4.2","artifacts":[{"name":"scala3-library_3","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar"},{"name":"scala3-library_3","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2-sources.jar"}]},{"organization":"org.scala-lang","name":"scala-library","version":"2.13.12","artifacts":[{"name":"scala-library","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar"},{"name":"scala-library","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12-sources.jar"}]}]},"tags":["library"]}}
6 changes: 6 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"files.watcherExclude": {
"**/target": true
},
"java.compile.nullAnalysis.mode": "disabled"
}
79 changes: 79 additions & 0 deletions examples/ML+DL-Examples/Spark-DL/dl_inference/aws-emr/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Spark DL Inference on AWS EMR

The instructions assume you already have a AWS EMR account.
**Note**: fields in \<brackets\> require user inputs.

## Setup

#### Setup AWS CLI

1. Install the [AWS CLI](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-cli.html).

2. Initialize the CLI via `aws configure`. You may need to create access keys by following [Authenticating using IAM user credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-user.html). You can find your default region name (e.g. Ohio) on the right of the top navigation bar. Clicking the region name will show the region code (e.g. us-east-2 for Ohio).

```shell
aws configure
AWS Access Key ID [None]: <your_access_key>
AWS Secret Access Key [None]: <your_secret_access_key>
Default region name [None]: <region-code>
Default output format [None]: json
```

#### Copy Files to S3

3. Create an S3 bucket if you don't already have one.
```shell
export S3_BUCKET=<your_s3_bucket_name>
aws s3 mb s3://${S3_BUCKET}
```
4. Upload the initialization script to S3.
```shell
aws s3 cp spark_rapids_ml.zip s3://${S3_BUCKET}/spark_rapids_ml.zip
cd ../../notebooks/aws-emr
aws s3 cp init-bootstrap-action.sh s3://${S3_BUCKET}/init-bootstrap-action.sh
```
- Print out available subnets in CLI then pick a SubnetId (e.g. subnet-0744566f of AvailabilityZone us-east-2a).
```shell
aws ec2 describe-subnets
export SUBNET_ID=<your_SubnetId>
```
- Create a cluster with at least two single-gpu workers. You will obtain a ClusterId in terminal. Noted three GPU nodes are requested here, because EMR cherry picks one node (either CORE or TASK) to run JupyterLab service for notebooks and will not use the node for compute.
```shell
export CLUSTER_NAME="spark_rapids_ml"
export CUR_DIR=$(pwd)
aws emr create-cluster \
--name ${CLUSTER_NAME} \
--release-label emr-7.3.0 \
--ebs-root-volume-size=32 \
--applications Name=Hadoop Name=Livy Name=Spark Name=JupyterEnterpriseGateway \
--service-role EMR_DefaultRole \
--log-uri s3://${S3_BUCKET}/logs \
--ec2-attributes SubnetId=${SUBNET_ID},InstanceProfile=EMR_EC2_DefaultRole \
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.2xlarge \
InstanceGroupType=CORE,InstanceCount=3,InstanceType=g4dn.2xlarge \
--configurations file://${CUR_DIR}/init-configurations.json \
--bootstrap-actions Name='Spark Rapids ML Bootstrap action',Path=s3://${S3_BUCKET}/init-bootstrap-action.sh
```
- In the [AWS EMR console](https://console.aws.amazon.com/emr/), click "Clusters", you can find the cluster id of the created cluster. Wait until all the instances have the Status turned to "Running".
- In the [AWS EMR console](https://console.aws.amazon.com/emr/), click "Workspace(Notebooks)", then create a workspace. Wait until the status becomes ready and a JupyterLab webpage will pop up.
- Enter the created workspace. Click the "Cluster" button (usually the top second button of the left navigation bar). Attach the workspace to the newly created cluster through cluster id.
- Use the default notebook or create/upload a new notebook. Set the notebook kernel to "PySpark".
- Add the following to a new cell at the beginning of the notebook. Replace "s3://path/to/spark\_rapids\_ml.zip" with the actual s3 path.
```python
%%configure -f
{
"conf":{
"spark.submit.pyFiles": "s3://path/to/spark_rapids_ml.zip"
}
}
```
- Run the notebook cells.
**Note**: these settings are for demonstration purposes only. Additional tuning may be required for optimal performance.
Loading

0 comments on commit 0e09d33

Please sign in to comment.