-
Notifications
You must be signed in to change notification settings - Fork 53
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
14 changed files
with
2,417 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"version":"1.4.0","project":{"name":"spark-rapids-examples_f4dd477a3a-test","directory":"/home/rishic/spark-rapids-examples/.scala-build","workspaceDir":"/home/rishic/spark-rapids-examples","sources":["/home/rishic/spark-rapids-examples/examples/MIG-Support/device-plugins/gpu-mig/src/test/java/com/nvidia/spark/TestNvidiaGPUMigPluginForRuntimeV2.java"],"dependencies":["spark-rapids-examples_f4dd477a3a"],"classpath":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/main","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/com/sourcegraph/semanticdb-javac/0.10.0/semanticdb-javac-0.10.0.jar"],"out":"/home/rishic/spark-rapids-examples/.scala-build/.bloop/spark-rapids-examples_f4dd477a3a-test","classesDir":"/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/test","scala":{"organization":"org.scala-lang","name":"scala-compiler","version":"3.4.2","options":["-Xsemanticdb","-sourceroot","/home/rishic/spark-rapids-examples"],"jars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.4.2/scala3-compiler_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.4.2/scala3-interfaces-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.4.2/tasty-core_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-native/3.25.1/jline-native-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar"],"bridgeJars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.2/scala3-sbt-bridge-3.4.2.jar"]},"java":{"options":["-Xplugin:semanticdb -sourceroot:/home/rishic/spark-rapids-examples -targetroot:javac-classes-directory","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"]},"test":{"frameworks":[{"names":["com.novocode.junit.JUnitFramework"]},{"names":["org.scalatest.tools.Framework","org.scalatest.tools.ScalaTestFramework"]},{"names":["org.scalacheck.ScalaCheckFramework"]},{"names":["org.specs.runner.SpecsFramework","org.specs2.runner.Specs2Framework","org.specs2.runner.SpecsFramework"]},{"names":["utest.runner.Framework"]},{"names":["munit.Framework"]}],"options":{"excludes":[],"arguments":[]}},"platform":{"name":"jvm","config":{"home":"/usr/lib/jvm/java-8-openjdk-amd64","options":[]},"mainClass":[]},"resolution":{"modules":[{"organization":"org.scala-lang","name":"scala3-library_3","version":"3.4.2","artifacts":[{"name":"scala3-library_3","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar"},{"name":"scala3-library_3","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2-sources.jar"}]},{"organization":"org.scala-lang","name":"scala-library","version":"2.13.12","artifacts":[{"name":"scala-library","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar"},{"name":"scala-library","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12-sources.jar"}]}]},"tags":["test"]}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{"version":"1.4.0","project":{"name":"spark-rapids-examples_f4dd477a3a","directory":"/home/rishic/spark-rapids-examples/.scala-build","workspaceDir":"/home/rishic/spark-rapids-examples","sources":["/home/rishic/spark-rapids-examples/examples/MIG-Support/device-plugins/gpu-mig/src/main/java/com/nvidia/spark/NvidiaGPUMigPluginForRuntimeV2.java","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/main.scala","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/scala/src/com/nvidia/spark/examples/pca/Main.scala","/home/rishic/spark-rapids-examples/examples/ML+DL-Examples/Spark-cuML/pca/target/classes/com/nvidia/spark/examples/pca/Main.scala","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/DecimalFraction.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/StringWordCount.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/URLDecode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/hive/URLEncode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/CosineSimilarity.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/DecimalFraction.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/NativeUDFExamplesLoader.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/URLDecode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/URLEncode.java","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLDecode.scala","/home/rishic/spark-rapids-examples/examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLEncode.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/agaricus/scala/src/com/nvidia/spark/examples/agaricus/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/agaricus/target/classes/com/nvidia/spark/examples/agaricus/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/Mortgage.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/scala/src/com/nvidia/spark/examples/mortgage/XGBoostETL.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/Mortgage.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/mortgage/target/classes/com/nvidia/spark/examples/mortgage/XGBoostETL.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/scala/src/com/nvidia/spark/examples/taxi/Taxi.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/CrossValidationMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/ETLMain.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/Main.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/taxi/target/classes/com/nvidia/spark/examples/taxi/Taxi.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/Benchmark.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/SparkSetup.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/Vectorize.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/scala/src/com/nvidia/spark/examples/utility/XGBoostArgs.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/Benchmark.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/SparkSetup.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/Vectorize.scala","/home/rishic/spark-rapids-examples/examples/XGBoost-Examples/utility/target/classes/com/nvidia/spark/examples/utility/XGBoostArgs.scala"],"dependencies":[],"classpath":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/com/sourcegraph/semanticdb-javac/0.10.0/semanticdb-javac-0.10.0.jar"],"out":"/home/rishic/spark-rapids-examples/.scala-build/.bloop/spark-rapids-examples_f4dd477a3a","classesDir":"/home/rishic/spark-rapids-examples/.scala-build/spark-rapids-examples_f4dd477a3a/classes/main","scala":{"organization":"org.scala-lang","name":"scala-compiler","version":"3.4.2","options":["-Xsemanticdb","-sourceroot","/home/rishic/spark-rapids-examples"],"jars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-compiler_3/3.4.2/scala3-compiler_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-interfaces/3.4.2/scala3-interfaces-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/tasty-core_3/3.4.2/tasty-core_3-3.4.2.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/modules/scala-asm/9.6.0-scala-1/scala-asm-9.6.0-scala-1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/compiler-interface/1.9.6/compiler-interface-1.9.6.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-reader/3.25.1/jline-reader-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal/3.25.1/jline-terminal-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-terminal-jna/3.25.1/jline-terminal-jna-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-sbt/util-interface/1.9.8/util-interface-1.9.8.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline-native/3.25.1/jline-native-3.25.1.jar","/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.14.0/jna-5.14.0.jar"],"bridgeJars":["/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-sbt-bridge/3.4.2/scala3-sbt-bridge-3.4.2.jar"]},"java":{"options":["-Xplugin:semanticdb -sourceroot:/home/rishic/spark-rapids-examples -targetroot:javac-classes-directory","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED","-J--add-exports","-Jjdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED"]},"platform":{"name":"jvm","config":{"home":"/usr/lib/jvm/java-8-openjdk-amd64","options":[]},"mainClass":[]},"resolution":{"modules":[{"organization":"org.scala-lang","name":"scala3-library_3","version":"3.4.2","artifacts":[{"name":"scala3-library_3","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2.jar"},{"name":"scala3-library_3","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.4.2/scala3-library_3-3.4.2-sources.jar"}]},{"organization":"org.scala-lang","name":"scala-library","version":"2.13.12","artifacts":[{"name":"scala-library","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12.jar"},{"name":"scala-library","classifier":"sources","path":"/home/rishic/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.12/scala-library-2.13.12-sources.jar"}]}]},"tags":["library"]}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"files.watcherExclude": { | ||
"**/target": true | ||
}, | ||
"java.compile.nullAnalysis.mode": "disabled" | ||
} |
79 changes: 79 additions & 0 deletions
79
examples/ML+DL-Examples/Spark-DL/dl_inference/aws-emr/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
# Spark DL Inference on AWS EMR | ||
|
||
The instructions assume you already have a AWS EMR account. | ||
**Note**: fields in \<brackets\> require user inputs. | ||
|
||
## Setup | ||
|
||
#### Setup AWS CLI | ||
|
||
1. Install the [AWS CLI](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-cli.html). | ||
|
||
2. Initialize the CLI via `aws configure`. You may need to create access keys by following [Authenticating using IAM user credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-user.html). You can find your default region name (e.g. Ohio) on the right of the top navigation bar. Clicking the region name will show the region code (e.g. us-east-2 for Ohio). | ||
|
||
```shell | ||
aws configure | ||
AWS Access Key ID [None]: <your_access_key> | ||
AWS Secret Access Key [None]: <your_secret_access_key> | ||
Default region name [None]: <region-code> | ||
Default output format [None]: json | ||
``` | ||
|
||
#### Copy Files to S3 | ||
|
||
3. Create an S3 bucket if you don't already have one. | ||
```shell | ||
export S3_BUCKET=<your_s3_bucket_name> | ||
aws s3 mb s3://${S3_BUCKET} | ||
``` | ||
4. Upload the initialization script to S3. | ||
```shell | ||
aws s3 cp spark_rapids_ml.zip s3://${S3_BUCKET}/spark_rapids_ml.zip | ||
cd ../../notebooks/aws-emr | ||
aws s3 cp init-bootstrap-action.sh s3://${S3_BUCKET}/init-bootstrap-action.sh | ||
``` | ||
- Print out available subnets in CLI then pick a SubnetId (e.g. subnet-0744566f of AvailabilityZone us-east-2a). | ||
```shell | ||
aws ec2 describe-subnets | ||
export SUBNET_ID=<your_SubnetId> | ||
``` | ||
- Create a cluster with at least two single-gpu workers. You will obtain a ClusterId in terminal. Noted three GPU nodes are requested here, because EMR cherry picks one node (either CORE or TASK) to run JupyterLab service for notebooks and will not use the node for compute. | ||
```shell | ||
export CLUSTER_NAME="spark_rapids_ml" | ||
export CUR_DIR=$(pwd) | ||
aws emr create-cluster \ | ||
--name ${CLUSTER_NAME} \ | ||
--release-label emr-7.3.0 \ | ||
--ebs-root-volume-size=32 \ | ||
--applications Name=Hadoop Name=Livy Name=Spark Name=JupyterEnterpriseGateway \ | ||
--service-role EMR_DefaultRole \ | ||
--log-uri s3://${S3_BUCKET}/logs \ | ||
--ec2-attributes SubnetId=${SUBNET_ID},InstanceProfile=EMR_EC2_DefaultRole \ | ||
--instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.2xlarge \ | ||
InstanceGroupType=CORE,InstanceCount=3,InstanceType=g4dn.2xlarge \ | ||
--configurations file://${CUR_DIR}/init-configurations.json \ | ||
--bootstrap-actions Name='Spark Rapids ML Bootstrap action',Path=s3://${S3_BUCKET}/init-bootstrap-action.sh | ||
``` | ||
- In the [AWS EMR console](https://console.aws.amazon.com/emr/), click "Clusters", you can find the cluster id of the created cluster. Wait until all the instances have the Status turned to "Running". | ||
- In the [AWS EMR console](https://console.aws.amazon.com/emr/), click "Workspace(Notebooks)", then create a workspace. Wait until the status becomes ready and a JupyterLab webpage will pop up. | ||
- Enter the created workspace. Click the "Cluster" button (usually the top second button of the left navigation bar). Attach the workspace to the newly created cluster through cluster id. | ||
- Use the default notebook or create/upload a new notebook. Set the notebook kernel to "PySpark". | ||
- Add the following to a new cell at the beginning of the notebook. Replace "s3://path/to/spark\_rapids\_ml.zip" with the actual s3 path. | ||
```python | ||
%%configure -f | ||
{ | ||
"conf":{ | ||
"spark.submit.pyFiles": "s3://path/to/spark_rapids_ml.zip" | ||
} | ||
} | ||
``` | ||
- Run the notebook cells. | ||
**Note**: these settings are for demonstration purposes only. Additional tuning may be required for optimal performance. |
Oops, something went wrong.