add jupyter notebook #1

Tianhao-Gu · 2024-05-16T00:50:11Z

No description provided.

github-advanced-security · 2024-05-16T00:52:59Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

MrCreosote · 2024-05-16T18:06:11Z

README.md

+
+```bash
+docker-compose up --build


You can daemonize with -d

I know. But I usually want to see the logs.

docker compose logs

but if you want to leave it as is that's fine

I know you can view logs that way. I think it's convenient to view it as it's happening. I will just leave it as is for now. It's individual dev's choice I guess.

MrCreosote · 2024-05-16T18:12:46Z

README.md

+    sh -c '
+	    /opt/bitnami/spark/bin/spark-submit \
+	    --master spark://spark-master:7077 \


Can you make the spark master arg use the environment variable in docker-compose?

MrCreosote · 2024-05-16T18:14:58Z

README.md

+    .master(os.environ['SPARK_MASTER_URL']) \
+    .appName("TestSparkJob") \
+    .config("spark.driver.host", os.environ['SPARK_DRIVER_HOST']) \


We should just add this to the config in the image in the entrypoint so the user doesn't have to worry about it

are you referring to make changes to the Spark configuration file? I can do it in my next PR. I think it will involve a lot of testing commit since locally it always works.

That's what I was thinking, but regardless of the implemenation the behavior should be that we set some environment var in rancher to the notebook hostname and then the user doesn't need to worry about the host name at all, it Just Works

next PR is fine

MrCreosote · 2024-05-16T18:57:15Z

README.md

+conf = SparkConf(). \
+    setMaster( os.environ['SPARK_MASTER_URL']). \
+    setAppName("TestSparkJob"). \
+    set("spark.driver.host", os.environ['SPARK_DRIVER_HOST'])
+sc = SparkContext(conf=conf)


Do you have to set up both the session and the context? Since they're in different code blocks it seems like they're different things

No, you don't need to set up both. SparkContext has more control over spark but more complicated to config. Some people have a preference for either option. So I just give 2 examples.

Maybe put "OR" between them?

MrCreosote · 2024-05-16T18:58:07Z

docker-compose.yaml

+
+  spark-master:
+    image: bitnami/spark:3.5.1
+    container_name: spark-master


Is there a reason for providing container names that appear to be redundant?

Otherwise the container name is _1. I just don't like the suffix when I need to docker exec -it

docker-compose.yaml

MrCreosote · 2024-05-16T19:00:18Z

docker-compose.yaml

+      - SPARK_MASTER_URL=spark://spark-master:7077
+      - SPARK_WORKER_CORES=2
+      - SPARK_WORKER_MEMORY=1G
+      - SPARK_WORKER_PORT=8091


I don't think there's any need to specify the worker ports

MrCreosote · 2024-05-16T19:00:41Z

docker-compose.yaml

+      dockerfile: Dockerfile
+    container_name: spark-notebook
+    ports:
+      - "4040:4040"


Why is port 4040 mapped?

removed. I must test something before and I forget to remove it.

Co-authored-by: MrCreosote <[email protected]>

MrCreosote · 2024-05-16T19:23:52Z

Dockerfile

+RUN chmod a+x /opt/entrypoint.sh
+
+# Switch back to the original user
+#USER ${ORI_USER}


is this supposed to be commented out now?

oh oh. I was testing things and did add .

MrCreosote · 2024-05-16T19:33:48Z

Dockerfile


-ENTRYPOINT ["sleep 10s"]
+ENV PYTHON_VER=python3.11


Now I think about it I'm pretty sure bitnami installs python in the image. Is there a reason you're not using that python, assuming I'm right? If the python version here doesn't match what's in the cluster the job won't run

yea. I can use python from the image. But I remember you want to control which python version we are using. I have no strong option on that. I am fine either way.

In the prior repo the version was set for both the cluster and the test image (assuming you were using the same image). Here the python version could be different for the cluster and the notebook which would break things

👍 Switched to use image python. Both base image has python installed. I don't see the differences.

Both base image has python installed.

What do you mean by both base images?

I don't see the differences.

Not sure what you mean here either

both bitnami/spark and the image you used previously has python. I don't see why we spent a lot of time trying to install python before. We could just use the python form the image before.

The old image had 3.10. If it had 3.11 I wouldn't have bothered

add jupyter notebook

1e98bb4

Tianhao-Gu added 3 commits May 16, 2024 08:20

add container name for notebook

a1047ed

add instruction for Rancher deployment

7df710e

update rancher env variable instruction

68d386a

MrCreosote reviewed May 16, 2024

View reviewed changes

using env var

8e28235

MrCreosote reviewed May 16, 2024

View reviewed changes

Tianhao-Gu and others added 3 commits May 16, 2024 14:05

Update docker-compose.yaml

6bfee7e

Co-authored-by: MrCreosote <[email protected]>

remove 4040 port for notebook

7d5e88b

update readme

a13d27f

MrCreosote reviewed May 16, 2024

View reviewed changes

uncomment using ori user

908f340

MrCreosote reviewed May 16, 2024

View reviewed changes

using image python

f9754a1

MrCreosote approved these changes May 16, 2024

View reviewed changes

Tianhao-Gu merged commit bfbc095 into main May 16, 2024
6 checks passed

Tianhao-Gu deleted the dev_notebook branch May 16, 2024 20:36


		```bash
		docker-compose up --build

add jupyter notebook #1

add jupyter notebook #1

Conversation

Tianhao-Gu commented May 16, 2024

github-advanced-security bot commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCreosote May 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCreosote May 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrCreosote May 16, 2024 •

edited

Loading

MrCreosote May 16, 2024 •

edited

Loading