Merge pull request astrolabsoftware#1 from astrolabsoftware/notebook

Add Notebook & explanations
abualia4 · Feb 8, 2019 · e9fc7b6 · e9fc7b6
2 parents a9f1694 + b72e699
commit e9fc7b6
Show file tree

Hide file tree

Showing 2 changed files with 777 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -1,2 +1,32 @@
-# 1807.03078
-Supplementary material for arXiv:1807.03078
+# Analysing billion-objects catalogue interactively: Apach Spark for physicists
+
+This repository contains supplementary material for arXiv:[1807.03078](https://arxiv.org/abs/1807.03078).
+
+## How to run the notebook
+
+You must have [Apache Spark](http://spark.apache.org/) and [Jupyter notebook](https://jupyter.org/) installed on your machine or your cluster. 
+
+### On a local machine
+
+```bash
+PACK="com.github.astrolabsoftware:spark-fits_2.11:0.7.2"
+PYSPARK_DRIVER_PYTHON_OPTS="jupyter-notebook" pyspark \
+     --master local[*] \
+     --packages $PACK 
+```
+
+### On a cluster
+
+Standalone mode:
+
+```bash
+PACK="com.github.astrolabsoftware:spark-fits_2.11:0.7.2"
+PYSPARK_DRIVER_PYTHON_OPTS="jupyter-notebook --debug --no-browser --port=$PORT1" pyspark \
+     --master $SPARKURL \
+     --packages $PACK \
+     --driver-memory $MEMDRIVER --executor-memory $MEMEXEC --executor-cores $EXECCORES --total-executor-cores $TOTALCORES
+```
+
+### DESC members: working at NERSC
+
+Source your DESC environement. Then go to the Jupyter Lab web [interface](https://jupyter-dev.nersc.gov/), and execute the notebook with the desc-pyspark kernel.