03-01_01_Install_Jupyter_Scala.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Spark in Windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\* Source: https://github.com/Cheng-Lin-Li/Spark/wiki/How-to-install-Spark-2.1.0-in-Windows-10-environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download necessary packages\n",
    "- Java 1.8\n",
    "http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-windows-x64.exe\n",
    "- Scala 2.11.8\n",
    "https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.msi\n",
    "- Spark 2.1.1\n",
    "https://archive.apache.org/dist/spark/spark-2.1.1/spark-2.1.1-bin-hadoop2.7.tgz\n",
    "- winutils \n",
    "https://github.com/steveloughran/winutils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- ConEmu https://github.com/Maximus5/ConEmu/releases/download/v18.06.17/ConEmuSetup.180617.exe (This is the alternative for windows cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Package Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Java\n",
    "Please install Java to a path without space. For example: by default it will install into c:\\Program Files\\Java. Please change it to c:\\Java\n",
    "<br><img src=\"img/java-version.png\" width=\"250\" height=\"auto\" align=\"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scala\n",
    "Install scala with downloaded package and make sure scala is installed by type \"scala -version\" in the cmd/ConEmu\n",
    "<br><img src=\"img/scala-version.png\" width=\"500\" height=\"auto\" align=\"left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spark\n",
    "unzip the downloaded spark package to a folder, eg: c:\\spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Winutils\n",
    "unzip the package to a folder, eg: c:\\winutils. Delete unnecessary version in the folder. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup Environment Path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. In windows Control Panel, search \"environment\" and go to \"Edit the system environment variables\" option.\n",
    "2. When a new window \"System Properties\" with tab \"Advanced\" is shown.\n",
    "<br><img src=\"img/env-window.png\" width=\"300\" height=\"auto\"><br>\n",
    "3. Click on the \"Environment Variables\".\n",
    "4. When a new window \"Environment Variables\" is shown.\n",
    "5. Click on the \"New\" and fill in \"JAVA_HOME\"\n",
    "<br><img src=\"img/setup-env-window.png\" width=\"400\"><br>\n",
    "6. Similarly, setup the \"SPARK_HOME\" where the the spark folder is located. Eg. c:\\spark\n",
    "7. Setup \"HADOOP_HOME\" where the winutils folder is located. Eg c:\\winutils\\hadoop-2.7.1\n",
    "8. Add all below parameter in the path\n",
    " - %JAVA_HOME%\\bin\n",
    " - %HADOOP_HOME%\\bin\n",
    " - %SPARK_HOME%\\bin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup the permission for /tmp/hive\n",
    "1. go to the winutils folder and further bin folder\n",
    "2. issue below command in the cmd/ConEmu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Change the spark config "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Spark configuration file is located at %SPARK_HOME%\\conf, eg: c:\\spark\\conf\n",
    "2. common change is to change the log level to \"ERROR\" from \"WARN\"\n",
    "3. make a copy of log4j.properties.template and rename it to \"log4j.properties\"\n",
    "4. change the line to \"log4j.logger.org.apache.spark.repl.Main=WARN\" to \"log4j.logger.org.apache.spark.repl.Main=ERROR\"\n",
    "5. save the \"log4j.properties\"\n",
    "\n",
    "\\* spark log level can be changed in the spark code also"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open spark-shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything goes smooth following above steps, you can now use spark-shell now.\n",
    "\n",
    "1. go to cmd/ConEmu and type \n",
    "```shell\n",
    "spark-shell --master local[2]\n",
    "```\n",
    "2. Below window would appear\n",
    "<br><img src=\"img/spark-appear.png\" width=\"500\" height=\"auto\"><br>\n",
    "3. Now you can start your spark development work in the spark-shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install jupyter-scala to enable scala/spark in the jupyter notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This method is currently only available for your personal PC as some part of this method is not feasible due to company firewall and sbt downloading is not successful yet. \n",
    "\\* Source: https://github.com/jupyter-scala/jupyter-scala\n",
    "\n",
    "1. Install spark as mentioned above.\n",
    "2. Install sbt from https://piccolo.link/sbt-1.1.6.msi\n",
    "3. Make sure jupyter is installed in the pc (if not, download from Anaconda)\n",
    "4. Download jupyter-scala from github\n",
    "```shell\n",
    "cd c:\\\n",
    "git clone https://github.com/alexarchambault/jupyter-scala.git\n",
    "cd jupyter-scala\n",
    "sbt publishLocal\n",
    "```\n",
    "5. open Git Bash\n",
    "```shell\n",
    "cd c:\\\\jupyter-scala\n",
    "./jupyter-scala --id scala-develop --name \"Scala (develop)\" --force\n",
    "```\n",
    "\n",
    "6. This will install the scala kernel in the jupyter notebook\n",
    "7. when you open the jupyter notebook by \n",
    "```shell\n",
    "jupyter notebook\n",
    "```\n",
    "8. An new kernel is there \"Scala (develop)\"\n",
    "<img src=\"img/scala-kernel.png\" width=\"500\" height=\"auto\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}