Skip to content

Latest commit

 

History

History
46 lines (31 loc) · 1.71 KB

spark-tips-and-tricks-running-spark-windows.adoc

File metadata and controls

46 lines (31 loc) · 1.71 KB
Raw

Running Spark on Windows

Running Spark on Windows is not different from other operating systems like Linux or Mac OS X, but there are few minor issues due to the way Hive works on Windows, among them is a permission error when running Spark Shell.

Note
 The issue is due to the way Hive works on Windows. You need no changes if you need no Hive integration in Spark SQL. 
15/01/29 17:21:27 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
  at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
  at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
  at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
  at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
Note
 You need to have Administrator rights on your laptop. All the following commands must be executed in a command-line window (cmd) ran as Administrator, i.e. using Run As Administrator option while executing cmd.

Download winutils.exe and save it to a directory of your choice, say c:\hadoop\bin.

Set HADOOP_HOME to reflect the directory with winutils (without bin).

set HADOOP_HOME=c:\hadoop

Set PATH environment variable to include %HADOOP_HOME%\bin as follows:

set PATH=%HADOOP_HOME%\bin;%PATH%
Tip
 Define HADOOP_HOME and PATH environment variables in Control Panel.

Create c:\tmp\hive folder and execute the following command:

winutils.exe chmod -R 777 \tmp\hive

Check the permissions:

winutils.exe ls \tmp\hive

Open spark-shell and report SUCCESS!