hadoop developer's guide

hdfs (namenode and datanode) only at the moment – not map-reduce (jobtracker)

This assumes that there are three machines with the IPs as follows:

1. 192.168.56.1
2. 192.168.56.10
3. 192.168.56.20

We’ll run the namenode on 192.168.56.10 and a datanode on all three nodes. Our workstation is 192.168.56.1, where we’ll do development on, so that datanode will come up and down a lot.

1. on 192.168.56.10, start namenode:

 screen -S namenode
 cd hadoop-hdfs 
 bin/hdfs namenode

2. on 192.168.56.10, 192.168.56.20, and 192.168.56.1 (test/dev datanode):

screen -S datanode
bin/hdfs datanode

3. see how we are doing (on any host) :

bin/hdfs dfsadmin -report

Should show: “Datanodes available: 3 (3 total, 0 dead)”

Dev datanode or namenode (192.168.56.1) SETUP:

cd hadoop-common
ant clean
ant compile
cd hadoop-hdfs
ant clean 
ant compile

Development Cycle

cd hadoop-common (or hadoop-hdfs)
git pull, merge,etc
do { 
  (edit some files)
  ant compile
  screen -ls | grep datanode | cut.. (find the datanode screen)
  ctrl-c, restart datanode with cursor-up key) (bin/hdfs datanode)
} while !(feature_or_bug.fixed) 
git commit, push, etc

Running Hadoop with explicit CLASSPATHs.

For rapid development (a faster edit/compile/debug cycle), or for expository purposes, you might want to set your CLASSPATH explicitly, to load particular .class files, rather than requiring recompilation of the entire .jars.

FS

export HADOOP_CLASSPATH=/home/ekoontz/hadoop-hdfs/build/hadoop-hdfs-0.22.0-SNAPSHOT.jar

java org.apache.hadoop.fs.FsShell -ls hdfs://192.168.56.10/

DFSIOTest

export CLASSPATH=/Users/ekoontz/hadoop-common/build/classes:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/commons-logging-1.1.1.jar:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/commons-cli-1.2.jar:\
/Users/ekoontz/hadoop-hdfs/build/classes:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/junit-4.8.1.jar:\
/Users/ekoontz/hadoop-mapreduce/build/classes:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/log4j-1.2.15.jar:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/avro-1.3.2.jar:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/jackson-mapper-asl-1.4.2.jar:\
/Users/ekoontz/hadoop-common/build/ivy/lib/Hadoop-Common/common/jackson-core-asl-1.4.2.jar

java org.apache.hadoop.util.RunJar \
/Users/ekoontz/hadoop-mapreduce/build/hadoop-mapred-test-0.22.0-SNAPSHOT.jar \
TestDFSIO -write -nrFiles 10 -fileSize 1000