Running an Application with IterativeReduce

Running an Application with IterativeReduce

To run an IterativeReduce application, you will need 4 things:

1. A YARN cluster
2. The IterativeReduce JAR file
3. Your application JAR file
4. An application properties file

Create the properties file

Once your application has been built and is ready to deployed you will need to create the application properties file. A sample app.properties file is located at the root of the IterativeReduce repository.

The properties file needs to (at least) contain the following properties:

• iterativereduce.jar.path
  This is the path on the local filesystem for the IterativeReduce JAR file your application was compiled against. If the path is relative, it is relative to the local working directory.

• app.jar.path
  This is the path on the local filesystem for your application JAR file. If the path is relative, it is relative to the local working directory.

• app.lib.jar.path
  This is a comma-separated list of paths to JAR files on the local filesystem. These JAR files will be deployed to the YARN cluster when the application is launched. Can be omitted or left blank if not applicable.

• app.input.path
  This is the path to the input file (typically an HDFS) your application will be working on.

• app.output.path
  The output path where your application will emit its final result (typically on HDFS).

• yarn.memory
  The amount of memory that will be allocated for the application master and workers. Adjust this if your application workers require more memory. The default is 512mb.

• yarn.master.main
  This is the fully-qualified class name that contains a main() method for your (application) master.

• yarn.master.args
  Any additional arguments (such as debug, profile, etc.) to be passed to your application master's JVM.

• yarn.worker.main
  The fully-qualified class name that contains a main() method for your worker.

• yarn.worker.args
  Any additional arguments to be passed to the worker's JVM.

• app.conf1
  Any arbitrary configuration elements that will be passed to your application. This does not need to conform to app. convention. It can be any property name (aside from the reserved ones above).

Launch the IterativeReduce Application

On a YARN-client execute the following:

# yarn jar <path-to-IR-jar> <path-to-properties>

Where <path-to-IR-jar> is the local filesystem path to the IterativeReduce JAR file, and <path-to-properties> is the local filesystem path that contains your application properties file. If not specified, IterativeReduce looks for a file ./app.properties