Tutorial

This tutorial describes how to set up Elastic Phoenix on a cluster of Linux KVM virtual machines running on a single physical host.

1. Virtual Machine Setup

Elastic Phoenix has been tested under Ubuntu 10.04, and requires the Nahanni shared memory mechanism, which is available in QEMU 0.13.50 and newer. For this tutorial, we will assume you have three Ubuntu 10.04 virtual machines, called ubuntu-c1, ubuntu-c2, and ubuntu-c3. You will also need the Nahanni UIO driver. We will assume that your POSIX shared-memory object is called phoenix and that in the VMs the Nahanni device is /dev/uio0.

In order to use a large amount of memory and Nahanni shared memory, you will need Linux kernel 2.6.36 or newer. This is due to a bug in the kernel that returns bad addresses for 64-bit PCI BARs. We have tested Elastic Phoenix on VMs with 4GB of main memory and 4GB of Nahanni shared memory.

You will run each of your VMs using a command like the following:

ubuntu-c1$ qemu-system-x86_64 -hda ubuntu_c1.img -net nic,model=virtio,macaddr=52:54:00:12:34:97 -net vde \
    -m 4096 -smp 2 -device ivshmem,size=4096,shm=phoenix -vnc :97 2>&1 > c1.log &

At this point, you should have three VMs running, and should be able to SSH into them. The Nahanni tests should run successfully.

2. Getting and Compiling Elastic Phoenix

First, check out Elastic Phoenix from GitHub:

ubuntu-c1$ git clone git://github.com/adamwg/elastic-phoenix.git

Change into the elastic-phoneix directory, and run make. This should build Elastic Phoenix and all the sample applications. If you have libraries in non-standard locations, or want to enable debugging, you may need to edit Defines.mk.

3. Running the Samples

The sample applications live in the tests directory. For this tutorial, we'll use the histogram application, so change to the tests/histogram directory. (All the sample applications are described on the Sample Applications wiki page.)

Download some histogram data from the Phoenix webpage. Extract it to get small.bmp, med.bmp, and large.bmp. With 4GB of Nahanni shared memory, you should be able to run histogram on any of the three inputs. With 1GB, you should be able to run small and med.

For any Elastic Phoenix application, you must first start the master process. (Run histogram with no arguments to see the usage.) We'll run it on ubuntu-c1:

ubuntu-c1$ ./histogram -v -m -d /dev/uio0 -s 4096 -- large.bmp

You should see some output like:

Initializing as the master.
Histogram: Running...
Histogram: Calling MapReduce Scheduler
No need to swap
This file has 1406250002 bytes of image data, 468750000 pixels
There are 468750000 units. Do 14305 at a time.
32 * 32
Splitting data with chunk_size = 14305
Done splitting data.

For now, run just one worker, on ubuntu-c2:

ubuntu-c2$ ./histogram -v -w -d /dev/uio0 -s 4096

The job should run and you'll see a histogram of the image printed out. You can verify correctness by running the histogram application from the original Phoenix framework.

4. Elastic Scaling

Now try adding a second worker. Run the master and the first worker as before:

ubuntu-c1$ ./histogram -v -m -d /dev/uio0 -s 4096 -- large.bmp
ubuntu-c2$ ./histogram -v -w -d /dev/uio0 -s 4096

When the map phase starts, run another worker in ubuntu-c3:

ubuntu-c3$ ./histogram -v -w -d /dev/uio0 -s 4096

Each worker will print the number of tasks each of its threads executed. You should see each thread get some tasks.

Note that since the application runs quite fast, you'll need to be quick to add a second worker. Try flushing the cache (for example, by running a program that allocates all of the system's memory and touches every word) to slow it down a bit.

5. The submit_phoenix.sh Script

If you're running Elastic Phoenix on a shared machine with a job scheduler, or just want to simplify the process of running a job with multiple workers, the submit_phoenix.sh script in the bin directory might be useful. It starts up your VMs, runs (and times) a Phoenix application, then shuts down the VMs. You will likely need to modify the script a bit to work in your environment; it is written to work from the commandline or under the PBS batch scheduler.