Example: Remote Code Runner

About

This example demonstrates a Helix system that contains Python participants and a Python spectator. Through a web frontend, we can submit Python programs that will run on machines that register as "code runners." There are many applications for this in real systems:

1. We want to make sure the program runs the same in different environments
2. We want to distribute the load of programs coming in that need to be run
3. We want to run the same code on a set of machines to change some state on those machines
4. We want to run code, but the machine we have direct access to is underpowered
5. many others

The code here is a bare-bones example of how this might be implemented. It's not recommended that this run publicly in production as it is capable of running arbitrary code, which could lead to disastrous situations.

Package Structure

• coderunner.py -- the coderunner frontend
• coderunner_cluster.yaml -- YAML file describing the cluster structure
• coderunner_node.yaml -- process that can run code
• index.tpl -- home page template of the frontend
• runnermodel.py -- the Helix state model of a coderunner

Preliminaries

Get bottle

sudo easy_install bottle

Get pyhelix

sudo easy_install pyhelix

Get Helix

wget http://www.us.apache.org/dist/incubator/helix/0.6.2-incubating/binaries/helix-core-0.6.2-incubating-pkg.tar
tar xf helix-core-0.6.2-incubating-pkg.tar
cd helix-core-0.6.2-incubating/bin

Running everything locally

This is a basic setup to get some code runners and a frontend running on a local machine. To make this distributed, just run the processes on different machines with different hosts and ports.

Start ZooKeeper

./start-standalone-zookeeper.sh

Set up the cluster

./yaml-cluster-setup.sh localhost:2199 /path/to/pyhelix/examples/coderunner_cluster.yaml

Start a controller

./run-helix-controller --zkSvr localhost:2199 --cluster coderunner-cluster

Start some code runners

# Run each of these in the background or in separate shells
python coderunner_node.py --host localhost --port 8001 --zkSvr localhost:2199
python coderunner_node.py --host localhost --port 8002 --zkSvr localhost:2199
python coderunner_node.py --host localhost --port 8003 --zkSvr localhost:2199

Start the code runner frontend

python coderunner.py --host localhost --port 8000 --zkSvr localhost:2199

Try it out!

1. Browse to http://localhost:8000
2. Run some Python programs