Skip to content

Latest commit

 

History

History
68 lines (50 loc) · 4.23 KB

README.markdown

File metadata and controls

68 lines (50 loc) · 4.23 KB
Raw

Fast tool to deploy Storm on Amazon EC2, written entirely in Java. Download latest stable release here.

Please don't hesitate to contact me. Your feedback will help to further improve this tool.

Notice

I am working on adding support for deploying on Apache CloudStack. I have had to change the credential specification a bit. When updating to the newest version, please change your credential.yaml according to credential_example.yaml.

Features

  • Runs Storm and Zookeeper daemons under supervision (automatically restarted in case of failure)
  • Only fetch and compile what is needed (can deploy on prepared images in a few minutes)
  • Supports executing user-defined commands both pre-config and post-config
  • Automatically sets up s3cmd, making it easy to get/put files on Amazon S3
  • Automatically sets up Ganglia, making it easy to monitor performance
  • Automatically sets up Amazon EC2 AMI Tools on new nodes
  • Supports Zookeeper versions: 3.3.3 & 3.4.5
  • Supports Storm versions: 0.8.2 & 0.9.0.1

Configuration

This tool, requires two configurationfiles: conf/credential.yaml and conf/configuration.yaml. Put your credentials into the file conf/credential.yaml.

Below is an example of a single cluster configuration, for conf/configuration.yaml

mycluster:
    - m1.medium {ZK, WORKER, MASTER, UI}
    - m1.medium {WORKER}
    - storm-version "0.8.2"
    - zk-version "3.3.3"
    - image "eu-west-1/ami-480bea3f" 	#official Ubuntu 13.10 AMI
    - region "eu-west-1"
    - exec-preconfig {cd ~, echo hey > hey.txt}
    - exec-postconfig {}
  • WORKER is the Storm Supervisor daemon
  • MASTER is the Storm Nimbus daemon
  • UI is the Storm and Ganglia User-Interface
  • ZK is the Zookeeper daemon

Please ensure the image resides in the same region as specified.

Usage

Deploy

Execute java -jar storm-deploy-alternative.jar deploy CLUSTER_NAME

After successful deployment, a small file is written to $HOME/.storm/, which allows you to interact with the cluster directly from the bin/storm script. For details on how to use the bin/storm script, please refer to the Storm wiki.

Kill

Execute java -jar storm-deploy-alternative.jar kill CLUSTER_NAME

Kills all nodes belonging in the cluster with name CLUSTER_NAME.

Attach

Execute java -jar storm-deploy-alternative.jar attach CLUSTER_NAME

Attaches the bin/storm script to a cluster with name CLUSTER_NAME.

Scaling

Execute java -jar storm-deploy-alternative.jar scaleout CLUSTER_NAME #NumInstances INSTANCE_TYPE

Adds new worker instances to an already running cluster. For example, you could execute java -jar storm-deploy-alternative.jar scaleout test 2 m1.medium, to add two new instances of the type m1.medium to the cluster called test. When completed, you can see the new nodes in the Storm UI.

FAQ

  • I am seeing the error: net.schmizz.sshj.userauth.UserAuthException: publickey auth failed. This error means the software could not connect to the newly launched instances using SSH (for configuring them). There can be multiple reasons why this error happens. Please ensure you have ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub and that both files are valid. Furthermore, please go to AWS EC2 interface -> Key Pairs, and delete the jclouds#CLUSTER_NAME keypair. If deploying the same cluster, using multiple machines, please ensure the same keypair exists on all machines. In case problems persist, please try generating a new keypair by executing ssh-keygen -t rsa, then delete old keypair from AWS EC2 interface and retry deployment.
  • I am seeing the warning: cipher strengths apparently limited by JCE policy. You can improve your security by installing Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files.

Limitations

Currently, only deploying to Ubuntu AMIs on Amazon EC2 is supported.