Add workflow

README.md

@@ -1,6 +1,6 @@
 # OmniDataHouse
 
-Utilities for OMNILab data warehouse.
+Tool set for OMNILab data warehouse.
 
 
 ## Architecture
@@ -25,12 +25,14 @@ to add new data sources to Layer0 or Layer1.
 
 * `etlers`: ETL tools for each data repo.
 
-* `porters`: automatic scripts to port a new repo perodically with ETL tools.
+* `porters`: automatic scripts to port a new repo periodically with ETL tools.
 
 * `repos`: documentation for each repo.
 
 * `global_config.sh`: global settings used by porters.
 
+* `workflow.sh`: global workflow to run periodically.
+
 * `README.md`: this file.
 
 
@@ -48,3 +50,23 @@ to add new data sources to Layer0 or Layer1.
 * [WifiSyslog](https://github.com/OMNILab/OmniDataHouse/blob/master/repos/wifi_syslog.md)
 
 * [WifiTraffic](#)
+
+
+## Instructions to add a new repo.
+
+1. Add a related ETL program or script to `etlers`. Each program deserves an independent folder.
+
+2. Add a shell script to call your ETL program automatically.
+
+3. Append the shell script to right position in the workflow.
+
+4. Add documentation of the new repo to `repos`.
+
+5. Contact admin to redeploy this tool set.
+
+
+## Contact
+
+* Xiaming Chen <[email protected]\>
+
+* Haiyang Wang <[email protected]\>

porters/wifi_syslog_session.sh

@@ -36,10 +36,6 @@ year=`date -d "yesterday" '+%Y'`
 month=`date -d "yesterday" '+%m'`
 day=`date -d "yesterday" '+%d'`
 
-year=2013
-month=04
-day=23
-
 INPUT=$HDFS_WIFI_SYSLOG/wifilog$year-$month-$day
 OUTPUT=$HDFS_WIFI_SYSLOG_SESSION/wifilog$year-$month-$day
 

workflow.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+function clean_trash () {
+  hadoop fs -rm -r .Trash/Current > /dev/null
+}
+
+function die () {
+    echo "${@}"
+    exit 1
+}
+
+# Check permission
+if [ `whoami` != 'omnilab' ]; then
+    die "Need permission of OMNILAB to run. Try user omnilab."
+fi
+
+# Global vars
+BASEDIR=$(dirname $0)/..
+source $BASEDIR/global_config.sh
+
+## Run WifiSyslog cleansing
+exec '$BASEDIR/porters/wifi_syslog.sh'
+
+## Run WifiSyslogSession extraction
+exec '$BASEDIR/porters/wifi_syslog_session.sh'