GENERAL NOTE:
-------------

These tools only work with MySQL 5.5 and above. In order to use them 
and compile them, you need the complete source code of the MySQL installation
you are using. If you are using an installation from a Linux distribution
obtain the source code from the distribution. Another option is to compile
MySQL yourself. Instructions (sketchy ones at most) are given below for
ubuntu. If you are using the source of your distribution, you can skip
this part.

PaQu - INTRODUCTION:
--------------------

PaQu is a parallel querying facility that works together with the MySQL Spider
engine. It is built on a similar project using Gearman 
(shard-query: http://code.google.com/p/shard-query/) and has been forked from there
at an early stage. It has been heavily extended and altered to suite our needs.

The general idea of PaQu is to take a query, analyse it and automatically shard/parallelise
it to be run on the different Spider nodes. This is achieved in a map/reduce kind of
fashions. Results are then gathered on the head node and further reduced if aggregation is
needed.

To have an idea how to use PaQu, have a look at paraQuery.php until I find the time to
write a detailed introduction on how to use PaQu. Or use the Daiquiri web framework
(escience.aip.de/daiquiri) to interface to your Spider database from the web using PaQu.

A command line tool is also provided to run queries against a predefined server setting. The command
line tool is in paqu.php.


In order to use PaQu you need to compile the following MySQL extensions (see below for instructions
or the README in the respective directories):

- mysql_node_jobMon: a deamon to be installed on the Spider nodes to monitor the head node. This is
                     needed to detect any job kill on the head node, killing the associated job on the
                     Spider node

- mysql_udf: some UDFs needed by PaQu to aggregate STDEV queries

- (optional) direct_paraQuery: this UDF will execute PaQu from within MySQL using libPhpEmbed to call
                               PHP code from within MySQL. Decide for yourself if this is a good idea.
                               To compile, have a look at build.sh and alter accordingly. Also alter the
                               path in udf_paraQuery.cc for finding PaQu. If this is functionality is
                               realy needed, let me know and I will improve the code...

KNOWN LIMITATIONS:
------------------

- Sorting is not necessarily done in parallel - ORDER BYs are pushed to the nodes and executed
  there, however anohter sort is required on the head node. On large tables this is highly
  inefficient. One would need to implement some kind of merge sort in MySQL on the head node
  to achieve the desired behaviour...


HOW TO COMPILE MYSQL FROM SCRATCH:
----------------------------------

Legal prerequisite:
We donot take any responisibilty for wrongly configured and compiled
server versions. IT IS YOUR RESPONIBILITY TO THINK, read the documentation
and decide what you need and how you need it. Only the simplest instructions
are given below and I highly recommend reading a 
book/blog/how-to/tutorial/forum on how to securely compile MySQL.

Software prerequistes:
build-essential
cmake
cmake-gui
libncurses5-dev



Installation instructions:

We will assume, that you already installed MySQL using the package management
system and you want to have your installation besides the official one.

1) Download MySQL source code (Source code - tar.gz version, not RPM)

2) Untar in directory of your convenience:

tar -xf mysql-5.5.28.tar.gz

3) cd mysql-5.5.28

4) mkdir build
   cd build
   cmake-gui ..

5) press configure and walk through wizard (Set generator to Makefile and 
   use native compiler)

6) after configure has finished, adjust the settings in the table. You
   might want to change: CMAKE_INSTALL_PREFIX, MYSQL_DATADIR, 
   WITH_FEDERATED_STORAGE_ENGINE (make sure PARTITIONED is enabled)

7) press configure again and click on generate and close cmake-gui

8) make 
   sudo make install

9) copy package my.cnf file to new location and adjust to new setup.
   especially change port and paths to socket, pid, data, logs to become
   independant of any other mysql installation

   sudo cp /etc/mysql/my.cnf /etc/mysql/my2.cnf
   emacs /etc/mysql/my2.cnf (adjust port, paths and such)

10) Copy and adjust Upstart/ini.d script
    (adjust env HOME if needed, replace my.cnf with my2.cnf, all
     refeneces to the binary files). Specify the new my2.cnf config
     file in the execuptabel (eg:
     
     exec /opt/mysql/bin/mysqld --defaults-file="${HOME}"/my2.cnf

     )

    sudo cp /etc/init/mysql.conf /etc/init/mysql2.conf
    sudo emacs /etc/init/mysql2.conf

11) setup mysql grant table (change paths if needed):
   
    /opt/mysql/scripts/mysql_install_db --basedir=/opt/mysql --datadir=/usr/local/mysql/data

12) sudo start mysql2

if things dont work, look at the log file. check if all directories (especially
the ones for the data and logs) are there. Check the priviledges, these need
to belong to the mysql user and group (in general 

chown -R mysql:mysql /foo/bar

)


COMPILING ADDONS AND PLUGINS FROM PAQU
--------------------------------------

At the moment, the following projects are usable:
mysql_query_queue, mysql_validateSQL

Each of these projects can be compiled and installed in a similar fashion.
The only requirement is, that cmake is available and the source code of
the used MySQL server is present. These tools require heavily on the original
source code.

1) cd into directory

2) mkdir build
   cd build

3) edit the CMakeLists.txt file, to properly refere to the source path of the
   MySQL server

   emacs ../CMakeLists.txt

   set MYSQL_PATH and MYSQL_SOURCES_PATH to the location of the MySQL 
   installation directory and the MySQL sources respectively.

4) cmake ..

5) make
   sudo make install

   (it might be, that mysql messes up the location of where to look for
    the plugins. On ubuntu you might need to copy from where "make install"
    installed the .so file, to /usr/lib/mysql/plugin...

    no clue yet how to change that path...)

6) install UDF/Plugin in MySQL server using install_FOO.sql

   (as root copy commands over into mysql prompt)