Suitable for syncing?

[wizonesolutions]

Hey there,

So I guess my use case is essentially deployment, but I wanted to bounce it off someone just to get my thoughts in order.

I'm doing up some scripts to sync some databases and files...and it'd be nice to be able to issue different commands depending on which of them I actually want to run. Much like Capistrano.

I probably don't need any of the deployment magic (i.e. creation of release directories and whatnot), but I can turn that off anyway right? Maybe it's overkill and I should just use shell scripts that run other shell scripts. I tend to overkill things.

Thoughts appreciated!

[jessegreathouse]

Hi,
Trying to get a clear picture of your use case. From what i can tell you have scripts and you want to be able to run them remotely on your deployment servers? But you dont want the full on deployment right?

I frequently need to run scripts on remote servers and it's a pain because first I have to look up what servers the application actually deploys to, and then I have to get all of their info like IP, login, pass, deployment path, etc...

I got so tired of doing this I added a feature to phingistrano called "remote". It's in the deploy submodule but you dont actually have to do a full deployment to use it. What I do is I call from the command line:

jesse@picard $ phing deploy.remote
> /etc/init.d/apache2 restart

That will bring up an interactive prompt where you can enter a command, and when you enter it, it will be performed on all of your production servers defined in your properties.

So for instance if you had a database script called syncdb.sh I would put the script on the deployment server (assuming its on a uniform path in every server)

 jesse@picard $ phing deploy.remote
> /path/to/my/scripts/syncdb.sh

The code behind the scenes is very simple, you dont even need the whole phingistrano repository to do something like this --- (but you need the libssh dependency unfortunately... thats a dependency that the phing ssh target requires)

<?xml version="1.0" encoding="UTF-8"?>
<project name="myProject" default="help">
<property name="deploy.user"       value="jesse" />
    <property name="deploy.password"   value="jiveturkey" />
    <property name="deploy.path"       value="/var/www/deployments/myProject/${deploy.branch}" />
    <property name="deploy.log"        value="2&gt;&amp;1 | tee ${deploy.path}/deploy.log" />
    <property name="deploy.servers"    value="172.99.99.99, 172.99.99.98" />

    <!-- ============================================  -->
    <!-- Target: deploy.remote                                -->
    <!-- ============================================  -->
    <target name="deploy.remote" >
        <input propertyname="command">Enter command: </input>
        <foreach list="${deploy.servers}" param="deploy.server" target="deploy.remotecmd" />
    </target>

    <!-- ============================================  -->
    <!-- Target: deploy.remotecmd                                -->
    <!-- ============================================  -->
    <target name="deploy.remotecmd" >
        <echo msg="${deploy.user}&#64;${deploy.server}&#36; ${command}" />
        <ssh username="${deploy.user}"
             password="${deploy.password}"
             host="${deploy.server}"
             command="echo ${deploy.password} | ${command}" />
    </target>
</project>

That code is just an example for demonstration purposes but it should be pretty close. The deploy.remote target is part of phingistrano but if you're not doing deployments then I'd recommend just using the simple snippet I posted.

I don't have code for explicitly syncing databases yet but from what it sounds like you already have the utility to do that and just need a way to automate it. Let me know if I can help any further.

[jessegreathouse]

oh I forgot to mention the reason I pipe the password in the command is that if your deployment user has sudo privelages on debian you can put your command with sudo -S like this

jesse@picard $ phing deploy.remote
> sudo -S /etc/init.d/memcached restart

If you want to do something like this be sure to check out the SSH phing extension and make sure you have the dependencies it can be tricky to use:
http://www.phing.info/docs/guide/stable/chapters/appendixes/AppendixC-OptionalTasks.html#SshTask

Install ssh2 library on Fedora 13+ &RHEL 6+

yum install libssh2-devel
Install ssh2 library on Ubuntu 10.10+

Install the PHP Bindings for libssh2
sudo apt-get install libssh2-1-dev
then download and install the ssh2 library with git
sudo apt-get install libssh2-php
SSH2 PHP Extension (all operating systems)

Once you've installed the SSH2 library, install the PHP extension with PECL

sudo pecl install channel://pecl.php.net/ssh2-0.11.2
Configure extension
The ssh2 extension might not add itself to the php ini. I've found that I usually have to link it up manually.

[wizonesolutions]

Interesting. Thanks for all the info...I'll have to read up on this. Basically, I'm only trying to go one way at a time, and yeah, I'm not really doing a deployment per se. I'm running scripts that I want to do stuff like run local commands (rsync and something called drush and stuff) that will go to a central server, and it'd be great if I could pass some arguments to them based on my Phingistrano configuration...so that I can change stuff in one place, you know? Then as part of the same or a different command (so being able to combine them is helpful) I'd want to run a command on a particular remote server (probably never several at a time, but maybe) that pulls stuff from that central server via the same sorts of commands or scripts.

So to sum it up...right now the main thing I'd like is to have a central place for my configuration. I could just as easily do up a conf file, I guess, but it's like...there's a tool that already does this...plus more that maybe I'll want later...and I want to avoid rolling my own stuff as much as possible, even though a lot of the actual tasks that run will be custom to me.

To put some context to it, I'm trying to get something in place so I can sync up my development environments pretty fast. Usually I have my stuff on a server that I remote into, but if traveling I might want to put everything on a VirtualBox image in short order for when I don't have Internet...etc.

[jessegreathouse]

So if you're running the Phingistrano and want to add some customized commands/configurations I've found the best way to do it is to import a custom phing configuration file for that project. In my example I call it "helpers". I just did something nearly exactly what you're talking about for a symfony project I've worked on. Basically I keep the helpers file around to add customizations because I dont want to put project level customizations in the build repository. So the phing confiig loads an external file with the import target like this:

<?xml version="1.0" encoding="UTF-8"?>
<project name="EDUDirect" default="help">

    <!-- Imports -->
    <import file="${project.basedir}/build.helpers.xml" />
    <import file="${project.basedir}/vendor/Phingistrano/build.xml" />

In the build helpers I had a lot of remote commands that were needed for the project but I didnt want to type in the commands all the time because they were all the same set of symfony2 commands so it was convenient to set up additional targets for each command.

    <!-- migrate -->
    <target name="migrate"
                  description="Run migrations on production servers" >
        <property name="command"
                  value="(
cd ${deploy.path}/current/app &amp;&amp;
./console doctrine:migrations:migrate ${pd.log}
)"
                  override="true" />
        <foreach list="${deploy.servers}"
                  param="deploy.server"
                  target="deploy.remotecmd" />
    </target>


    <!-- warm assets -->
    <target name="warm_assets"
            description="Warm assets on production servers" >
            <property name="command"
                value="(
cd ${deploy.path}/current/app &amp;&amp;
./console assets:warm ${pd.log}
)"
                override="true" />
            <foreach list="${deploy.servers}"
                param="deploy.server"
                target="deploy.remotecmd" />
    </target>

    <!-- dump cache -->
    <target name="dump_cache"
            description="dump and warm cache on production servers" >
            <property name="command"
                value="(
cd ${deploy.path}/current/app &amp;&amp;
./console cache:clear ${pd.log} &amp;&amp;
./console cache:warm ${pd.log}
)"
                override="true" />
            <foreach list="${deploy.servers}"
                     param="deploy.server"
                     target="deploy.remotecmd" />
    </target>

Looks kinda messy but its just a set of customized commands used in the same way as my first example, except they're static commands nested in custom targets.

The best part about doing this is the ability to aggregate your custom targets. Like for instance, every time I deploy, i migrate the database, warm the assets and clear the cache. So in order to make that very simple I can aggregate all those targets with depends like so:

    <!-- postdeploy  -->

    <target name="post_deploy"
            depends=" migrate, warm_assets, dump_cache"
            description="Execute post deployment utilities on production" />

Ive found target aggregation to be a very powerful thing with phing because it allows you to use the commands on an individual basis but you can also chain them into routines.

[wizonesolutions]

Ooh...I have to get more familiar with the Phing XML, but this does look
pretty sweet. A project can be anything, right? So I could just have a
separate directory on the servers concerned (maybe managed by Git) and put
in remote commands that act on files not necessarily within that folder
tree.

Greatly appreciate you taking the time to explain that! And interesting
(though not surprising :)) that you've done something similar. Good
deployments and the like seem to be problems constantly being re-solved and
refined, huh.

On Tue, Jun 21, 2011 at 3:33 PM, jessegreathouse <
[email protected]>wrote:

So if you're running the Phingistrano and want to add some customized
commands/configurations I've found the best way to do it is to import a
custom phing configuration file for that project. In my example I call it
"helpers". I just did something nearly exactly what you're talking about for
a symfony project I've worked on. Basically I keep the helpers file around
to add customizations because I dont want to put project level
customizations in the build repository. So the phing confiig loads an
external file with the import target like this:

<project name="EDUDirect" default="help">

   <!-- Imports -->
   <import file="${project.basedir}/build.helpers.xml" />
   <import file="${project.basedir}/vendor/Phingistrano/build.xml" />

In the build helpers I had a lot of remote commands that were needed for
the project but I didnt want to type in the commands all the time because
they were all the same set of symfony2 commands so it was convenient to set
up additional targets for each command.

   <!-- migrate -->
   <target name="migrate"
                 description="Run migrations on production servers" >
       <property name="command"
                 value="(

cd ${deploy.path}/current/app &&
./console doctrine:migrations:migrate ${pd.log}
)"
override="true" />

    <!-- warm assets -->
   <target name="warm_assets"
           description="Warm assets on production servers" >
           <property name="command"
               value="(

cd ${deploy.path}/current/app &&
./console assets:warm ${pd.log}
)"
override="true" />

    <!-- dump cache -->
   <target name="dump_cache"
           description="dump and warm cache on production servers" >
           <property name="command"
               value="(

cd ${deploy.path}/current/app &&
./console cache:clear ${pd.log} &&
./console cache:warm ${pd.log}
)"
override="true" />

Looks kinda messy but its just a set of customized commands used in the
same way as my first example, except they're static commands nested in
custom targets.

The best part about doing this is the ability to aggregate your custom
targets. Like for instance, every time I deploy, i migrate the database,
warm the assets and clear the cache. So in order to make that very simple I
can aggregate all those targets with depends like so:

   <!-- postdeploy  -->

   <target name="post_deploy"
           depends=" migrate, warm_assets, dump_cache"
           description="Execute post deployment utilities on

production" />

Ive found target aggregation to be a very powerful thing with phing because
it allows you to use the commands on an individual basis but you can also
chain them into routines.

Reply to this email directly or view it on GitHub:
#1 (comment)

[jessegreathouse]

yea you could do all of the above. np glad to help out. GL.

[wizonesolutions]

So I started setting this up today (as you have no doubt guessed). You were saying I don't even necessarily need the submodule to do what I want; is that to say that Phing itself would probably suffice? Is Phingistrano basically just an additional set of re-usable Phing targets? Trying to get it all together in my head as to what plays what role. It's interesting stuff :)

[jessegreathouse]

That's correct. Phing is it's own application that provides a way to
abstract some php language into XML. Phing can be used to create scripts
and such. It has a really solid documentation here:

http://www.phing.info/trac/

Phingistrano is a utility that I and my colleagues created with phing to
replace capistrano in our enterprise software projects. We had been
using capistrano for deployment but we replaced it with the collection
of phing code that you see in this repository.

The thing that makes Phingistrano helpful is that once you've written a
bunch of phing scripts for a bunch of different projects, it becomes a
headache to maintain and keep track of what all those scripts are doing.
Phingistrano attempts to provide a one-size-fits all approach to
deployment, and other build activities, by keeping some standard targets
and importing them into a customized central project build.xml. And you
can add it to a git project as a git submodule so if there are updates
to the phing scripts they are immediately available to all of your
projects that use them via git.

On 6/23/2011 9:04 PM, wizonesolutions wrote:

So I started setting this up today (as you have no doubt guessed). You were saying I don't even necessarily need the submodule to do what I want; is that to say that Phing itself would probably suffice? Is Phingistrano basically just an additional set of re-usable Phing targets? Trying to get it all together in my head as to what plays what role. It's interesting stuff :)