README.dev-and-workflow

This file documents dev, the workflow helper tool for Crowbar
development.

dev is designed to help automate the workflow and infrastructure
related tasks associated with development in general with the Crowbar
git tree layout.  Dev is designed around the following assumptions:

 * Everyone has their own forks on Github of the Crowbar repository
   and all of the barclamps.
 * Your development workflow will involve regular synchronization
   against your upstream repositories.
 * The only path for getting code into upstream repositories is via
   pull request.

Requirements:

 * A checkout of Crowbar.
 * Bash 4, ruby, rubygems, and the json gem.
   If running under Windows, make sure that all these operate in your
   cygwin or msysgw enviromnent.
 * A github username and password.

Releases, Builds, and Barclamps:

A release is a collection of builds, which in turn are collections of
barclamps and some associated build-specific metadata.

Releases are intended to be long-running, primary units of maintenance
and development for a collection of builds. By convention, every
release has a master build that contains references to the core
crowbar barclamps, along with other builds that may add other
barclamps to add extra capabilities.

Builds how a specific product in a release is built and what it
includes.  It includes references to barclamps, and any build-specific
metadata and infrastructure.

Barclamps enable Crowbar to manage sets of services across a
cluster. All of Crowbar's functionality is implemented in terms of
barclamps. Barclamps consist of independent git repositories with a
well-defined and dev-controlled branching structure.

Crowbar expects release and build names in the following formats:

release
release/build

Any exceptions for specific commands will be documented in the help
for that command.

Release Branching Structure:

The dev tool has the notion of a release, which it manages in terms of
a directory hierarchy rooted at the releases/ directory in the main
Crowbar repository.  Currently implemented release types are:

 * Development is where the day-to-day development on new features and
   bugfixes happens.  All master branches in the barclamps are
   considered to be in the development release.
 * Releases, which consist of released versions of Crowbar.  All
   releases but development are considered to be stable, and any code
   added to a release is either a bugfix or a backwards-compatible
   functionality enhancements. Releases are named with a unique name
   (similar to Ubuntu or Debian release names).

   All barclamps which are members of a release must have a branch named
   release/<release name>/master.
 * Feature bundles, which are designed to be relatively independent
   development streams in the main development stream.  Features are
   intended to be used to implement a specific bugfix, enhancement,
   or new feature that is relatively independent of the rest of the
   development stream, but that will be merged back into development
   when it is code complete. Features must have unique names, and
   should be descriptive of what the feature is doing.

   All barclamps which are members of a feature must have a branch
   named feature/<feature name>/master.

How Releases are Tracked:

All releases are tracked in the following directory structure:

$CROWBAR_DIR/
  releases/
    <release name>/
       <build name>/
          barclamp-<barclamp name>
          ...
          crowbar.json
          <other build-specific .json files>
          extra/
            <build-specific scripting and install bits>
          change-image/
            <build-specific ancillary commands>

For a build to be considered valid by the dev tool, it must have a
barclamp-crowbar entry that points at a valid branch in the crowbar
barclamp.

For a release to be considered valid by the dev tool, it must contain
a valid master build.

Remotes:

Dev expects to manage the various remotes that you work with to pull
changes from and issue pull requests against.  dev wraps the git add,
rm, and set-remote commands to ensure that remotes are named
identically across the main Crowbar repo and the barclamps.  dev also
adds the concepts of the remotes having priorities, which are assigned
in inverse numerical order -- a remote with priority 1 us higher
priority than a remote with priority 50, and so on.  Remote priorities
also determine where barclamps are initially cloned from, and how
remote tracking branches are configured.

By default, the remote you cloned dev from will be assigned priority
5, the remote named personal will have priority 95, and everything
else will get priority 50.  You can change the priority of a remote at
any time with the dev remote priority command.

When ./dev setup is run, dev will create consistently-named remotes
across all the Crowbar repositories, add a personal remote based on
your github ID, and delete any remotes named origin in any of the
Crowbar repositories -- since we calculate what the upstream remote is
on a branch-by-branch basis, we don't need a per-repository origin remote
that can be inconsistent from repository to repository.

If there is a remote named personal, it will be used as the default
target for backing up local changes and for creating placeholder
branches for pull requests. dev setup will create a personal remote by
default -- if you delete it you will no longer be able to use the dev
tool to back up local changes or issue pull requests.


Day to Day Workflows:

Initial Setup:

 1: Clone the Crowbar repository from you preferred upstream fork of
    Crowbar.  If you are not sure where to clone from, use
    https://github.com/crowbar/crowbar.git.
 2: Run crowbar/dev setup.
    a: Provide your github login ID and password.  Dev will need it to
       to handle talking to Github.
    b: dev will set up a remote for the upstream account you forked
       from and a personal remote.  Both of these remotes will point
       at the appropriate urls from Github.
    c: dev will clone any Crowbar repositories from your upstream
       Crowbar repositories on to your local system.
    d: dev will also create personal forks of all the upstream
       repositories into your personal Github account
 3: Create your local build cache.  See README.build for more
    information.

Regular Development:

 1: Run dev fetch followed by dev sync to fetch and merge changes from
    your upstream repositories.
    a: Dev will fetch all changes from all upstream remotes for all
       repositories.
    b: Dev will attempt to synchronize all the local branches with
       their corresponding remote branches.  If there are merge
       conflicts, the sync process will stop at the branch that was
       having problems.  From there, you can manually fix the
       conflicts and rerun dev sync. If the result of merging a branch
       and rebasing it are conflict-free and result in identical
       working trees, dev will prefer the rebased version of the branch.
 2: Run dev new-feature <featurename> to start working on a new
    feature.
    a: This command should be run from the development release.
    b: Dev will create branches in the feature/<featurename> namespace
       in the main Crowbar repository and in all the barclamps.  The
       initial starting point for these branches will be the tips of
       the corresponding branches in the development namespace.
 3: Hack/build/test/commit.
 4: Run dev backup to back up your changes.  This force-pushes your
    changes to your personal forks of the crowbar repositories on
    Github.
 5: If you are not ready to create a pull request for your changes, go
    to 1.

Collaborating on a Feature:
Make Feature available for fetch:

 1: Run dev push-release feature/<featurename>
    a: dev will create branches for this release on github/crowbar
 2: Run git push crowbar master
    a: dev will update the github/crowbar/Crowbar main repo with the
       metadata necessary for others to manage this feature

Grabbing a feature and starting to hack on it:

 1: Normal dev fetch and dev sync will make the feature available in your
     local repo.
 2: dev releases will show you the available releases and features
 3: dev switch feature/<featurename> will checkout that feature in your working
    copy

Merging changes from parent into feature/<featurename>

 1: Commit your current work to your local repo. git commit -am 'cool message'
 2: Run ./dev find-parent returns the name of your working copy's parent.
 3: Run ./dev sync <parent>  (no branch name allowed)

Merging changes from your now complete feature back into parent:

 1: ./dev switch <parent>
 2: ./dev sync feature/<featurename>

Ready for pull request:

 1: Run dev pull-requests-prep
    a: dev will verify that all the local Crowbar repositories are "clean".
    b: dev will perform a fetch and a sync, and abort if there were any
       merge conflicts.
    c: dev will figure out what barclamps and what main Crowbar
       branches are candidates for a pull request in the current release.
    d: dev will print a command line with all the branches and
       barclamps that are candidates for pull requests in the current release.
 2: Run dev pull-requests-gen
    a: dev will ask for a title to be used for all the pull requests.
    b: dev will open an editor for you to type in a body that will be
       used for all the pull requests that will be generated.
    c: dev will work out the proper order that a reviewer will need to
       merge the pull requests in based on barclamp submodule and
       branch dependencies.
    d: dev will issue pull requests in the order worked out
       earlier. Each individual pull request will have a sequence
       number added to the title indicating the proper merge order.
    e: Reviewers for your upstream repository will review the pull
       requests and merge them as appropriate.

Review pull request:

  See README.dev-and-code-review

Release Workflows:

Getting a list of known releases:

 1: Run dev releases

Getting the release you are currently on:

 1: Run dev release

Switching to a different release:

 1: Run dev switch <release name>
    a: dev will verify that the barclamp repositories are "clean", and
       it will refuse to do anything if they are not.
    b: dev will checkout the appropriate master branch for the release
       in all the barclamps.

Cutting a new release:

  1: Ensure that all the crowbar repositories are in the exact state
     you want the new release to start out in.
  2: Run dev cut_release <new release name>.
     a: dev will verify that the release name is not already in use in
        the current repository.
     b: dev will create the master branch for the new release in each
-        of the barclamps.
     c: dev will create a branch structure for the new release based
        on the branch structure of the current release.

Managing multiple remotes:

  Key concepts:
    * There is no such thing as an "origin" remote -- instead, the
      "best" origin remote depends on what branch of a release you
      happen to be working on at that time.
    * Instead, remotes have priorities, and these priorities control
      what the "best" upstream for any given branch is on a branch by
      branch basis.
    * dev owns and controls the tracking branches, and changes what
      branches track what remotes based on the priority of the remotes
      that the remote tracking branches are in.

The dev script knows how to manage releases spread out across multiple
remotes.  When dev initially set things up, it created 2 remotes:
 * 1 named after the name of the repository Crowbar was forked from, and
 * 1 named personal that pointed at your fork of Crowbar.

The dev script knows how to keep all the remotes in sync across the
barclamps -- it can add, remove, change source URLs, and synchronize
barclamp remotes across all your local clones of the Crowbar
repositories in a consistent fashion.

Whenever you do anything that changes the total set of remote branches
using dev, it will recalculate which local branch should track what
remote branch based on the priorities of the remotes involved.

If your working circumstances involve not always having access to all
the remotes you have configured, you can set the DEV_AVAILABLE_REMOTES
environment variable to a list of remotes that are available.  dev
will only try to operate on remotes that it currently thinks are
reachable, but things will probably not work too well if origin gets
excluded.

You can see what commands dev has by running dev with no arguments,
and you can get detailed help on each command by running dev help with
the command name.

Commands that expect to talk remotely:

backup: Force-pushes data to your personal remote.
clone-barclamps: Clones remote barclamps on to your local system and
  into your personal remote (if configured.)
fetch: Fetches changes from remotes into your local Got repository.
  Does not change working tree.
pull-requests-gen: Pushes placeholder branches for Github pull
  requests, issues pull requests using the Github API.
pull-requests-prep: Calls fetch internally.
push: Pushes branches to personal remote.
push-release: Pushes a release to a remote.
remote add: Probes for the existence of remotes, fetches changed from
  newly-configured remote.
remote set-url: Ditto
remote sync: Ditto.
scrub-merged-pulls: Erases placeholder branches for Github pull
  requests that have been merged.  Fetch calls this internally as needed.
setup: Clones barclamps, adds and syncs remotes, verifies Github ID
  and passwords.

All other commands operate just on your local repository.