A CLI tool for building, testing and composing OpenShift repositories.
The origin-ci-tool allows for developers to provision virtual machine environments locally using a hypervisor on the
development host and remotely in a cloud computing environment. These environments contain all of the major source code
repositories for the OpenShift ecosystem and can be configured to check out these repositories in whatever state necessary.
Release artifacts can be built from all of these repositories in concert and used to start an OpenShift cluster.
This tool is a successor to the OpenShift Vagrant plugin and is intended to support a superset of use-cases. Consult the Sample Use-Cases section to see some examples of specific workflows that are supported.
Prospective contributors should read the Contributing section to determine the best way to extend the functionality of this tool for their needs.
Table of Contents
This project requires Python 2.7 and the pip package manager for installation. On RHEL, CentOS and Fedora, install them with:
$ sudo yum install python2 python-pipOn Mac OS, the preferred method uses Homebrew. Installation instructions for Homebrew are on their
website. Using Homebrew, install Python (
brew will automatically install pip) with:
$ brew install pythonToday, full functionality of this tool requires a development version of Ansible to benefit from the following patches:
- New ec2_group_facts module to be able to get facts from EC2 security groups
- Factored polling std{out,err} reads into a function
- Add support to the make module for targets with options
- ec2_group_facts: Fail correctly when boto3 is not installed
- Only read EC2 regions_exclude list if necessary
Due to this requirement, it is highly suggested that you install this tool inside of a virtual environment for Python as the
installation requires source checkouts of Ansible and this tool. To create the virtual environment, the virtualenv package
will be necessary. Install it with:
$ sudo pip install virtualenvNow, create a virtual environment. In the following examples, the environment is named and created in a directory venv. On
Linux systems, you will want to allow the virtual environment to access system site packages, as running Ansible against the
local host requires Python bindings for yum and dnf, which cannot be installed in the virtual environment:
$ virtualenv venv --system-site-packagesOn Mac OS, the virtual environment can be created without this option:
$ virtualenv venvActivate the virtual environment:
$ source ./venv/bin/activateInstall Ansible and the origin-ci-tool in the virtual environment:
$ pip install git+https://github.com/openshift/origin-ci-tool.git --process-dependency-linksYou are now ready to use the oct CLI tool. If you want to use this virtual environment and get access to oct every time
you open a shell, add the activate line to your ~/.bashrc:
$ echo "source '$( pwd )/venv/bin/activate'" >> ~/.bashrcIf you want to exit the virtual environment in a shell, use the deactivate function:
$ deactivateOn Mac OS, the Python bindings for the AWS API are also necessary:
$ pip install boto boto3On Linux, some system dependencies are furthermore necessary. Install them using:
$ oct bootstrap selfIf you wish to develop and package VM images, further dependencies are required and can be installed with:
$ oct bootstrap self --for-imagesThe origin-ci-tool will place a directory of configuration files and runtime metadata to persist state between CLI
invocations. By default, this will be placed under ~/.config but can be configured to be under a custom directory by setting
the ${OCT_CONFIG_HOME} environment variable. Remember to add the ${OCT_CONFIG_HOME} environment variable to your
~/.bashrc if you are using a custom setting.
In general, configuration options for the origin-ci-tool can be accessed and changed with the following invocation, where
COMPONENT is a semantic grouping of configuration options like aws-client or ansible-defaults and KEY and
VALUE are the key-value pair to configure:
$ oct configure COMPONENT KEY VALUEConfiguration for a component can be reviewed with:
$ oct configure COMPONENT --viewCommunicating with the AWS API to provision virtual machines in EC2 requires a set of credentials. The origin-ci-tool uses
the same credential store as the AWS CLI. Detailed instructions for configuring the credential file are at the AWS User Guide, but the general flow is simple. If you have
the AWS CLI installed, simply run:
$ aws configureIf not, you'll want to place a file at ~/.aws/credentials with the following content:
[default] #<1>
aws_access_key_id=XXXXXXXX #<2>
aws_secret_access_key=XXXX #<3>- The name of the AWS credential profile. If this is not set to
default,${AWS_PROFILE}will need to be set to choose the correct profile to use. - The AWS secret access key ID. Consult the AWS documentation for more details.
- The AWS secret access key. Consult the AWS documentation for more details.
When setting up the SSH configuration for virtual machines provisioned in AWS EC2, the name and location of the private key used
to reach the instance need to be known by the origin-ci-tool. Configure them with:
$ oct configure aws-client keypair_name KEY_NAME
$ oct configure aws-client private_key_path /path/to/KEY_NAME.pemThe region in which to provision the cluster can be configured with:
$ oct configure aws-defaults region REGION_NAMEThe instance type to use for a master can be configured with:
$ oct configure aws-defaults master_instance_type TYPEWhen provisioning in AWS EC2, a number of high-level objects like virtual private clouds, subnets, security groups and elastic
load-balancers are necessary. By default, the origin-ci-tool does not create these objects when provisioning instances in EC2
to reduce the minimum permission level necessary to provision a cluster. Instead, objects of the correct type that are visible
are used if they have the correct tag. By default, the origin_ci_aws_cluster_component tag is used, but this can be changed
with:
$ oct configure aws-defaults identifying_tag_key KEY_NAMEThe acceptable value for this identifying tag for each component can also be configured. For instance, the default value for a
subnet that can be used as a master subnet is master_subnet. This configuration can be changed with:
$ oct configure aws-defaults master_subnet_tag_value KEY_VALUEInstead of determining the correct cluster component by searching through tags, it is possible to provide a comma-delimited list of literal identifiers to use:
$ oct configure aws-defaults master_subnet_ids subnet-XXXXXXXX,subnet-XXXXXXXX,subnet-XXXXXXXXThe root volume size in GB to use for a master (default: 35) can be configured with:
$ oct configure aws-defaults master_root_volume_size NUMBER_OF_GBWhen provisioning a local All-in-One VM, make sure that your local environment has the storage, CPU and memory required to support the VM, then run:
$ oct provision local all-in-one --os OS \ #<1>
--provider NAME \ #<2>
--stage STAGE #<3>- Select the operating system you would like to use with
--os. Fedora, Fedora-cgroupv2 and CentOS are supported. - Choose the virtualization provider to use. LibVirt, VirtualBox and VMWare Fusion are supported.
- Determine the image stage to base the virtual machine on. Valid image stages are
bare,baseandinstall. Only the bare OS stage is supported for VMWare Fusion.
By default, about five gigabytes of storage are necessary to start the machine; six gigabytes of RAM and two CPUs are
made available to the virtual machine. Fewer resources can be provided to the machine by providing the --memory and/or
--cpus flags to oct provision local all-in-one, but this is not recommended for workflows that compile the Origin
repository.
| Warning | The implementation of user-configured virtual machine memory and CPU limits is not complete. The above section will be relevant once issue #31 is finished. |
To access the machine, use SSH:
$ ssh openshiftdevelTo remove the VM, use:
$ oct deprovision| Note | Configure your AWS Credentials and Configuration before trying the following steps. |
To provision an All-in-One VM in the cloud, run:
$ oct provision remote all-in-one --os OS \ #<1>
--provider NAME \ #<2>
--stage STAGE \ #<3>
--name VM-NAME #<4>- Select the operating system you would like to use with
--os. Fedora, CentOS and RHEL are supported. - Choose the cloud provider to use. Only AWS is supported.
- Determine the image stage to base the virtual machine on. Valid image stages are
bare,baseandinstall. - Provide the identifier to use when naming the virtual machine on the cloud platform.
To access the machine, use SSH:
$ ssh openshiftdevelTo remove the VM, use:
$ oct deprovision| Note | Configure your AWS Credentials and Configuration before trying the following steps. |
Only certain configurations of clusters are available for provisioning using the origin-ci-tool. If a more fine-tuned setup
is necessary, direct interfacing with the OpenShift Ansible AWS reference architecture and/or provisioner is necessary.
To provision an OpenShift cluster, use:
$ oct provision remote cluster| Warning | The implementation of the full cluster provisioning logic is not complete. The above section will be relevant once issue #41 is finished. |
First, follow the steps in Provisioning an OpenShift Origin All-in-One Locally or Provisioning an OpenShift Origin
All-in-One in the Cloud. Then, make changes to a local checkout of a repository supported in the VM. For this example, we will
use the origin repository.
$ cd "${GOPATH}"/src/github.com/openshift/origin
# make some changes, optionally stage and/or commit them
$ oct sync local origin --branch BRANCH #<1>
$ oct make origin test-unit --env WHAT=pkg/changed/path #<2>- Sync the state of the repository on the local host to the remote host. Changes will be synced regardless of whether they are staged or committed. On the remote, a branch will be made with the same name and state as your local checkout.
- Interact with the repository on the remote host in some way.
First, set up a virtual machine and make some changes as described in Running Tests on Updated Code. Then, run:
$ oct build origin --follow-dependencies #<1>
$ oct install origin #<2>- Re-build the
originrepository and use the build artifacts (RPMs, binaries, container images) to re-build any other repositories that are downstream consumers of those artifacts. - Use the new artifacts to re-install the OpenShift Origin instance. If you need to re-install other downstream projects, use
separate
oct installdirectives.
| Note | If packaging local virtual machine images, install the required dependencies image_prerequisites_ before trying the following steps. |
To package a local virtual machine into a re-useable image, use:
$ oct package vagrant --update \ #<1>
--bump-version TYPE \ #<2>
--serve-local #<3>- Update the current image stage, or
--upgradeto create an image for the next stage. - Strategy for updating the Vagrant box semantic version, can be
major,minor,patchornone. - Configure the Vagrant box to pull the new image from it's location on disk, or
--serve-remoteto write the URL under the OpenShift mirror.
| Warning | The implementation of provisioning from a local image source file is not complete. The above section will be relevant once issue #30 is finished. |
| Note | Configure your AWS Credentials and Configuration before trying the following steps. |
| Note | Packaging images from virtual machines in the cloud is only supported when there is only one virtual machine provisioned. |
To package a remote virtual machine into a re-useable image, use:
$ oct package ami --update #<1>- Update the current image stage, or
--upgradeto create an image for the next stage.
When a new image is created for the bare or base image stages, it is not known if the image will support the full
OpenShift build and install. Therefore, it is possible to execute whatever build, installation or test actions are necessary on
the virtual machine, then use the following command to mark the image previously created from the virtual machine as ready for
consumption:
$ oct package ami --mark-readyThis action will change the ready tag value from no to yes on the remote image.
The core design principle behind the origin-ci-tool is that it should contain the smallest amount of logic possible. The
largest lesson learned from the Vagrant plugin for OpenShift was that
internalizing repository-specific logic led to a bloated code-base that could neither support all of the use-cases that the
repositories wanted nor could be update quickly when repositories needed changes in behavior. For this reason, all of the
interaction that the origin-ci-tool has with repositories is through make targets. This allows the origin-ci-tool
to provide a low-level oct make REPO TARGET command that can be utilized to support whatever custom workflow any repository
needs.
A second but nonetheless critical design principle is dog-food. In
the past, a large proliferation of provisioning, installation and configuration solutions was created by members of the
OpenShift community because no simple central utilities existed. The OpenShift Productization team now supports a full-featured installation and configuration path using Ansible
in their OpenShift Ansible repository. Reference architectures and
implementations of provisioning solutions exist in the OpenShift Ansible contributions repository. The origin-ci-tool utilizes these tools to ensure that we
eat our own dog-food.
- When adding to this project, therefore, it is necessary to ask:
- is this change adding repository-specific business logic to the
origin-ci-tool? - should this change instead be contributed to an upstream solution for the OpenShift community to share?
- is this change adding repository-specific business logic to the
It is not certain that the origin-ci-tool can support any repository generically, so integrating with a new repository
requires changes to the codebase. New repositories need to be added to the Repository enumeration in
cli/util/repository_options.py:
class Repository(object):
"""
An enumeration of repository names that are currently
supported as a part of the OpenShift ecosystem.
"""As the origin-ci-tool interacts with repositories using make, your repository will need a Makefile in the repository
root with whatever targets are necessary. If you wish for the origin-ci-tool to support helpful commands like oct
build, oct install, oct test, and/or oct download, you will need to place a .oct-config.yml file in your
repository root. The file as described below contains four lists in normal YAML syntax. The
build, install, and test entries list make targets. The download list contains directories or files that the
origin-ci-tool will download from a remote host.
---
build: <1>
- build-release <2>
ENVAR: value <3>
install: <4>
- install-onto-cluster
test: <5>
- test -o build-release <6>
download: <7>
- /tmp/myrepo- If present, this list of
maketargets will be called when a user invokesoct build REPO. - A
maketarget can be presented in-line. - Options for the
maketarget are provided as key-value pairs, descendant from the target entry. - If present, this list of
maketargets will be called when a user invokesoct install REPO. - If present, this list of
maketargets will be called when a user invokesoct test REPO. - Complicated
makeinvocations can be provided for the target. - If present, this list of absolute paths will be downloaded from the remote host when a user invokes
oct download REPO-artifacts
| Warning | The implementation of the .oct-config.yml configuration file is not complete. The above section will
be relevant once issue #29 is finished. |
The main means by which automated tests verify that the origin-ci-tool functions is by ensuring that a specific CLI
invocation results in the correct Ansible playbook being called with the correct variables. In order to run the unit tests,
install the test-specific dependencies first. To get the dependencies and a version of oct that tracks the source, run the
following command from the origin-ci-tool source directory:
$ make install-developmentStatic analyzers for source code style, format and common errors can be run with:
$ make verifyAll of the unit tests can be run with:
$ make testIn order to see stack traces from the test harness, use the environment ${SHOW_STACK_TRACE}:
$ SHOW_STACK_TRACE=true make testSpecific modules, test classes and test methods can be supplied using ${TARGET} as they would be supplied to the unittest
module normally:
$ TARGET=oct.cli.provision.local.tests.test_all_in_one.ProvisionVagrantTestCase make testThe code coverage report can be generated to htmlcov/index.html with:
$ coverage html