Skip to content

Latest commit

 

History

History
 
 

ci-operator

README.md

ci-operator

This document describes how to create CI jobs for Openshift components using ci-operator and is intended for component developers who want to add tests to their CI process.

After editing the files under this directory, make sure to run the generator to ensure that your changes are compliant to our conventions and pass the CI tests that will run when you submit your changes as a PR:

docker pull registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest
docker run -it -v $(pwd)/ci-operator:/ci-operator:z           \
  registry.svc.ci.openshift.org/ci/ci-operator-prowgen:latest \
  --from-dir /ci-operator/config/ --to-dir /ci-operator/jobs

Make sure to pull the latest image to ensure that you do not use a stale version of the generator. Pre-submit tests on this repository will ensure that a run of the latest generator does not error on proposed configuration changes and also does not generate any new configuration.

Conventions

Under this directory, we have three main directories:

  • config/$org/$repo/$org-$repo-$branch.yaml Contains your ci-operator definition which describes how the images and tests in your repo works. These files are copied into config maps on the CI cluster and referenced by Prow jobs. If you are building branches within a fork of a repo in another organization, $repo should point to the fork that holds the branch (for example github.com/openshift/kubernetes-metrics-server instead of k8s.io/metrics-server).
  • jobs/$org/$repo/$org-$repo-$branch-(presubmit|postsubmit|periodic).yaml Contains Prow job definitions for each repository that are run on PRs, on merges, or periodically. When we branch jobs, we will copy the current master jobs into a release branch specific job. Each prow job calls into the appropriate subset of the tests defined in your ci-operator config and passes in the secrets and infrastructure info specific to our CI environment
  • templates/*.yaml These templates are used for more complicated jobs that don't run in a single pod. The templates are referenced by the Prow jobs and are instantiated by the ci-operator using parameters generated by the build (references to images usually).

End-to-end tests

This section describes how to configure end-to-end tests using ci-operator. In this context, "end-to-end" means the functionality of the application is being tested on top of a Kubernetes cluster from an end-user perspective.

The preferred way to write this type of tests is using ci-operator. See the documentation for details on how to download, build, and execute it:

https://github.com/openshift/ci-tools.git

ci-operator requires a configuration file for the repository being tested. These files are located in the config directory. The ci-operator repository has documentation for adding a new configuration file in case one doesn't already exist. These files take care of most of the CI process: downloading the source, building binaries, building RPMs, creating images, executing unit tests, etc., and can be built upon for e2e tests with little or no modification.

To add an e2e test:

  1. Determine the pre-requisites for the test. Practically, this means choosing the ci-operator template according to the type of test. There are already files in the templates directory for the most common cases, see Using a template below.
  2. Determine and configure the template's inputs. This is specific to each template and should be documented in its parameters. The e2e test might need minor modifications to fit the environment created by the template.
  3. Add one or more Prow jobs to Prow's configuration file with the information gathered in the previous steps.

Provisioning a cluster

Contrary to other types of tests, e2e tests usually require a cluster, not just a single container. While there aren't yet native primitives in ci-operator for cluster provisioning, it provides one open-ended feature that can be leveraged to accomplish that: template steps.

Template steps allow the creation of arbitrary objects in the cluster where the CI pipeline is executed. This is used to start a pod that will then provision a separate cluster for the tests. This directory already contains a few templates that can be used either directly or (rarely necessary, in practice) as a reference:

  • cluster-launch-e2e.yaml: launches a cluster in GCP using openshift-ansible and runs Origin e2e tests on it, parameterized by test focus.
  • cluster-launch-src.yaml: launches a cluster in GCP using openshift-ansible and runs a script from the repository being tested with the resulting $KUBECONFIG, parameterized by test script.
  • cluster-launch-installer-e2e.yaml: same as cluster-launch-e2e.yaml, but uses openshift-installer instead of openshift-ansible.
  • cluster-launch-installer-src.yaml: same as cluster-launch-src.yaml, but uses openshift-installer instead of openshift-ansible.
  • master-sidecar-4.2.yaml: spins up a simple openshift control plane as a sidecar and waits for the COMMAND specified to the template to be executed, before itself exiting. The test container is given access to the generated configuration and the admin.kubeconfig.

To access the cluster, the test should use the standard configuration loading rules, which are described in the upstream documentation:

https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api

Using a template

The preferred way to add a test that uses a template is to add it to the ci-operator configuration file and use the configuration generator to generate the job. The list of supported test types can be found in the configuration documentation.

The process for adding jobs manually is significantly more complex. For templates that are expected to be used by many jobs, it may be easier to add support for automatic generation. The example job in the next section can be used as a reference for jobs that are not in that category.

Writing a template

This section covers the process of creating a new template when none of the existing ones provide the workflow required for a particular type of test — e.g. when a new installer needs to be supported. It supplements the ci-operator template documentation.

From the perspective of an end-to-end test, a ci-operator template is simply a way to create one or more pods and auxiliary objects to setup and clean up the environment and execute the test.

While users should deal mostly with the ci-operator configuration file and generate Prow jobs automatically from it, the structure of the Prow jobs has to be taken into consideration when writing a template. The e2e-conformance-k8s job in openshift/origin, which uses the cluster-launch-src.yaml will be used as an example:

https://github.com/openshift/release/blob/master/ci-operator/config/openshift/origin/openshift-origin-master.yaml https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/origin/openshift-origin-master-presubmits.yaml https://github.com/openshift/release/blob/master/ci-operator/templates/cluster-launch-src.yaml

The CI process begins when a webhook from Github triggers the creation of one or more Prow jobs. For a complete description of Prow jobs, see the upstream documentation.

# ci-operator/jobs/openshift/origin/openshift-origin-master-presubmits.yaml
presubmits:
  openshift/origin:
  # …
  - agent: kubernetes
    always_run: true
    # Each branch needs its own ci-operator configuration file.
    branches:
    - master
    context: ci/prow/e2e-conformance-k8s
    decorate: true
    # The name should follow the format used for auto-generated jobs:
    # pull-ci-$org-$repo-$branch-$name or branch-ci-$org-$repo-$branch-$name.
    # "e2e-conformance-k8s" is used as a unique identifier for for this job
    # thoughout the job definition (e.g. in `context` above).
    name: pull-ci-openshift-origin-master-e2e-conformance-k8s
    rerun_command: /test e2e-conformance-k8s
    # ci-operator doesn't require the source code of the repository, it will
    # be cloned in a separate container.
    skip_cloning: true
    spec:
      containers:
      # The names passed to `--secret-dir`, `--target`, and `--template` are
      # important and should follow the format presented here.
      - args:
        - --artifact-dir=$(ARTIFACTS)
        - --give-pr-author-access-to-namespace=true
        # `--secret-dir` references a directory that is volume-mounted in the
        # container by combining secrets and configmaps from the cluster. This
        # is one way of passing extra configuration as input to the template.
        - --secret-dir=/usr/local/e2e-conformance-k8s-cluster-profile
        - --target=e2e-conformance-k8s
        # The template is stored in a configmap in the cluster and
        # volume-mounted in the container.
        - --template=/usr/local/e2e-conformance-k8s
        command:
        - ci-operator
        # Other than CONFIG_SPEC, these are specific to the template being
        # used.
        env:
        - name: CLUSTER_TYPE
          value: gcp
        # The ci-operator configuration stored in a configmap in the cluster.
        - name: CONFIG_SPEC
          valueFrom:
            configMapKeyRef:
              key: openshift-origin-master.yaml
              name: ci-operator-master-configs
        - name: JOB_NAME_SAFE
          value: e2e-conformance-k8s
        - name: TEST_COMMAND
          value: test/extended/conformance-k8s.sh
        image: ci-operator:latest
        imagePullPolicy: Always
        name: ""
        resources:
          requests:
            cpu: 10m
        volumeMounts:
        - mountPath: /usr/local/e2e-conformance-k8s-cluster-profile
          name: cluster-profile
        - mountPath: /usr/local/e2e-conformance-k8s
          name: job-definition
          subPath: cluster-launch-src.yaml
      serviceAccountName: ci-operator
      # Specific to the template being used. Combine a secret and a configmap
      # into a directory that will be copied to the namespace created by
      # ci-operator using the `--secret-dir` option.
      volumes:
      - name: cluster-profile
        projected:
          sources:
          - secret:
              name: cluster-secrets-gcp
          - configMap:
              name: cluster-profile-gcp
      # The template stored in a configmap in the cluster.
      - configMap:
          name: prow-job-cluster-launch-src
        name: job-definition
    trigger: ((?m)^/test( all| e2e-conformance-k8s),?(\s+|$))

The Secrets and ConfigMaps referenced by the job reside in the ci namespace. cluster-profile-* are ConfigMaps that contain the cluster profiles in this repository. cluster-secrets-* are Secrets that contain credentials to provision and access clusters in a specific cloud provider (the contents can be seen in the script that populates them.

Inputs

When instantiating the template, data about the pipeline is provided as parameters. The location of images and RPMs from both the release and the CI pipeline is available this way. Extra parameters can be provided via environment variables, which will have to be set by the Prow job.

External access to the images that were built in the test namespace is required by most end-to-end tests, so templates often create this role binding:

- kind: RoleBinding
  apiVersion: authorization.openshift.io/v1
  metadata:
    name: ${JOB_NAME_SAFE}-image-puller
    namespace: ${NAMESPACE}
  roleRef:
    name: system:image-puller
  subjects:
  - kind: SystemGroup
    name: system:unauthenticated

The template can reference objects from any namespace, but Kubernetes requires them to be in the same namespace to be used as volume mounts. As described in the section above, the Prow job definition and ci-operator's --secret-dir can be used to combine objects into a volume mount and make them available in the test namespace.

Outputs

The outputs of a template test are:

  • Success/failure status, determined from the test pod.
  • The pod's stdout and stderr, reflected in ci-operator's output in case of failure.
  • Artifacts.

These are described in more detail in the ci-operator documentation.

Adding a template

With the template file ready, the steps required to add it to the repository and make it available for CI jobs are:

  1. Create the yaml file in the templates/ directory.
  2. Add the files to the config-updater section of Prow's configuration file to ensure they are added to a ConfigMap in the CI cluster.
  3. Optional: add a test type to ci-operator to enable automatic generation of jobs that use this template.
  4. Add necessary secrets (if any) to the deployment configuration in this repository and apply it to the cluster.

Because the configuration updater configuration has to be updated before a PR with the files is merged, those changes have to be merged previously in a separate PR.

Testing a template

A job that uses a template can be tested in two different ways. The mkpj and mkpod tools can be used to create a pod that will reproduce the setup used in CI, although some customization is still possible by editing the job locally. For complete control of the execution, the more manual process of assembling the ci-operator call can be used.

Testing with mkpj and mkpod

Prow has a utility to generate a ProwJob from its configuration files and another to turn that into a Pod ready to be executed. These utilities are available in upstream container images (gcr.io/k8s-prow/mkpj and gcr.io/k8s-prow/mkpod) or can be compiled from the upstream repository.

The mkpjpod.sh script can be used to streamline that process. This script outputs the yaml for a pod that runs the specified test job and reports the status back into the referenced pull request. You can pipe the output of the script to oc create -f - and launch the pod in a cluster somewhere. For Red Hat employees working on OpenShift, you can use the ci-stg namespace in the CI cluster to run test jobs.

For example, say that we want to test the pull-ci-openshift-origin-master-e2e-aws job against a pull request to openshift/origin. We'll specify the following pieces of information:

  • the repository: openshift/origin
  • the base branch of the repository to test: master
  • the pull request to merge onto the base branch: 21526
  • the github user that should have access to the namespace where the tests are run: yourgithubname
  • the job to run: pull-ci-openshift-origin-master-e2e-aws
hack/pj_env.py \
    openshift/origin \
    master \
    21526 \
    'yourgithubname' \
    hack/mkpjpod.sh pull-ci-openshift-origin-master-e2e-aws \
    | oc -n ci-stg create -f -

The script produces a log like:

time="2019-01-18T14:53:08Z" level=warning msg="No BuildID found in ProwJob status or given with --build-id, GCS interaction will be poor."
pod/c47e597f-1b30-11e9-8b3b-661cd79effa8 created

From which you can follow the job logs:

oc -n ci-stg logs -c test -f c47e597f-1b30-11e9-8b3b-661cd79effa8

where you can determine the namespace where the ci-operator is running the job. The logs have a line like:

2019/01/18 14:55:20 Using namespace ci-op-3167h4v3

This will allow to determine the installer logs:

oc logs -f e2e -c setup  -n ci-op-3167h4v3

Note, it is important to supply the name of your github account in order to be granted RBAC permission to view pods in the created test namespace.

You'll be able to see the test logs in the pod where ci-operator runs the job:

oc logs -f e2e -c test -n ci-op-3167h4v3

You can also see the documentation in the script file and the following screencasts for details:

  • Basic usage of the script to reproduce container and template tests: asciicast
  • Making changes to the job or ci-operator configuration file: asciicast

Testing manually

ci-operator tests can be executed locally with little effort, but setting up the dependencies for template tests is more involved. The typical end-to-end test requires:

  • A kubeconfig pointing to a cluster with external access.
  • The ci-operator configuration file.
  • The template file.
  • The secrets required for the --secret-dir option, if applicable.
  • The environment variables required, if applicable.

The simplest way to get started is to create a personal namespace in the CI cluster. Substitute mynamespace below with the name of that namespace.

The secrets and environment variables are very specific to the template in use, but the e2e-conformance-k8s can be used as a general example. The template it uses (cluster-launch-src) requires two parameters (the others are all provided by ci-operator): CLUSTER_TYPE determines the cloud provider used to provision the cluster, and TEST_COMMAND is the command that executes the test.

This template also requires a secret containing the cluster profile and credentials. In the CI cluster, it is created using a volume that combines the projection of a Secret and a ConfigMap. Locally, it has to be assembled into a directory manually. How these objects are composed is described in the "writing a template" section above. One final note: because the name of the secret is determined by the argument passed to --secret-dir, the directory has to be named in a way that reflects the secret name expected by the template.

Putting this all together, to execute the e2e-conformance-k8s test the following command can be used:

name=mytestname
CLUSTER_TYPE=gcp
mkdir artifacts/ "$name-cluster-profile"/
ln -s "$PWD/cluster/test-deploy/$CLUSTER_TYPE/"* "$name-cluster-profile"/
# populate the following files in the $name-cluster-profile directory:
# - gce.json
# - ops-mirror.pem
# - ssh-privatekey
# - ssh-publickey
# - telemeter-token
export CLUSTER_TYPE JOB_NAME_SAFE=$name TEST_COMMAND=test/extended/conformance-k8s.sh
ci-operator \
    --artifact-dir artifacts/ \
    --config ci-operator/config/openshift/origin/openshift-origin-master.yaml \
    --git-ref openshift/origin@master \
    --template ci-operator/templates/cluster-launch-src.yaml \
    --target cluster-launch-src \
    --secret-dir "$name-cluster-profile/" \
    --namespace mynamespace