Lab 7: Implementing DevSecOps to Build and Automate Security into the Application in a Secure CI/CD Pipeline
Lab Length: Medium/Average
The goal of this lab is to learn how to automate and build security into the application and build a secure software factory that orchestrates a combination of security tools including SonarQube, OWASP Zed Attack Proxy (ZAP), Red Hat Quay, Clair and the Open Security Content Automation Protocol (OpenSCAP) in a Continuous Integration / Continuous Delivery (CI/CD) pipeline (via Jenkins). We will also automatically detect security and compliance issues in a controlled way for automated security compliance at scale.
The benefits of DevOps and Continuous Integration / Continuous Delivery (CI/CD) have been demonstrated with great success over the years. Security was initially treated as a second class citizen in many cases - left until the end usually to much detriment. With the advent of DevSecOps, security is put front and center - it is addressed in terms of people, processes and technology. Security tools are integrated right into the build process and you could easily break the build if security requirements are not met with the security gates that you build into your CI/CD pipeline.
This lab exercise will focus on some of the technologies used to implement automated security compliance controls within a typical CI/CD application pipeline.
A number of tools have been installed and pre-configured to support the DevSecOps pipeline. Most of these tools are running containerized within the Red Hat OpenShift cluster. Here is the pipeline we will be stepping through during our lab:
The Ansible role that deploys this entire pipeline can be found here. This pipeline will checkout java web application source code from a git repository, build it, run automated tests, perform a static code analysis scan with SonarQube, send the produced artifact to Sonatype Nexus for storage, build a docker image containing our application, push it to quay for additional security scanning, perform OpenSCAP scans, deploy the application to OpenShift so that it can be scanned for web vulnerabilities by OWASP ZAP, then we have the opportunity to promote the application to other OpenShift projects if we are satisfied with the results. The following sections describe the tools we will be using in this pipeline lab.
Gogs is the Go Git Service- a github-like git server written in the Go programming language.
The goal of the Gogs open source project is to make the easiest, fastest, and most painless way of setting up a self-hosted Git service. With Go, this can be done with an independent binary distribution across ALL platforms that Go supports, including Linux, Mac OS X, Windows and ARM.
Gogs is great ligtweight git server for use in lab environments such as this. Most enterprise projects will use more fully-featured git repositories such as GitHub Enterprise, Gitlab, Atlassian BitBucket, etc.
Jenkins is an open-source continuous integration software tool written in the Java programming language for testing and reporting on isolated changes in a larger code base in real time. The software enables developers to find and solve defects in a code base rapidly and to automate testing of their builds.
While there are other tools available for implementing CI/CD pipelines, jenkins is the most widely used. A containerized version is provided with OpenShift and integrates natively with it.
Apache Maven is a software project management and comprehension tool. Based on the concept of a project object model (POM), Maven can manage a project’s build, reporting and documentation from a central piece of information.
Maven provides an opinionated framework defining a structure and layout for Java source code. By following this structure, a myriad of plugins can be used to simplify the build, documentation, testing and deployment of source code. The pipeline in this lab uses maven to build and package our source code as well as transmit the application to nexus (described below) and submit static code analysis requests to SonarQube (also described below).
Sonatype Nexus is a repository manager. It allows you to proxy, collect, and manage your dependencies so that you are not constantly juggling a collection of Java ARchives (JARs). It makes it easy to distribute your software. Internally, you configure your build to publish artifacts to Nexus and they then become available to other developers.
When we build our application as a part of the lab, the resulting JAR file will be stored within nexus.
SonarQube is a static code analysis system that provides continuous inspection including the detection of common code issues, including in the areas of security.
In this lab we will upload the source code to SonarQube, which will perform static code analysis and produce various metrics, such as unit test code coverage.
Red Hat Quay is a private container registry that stores, builds, and deploys container images. It analyzes your images for security vulnerabilities (using Clair), identifying potential issues that can help you mitigate security risks. The pipeline will send our application image to quay where it will be automatically scanned for security vulnerabilities by Clair.
The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications. Its also a great tool for experienced pentesters to use for manual security testing.
In our pipeline, we will stand up an instance of our web application then point OWASP ZAP to that instance so that it can perform a baseline security scan.
For the purposes of this lab, we will be examining a simple, all-in-one spring boot application. The application presents a simple e-commerce web user interface, which is backed by a self-contained restful API.
-
After following Lab 0 and choosing TC9818 -AD - Implementing proactive security and compliance automation and DevSecOps from the drop down, you will be presented with your user id and GUID.
-
Log in to each of the tools that we previously mentioned with the credentials shown below. Replace {GUID} with your provided lab cluster GUID and replace {USERID} with your provided user id - e.g., user1 . Both the lab cluster GUID and USERID can be found on the Lab Information page where you got your assigned GUID and USERID. (see previous step for more details)
-
Red Hat OpenShift console - https://master.{GUID}.example.opentlc.com
login: {USERID}
password: r3dh4t1!
-
Gogs - http://gogs-ocp-workshop.apps.{GUID}.example.opentlc.com
login: {USERID}
password: openshift
-
Jenkins - https://jenkins-{USERID}.apps.{GUID}.example.opentlc.com (Click on Log In with OpenShift)
login: {USERID}
password: r3dh4t1!
When prompted, press the Allow selected permissions button.
The main Jenkins page will appear as below:
-
Nexus - http://nexus-ocp-workshop.apps.{GUID}.example.opentlc.com
no login necessary
-
Sonarqube - http://sonarqube-ocp-workshop.apps.{GUID}.example.opentlc.com
no login necessary
-
Quay - http://quayecosystem-quay-quay-enterprise.apps.{GUID}.example.opentlc.com
login: admin
password: admin123
-
The following sections describe each of the stages in the CI/CD pipeline that we will be running during this lab.
This stage checks the source code out the gogs git repository. Once the code is checked out, we can proceed with the rest of the pipeline.
This stage uses Apache Maven to build our spring boot Java application from source. If the build fails, our pipeline will terminate since we cannot proceed with the remaining steps
This stage again uses Maven to run any unit tests that were written. Unit tests are important in that they prevent our code from becoming brittle. Without unit tests we can not be confident that any changes made to the source code will not adversely affect other parts of the code. By running unit tests we can also collect metrics in terms of what percentage of the code is being exercised by the tests. This is called code coverage and will be visible within SonarQube.
During this stage we run the Maven SonarQube plugin to send the code, unit test results and code coverage metrics to our sonarqube server for static code analysis.
The result of our build is the creation of a Java ARchive, or JAR file. This file is the Java bytecode that is run to deploy our web application. We send the JAR file to nexus for storage so that it can be retrieved in the future if needed (rather than having to rebuild the same version of code). This also allows our artifacts to be shared easily to other interested parties. The supported version of SonaType Nexus also has the ability to analyze any dependencies (libraries) used by our uploaded code to detect potential known vulnerabilities they may introduce.
During this step, we use the Dockerfile provided along with our source code to construct a new docker image. A Dockerfile contains the instructions for building a docker image. The Dockerfile we are using to deploy the ecommerce application takes a base, Red Hat provided OpenJDK 1.8 image (Open JDK is the Java Development Kit - the runtime required to execute a Java application) and superimposes our executable JAR file on top of it to form a new Docker image, capable of running our application.
In this stage we take the docker image that we just built and push it into Red Hat Quay - our centralized image registry. Once the image is pushed to a registry, it can be made available to interested parties and promoted to different OpenShift or kubernetes based clusters. Quay has also been configured with Clair - which will perform a vulnerability scan of any images pushed to Quay. The Clair report will become available shortly after the image has been pushed into Quay.
This will perform two OpenSCAP scans. A vulnerability scan and a Defense Information Systems Agency (DISA) Security Technical Implementation Guideline (STIG) compliance scan. Both reports will be collected by Jenkins and made available for viewing.
This step will deploy the application into your Openshift user project, which will serve as a development environment. This environment is typically used by developers to verify applications come up as expected and also for performing automated integration tests.
During this stage a Jenkins Agent Pod will be started within OpenShift for the purposes of performing an OWASP ZAP baseline scan against the running container image. The scan will analyze and crawl starting from the main page of our web application, searching for potential vulnerabilities lurking within javascript including things such as potential cross-site scripting (XSS) exploits. The resulting report will also be published by Jenkins.
Configures the OpenShift staging project (i.e., userid-stage) so that the image can be promoted into it
This step pauses the pipeline and waits for an authorized person (e.g., a QA test lead) to confirm promotion into a higher level environment.
-
Navigate back to the jenkins user interface https://jenkins-{USERID}.apps.{GUID}.example.opentlc.com
-
Click the folder label with your user id (e.g., user1)
-
Click the pipeline (e.g., user1/user1-ecommerce-pipeline)
-
Click the Build with Parameters link in the left menu
-
Keep the default values and press the Build button
-
Click the build label (e.g., #1) next to the build in the Build History pane on the left
-
Click the Console Output link to monitor the build progress
-
Once the build has completed successfully (and is prompting you to proceed or abort), click the user#/user#-ecommerce-pipeline link at the top of the display. We will inspect the security scan artifacts before allowing the pipeline to promote our code into the staging project. Do not press either of the Proceed or Abort links at this time.
-
Click the the drop down arrow next to the build number in the left hand menu. Select Open Blue Ocean
Click Artifacts in the blue ocean view:
-
Then view each of the reports. You may need to refresh the browser.
-
OpenSCAP Compliance Report
Experiment with the filters to adjust the output of the report.
-
OpenSCAP Vulnerability Report
Note the number of vulnerabilities identified by the OpenSCAP Report:
-
OWASP ZAP Baseline Report
-
-
Navigate to the SonarQube url http://sonarqube-ocp-workshop.apps.{GUID}.example.opentlc.com Click on the number link above Projects Analyzed section:
-
Find the project prefixed with your userid. Note 3 vulnerabilities were found and that the unit test code coverage has been recorded. SonarQube also provides code metrics on items including potential bugs and code smells. Click the Project prefixed with your userid and determine where the vulnerabilities are in the source code.
-
Navigate to the nexus url http://nexus-ocp-workshop.apps.{GUID}.example.opentlc.com, click browse then maven-snapshots
-
Navigate the folder structure and verify your JAR file exists within it.
-
Navigate to the quay url http://quayecosystem-quay-quay-enterprise.apps.{GUID}.example.opentlc.com, login as admin with the password admin123 if you haven’t already
-
Navigate to the image tagged with your userid by clicking the ecommerce repository
-
Click the tag icon on the left (second icon from the top) then click the image hash id
-
Click the bug icon to see the vulnerabilities detected by Clair. Make a note of the number of vulnerabilities
-
Click the package icon to see the packages that are affected
-
You can also view the deployed application at http://ecommerce-{USERID}.apps.{GUID}.example.opentlc.com/
An old base image was used to build the application, we will update the base image to use a newer version. This will reduce the number of vulnerabilities that are detected. We will also update the source code to remove the vulnerabilities detected by SonarQube.
-
Navigate to your source code repository in gogs http://gogs-ocp-workshop.apps.{GUID}.example.opentlc.com and login if you haven’t already (userid / openshift)
-
Click on the SecurityDemos repository under My Repositories
-
Click the Dockerfile
-
Click the edit icon (small pencil) and change the image version from 1.0 to latest.
The current line in the Dockerfile is referring to an older image, with associated vulnerabilities. The latest image we are using will resolve many (potentially all) of these known vulnerabilities. You can explore the Red Hat Container Catalog to search for images provided by Red Hat as well as inspect their current security posture.
FROM registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift:latest
-
Click the Commit Changes button
NoteFor the purposes of this lab we are committing changes directly into the master branch of our git repository for demonstration purposes. A true development environment will have processes and procedures for submitting and peer reviewing code changes before they are accepted into a master branch. 
-
Navigate back to the root of the SecurityDemos folder
-
Click down through the following folders src → main → java → com → baeldung → ecommerce → controller → OrderController.java
-
Edit the file and remove line 28, which has a hardcoded password. This password is not actually used by the application so it is safe to remove it.
-
Click the commit changes button
-
Navigate back to jenkins and abort the current build, then navigate to the pipeline page
-
Start another build by clicking Build with Parameters
-
Then click the Build button
-
Once the build has started,navigate to the blue ocean view of the build. First select the current build
-
Click the Open Blue Ocean Link
-
View the build progress
-
Once the build is complete review the reports, quay vulnerability scan and sonarqube to verify that the number of vulnerabilities has been reduced. Click Artifacts in the blue ocean view:
-
Then view each of the reports. You may need to refresh the browser.
-
After viewing the reports, feel free to promote the application image into the staging and prod projects:
-
The staging version of the app can be viewed here:
-
The production version of the app can be viewed here:
-
Time permitting, go back to your gogs repository, open up the Jenkinsfile and take a look at some of the pipeline implementation details.
