Skip to main content

10. Building Out Our Pipeline

Will DowerAbout 7 min

More Pipeline Steps

Let's make this pipeline deploy, harden, validate, and verify an NGINX container.

Prep Steps

First, we need to make sure that the node that runs our pipeline will have access to the executables it needs. By default, Gitub's runners have quite a bit of software pre-installed, including Docker and Ansible (see the full list hereopen in new window). However, the Ubuntu image we are using does not have InSpec installed, nor does it have a copy of our test code. Let's add to our pipeline file to fix this.

Adding More Steps
- name: PREP - Install InSpec executable 
  run: curl https://omnitruck.chef.io/install.sh | sudo bash -s -- -P inspec -v 5

- name: PREP - Check out this repository  # because that's where our profile is!
  uses: actions/checkout@v3

The first new step installs the InSpec executable using the install instructions for Ubuntu as given hereopen in new window. Remember that GitHub gives us a brand-new runner node every time we execute the pipeline; if we don't install it and it isn't on the pre-installed software list, it won't be available!

Actions

The next step ("PREP - Check out this repository") is our first one to use an Action. Actions are pre-packaged pipeline steps published to the GitHub Marketplaceopen in new window. Any project or developer can publish an Action to the Marketplace as part of the GitHub Actions ecosystem. Most other orchestration tools for pipelines have a similar plugin system.

We can use Actions as shortcuts in the same way we use InSpec resources to abstract out quite a bit of test code logic. Actions are invoked with the uses attribute in a step in place of the run attribute we have been using so far, which simply executes a terminal command.

This Action in particular is one of the most common -- checkoutopen in new window. If called with no other attributes attached to it, it simply checks out and changes directory into the repository where the workflow file lives to the runner that is currently executing the workflow. We need to do this to make sure we have access to InSpec profile you created earlier!

Linting

Most CI pipelines will also include a lint step, where the code is statically tested to make sure that it does not contain errors that we can spot before we even execute it, and to make sure it is conforming to a project style guide. For our purposes, it's a good idea to run the inspec check command to ensure that InSpec can recognize our tests as a real profile.

::: Note We can run InSpec inside this runner now because we installed it in a prior step!
:::

Let's add the lint step:

Adding Lint Step
- name: LINT - Run InSpec Check           # double-check that we don't have any serious issues in our profile code
  run: inspec check $PROFILE

Deploy Test Container

We now have inspec exec and the my_nginx profile available in our pipeline. Now we need the image we're going to harden.

Luckily, the Ubuntu runner we are using already has the Docker Engine installed, so we can deploy a container easily. We will deploy the same container image we have been using in this class so far. We will also name it nginx to keep things consistent, but recall that this container is running on a GitHub cloud runner, not inside your codespace like your local containers we've been using for prior classwork.

We'll also need to make sure that our test target has Python installed, since that's how Ansible will connect to it later to harden it.

(You didn't have to do that for your local NGINX container because the build-lab.sh script did all that config for you.)

Adding Deploy Steps
- name: DEPLOY - Run a Docker container from nginx
  run: docker run -dit --name nginx nginx:latest

- name: DEPLOY - Install Python for our nginx container
  run: |
    docker exec nginx apt-get update -y
    docker exec nginx apt-get install -y python3

Multiline `run` commands

You can run a multiline script in a single run step by passing it as a multiline string, which is denoted with a pipe character (|).

Hardening

Alright, if our pipeline makes it this far, then we have installed InSpec, pulled our profile, checked it for errors, and deployed a test target. It's time to harden the target.

In our case, we're going to borrow an open-source Ansible role for NGINX that is part of the SAF Hardening Libraryopen in new window. If you took the SAF User Class, you might recognize this role as what you ran manually during the Hardening section of that class. Again, we are borrowing some of the steps from the lab setup script and executing them against our runner system, for convenience.

Let's add the Hardening steps now.

Adding Harden Steps
- name: HARDEN - Fetch Ansible role
  run: |
    git clone --branch docker https://github.com/mitre/ansible-nginx-stigready-hardening.git || true
    chmod 755 ansible-nginx-stigready-hardening

- name: HARDEN - Fetch Ansible requirements
  run: ansible-galaxy install -r ansible-nginx-stigready-hardening/requirements.yml

- name: HARDEN - Run Ansible hardening
  run: ansible-playbook --inventory=nginx, --connection=docker ansible-nginx-stigready-hardening/hardening-playbook.yml

Validation

Time for the main event. Our gold image container should now be hardened, but we need to prove it.

Let's run InSpec:

Adding Validate Steps
- name: VALIDATE - Run InSpec
  continue-on-error: true                 # we dont want to stop if our InSpec run finds failures, we want to continue and record the result
  run: |
    inspec exec $PROFILE \
    --input-file=$PROFILE/inputs.yml \
    --target docker://nginx \
    --reporter cli json:results/pipeline_run.json

- name: VALIDATE - Save Test Result JSON  # save our results to the pipeline artifacts, even if the InSpec run found failing tests
  uses: actions/upload-artifact@v3
  with:
    path: results/pipeline_run.json

You may notice that the step that runs InSpec sets an attribute called continue-on-error to true. We'll discuss why we do that in the next section.

Where are we in the directory structure right now?!

Remember that we used the checkout action earlier, so the pipeline is currently running inside the root of our repo as it exists on the runner system. That's why we can refer to files in this repo by local paths (like the profile repo itself, and the results subdirectory).

Artifacts

We used the --reporter json flag when we invoked InSpec, so we should now have a report file sitting on the runner. We want to be able to access that file -- both so that we can read it ourselves, and so that we can do some later processing on it in later jobs if we want to.

That's why we used upload-artifact, another extremely common Action. This one makes whatever file or files you pass it available for download through the browser when we examine the pipeline run later, and also makes those files available to later jobs even if they take place on different runners in this workflow (by default, any files created by a runner do not persist when the workflow ends).

Any Other Steps?

Let's do some brainstorming -- are there any other steps you'd like to insert into the pipeline? What else do you want to know about the profile or do with it?

Apache-2.0 | Copyright © 2022 - The MITRE Corporation
Copyright © 2024 Will Dower