Skip to main content

11. Verifying Results With The SAF CLI

Will DowerJune 28, 2022About 5 min

Verification

At this point we have a much more mature workflow file. We have one more activity we need to do -- verification, or checking that the output of our validation run met our expectations.

Note that "meeting our expectations" does not automatically mean that there are no failing tests. In many real-world use cases, security tests fail, but the software is still considered worth the risk to deploy because of mitigations for that risk, or perhaps the requirement is inapplicable due to the details of the deployment. With that said, we still want to run our tests to make sure we are continually collecting data; we just don't want our pipeline to halt if it finds a test that we were always expecting to fail.

By default, the InSpec executable returns a code 100 if any tests in a profile run fail. Pipeline orchestrators, like most software, interpret any non-zero return code as a serious failure, and will halt the pipeline run accordingly unless we explicitly tell it to ignore errors. This is why the "VALIDATE - Run InSpec" step has the continue-on-error: true attribute specified.

Our goal is to complete our InSpec scan, collect the result as a report file, and then parse that file to determine if we met our own threshold of security. We can do this with the SAF CLI.

Updating the Workflow File

Let's add two steps to our pipeline to use the SAF CLI to understand our InSpec scan results before we verify them against a threshold.

Adding Verify Steps
- name: VERIFY - Display our results summary 
  run: |
    saf view summary -i results/pipeline_run_attested.json

# check if the pipeline passes our defined threshold
- name: VERIFY - Ensure the scan meets our results threshold           
  run: |
    saf validate threshold -i results/pipeline_run_attested.json -F threshold.yml

A few things to note here:

  • We added the summary step because it will print us a concise summary inside the pipeline job view itself. That command takes one file argument; the results file we want to summarize.
  • The validate threshold command, however, needs two files -- one is our report file as usual, and the other is a threshold file.

Threshold Files

Threshold files are what we use to define what "passing" means for our pipeline, since like we said earlier, it's more complicated than failing the pipeline on a failed test.

Consider the following sample threshold file:

# threshold.yml file
compliance:
  min: 80
passed:
  total:
    min: 1
failed:
  total:
    max: 2

This file specifies that we require a minimum of 80% of the tests to pass. We also specify that at least one of them should pass, and that at maximum two of them can fail.

Threshold Files Options

To make more specific or detailed thresholds, check out this documentation on generating theshold files.

NOTE: You can name the threshold file something else or put it in a different location. We specify the name and location only for convenience.

This is a sample pipeline, so we are not too worried about being very stringent. For now, let's settle for running the pipeline with no errors (that is, as long as each test runs, we do not care if it passed or failed, but a source code error should still fail the pipeline).

Create a new file called threshold.yml in the main directory to specify the threshold for acceptable test results:

error:
  total:
    max: 0

How could we change this threshold file to ensure that the pipeline run will fail?

And with that, we have a complete pipeline file. Let's commit our changes and see what happens.

Committing And Pushing Code
git add .github
git commit -s -m "finishing the pipeline"
git push origin main

Let's hop back to our browser and take a look at the output:

The Completed Pipeline Run
The Completed Pipeline Run

There we go! All validation tests passed!

Note in the SAF CLI Summary step, we get a simple YAML output summary of the InSpec scan:

The Summary
The Summary

We see six critical-severity tests (remember how we set them all to impact 1.0?) passing, and no failures:

- profileName: my_nginx
  resultSets:
    - pipeline_run_attested.json
  compliance: 100
  passed:
    critical: 6
    high: 0
    medium: 0
    low: 0
    total: 6
  failed:
    critical: 0
    high: 0
    medium: 0
    low: 0
    total: 0
  skipped:
    critical: 0
    high: 0
    medium: 0
    low: 0
    total: 0
  error:
    critical: 0
    high: 0
    medium: 0
    low: 0
    total: 0
  no_impact:
    critical: 0
    high: 0
    medium: 0
    low: 0
    none: 0
    total: 0

Note also that our test report is avaiable as an artifact from the overall pipeline run summary view now:

The Artifact
The Artifact

From here, we can download that file and manually drop it off in something like Heimdall or feed into some other security process at our leisure. Luckily, we already automated that step. Let's check in with Heimdall Demo's Database View:

Results in Heimdall
Results in Heimdall

And voilà! We have a completed pipeline. We are launching our application, hardening it, testing it, processing the resulting data, and aggregating it into a dashboard.

Viewing the file in Heimdall

Heimdall is a very powerfull tool for examining security data. See more details in the SAF User Class

What Else Can We Do With A Pipeline?

In a real use case, if our pipeline passed, we would next save our bonafide hardened image to a secure registry where it could be distributed to users. If the pipeline did not pass, we would have already collected data describing why, in the form of InSpec scan reports that we save as artifacts.

Other pipelines might flat out refuse to permit merging code into a repository branch, or alert the developer team if issues occur. The exact implementation is up to you and your needs for your work.

Apache-2.0 | Copyright © 2022 - The MITRE Corporation
Copyright © 2024 Will Dower