4. Tools for Automation

4. SAF CLI

The SAF supports tools that can help you automate your security validation. This includes the SAF CLI tool.

If you are working in the course development lab environment, this tool is provided for you. If not, reference the page for installation guidance.

To update the SAF CLI version with npm, run:

npm update -g @mitre/saf

4.1. SAF CLI Capability

In addition to the documentation site, you can view the SAF CLI capabilities by running:

saf help

This will output:

The MITRE Security Automation Framework (SAF) Command Line Interface (CLI) brings together applications, techniques, libraries, and tools developed by MITRE and the security community to streamline security automation for systems and DevOps pipelines

VERSION
  @mitre/saf/1.1.10 linux-x64 node-v16.14.2

USAGE
  $ saf [COMMAND]

TOPICS
  convert   [Normalize]    Convert security results from all your security tools between common data formats
  generate  [Validate]     Set pipeline thresholds
  harden    [Harden]       Implement security baselines using Ansible, Chef, and Terraform content: Visit
            https://saf.mitre.org/#/harden to explore and run hardening scripts
  plugins   List installed plugins.
  scan      [Validate]     Scan to get detailed security testing results: Visit https://saf.mitre.org/#/validate to explore and run inspec profiles
  validate  [Validate]     Verify pipeline thresholds
  view      [Visualize]    Identify overall security status and deep-dive to solve specific security defects

COMMANDS
  convert ckl2poam  Translate DISA Checklist CKL file(s) to POA&M files
  harden            Visit https://saf.mitre.org/#/harden to explore and run hardening scripts
  heimdall          Run an instance of Heimdall Lite to visualize your Data
  help              Display help for saf.
  plugins           List installed plugins.
  scan              Visit https://saf.mitre.org/#/validate to explore and run inspec profiles
  summary           Get a quick compliance overview of an HDF file
  version

You can get more information on a specific topic by running:

saf [TOPIC] -h

4.2. View Summary of Results

Let's try out using the SAF CLI to understand our InSpec scan results.

You can view a summary of the results by running:

saf view summary -i path/to/output.json

This will show the high level information about the test results:

- profileName: my_nginx
  resultSets:
    - output.json
  compliance: 67
  passed:
    critical: 2
    high: 0
    medium: 0
    low: 0
    total: 2
  failed:
    critical: 1
    high: 0
    medium: 0
    low: 0
    total: 1
  skipped:
    critical: 0
    high: 0
    medium: 0
    low: 0
    total: 0
  error:
    critical: 0
    high: 0
    medium: 0
    low: 0
    total: 0
  no_impact:
    none: 0
    total: 0

4.3. Validate Results Against a Threshold

We can also generate a threshold for the results to be validated against. We will later see how this can be helpful in automated testing pipelines to indicate whether the pipeline should be considered a failure or a success.

Create a new file in your my_nginx folder to specify the thresholds for acceptable test results.These file contents specify some basic thresholds, such as the minimum compliance level, minimum number of tests passed, and maximum number of tests failed.

# threshold.yml file
compliance:
  min: 50
passed:
  total:
    min: 1
failed:
  total:
    max: 2

Threshold Files

To make more specific or detailed thresholds, check out this documentation on generating theshold files.

Now, validate the InSpec results against this threshold by running:

saf validate threshold -i output.json -F threshold.yml

This should result in:

All validation tests passed

If you try changing the threshold values to be more strict, you can see failing results. For example, update the threshold file like this:

# threshold.yml file
compliance:
  min: 50
passed:
  total:
    min: 1
failed:
  total:
    max: 0

To receive this output:

Error: failed.total.max: Received 1 > Expected 0

Using Threshold Files for Automation

Threshold files are used in CI to ensure minimum compliance levels and validate control severities and statuses using the saf validate threshold command.