5. AWS Testing Suite

Aaron LippoldJanuary 5, 2024About 2 min

AWS Testing Setup

Note

This guide assumes basic familiarity with AWS. If you're new to AWS, please review the AWS Getting Started Guide first.

Configure your AWS CLI and set up your AWS credentials
- If you haven't installed AWS CLI, download it here
- Run aws configure to set up your credentials
Verify AWS CLI access: aws s3 ls
- If this fails, check your credentials and permissions
Clone the repository
Navigate to the root directory of the profile repository
Configure Kitchen for AWS EC2:
```
export KITCHEN_LOCAL_YAML=kitchen.ec2.yml
```
The kitchen-ec2 driver enables TestKitchen to create and manage AWS EC2 instances for testing
(Optional) Target a specific control:

export INSPEC_CONTROL='SV-230222'

Running Through the AWS Test Suite

Understanding the Test Workflow:

vanilla represents an unmodified baseline system while hardened represents a system with security controls applied.

List the kitchen instances:

bundle exec kitchen list

You should see something like this:

Instance         Driver  Provisioner      Verifier  Transport  Last Action  Last Error
vanilla-rhel-8   Ec2     AnsiblePlaybook  Inspec    Ssh        Verified     None
hardened-rhel-8  Ec2     AnsiblePlaybook  Inspec    Ssh        Verified     None

Key Testing Steps Explained

Create the test instance:

bundle exec kitchen create vanilla

Executing that line launches a fresh EC2 instance for testing:

➜  redhat-enterprise-linux-8-stig-baseline git:(main*) bundle exec kitchen create vanilla
-----> Starting Test Kitchen (v3.5.1)
-----> Creating <vanilla-rhel-8>...
    < OTHER OUTPUT >
    Finished creating <vanilla-rhel-8> (0m0.00s).
-----> Test Kitchen is finished. (0m1.21s)

Converge the instance:

bundle exec kitchen converge vanilla

Convergence applies the necessary configurations to prepare the system for testing:

➜  redhat-enterprise-linux-8-stig-baseline git:(main*) bundle exec kitchen converge vanilla
-----> Starting Test Kitchen (v3.5.1)
      NOTICE - Installing needed packages
      Updating Subscription Management repositories.
      Unable to read consumer identity

      This system is not registered with an entitlement server. You can use subscription-manager to register.

      39 files removed
      < LOTS OF OTHER OUTPUT >
      Downloading files from <vanilla-rhel-8>
      Finished converging <vanilla-rhel-8> (0m21.36s).
-----> Test Kitchen is finished. (1m13.52s)

Verify that the Kitchen instance meets our requirements by using InSpec:

bundle exec kitchen verify vanilla

You'll see the same InSpec CLI output that we've gotten familiar with in our other classes:

    ➜  redhat-enterprise-linux-8-stig-baseline git:(main*) bundle exec kitchen verify vanilla
  -----> Starting Test Kitchen (v3.5.1)
  -----> Setting up <vanilla-rhel-8>...
        Finished setting up <vanilla-rhel-8> (0m0.00s).
  -----> Verifying <vanilla-rhel-8>...
        Loaded redhat-enterprise-linux-8-stig-baseline
  Could not determine patch status.
  Profile:   redhat-enterprise-linux-8-stig-baseline (redhat-enterprise-linux-8-stig-baseline)
  Version:   1.12.0
  Target:    ssh://ec2-user@34.229.216.179:22
  Target ID: 4c62a305-69eb-5ed6-9ee7-723cdc21c578

    ✔  SV-230222: RHEL 8 vendor packaged system security patches and updates must be installed and up to date.
      ✔  List of out-of-date packages is expected to be empty
  Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
  Test Summary: 1 successful, 0 failures, 0 skipped
        Finished verifying <vanilla-rhel-8> (0m5.38s).
  -----> Test Kitchen is finished. (0m6.62s)

Destroy the Kitchen instance:

bundle exec kitchen destroy vanilla

💸

Always remember to pause or destroy your test instances after testing to avoid unnecessary AWS charges. You can do it via kitchen or by going into the AWS Console.

Repeat these steps but replace vanilla with hardened.

Analyzing Results

Results Location: ./spec/results/rhel-8_*
Use Heimdall Lite to compare results:
1. Load both the vanilla and hardened results
2. Use the Comparison View (look for the toggle underneath the files list after selecting the hamburger menu in the top left) to compare the results sets and verify expected passes and failures
3. Review corner cases for complete coverage