Understanding Control Completion in Security Automation

Learning Objectives

By the end of this section, you will be able to:

• Define the criteria for a "done" security control
• Apply the MITRE SAF yardstick to evaluate controls
• Implement effective control testing strategies
• Create and maintain progress tracking systems
• Debug common control implementation issues

Knowledge Check Questions

Before we begin, consider these questions:

1. What makes a security control "complete"?
2. How do you verify a control works in different environments?
3. What's the difference between "passing well" and just passing?

Introduction

Understanding when a security control is truly "done" is crucial for security automation engineers. This section will guide you through the criteria, best practices, and practical approaches to ensure your controls are complete and effective.

When is a Control Considered 'Done'

You and your team might be wondering what 'done' means for a security control in your profile. Here are a few things to consider:

1. The security automation content and its tests are essentially an alternative definition of the 'validation' and 'remediation' guidance already established by the benchmark.
2. The security automation content tests should fully capture the spirit - or intention - of the guidance, including its caveats, notes, discussion, and 'validation' and 'remediation' content.
3. The tests can - and usually do - capture known corner cases and security best practices which are sometimes only indirectly addressed by the benchmark but implied by the spirit of the security requirement being addressed.
4. These tests, like all human-written code, may not be perfect. They will need updates and will evolve as our knowledge of the system and benchmark grows. We use the profile in production and real-world environments. In other words, don't let the pursuit of perfection hinder progress.

The 'is it done' litmus test is not solely determined by a perfect InSpec control or describe and expect blocks. It also heavily relies on you, the security automation engineer. Your experience, understanding of the platform you're working on, and the processes that you and your team have collectively agreed upon are all vital components.

Trust your established expected test outcomes, the guidance document, and the CI/CD testing framework. They will help you know that, to the best of your ability, you have captured all requirements and the spirit of the testing specified by the benchmark.

The MITRE SAF Testing Framework

Our framework provides a comprehensive approach to testing controls. We call this the "SAF Yardstick":

We consider a control effectively tested when:

1. All aspects of the 'validation' - also known as 'check text' - have been addressed. This is usually straightforward.
2. Any aspects of the 'remediation' - also known as 'fix text' - that are part of the 'validation' process have been captured. Sometimes guidance authors mix the type of information between the 'check' and 'fix' text areas, so we need to comprehensively read all of the guidance to ensure that we've extracted all of the 'validation' content wherever it might be.
3. Any documented conditions that are Not Applicable, as outlined in the 'discussion', 'check', or 'fix' text, have been addressed.
4. Any documented conditions that require manual review, as outlined in the 'discussion', 'check', or 'fix' text, have been addressed such that the control is marked as Not Reviewed on execution of the profile.
5. The conditions for Not Applicable and Not Reviewed are assessed early in the control to ensure the control is as efficient as possible.
6. The control uses the only_if block vs 'if/else' logic when possible to ensure that the control is as clear, direct, and maintainable as possible from a coding perspective.
7. The control has been tested on both 'vanilla' and 'hardened' instances, ensuring that:
   1. The test communicates effectively and passes as expected on both the 'vanilla' and hardened testing targets which were correctly configured.
   2. The test communicates effectively and fails as expected on both the vanilla and hardened testing targets which were misconfigured.
   3. The test communicates effectively and fails as expected on a misconfigured vanilla target, but then passes as expected on a properly configured hardened target.
   4. The test communicates effectively and clearly articulates the Not Applicable condition for both 'vanilla' and 'hardened' testing targets.
   5. The test communicates effectively and clearly articulates the Not Reviewed condition for both the 'vanilla' and 'hardened' testing targets.
   6. The tests have been constructed in a way that they do not produce Profile Errors when looping, using conditional logic, or when system conditions - such as missing files, directories, or services - are not in the expected locations.

Best Practices for Test Implementation

Passing Tests (Passing Well)

A well-implemented passing test should:

• Clearly communicate success conditions
• Use simple, direct language
• Include validation of edge cases

For example:

✔  SV-230222: RHEL 8 vendor packaged system security patches and updates must be installed and up to date.
    ✔  All system security patches and updates are up to date and have been applied

Passes as Expected also encompasses:

• The conditions for the Not Reviewed and Not Applicable states for the control, if any.

Failing Tests (Failing Well)

When implementing failure scenarios, ensure:

• Clear error messages
• Actionable feedback
• Proper error handling

For example:

✔  SV-230222: RHEL 8 vendor packaged system security patches and updates must be installed and up to date.
    x  The following packages have security patches and need to be updated:
          - package 1
          - package 2
          - package 3
          - package 4

Fails as Expected also encompasses:

• Misconfigurations, extra lines in files, extra settings, missing files, etc.

Hands-on Exercise 1: Creating Your First Control

Let's practice implementing a basic control:

Create a basic control test

control 'SV-257844' do
  title 'RHEL 9 must use a separate file system for /tmp.'
  desc 'The "/tmp" partition is used as temporary storage by many programs. Placing "/tmp" in its own partition enables the setting of more restrictive mount options, which can help protect programs that use it.'
  desc 'check', 'Verify that a separate file system/partition has been created for "/tmp" with the following command:

$ mount | grep /tmp

tmpfs /tmp tmpfs noatime,mode=1777 0 0

If a separate entry for "/tmp" is not in use, this is a finding.'
  desc 'fix', 'Migrate the "/tmp" path onto a separate file system.'
  impact 0.5
  ref 'DPMS Target Red Hat Enterprise Linux 9'
  tag severity: 'medium'
  tag gtitle: 'SRG-OS-000480-GPOS-00227'
  tag gid: 'V-257844'
  tag rid: 'SV-257844r925519_rule'
  tag stig_id: 'RHEL-09-231015'
  tag fix_id: 'F-61509r925518_fix'
  tag cci: ['CCI-000366']
  tag nist: ['CM-6 b']
  tag 'host'
end

Add passing and failing scenarios

control 'SV-257844' do
  title 'RHEL 9 must use a separate file system for /tmp.'
  desc 'The "/tmp" partition is used as temporary storage by many programs. Placing "/tmp" in its own partition enables the setting of more restrictive mount options, which can help protect programs that use it.'
  desc 'check', 'Verify that a separate file system/partition has been created for "/tmp" with the following command:

$ mount | grep /tmp

tmpfs /tmp tmpfs noatime,mode=1777 0 0

If a separate entry for "/tmp" is not in use, this is a finding.'
  desc 'fix', 'Migrate the "/tmp" path onto a separate file system.'
  impact 0.5
  ref 'DPMS Target Red Hat Enterprise Linux 9'
  tag severity: 'medium'
  tag gtitle: 'SRG-OS-000480-GPOS-00227'
  tag gid: 'V-257844'
  tag rid: 'SV-257844r925519_rule'
  tag stig_id: 'RHEL-09-231015'
  tag fix_id: 'F-61509r925518_fix'
  tag cci: ['CCI-000366']
  tag nist: ['CM-6 b']
  tag 'host'

  tmp = mount('/tmp')
  is_mounted = tmp.mounted?
  describe is_mounted do
    it { should cmp true }
  end

  describe etc_fstab.where { mount_point == '/tmp' } do
    it { should exist }
  end
end

Implement clear communication

control 'SV-257844' do
  title 'RHEL 9 must use a separate file system for /tmp.'
  desc 'The "/tmp" partition is used as temporary storage by many programs. Placing "/tmp" in its own partition enables the setting of more restrictive mount options, which can help protect programs that use it.'
  desc 'check', 'Verify that a separate file system/partition has been created for "/tmp" with the following command:

$ mount | grep /tmp

tmpfs /tmp tmpfs noatime,mode=1777 0 0

If a separate entry for "/tmp" is not in use, this is a finding.'
  desc 'fix', 'Migrate the "/tmp" path onto a separate file system.'
  impact 0.5
  ref 'DPMS Target Red Hat Enterprise Linux 9'
  tag severity: 'medium'
  tag gtitle: 'SRG-OS-000480-GPOS-00227'
  tag gid: 'V-257844'
  tag rid: 'SV-257844r925519_rule'
  tag stig_id: 'RHEL-09-231015'
  tag fix_id: 'F-61509r925518_fix'
  tag cci: ['CCI-000366']
  tag nist: ['CM-6 b']
  tag 'host'

  describe mount('/tmp') do
    it { should be_mounted }
  end

  describe etc_fstab.where { mount_point == '/tmp' } do
    it { should exist }
  end
end

Test edge cases

control 'SV-257844' do
  title 'RHEL 9 must use a separate file system for /tmp.'
  desc 'The "/tmp" partition is used as temporary storage by many programs. Placing "/tmp" in its own partition enables the setting of more restrictive mount options, which can help protect programs that use it.'
  desc 'check', 'Verify that a separate file system/partition has been created for "/tmp" with the following command:

$ mount | grep /tmp

tmpfs /tmp tmpfs noatime,mode=1777 0 0

If a separate entry for "/tmp" is not in use, this is a finding.'
  desc 'fix', 'Migrate the "/tmp" path onto a separate file system.'
  impact 0.5
  ref 'DPMS Target Red Hat Enterprise Linux 9'
  tag severity: 'medium'
  tag gtitle: 'SRG-OS-000480-GPOS-00227'
  tag gid: 'V-257844'
  tag rid: 'SV-257844r925519_rule'
  tag stig_id: 'RHEL-09-231015'
  tag fix_id: 'F-61509r925518_fix'
  tag cci: ['CCI-000366']
  tag nist: ['CM-6 b']
  tag 'host'

  only_if('This control is Not Applicable to containers', impact: 0.0) {
    !virtualization.system.eql?('docker')
  }

  describe mount('/tmp') do
    it { should be_mounted }
  end

  describe etc_fstab.where { mount_point == '/tmp' } do
    it { should exist }
  end
end

Key Takeaways

• Control completion is more than just passing tests
• Use the SAF Yardstick as your guide
• Clear communication is essential
• Track progress consistently
• Group similar controls for efficiency

Tracking Your Progress

Updating a 'Patch', 'Release', or 'Major Version' can be a challenging task. However, there are several methods a team can use to track progress.

You might use a Spreadsheet or CSV file, a Markdown or RST Table, or even a 'checklist' on the 'Pull Request' that your team updates as progress is made.

All these methods are acceptable. The important thing is to choose one method and use it consistently.

When working with multiple team members, it's crucial to have an effective way to communicate progress, understand who is working on which parts of the security guide, and know what still needs work without constant direct communication.

The tracking method will also be influenced by the PR process you and your team select - either 'Macro' or 'Micro' - as discussed in the Micro vs Macro section.

Example Tracking Table

The key to effective tracking is simplicity. This ensures:

1. Each team member can easily understand what and how they need to document their progress, and
2. They will actually document their progress.

Simple Tracking Table

Assignee	Control	Priority	Reviewed	Tested	Text Updated	New Resource	Inputs
John	SV-230221	N	Y	Y	Y	N	None
John	SV-230222	Y	Y	Y	Y	N	disable_slow_controls
Jane	SV-230223	Y	Y	Y	Y	N	use_fips
Bob	SV-230224	Y	Y	Y	Y	N	data_at_rest_exempt

In this example, we used a simple markdown table. However, an Excel Spreadsheet, Google Sheet, or CSV might work better for your team.

Ultimately, the most important thing is to track the work. The method you choose to do so is a team decision.

Considerations for Grouping Work

The MITRE SAF team has found the following best practices effective for organizing our work:

1. Group Similar Controls: When working on a security guidance document, group the controls you are working on using the guidance indexes - such as the SRG ID in STIGs, and requirement major version in CIS Benchmarks. This allows for efficient reuse of repeated patterns of control implementation.

2. Tags, Status Columns, and State: As a team, decide on the method for tracking work progress and agree on the terminology for concepts such as 'reviewed', 'tested', and 'done'/'completed'. Refer to the Tracking Table example above to understand how both 'technical' and 'business' requirements are tracked and reported for each requirement in the profile.

3. Assign Priority, and Agree on an In/Out Approach: Every benchmark will have easy, medium, and complex requirements and tests that need implementation. You will need to review every control in the Profile, but choosing an 'easy first, hard last' or 'hard first, easy last' approach can help your team make efficient progress quickly and avoid continuous 'context switching' between straightforward and complicated testing.

4. Always Strive to Have a Full Test Suite: Ensuring the fidelity of testing is crucial. This principle applies to both the 'vanilla' and 'hardened' contexts, as well as to the primary deployment platforms that your profile supports. These platforms might include Virtual Machines, Cloud Instances, and Container Deployments. Your goal should be to have both 'hardened' and 'vanilla' baselines for each deployment target. This strategy allows for easy provisioning of each platform. It also facilitates easy testing of your control on each platform as you progress from one control to another. This practice ensures that you are crafting the best possible tests for each target platform and configuration.

5. Try to Test Locally First, with the Pipeline Second: One of the key patterns highlighted in this guidance is the combination of local and CI/CD-based testing. We advocate for both approaches for a specific reason. When you are working on multiple controls, it's more efficient to test each control on each platform locally. This method is quicker than waiting for the CI/CD pipeline to create a new deployment of the test and target platforms each time. Once you have configured your targets and platforms locally with Test Kitchen, you can be confident in their stability. You should prioritize these local targets for initial testing. After testing them and when you are ready to proceed to the next control, push those updates to the CI/CD pipeline. This step verifies that your controls still function in a clean environment. This approach promotes a more efficient workflow process and eliminates the need for continuous 'push and wait' for the pipeline.