What Is Done for a Control?

When is a Control Considered 'Done'

You and your team might be wondering what 'done' means for a security control in your profile. Here are a few things to consider:

• The security automation content and its tests are essentially a refactoring of the 'validation' and 'remediation' guidance already established by the benchmark.
• The security automation content tests should fully capture the spirit - or intention - of the guidance, including its caveats, notes, discussion, and 'validation' and 'remediation' content.
• The tests can - and usually do - capture known 'corner cases and security best practices' that are sometimes indirectly or not directly addressed by the benchmark but implied by the spirit of the security requirement being addressed.
• These tests, like all human-written code, may not be perfect. They will need updates and will evolve as our knowledge of the system and benchmark grows. We use the profile in production and real-world environments. In other words, don't let the pursuit of perfection hinder progress.

The 'is it done' litmus test is not solely determined by a perfect InSpec control or describe and expect blocks. It also heavily relies on you, the security automation engineer. Your experience, understanding of the platform you're working on, and the processes that you and your team have collectively agreed upon are all vital components.

Trust your established expected test outcomes, the guidance document, and the CI/CD testing framework. They will help you know that, to the best of your ability, you have captured the spirit of the testing required by the Benchmark.

The MITRE Security Automation Framework 'Yardstick'

We consider a control effectively tested when:

1. All aspects of the 'validation' - also known as 'check text' - have been addressed.
2. Any aspects of the 'remediation' - also known as 'fix text' - that are part of the 'validation' process have been captured.
3. Any documented conditions that are Not Applicable, as outlined in the 'discussion', 'check', or 'fix' text, have been addressed.
4. Any documented conditions that have Not Been Reviewed, as outlined in the 'discussion', 'check', or 'fix' text, have been addressed.
5. The conditions for Not Applicable and Not Reviewed are early in the control to ensure the control is as efficient as possible.
6. The control uses the only_if block vs 'if/else' logic when possible to ensure that the control is as clear, direct, and maintainable as possible from a coding perspective.
7. The control has been tested on both 'vanilla' and 'hardened' instances, ensuring that:
   1. The test communicates effectively and fails as expected on the 'vanilla' testing target.
   2. The test communicates effectively and passes on the 'hardened' testing target.
   3. The test communicates effectively and fails on a misconfigured 'vanilla' testing target.
   4. The test communicates effectively and fails on a misconfigured 'hardened' testing target.
   5. The test communicates effectively and clearly articulates the Not Applicable condition for both 'vanilla' and 'hardened' testing targets.
   6. The test communicates effectively and clearly articulates the Not Reviewed condition for both the 'vanilla' and 'hardened' testing targets.
   7. The tests have been constructed in a way that they do not produce Profile Errors when looping, using conditional logic, or when system conditions - such as missing files, directories, or services - are not in the expected locations.

Defining 'Passes as Expected'

'Passing as expected' is the most straightforward concept as it directly corresponds to the test conditions. When the test asserts a condition, it validates that assertion and reports it to the end user in a clear and concise manner.

We strive to ensure that when we report a 'pass', we do so in a language that is direct, simple, and easy to understand.

For example:

✔  SV-230222: RHEL 8 vendor packaged system security patches and updates must be installed and up to date.
    ✔  All system security patches and updates are up to date and have been applied

Passes as Expected also encompasses:

• The conditions for the Not Reviewed and Not Applicable states for the control, if any.

Defining Fails as Expected

'Failing as expected' is a less straightforward concept as it doesn't always directly correspond to the test conditions. When the test asserts a condition and it fails, the reason for that failure should be communicated to the end user in a clear and concise manner.

However, as we all know, a test may fail for more than one reason. Sometimes, the reason for failure might be connected to human error, conditions on the system such as extra lines, files, or housekeeping on the system that was not done, etc. All these factors may need to be accounted for in your tests and perhaps captured in your output and 'reasons' for failure.

This is where the above 'best practices' come into play. You don't just test in optional 'pass' and 'fail' conditions only, but also 'dirty things up' a bit and make sure that your 'failure' cases are robust enough to handle the real world and semi-perfect conditions.

For example:

✔  SV-230222: RHEL 8 vendor packaged system security patches and updates must be installed and up to date.
    x  The following packages have security patches and need to be updated:
          - package 1
          - package 2
          - package 3
          - package 4

Fails as Expected also encompasses:

• Misconfigurations, extra lines in files, extra settings, missing files, etc.

Defining Communicates Effectively

Clear communication from your testing suite may require you to use a combination of approaches, but the extra time and effort is well worth it.

Here are some methods you can employ and things to consider:

• Use expect vs describe statements in cases where you have multi-part or multi-phase test cases.
• Break up your describe statements into multiple layers so that the final output to the end user is clear and concise.
• Post-process and format both 'passing' and 'failures' so that they are useful to the end user later and clear for communication to other team members. This could be in the form of lists or bulleted lists.
• Collect 'failing results' as simple, clear lists or bullets that are easy to 'copy and paste'. This makes it easier for teams to know 'what they have to fix and where'.
• Consider assisting 'Manual Review'/'Not Reviewed' tests by collecting needed information, such as users, groups, or other elements that you are asking the user or another person to review. While we may not be able to fully automate the test, if the 'automation can help collect' then it still adds value.

Tracking Your Progress

Updating a 'Patch', 'Release', or 'Major Version' can be a challenging task. However, there are several methods a team can use to track progress.

You might use a Spreadsheet or CSV file, a Markdown or RST Table, or even a 'checklist' on the 'Pull Request' that your team updates as progress is made.

All these methods are acceptable. The important thing is to choose one method and use it consistently.

When working with multiple team members, it's crucial to have an effective way to communicate progress, understand who is working on which parts of the security guide, and know what 'still needs work' without constant direct communication.

The tracking method will also be influenced by the PR process you and your team select - either 'Macro' or 'Micro' - as discussed in the Micro vs Macro section.

Example Tracking Table

The key to effective tracking is simplicity. This ensures:

1. Each team member can easily understand what and how they need to document their progress, and
2. They will actually document their progress.

Simple Tracking Table

Assignee	Control	Priority	Reviewed	Tested	Text Updated	New Resource	Inputs
John	SV-230221	N	Y	Y	Y	N	None
John	SV-230222	Y	Y	Y	Y	N	disable_slow_controls
Jane	SV-230223	Y	Y	Y	Y	N	use_fips
Bob	SV-230224	Y	Y	Y	Y	N	data_at_rest_exempt

In this example, we used a simple markdown table. However, an Excel Spreadsheet, Google Sheet, or CSV might work better for your team.

Ultimately, the most important thing is to track the work. The method you choose to do so is a team decision.

Considerations for Grouping Work

The MITRE SAF team has found the following best practices effective for organizing our work:

1. Group Similar Controls: When working on a security guidance document, group the controls you are working on using the guidance indexes - such as the SRG ID in STIGs, and requirement major version in CIS Benchmarks. This allows for efficient reuse of repeated patterns of control implementation.

2. Tags, Status Columns & State: As a team, decide on the method for tracking work progress and agree on the terminology for concepts such as 'reviewed', 'tested', and 'done'/'completed'. Refer to the Tracking Table example above to understand how both 'technical' and 'business' requirements are tracked and reported for each requirement in the profile.

3. Assign Priority & Agree on an In/Out Approach: Every benchmark will have easy, medium, and complex requirements and tests that need implementation. You will need to review every control in the Profile, but choosing an 'easy first, hard last' or 'hard first, easy last' approach can help your team make efficient progress quickly and avoid continuous 'context switching' between straightforward and complicated testing.

4. Always Strive to Have a Full Test Suite: Ensuring the fidelity of testing is crucial. This principle applies to both the 'vanilla' and 'hardened' contexts, as well as to the 'primary deployment platforms' that your profile supports. These platforms might include Virtual Machines, Cloud Instances, and Container Deployments. Your goal should be to have both 'hardened' and 'vanilla' baselines for each deployment target. This strategy allows for easy provisioning of each platform. It also facilitates easy testing of your control on each platform as you progress from one control to another. This practice ensures that you are crafting the best possible tests for each target platform and configuration.

5. Try to Test Locally First, with the Pipeline Second: One of the key patterns highlighted in this guidance is the combination of local and CI/CD-based testing. We advocate for both approaches for a specific reason. When you are working on multiple controls, it's more efficient to test each control on each platform locally. This method is quicker than waiting for the CI/CD pipeline to create a new deployment of the test and target platforms each time. Once you have configured your targets and platforms locally with Test Kitchen, you can be confident in their stability. You should prioritize these local targets for initial testing. After testing them and when you are ready to proceed to the next control, push those updates to the CI/CD pipeline. This step verifies that your controls still function in a clean environment. This approach promotes a more efficient workflow process and eliminates the need for continuous 'push and wait' for the pipeline.