Recording Defense Outcomes

Test Cases

Defense Activity

VECTR is designed to record the activity and result of various cybersecurity test operations organized into Test Cases. Each Test Case has a Defense Activity section to record the defense outcome.

Defense Layers

Each VECTR Test Case should have a list of Defense Layers or Defense Tool Types that would be expected to defend against the Test Case’s described attack. Some examples of Defense Layers are Firewall, SIEM, or EDR. Some Test Cases may only have 1 expected Defense Layer while others may have many. Typically, this information is configured at the library level and should be prepopulate when you add a test case. If you need to modify this or are creating a Test Case ad-hoc, you can edit Defense Layers by clicking the cog in the top right corner of the Defense Activity control.

Test Case Outcome Recording

The Test Case outcome represents a successful or unsuccessful defense against the described test. Examples of outcomes include ‘Blocked’, ‘Logged – Centrally Logged’ and ‘None’

VECTR has two modes of recording Test Case Outcomes: * Per-tool Outcome Mode * Legacy Mode

Per-tool Outcome Mode

Introduced in VECTR 9.6, Per-tool Outcome Mode is the suggested and default mode for recording Test Case Outcomes for any new Assessments. As the name implies, this feature allows you select a different outcome for each individual tool that is or should be involved in the defense of a specific Test Case. Additionally, the Test Case outcome is used to determine if the overall defense against this Test Case was successful or not.

In Per-tool Outcome Mode, the Test Case outcome is not a selectable field by default. Its value is derived from the selected tool outcomes which are the selection boxes in each row of the Defense Activity control. The selected defense outcomes have a pre-defined priority order of Blocked, Alerted, Logged, None, N/A, TBD. 3 tools were involved in the defense of the Test Case in the above screenshot. With outcomes of Local Telemetry, Blocked Alerted, and Alerted High, the Test Case uses the outcome of Blocked Alerted based on the pre-defined outcome priority.

There are some instances where this behavior may not be sufficient to fully document the results of the Test Case. For those instances, the overall Test Case Outcome can be overridden by clicking the Override button in the Test Case Outcome control and then selecting an outcome in the Test Case Outcome selection control.

One scenario where this might be useful is when you have a Defense Tool like an EDR or Firewall that can generate alerts, but the alerts are either not logged centrally in your SIEM or the appropriate detection event is not firing in your SIEM. In that case you may want to note the overall Test Case outcome as “Logged – Local Telemetry” which is an unsuccessful defense and investigate this Test to improve your Detection and Incident Response capability.

Changing Modes

Click the cog in the top right corner of the Defense Activity control to access the outcome mode toggle setting.

Legacy Mode

If you used VECTR prior to version 9.6 you may be familiar with this Test Case Outcome UI

This recorded the test case Outcome as one of 6 primary outcomes with each having potential sub-outcome information, allowing you to record if the result of a Test Case was Blocked and Alerted like in this example. With this interface, users were able to select multiple Blue Tools involved in the defense of a Test Case. However, the limitations of this UI meant that the Test Case Outcome applied to all selected “Detecting Blue Tools.”

This Outcome mode can still be represented in the new 9.6+ UI:

The chain link icons signal that the Test Case is in Legacy Outcome Mode and data will be saved accordingly. Note that the Test Case Outcome and Outcomes for any tools are linked together in this mode. Adding a defense tool will mean it will automatically have the same outcome as the test case. Changing any Outcome in this mode will change the Outcome for all tools and the Test Case. All your existing previous VECTR Tests will display in this mode. This mode is not recommended for new Assessments and you will lose out on future VECTR functionality by continuing to record tests this way.

Assessments

Outcome Mode Please review documentation for the Test Case panel Outcome Mode. On the Assessment page, this setting controls the default outcome mode for any new Test Cases created within. This setting may be modified on the Assessment Dashboard by editing an Assessment.

Per-tool Outcome Mode Behavior

It’s recommended that you set the available Defense Tools for your active Environment prior to creating Assessments in VECTR. First, create or locate Defense Tools in the library. Do this by clicking the Library dropdown in the nav bar and selecting Defense Tools. From here, you should check to see if you already have a Defense Tool entry for each of the Defense Tools you will be testing against. If you’re missing any, create them here. (Note: If you can not find an appropriate Defense Tool Product, select Unknown) Next, click the Environment dropdown in the Nav Bar and select Defense Tools. Disable any that are not used in your environment. If any Tools are missing that you’d like to test, click “Import Tool” and select from the Library.

Why do I need to create a both Library Defense Tool and an Environment Defense Tool?

• Since Defense Tools can potentially be shared across environments, this allows for linking between environments and potential reporting in the future

At this point, your Library should be configured. When you create a new Assessment, it will be in per-tool outcome mode. If you click Advanced Options when creating an Assessment, you’ll notice that your Environment’s enabled Defense Tools are reflected as available in the Assessment.

This Defense Tool information is used to auto-populate tools for specific Defense Layers in your Test Cases when recording testing information.