Attack Simulation Frameworks

What are attack simulation frameworks?

Attack simulation frameworks are tools for running authorized, repeatable representations of adversary behavior so a security team can test its own controls. A framework may execute one focused technique, coordinate a multi-step operation, or schedule recurring validation across an approved target group.

In plain language: the framework creates known activity, then the team asks whether prevention, telemetry, analytics, alert routing, and human response behaved as expected. The goal is not to prove that an attacker can cause harm. It is to produce controlled evidence that helps defenders find and fix a specific gap.

The core invariant is execution success is not detection success. A command can complete while a sensor is offline, an analytic misses, an alert is routed incorrectly, or no analyst responds. A defensible result keeps those stages separate.

Review Red Teaming Fundamentals first if authorization, rules of engagement, or the difference between red and blue team responsibilities is unfamiliar.

Execute

Create ground truth

Record the exact test, target, time, parameters, and result.

Observe

Verify telemetry

Prove that the expected behavior reached a healthy sensor and data path.

Detect

Evaluate analytics

Match ground truth to the intended analytic and alert context.

Respond

Close the loop

Confirm routing, acknowledgement, containment, cleanup, and ownership.

Choose the framework by testing grain

Testing grain is the amount of behavior and system scope exercised by one run. A small grain isolates one observation. A larger grain tests sequencing, correlation, and response, but creates more dependencies and a larger blast radius.

Focused technique test

Atomic Red Team

Use a small, portable test when the question is narrow: does this implementation of one ATT&CK technique create the expected telemetry or control response on this host type? Review the test's prerequisites, command, input arguments, and cleanup before running it.

Orchestrated emulation

MITRE CALDERA

Use an operation when abilities must run through agents in a controlled group and order. CALDERA models abilities, adversary profiles, agents, planners, facts, and operations, so it fits multi-step hypotheses and repeatable purple-team exercises.

Continuous validation

Commercial BAS

Use a breach-and-attack-simulation platform when the operating model needs managed content, scheduling, integrations, fleet controls, and recurring reporting. Validate the vendor's execution safety, evidence model, environment coverage, and update process rather than comparing scenario counts alone.

Independent evaluation

ATT&CK Evaluations

MITRE Engenuity ATT&CK Evaluations publishes a methodology and evidence for evaluating participating security products. It is useful comparative input, but it is not a simulation framework deployed inside your environment and it does not replace local validation.

Atomic Red Team describes itself as a library of small tests mapped to ATT&CK. CALDERA describes abilities as executable technique implementations, adversary profiles as groups of abilities, and operations as abilities run against agent groups under a planner. Read the current Atomic Red Team project documentation and CALDERA terminology before adapting content to a real environment.

Bound the objective before choosing tests

A simulation objective is a falsifiable statement about one defensive outcome. "Test T1059.001" is only a behavior label. "Verify that a reviewed PowerShell behavior on the Windows canary group creates process telemetry, one correlated alert, and an analyst acknowledgement within the exercise window" names the target, evidence, and pass condition.

Use the lab to compare a focused visibility check, a correlated detection, and a response rehearsal. Change the target boundary and approval mode, then watch the host-actions, concurrency waves, evidence work, and review decision change together.

Loading the simulation scope model...

Read the model as a safety boundary

Host-actions equal targets multiplied by technique steps. This is a planning count, not a coverage score.
Concurrency controls immediate blast radius. The same total work is safer when a failed step stops before every target receives it.
Evidence checkpoints make review work visible. More automated execution without matching evidence capacity can produce activity faster than the team can interpret it.
Approval mode must fit the environment. A broad production segment should not become autonomous because a framework exposes an automation toggle.

Understand the CALDERA execution model

CALDERA separates what can run from where, when, and why it runs. That separation is the main design idea to preserve even when another framework uses different names.

1
What can run
Review an ability
Inspect the ATT&CK mapping, platform executor, command, variables, prerequisites, timeout, payloads, parser behavior, and cleanup. A mapping does not make an ability safe for every target.
2
Behavior sequence
Assemble an adversary
Select only the abilities needed for the hypothesis. Their order and fact dependencies should represent the intended behavior chain without adding unrelated activity.
3
Execution boundary
Configure an operation
Bind the adversary to an approved agent group, planner, fact source, approval mode, concurrency policy, maintenance window, and stop conditions.
4
Defensive result
Correlate evidence
Join operation links and timestamps to sensor health, telemetry, analytic decisions, alert delivery, analyst actions, and verified cleanup.

CALDERA creates a link when an ability can run on an agent and its required facts are satisfied. The planner determines order; the agent executes through a compatible executor; parsers may turn output into facts for later links. The official CALDERA documentation is the authority for the installed release and enabled plugins.

The original CALDERA /api/rest interface is deprecated. New integrations should use the current API v2 documentation shipped with the installed CALDERA version, authenticate remote requests, and verify request and response schemas before automating an operation. See CALDERA's REST API notice.

Trace the evidence chain

An evidence chain connects simulation ground truth to a defensive decision. It answers four different questions: did the test run, did the sensor observe it, did the analytic recognize it, and did the response workflow act on it?

Switch among the complete path, sensor gap, analytic gap, and response gap. The selected route, failed or degraded component, component responsibility, and conclusion update together.

Preserve the first failed stage

If execution fails, fix the test contract or target compatibility before judging defense.
If execution passes but telemetry is absent, record a visibility gap and sensor health evidence.
If telemetry exists but no intended alert appears, record a detection gap with the analytic version and expected condition.
If the alert appears but no owner acts, record a response gap with routing and acknowledgement evidence.

ATT&CK remains the shared behavior taxonomy, but a technique mapping is not a local control result. Since ATT&CK v18, MITRE has organized defensive guidance around Detection Strategies and analytics; the October 2025 ATT&CK v18 update also deprecated the older Data Sources objects. Pin the ATT&CK version used by reports so mappings remain reproducible.

Measure conditional outcomes, not one score

A stage rate uses only the previous successful stage as its denominator. This prevents a missing sensor from being mislabeled as an analytic miss and prevents an execution failure from lowering response performance.

For one operation, calculate:

Execution rate = completed test steps / attempted test steps.
Visibility rate = steps with expected telemetry / completed test steps.
Detection rate = steps with the intended alert / steps with expected telemetry.
Response rate = acknowledged alerts / intended alerts that fired.

Always publish counts beside percentages. A 100% detection rate from one visible test is not equivalent to a 100% rate across fifty reviewed variants, platforms, and sensor configurations.

Do not combine these rates into a generic "security effectiveness" score. The stages have different owners, denominators, and remediation paths. Keep the evidence granular enough to reproduce each miss.

Encode the plan and evaluate evidence offline

The plan example is framework-neutral on purpose. It captures the authorization, target group, concurrency cap, stop conditions, expected evidence, and cleanup contract before any product-specific operation object is created.

Validate the JSON with python3 -m json.tool simulation-plan.json. A production adapter can translate this reviewed contract into the installed framework's current API without putting credentials or live commands in the lesson.

Framework-neutral simulation plan

Run python3 evaluate-simulation.py. The dependency-free script loads the co-located simulation result fixture, validates stage invariants, calculates conditional rates, and lists the first failed stage for each test.

Evaluate simulation evidence with the Python standard library

The sample decision is hold because it contains one visibility gap, one detection gap, and one response gap. That outcome is more useful than a blended score because it routes each problem to the correct owner.

Operate with explicit stop and cleanup contracts

A stop contract defines the conditions that end execution before the original plan completes. Cleanup reverses test artifacts, but it does not make an unsafe scope acceptable.

Before the run:

Obtain written authorization naming owners, target groups, time window, allowed behaviors, data handling, and emergency contacts.
Review every command, input, prerequisite, payload, privilege requirement, timeout, and cleanup action against the exact platform image.
Exclude identities, datasets, systems, and network paths that are not required by the hypothesis.
Record sensor health, analytic version, alert route, and expected evidence before creating activity.
Rehearse the kill switch and choose who may invoke it independently of the framework operator.

During the run:

Attach one correlation ID to controller ground truth, target telemetry, analytics, alerts, and analyst notes.
Stop on target-group drift, unexpected privilege, sensor failure, cleanup failure, service impact, or missing ownership.
Keep retries bounded and idempotent. A timeout does not prove that a target-side action failed.

After the run:

Verify cleanup and target health independently of the framework's success flag.
Preserve minimal evidence needed to reproduce the decision without retaining credentials, sensitive payloads, or unrelated user data.
Assign each gap to a named owner with a retest condition and expiry date.
Re-run the same reviewed case after the fix, then compare evidence using the same ATT&CK and analytic versions.

Know what the result can claim

Evidence

A pass is narrow

A pass supports one claim about the tested implementation, target type, sensor state, analytic version, alert route, and exercise window. It does not prove complete coverage of an ATT&CK technique.

Ownership

A miss needs attribution

Route an execution miss to the test owner, a visibility miss to telemetry engineering, a detection miss to the analytic owner, and a response miss to the SOC workflow owner.

Scale

Automation increases duty

Scheduling more tests increases target activity, evidence volume, content drift, and cleanup work. Increase automation only when scope enforcement and evidence review can keep pace.

Limit

Manual work still matters

Frameworks reproduce known behaviors well. Human-led red teaming still explores novel paths, social context, chained assumptions, and adaptive decisions that scripted tests do not represent honestly.

No quiz questions available

Could not load questions file