The 23 AI security sub-metrics

Ability to maintain correct behavior when subjected to crafted adversarial inputs designed to cause misclassification, hallucination, policy bypass, unsafe tool invocation, or unauthorized disclosure.

Primary metric: Attack Success Rate (ASR)

Ability to preserve safe, calibrated, and useful behavior when inputs, users, languages, domains, tools, or data sources differ materially from the assessment baseline.

Primary metric: Out-of-Distribution Degradation Rate (OOD-DR)

Ability to produce stable, policy-consistent, and semantically equivalent outputs across repeated runs, paraphrased prompts, equivalent inputs, and supported operating conditions.

Primary metric: Output Variance Rate (OVR)

Resistance to training-time, fine-tuning-time, retrieval-corpus, memory, tool-description, or feedback-loop manipulation that degrades integrity, implants backdoors, or skews outputs.

Primary metric: Poisoning Attack Success Rate (PASR)

Consistency of outcome rates across protected groups where parity is legally, ethically, or operationally appropriate for the use case.

Primary metric: Demographic Parity Difference (DPD)

Consistency of confidence, uncertainty, refusal, and risk estimates across protected groups, languages, and relevant user populations.

Primary metric: Calibration Error Difference

Degree to which training, evaluation, retrieval, generated content, and embedding behavior overrepresent, underrepresent, stereotype, or erase relevant groups.

Primary metric: Stereotype Reproduction Rate (SRR)

Stability of materially relevant outputs when protected attributes are changed while all task-relevant non-protected attributes remain constant.

Primary metric: Counterfactual Output Change Rate (COCR)

Ability to provide explanations at the depth required by the decision context, including outcome rationale, evidence used, uncertainty, limitations, and escalation path.

Primary metric: Explanation Depth Coverage (EDC)

Degree to which confidence, probability, risk, refusal, and uncertainty signals correspond to observed correctness, safety, and reliability outcomes.

Primary metric: Expected Calibration Error (ECE)

Completeness and integrity of records needed to reconstruct inputs, outputs, prompts, retrieval context, tool calls, model versions, policy versions, and human interventions.

Primary metric: Audit Trail Completeness Rate (ATCR)

Completeness of disclosed lineage for models, datasets, fine-tunes, retrieval corpora, tool manifests, evaluation sets, and material configuration changes.

Primary metric: Lineage Disclosure Coverage (LDC)

Likelihood that the system reveals memorized or reconstructable training, fine-tuning, retrieval, or proprietary data through normal or adversarial interaction.

Primary metric: Training Data Extraction Rate (TDER)

Resistance to membership inference, model inversion, attribute inference, property inference, and related attacks that infer sensitive information from model behavior.

Primary metric: Membership Inference Attack AUC-ROC

Degree to which the system collects, stores, retrieves, logs, and exposes only the data necessary for documented purposes and retention periods.

Primary metric: Minimization Compliance Rate (MCR)

Likelihood that anonymized, aggregated, embedded, logged, or generated data can be linked back to individuals or protected groups using auxiliary information.

Primary metric: Re-identification Success Rate (RISR)

Ability to keep the system within authorized tasks, data boundaries, tool permissions, and network scope. Consumption volume, budgets, and termination behavior are scored separately under Cn-7.

Primary metric: Scope Violation Success Rate (SVSR)

Resistance to privilege escalation, authorization bypass, role confusion, unsafe delegation, and tool-permission amplification.

Primary metric: Escalation Success Rate (ESR)

Ability to detect, block, transform, or safely route unsafe, unauthorized, policy-violating, or context-leaking outputs under normal and adversarial conditions.

Primary metric: Unsafe Output Escape Rate (UOER)

Resistance to information leakage through timing, token probability, error messages, resource usage, cache behavior, logs, telemetry, GPU/accelerator sharing, or covert channels.

Primary metric: Side-Channel Leakage Rate (SCLR)

Strength of identity verification, authentication, authorization, delegation, and attestation across agents, tools, MCP servers, workloads, and sessions.

Primary metric: Identity Spoofing Success Rate (ISSR)

Cn-6 measures the fraction of automated actions whose reversibility class was determined and enforced before the system executed them, with multi-step or multi-agent chains governed by the worst-case (highest-impact) hop present anywhere in the chain. Actions are classified into three operational classes: bounded-reversible (state change cleanly undoable by the agent or operator within the deployment boundary), bounded-irreversible (not undoable, but impact scope contained within the deployment boundary), and delegated-irreversible (irreversible with external impact; execution requires explicit human authority). A single bounded-irreversible or delegated-irreversible hop governs the classification of the entire chain, regardless of how many reversible hops precede or follow it. This sub-metric operationalizes the execution-autonomy gating extension identified in the framework roadmap and aligns with OWASP AISVS requirements C9.2.3, C9.2.4, and C9.2.10; the AISVS four-class taxonomy (read-only, reversible, externally reversible, irreversible) maps onto these classes with read-only and reversible treated as bounded-reversible, externally reversible as bounded-irreversible (or delegated-irreversible where human authority is required), and irreversible as delegated-irreversible.

Primary metric: Action Reversibility Classification Rate (ARCR)

Cn-7 measures whether every applicable resource and execution-expansion class has a declared, identity- or root-task-bound limit that is enforced at an authoritative control point and terminates or degrades safely when challenged. Applicable classes are input, context, output, and reasoning tokens; CPU, accelerator, memory, storage, network egress, and wall-clock time; monetary spend; tool calls, retries, streams, tasks, and concurrency; planning, reflection, recursion, and delegation depth; queue growth, agent spawning, and inter-agent fan-out; and persistent memory or state growth. Unknown surfaces are not NOT APPLICABLE. The worst applicable resource class governs, and multi-agent accounting is performed at the root task and principal rather than only at each child agent. This sub-metric aligns with OWASP AISVS 1.0 C9.1.1/C9.1.2, C9.3.3/C9.3.4, and C11.2.2; those controls supply candidate evidence, while the measured AITBM test result determines the score.

Primary metric: Resource Boundary Violation Rate (RBVR)