AI-governance Published: Updated: By

What is SHAP and why does it matter for compliance AI?

Quick answer

SHAP (SHapley Additive exPlanations) is a framework that measures each variable's contribution to an individual AI model decision. EU and US regulators require this explainability for high-risk financial AI. The EU AI Act and SR 11-7 both treat decision-level explanation as a baseline requirement. ---

The full answer

SHAP stands for SHapley Additive exPlanations. The name references Lloyd Shapley, whose 1953 work on cooperative game theory provided the mathematical foundation. Stuart Lundberg and Su-In Lee formalized its application to machine learning in a 2017 NeurIPS paper that has since become one of the most cited ML papers in existence.

The core idea: for any individual prediction, SHAP assigns each input feature a numerical contribution score. If a model's average prediction score is 0.30 (a baseline suspicious-activity rate) and a particular transaction scores 0.82, SHAP explains how each feature moved the number from 0.30 to 0.82. Transaction amount: +0.21. Counterparty jurisdiction: +0.18. Account dormancy pattern: +0.16. Payroll cycle regularity: minus 0.08. The contributions add up to the gap.

This differs from global feature importance. Most models can tell you which variables matter most on average across all predictions. SHAP tells you which variables drove this specific decision. That's the distinction compliance functions need.

Two implementations dominate financial services. TreeSHAP is fast and exact, designed for gradient-boosted trees and random forests, and runs in milliseconds. KernelSHAP is slower but works with any model architecture. For most AML and fraud applications running on ensemble models, TreeSHAP is the practical choice.

The Federal Reserve and OCC's SR 11-7 guidance has required banks to understand and document model outputs since 2011. Examiners applying it now routinely ask whether AI-generated decisions can be explained at the record level. "The model flagged it" fails that standard.

The EU AI Act, Regulation (EU) 2024/1689, classifies credit scoring, fraud detection, and AML systems as high-risk AI. Deployers must ensure transparency and must provide explanations for individual decisions affecting individuals. SHAP is the most common technical path for meeting that obligation. Banks subject to both regimes face the stricter of the two standards, and the EU requirement on individual decision explanation is more prescriptive than SR 11-7.

Why this matters

The SAR filing workflow is where this becomes concrete. An analyst reviewing an AI-generated alert can't write "AI flagged this transaction" in a SAR narrative. She needs to identify specific conduct. SHAP feature contributions make that possible: rapid cross-border transfers contributed most to this alert, combined with a dormant-then-active pattern and round-number amounts. That's a SAR narrative. For the filing timeline, see how long do banks have to file a SAR. For what FinCEN considers suspicious, see how does FinCEN define suspicious activity.

What percentage of AML alerts are false positives? is the more fundamental question behind SHAP adoption. In institutions where 90-95% of AI-generated alerts are false positives, SHAP helps analysts dismiss bad alerts faster and document the rationale for good ones without writing free-form notes from scratch.

Examiners ask about explainability directly during reviews. See what triggers a regulatory exam for what puts a bank on the schedule, and what happens when a bank fails an AML exam for what's at stake. When a bank can't explain how its AI made a decision, examiners treat that as a model risk management failure under SR 11-7, not just a technology gap. That finding can contribute to a consent order or, in severe cases, a monitorship. See what is a monitorship and when is one imposed for the threshold where examiners move from findings to enforcement.

One point compliance teams often miss: SHAP explains what the model did, not whether the model was right. A model trained on biased historical data produces confident, well-documented wrong decisions. The Basel Committee's BCBS d564 paper identifies explainability, robustness, and fairness as three separate requirements. Passing SHAP-based explainability audits doesn't substitute for independent model validation. Banks need both.

For AI-assisted AML specifically, see can AI be used for AML transaction monitoring for the broader regulatory position on AI in the alert process.

Related questions

Related concepts and regulations

← All compliance questions