AI-governance Published: Updated: By

What is SHAP and why does it matter for compliance AI?

Quick answer

SHAP (SHapley Additive exPlanations) is a mathematical method that breaks down an AI model prediction into the contribution of each input feature. For compliance AI in banking, it's the primary technique for meeting explainability requirements under Federal Reserve SR 11-7 and the EU AI Act's high-risk AI provisions.

The full answer

SHAP stands for SHapley Additive exPlanations. It's a framework for explaining individual machine learning predictions, built on Shapley values from cooperative game theory. For any single prediction, SHAP calculates how much each input feature contributed to the output, relative to the model's average baseline.

Scott Lundberg and Su-In Lee published the machine learning formulation in 2017, unifying earlier approaches under one consistent framework. TreeSHAP, a fast exact variant for gradient-boosted models, followed in 2020. Most production AML and credit risk models in financial services use gradient-boosted trees, so TreeSHAP is now the standard in practice.

The concrete example: a transaction monitoring model scores a wire transfer at 0.91. SHAP breaks that down. Counterparty country risk: +0.29. Transaction size versus customer baseline: +0.24. Velocity in the prior 72 hours: +0.21. Time-of-day pattern: +0.08. Account tenure: -0.11. An investigator reads that in seconds and decides whether to pursue it. Without SHAP, they see 0.91 and rely on gut feel. That's one reason AML alert false positive rates remain above 90% at most banks.

SR 11-7: the US regulatory anchor

SR 11-7, the Federal Reserve and OCC's 2011 guidance on model risk management, requires institutions to document, validate, and understand every model used in risk decisions. Section 4 demands "effective challenge," meaning staff must be able to explain model outputs to examiners. Banks that can't demonstrate per-decision explainability have received formal MRAs (Matters Requiring Attention), and in several consent orders, model opacity has appeared as a contributing factor alongside the primary violations.

The OCC's 2021 model risk FAQ update reinforced the point: complexity doesn't excuse opacity. If a model is too complex to explain, it requires additional compensating controls.

EU AI Act: the new EU standard

The EU AI Act (Regulation 2024/1689) classifies AI used in credit scoring, fraud detection, and financial services risk assessment as high-risk under Annex III. Article 13 requires that high-risk systems be transparent enough for deployers to interpret outputs and act responsibly. Article 11 mandates technical documentation sufficient to assess compliance.

Anyone deploying or subject to the EU AI Act in financial services needs to satisfy these requirements. The high-risk AI obligations are already active for systems in production. SHAP is the method most accepted by regulators because it also covers GDPR Article 22, which grants individuals a right to explanation for automated decisions affecting them.

Adverse action notices

ECOA and Regulation B require US lenders to state specific reasons for adverse credit decisions. The CFPB's 2023 circular on AI credit models confirmed that "the model scored you low" doesn't satisfy this. The dominant practice now is mapping SHAP's top negative feature contributions to the approved adverse action reason codes. It works, though the mapping requires legal review: the raw SHAP value for "transaction count in last 90 days" isn't the same as the plain-language reason a borrower receives.

Why compliance teams need SHAP explainability

The practical problem is operational, not just regulatory.

AI-driven transaction monitoring generates high alert volumes. Without per-alert explanations, investigators triage by instinct. With SHAP, the top three contributing factors appear on screen and a trained investigator can assess relevance in under a minute. Banks that added SHAP to existing models have reported SAR productivity gains without adding headcount.

The regulatory risk is also direct. What triggers a regulatory exam often includes model-related audit flags or complaints. Examiners now routinely test model explainability as part of BSA/AML reviews. Institutions that can't produce a per-decision explanation on demand leave themselves exposed. Monitorships have been imposed on institutions where systemic model governance failures, including opacity, contributed to enforcement actions.

Customer risk rating decisions are a specific pressure point. If a model triggers enhanced due diligence for a long-standing corporate client, the relationship manager will ask why. "The system flagged it" damages both the client relationship and the audit trail. SHAP gives the compliance team a specific, defensible answer: these three factors crossed the threshold.

Explainability also aids model monitoring. If SHAP shows that "account tenure" suddenly carries three times its historical weight after a retrain, that's a signal worth investigating before it affects regulatory filings. For institutions that refresh customer risk ratings continuously, SHAP attribution patterns over time become part of model validation evidence.

The one thing SHAP doesn't fix: a bad model. Attaching SHAP to a model trained on biased or unrepresentative data produces confident-sounding explanations for wrong decisions. Explainability tooling and model quality are separate problems, and regulators expect both.

Related questions

Related concepts and regulations

← All compliance questions