SHAP (SHapley Additive exPlanations): Definition and Use in Compliance

What is SHAP (SHapley Additive exPlanations)?

SHAP assigns a numerical contribution to each input feature for every individual machine learning prediction. The value shows how much that feature pushed the model's output higher or lower, relative to a baseline. For a fraud risk score of 0.87, SHAP tells you which features contributed how much and in which direction.

The underlying math comes from Lloyd Shapley's 1953 work in cooperative game theory. The problem Shapley solved: given a coalition of players jointly producing an outcome, how do you fairly allocate credit? His axiomatic solution guarantees efficiency (contributions sum to the total output), symmetry (equal contributors receive equal credit), and additivity (combined effects work linearly). Scott Lundberg and Su-In Lee showed in their 2017 NeurIPS paper that the same logic applies to ML models: features are players, the prediction is the outcome, and SHAP values are the fair attribution.

For compliance teams, this matters because ML models now trigger decisions with regulatory consequences. Suspicious Activity Report (SAR) filings, customer risk tier assignments, fraud blocks on legitimate transactions. A risk score is a number. SHAP turns it into a statement: this specific transaction scored high because the wire amount was $94,000, the counterparty operates in a high-risk jurisdiction, and the customer opened this account 12 days ago.

SHAP produces explanations at two levels. Local explanations cover a single prediction: here is why this specific case scored the way it did. Global explanations aggregate SHAP values across thousands of predictions to show which features matter most to the model overall. Local explanations support analyst review of individual cases. Global explanations support Model Risk Management (MRM) and ongoing model oversight.

The dominant production implementations are TreeSHAP (fast and exact for gradient boosting and random forest models) and KernelSHAP (model-agnostic, slower). Banks running real-time scoring pipelines use TreeSHAP. Compliance audits of neural network outputs typically use KernelSHAP or approximate variants like DeepSHAP.

How is SHAP (SHapley Additive exPlanations) used in practice?

The most direct application is alert explanation. A transaction monitoring system flags a payment. The analyst sees the score and the breakdown: wire amount ($94,800) contributed +0.41, counterparty country contributed +0.29, customer account age contributed -0.12, time of day contributed +0.07. The analyst can agree with the model's reasoning, override it, or escalate, and each of those actions gets documented with the underlying feature evidence attached.

This isn't a minor workflow improvement. We've seen compliance teams where analysts were dismissing more than 80% of alerts without reviewing the transaction details, because scores were presented with no context. Adding SHAP breakdowns to the alert interface changed how analysts engaged with the queue. Dismissal patterns became more consistent, and case notes became substantive enough to satisfy examination.

Enhanced Due Diligence (EDD) workflows benefit in a similar way. When a customer's rating crosses the EDD threshold, the relationship manager needs to understand what drove the change. A SHAP output showing that 70% of the risk score came from a recent counterparty change tells the reviewer exactly where to focus additional scrutiny.

Model validation teams use SHAP to catch problems before they reach production. At one US mid-market bank, a fraud model's top SHAP contributor turned out to be a field that functioned as a proxy for customer ethnicity. The model never deployed. Without SHAP, that bias would have been invisible in aggregate accuracy metrics.

For examination preparation, SHAP logs let compliance officers pull any past decision and show the exact feature-level reasoning behind it. When an examiner asks why a customer received enhanced scrutiny or why a specific filing occurred, a SHAP record answers that question at the case level. Examiners at OCC and FinCEN have explicitly referenced the need for per-decision documentation in recent guidance, and SHAP is the most practical way to provide it.

SHAP (SHapley Additive exPlanations) in regulatory context

No regulation names SHAP by name. But the requirements that make SHAP necessary have been in place for more than a decade.

SR 11-7, the Federal Reserve and OCC's model risk management guidance, requires banks to understand and document how models reach conclusions. For a black-box ML model, that means individual-decision transparency. Validators must evaluate "conceptual soundness" and "check model outputs against intuition and other known results." For a gradient boosting model with 200 input features, SHAP is the only practical tool that satisfies those requirements at the case level, not just at the aggregate performance level.

The EU AI Act (2024) classifies credit scoring and AML systems as high-risk AI. It requires that affected individuals receive "meaningful information about the logic involved" in automated decisions. Generic feature importance rankings from a model summary don't satisfy this. SHAP's per-prediction outputs do, provided the bank translates numerical SHAP values into readable explanations before including them in customer-facing notices.

AI Governance programs at large banks now treat SHAP outputs as a required artifact in the model development lifecycle, sitting alongside training data documentation and validation reports. The question has shifted from "can you explain your model in general terms" to "can you produce a timestamped explanation for any production decision you made in the last seven years."

Fair Lending is the highest-stakes application. If SHAP analysis reveals that a protected characteristic, or a correlated proxy, is a top contributor to loan denials or elevated risk scores, that's a potential ECOA or Fair Housing Act violation. Several US banks have run SHAP audits proactively to identify and remove biased features before examiners found them. Discovering it during examination is considerably more expensive, and more damaging to the bank's relationship with its primary regulator.

Common challenges and how to address them

SHAP has real tradeoffs. Compliance teams should understand them before deploying it in production.

Computational cost is the first constraint. KernelSHAP requires sampling and can take several seconds per prediction, which is incompatible with real-time fraud scoring at scale. The practical solution is to use TreeSHAP for tree-based models, where it runs in milliseconds, and to batch SHAP calculations for transaction monitoring jobs rather than generating explanations on demand. For neural networks, GPU-accelerated approximate methods like DeepSHAP are fast enough for most production workflows, though they trade some precision for speed.

The second issue is correlated features. SHAP assumes feature independence, and financial data violates this constantly. Transaction count and transaction amount are correlated. When correlated features are present, SHAP distributes their combined contribution somewhat arbitrarily between them, which can mislead analysts reviewing individual explanations. The fix is to document known correlated pairs and train analysts to interpret grouped contributions rather than treating each SHAP value as an independent, standalone fact.

Misuse in customer-facing communications is a compliance risk in itself. Some institutions have passed raw SHAP outputs directly into adverse action notices. A value of "+0.31 for account_tenure_days" is technically accurate but doesn't satisfy a customer's right to understand why they were declined. A translation layer between SHAP output and customer-facing language is required, and that layer needs its own documentation for examination purposes.

Finally, SHAP doesn't replace analyst judgment. A high SHAP contribution from "counterparty_in_high_risk_jurisdiction" still requires a human to assess whether that counterparty is a legitimate importer or part of a mule network. SHAP identifies what the model noticed. Whether the model was right is a question for the alert disposition workflow.

Related terms and concepts

SHAP sits in a cluster of interpretability and model governance concepts that define current AI accountability standards in financial services.

LIME (Local Interpretable Model-Agnostic Explanations) is the main alternative. Both explain individual predictions, but LIME fits a local linear approximation around a single prediction's neighborhood, whereas SHAP uses game-theoretic attribution with theoretical guarantees. LIME is faster in certain configurations, but its explanations are less stable across repeated runs. In compliance contexts where explanation reproducibility is required for audit, SHAP is the more defensible choice.

Explainability is the broader category. It covers all techniques for making model behavior transparent: saliency maps for computer vision, attention weights for transformers, partial dependence plots for aggregate effects, and SHAP for individual feature attribution. When regulators ask for explainability in the context of AML or credit models, SHAP is the specific method most likely to satisfy that requirement for tabular data.

Model validation and model monitoring both depend on SHAP for ongoing model oversight. Validation uses SHAP to confirm a new model is using appropriate features before production approval. Monitoring uses SHAP drift analysis: if the top contributing features shift month over month without corresponding changes in the business environment, the model may be fitting noise from recent data. That's a stability problem requiring intervention before it becomes an examination finding.

AI Bias detection is where SHAP has its biggest downstream regulatory effect. Running SHAP analysis across demographic subgroups reveals whether different populations receive risk scores for different reasons. Aggregate metrics like AUC can look identical across groups while SHAP distributions differ, which indicates the model is solving the problem differently for different populations. That difference is where disparate impact analysis begins.

For ongoing model comparison, SHAP distributions factor into champion/challenger evaluation: challenger models are assessed in part by comparing the features driving their predictions against those of the current production model, with material differences in the top SHAP contributors flagged for reviewer attention.