F1 Score: Definition and Use in Compliance

What is F1 Score?

F1 Score is the harmonic mean of precision and recall, computed as 2 × (Precision × Recall) / (Precision + Recall). It ranges from 0 to 1. A score of 1 means the model catches every real positive and produces no false alerts. A score near 0 means it fails catastrophically on at least one dimension.

The harmonic mean, unlike a simple average, punishes extreme imbalances. A model with precision 0.95 and recall 0.10 would average to 0.525. Its F1 Score is 0.18. That number correctly communicates that the model is operationally useless despite high precision, because it's missing 90% of the crimes it's supposed to detect.

The components come directly from the confusion matrix: the four-cell table of true positives, false positives, true negatives, and false negatives. Precision is true positives divided by all predicted positives. Recall is true positives divided by all actual positives. Every precision-recall calculation traces back to those four counts. A model that generates a false negative on a genuine $2 million suspicious activity report creates more than an accuracy problem; it's a potential regulatory failure with exam consequences.

A concrete example: a regional bank processes 50,000 transactions per day. Its fraud model flags 300. Analysts confirm 240 are real fraud, giving precision of 0.80. That day, 280 actual fraudulent transactions occurred, so the model caught 240 of them, giving recall of 0.857. F1 = 2 × (0.80 × 0.857) / (0.80 + 0.857) = 0.828. A BSA officer reviewing that number knows the model is performing well but missing roughly 40 real fraud cases daily. Whether that gap is acceptable depends on the institution's risk profile and typology mix.

Why does this metric dominate compliance model evaluation over simpler alternatives? Because accuracy is structurally dishonest on imbalanced datasets. When fraud represents 0.05% of transactions, a model that never predicts fraud at all scores 99.95% accurate. F1 Score can't be fooled that way.

How is F1 Score used in practice?

Compliance teams touch F1 Score at three distinct points: initial model deployment, periodic model validation, and after rule or feature changes.

At deployment, the team splits their labeled historical dataset into training and test sets. After training, they evaluate the model against the hold-out test set and record F1 alongside other performance metrics. That number is the baseline. If subsequent threshold tuning to reduce alert volume drops F1 from 0.80 to 0.71, the team knows the operational improvement came at a real cost to detection quality.

Periodic validation is where F1 Score becomes a regulatory instrument. Model validators running annual reviews under the Federal Reserve's SR 11-7 guidance compare the current F1 against the deployment baseline. A meaningful decline triggers escalation. We've seen banks drop from F1 0.83 to F1 0.69 after a data pipeline change silently excluded a key behavioral feature from the model's input. Scheduled model monitoring caught that drift before the next examination cycle and prevented a documented model limitation finding that would have required formal remediation.

After configuration changes, F1 acts as the acceptance criterion. A compliance team adding a typology rule for cryptocurrency structuring attempts wants to see F1 hold steady or improve. If F1 drops from 0.82 to 0.76 after the rule addition, the new rule has degraded overall model performance regardless of how many incremental alerts it generated. That's the kind of finding that stops a rule deployment in its tracks, and rightly so.

Day-to-day, the metric appears in model governance committee packs, regulatory examination responses, and internal audit reports. MLROs reviewing AI system performance will see F1 alongside investigation conversion rates and analyst capacity utilization. It's the number that bridges data science outputs and the business-level question: "Is this model catching what it should?"

F1 Score in regulatory context

No regulation names F1 Score specifically. The regulatory framework around model risk management has still made it a de facto standard, because regulators require performance documentation and F1 is the most informative single metric for imbalanced classification problems.

The Federal Reserve's SR 11-7 guidance, issued in April 2011 and jointly adopted by the OCC through Bulletin 2011-12, requires institutions to document model performance, conduct independent validation, and maintain ongoing monitoring. When examiners ask for evidence that a model is performing as intended, F1 Score is one of the primary statistics institutions produce. A well-prepared model risk management package will show F1 at deployment, F1 at last validation, and F1 from the most recent monitoring cycle.

The Financial Action Task Force (FATF) addressed AI and machine learning in financial crime detection directly in its 2021 report "Opportunities and Challenges of New Technologies for AML/CFT." The report calls for institutions to validate AI model outputs and document detection performance. F1 Score satisfies those validation requirements because it captures both dimensions of model quality on the highly imbalanced datasets that AML and fraud environments produce.

In the EU, the AI Act (Regulation 2024/1689) classifies certain fraud detection and AML systems as high-risk AI. Article 9 requires ongoing performance monitoring, and Articles 11 and 16 require technical documentation of model performance. Compliance teams use F1 Score as the core metric in that technical documentation. It's specific, reproducible, and directly tied to the four fundamental outcomes (true positives, false positives, true negatives, false negatives) that regulators care about.

A concrete examination scenario: an OCC examiner reviewing a bank's model risk program asks for evidence that the AML scoring model is performing as stated. The bank produces a validation report showing F1 of 0.79 at deployment and 0.74 after 18 months in production. That 5-point decline becomes a documented model limitation, triggering a retraining obligation before the next examination.

Common challenges and how to address them

The most common problem teams face with F1 Score in financial crime is class imbalance. Fraud and money laundering events represent a tiny fraction of all transactions, sometimes 0.01% or less. Training on that raw data produces classifiers that almost never predict the positive class, achieving high accuracy and poor F1.

Standard fixes include oversampling the minority class (SMOTE is the most widely deployed method), undersampling the majority class, or adjusting class weights during training. All three approaches shift model attention toward rare positive events, typically at some cost to precision. The right balance depends on analyst capacity. A team that can handle 200 alerts per day doesn't benefit from a high-recall, low-precision model generating 800 alerts, even if the false positive rate looks acceptable on paper.

The second challenge is threshold selection. Most classification models default to a 0.50 probability threshold for the positive class. Lowering it to 0.30 increases recall (catch more true fraud) at the cost of precision (more false alerts). Plotting F1 Score across all possible thresholds, finding the peak, and setting the production threshold there is the correct approach. This isn't a one-time exercise: threshold tuning should run after every major model update and at least annually as part of validation.

Dataset contamination is a subtler problem. If labeled training data includes suspicious activity reports filed on transactions later cleared by law enforcement, the model learns from noisy labels. F1 Score calculated on a contaminated test set produces a falsely optimistic view of real-world performance. Periodic ground truth refresh, re-examining labels as investigative dispositions come in, corrects this over time but requires deliberate data governance that many compliance teams don't have in place.

Finally, F1 Score treats false positives and false negatives as equally costly. Missing a $50 million structuring scheme is categorically worse than generating a false alert on a $300 transfer. The F-beta score (Fβ) addresses this by weighting recall more heavily than precision when β > 1. Teams in high-risk environments, particularly correspondent banking or sanctions monitoring, often use F2 or F3 score to reflect the asymmetric cost of missed detections.

Related terms and concepts

F1 Score sits at the center of a cluster of metrics that compliance and model risk teams use together. Understanding the relationships helps interpret what F1 is actually measuring.

Precision and recall are the direct inputs. Precision measures alert quality; recall measures detection coverage. You can't optimize F1 without managing the trade-off between them. Increasing recall typically reduces precision, and the right balance depends entirely on the institution's operating environment. A high-volume retail bank with a large analyst team can tolerate more false positives to maintain high recall. A smaller institution with three analysts cannot.

The model validation process is the organizational context where F1 Score is most formally reviewed. Independent validators don't build models; they stress-test them. F1 Score is one of the primary statistics they challenge, particularly on out-of-time samples, which are datasets from periods not seen during training. A model that achieves F1 of 0.85 on the training-period test set but 0.67 on a two-year-old holdout has an overfitting problem that validation is designed to catch.

ROC AUC is a complementary metric. It's threshold-independent, measuring the model's ability to distinguish between classes across every possible operating point simultaneously. AUC is most useful for selecting the best model during development. F1 Score is most useful for evaluating the deployed model at its specific operating threshold. Production monitoring programs need both.

Explainability is increasingly linked to F1 Score in regulatory settings. An aggregate F1 of 0.82 tells a regulator the model is performing adequately overall, but it doesn't explain why the model missed the 18% it failed to detect. Full decision explanations, showing which input features drove each individual prediction, turn an aggregate metric into actionable intelligence for analysts investigating specific cases. Regulators in both the US and EU are moving toward requiring that explainability be demonstrated alongside performance metrics, not separately from them.

The three metrics together, F1 Score, ROC AUC, and explainability documentation, form the baseline evidence package that compliance teams produce for model examinations. F1 is the most concise summary of how the model behaves in the real world today.