Absence of Evidence ≠ Evidence of Absence: Rethinking Fraud Prevention KPIs

The Zero-Incident Paradox: Why Silence Isn't Always Golden in Fraud Prevention Using Bayesian reasoning to measure what you cannot directly observe

The Core Paradox

The Fundamental Challenge: Low incident rates in fraud prevention could indicate either:

• Strong defenses - The system is working effectively

• Lack of attempts - There simply haven’t been many fraud attempts

This ambiguity makes it difficult to assess the true effectiveness of fraud prevention measures.

The Unknown Unknowns Problem

Beyond measuring what we know, there’s a deeper challenge: you cannot directly measure what you are unaware of. This is the classic “unknown unknowns” dilemma in security and fraud prevention.

Bayesian Framework

To address these challenges, we can apply Bayesian reasoning:

\(P(\text{Fraud}|\text{No Incident}) = \frac{P(\text{No Incident}|\text{Fraud}) \cdot P(\text{Fraud})}{P(\text{No Incident})}\)

Interpreting the Evidence

• When the likelihood is high: P(No Incident∣Fraud) is high

  • The absence of fraud incidents strongly indicates effective fraud prevention

• When the likelihood is low: P(No Incident∣Fraud) is low

  • The Absence of incidents provides little information about fraud prevention effectiveness

Measuring the Unmeasurable

The Challenge

Evaluating the likelihoods requires:

• Understanding the threat landscape

• Assessing coverage of known fraud methods

• Accounting for unknown fraud methods (which cannot be directly measured)

Indirect Measurement Methods

Approaches to Discover Unknown Unknowns

When direct measurement is impossible, we can use these indirect methods:

1. Red Teaming - Simulate adversarial attacks to find vulnerabilities

2. External Threat Intelligence - Learn from industry patterns and breaches

3. Post-Mortem Analysis - Learn from failures when they occur

4. First Principles Decomposition - Break down attack surfaces systematically

Practical Framework

Measuring Fraud Prevention Success: To measure the success of fraud prevention in low incident scenarios, focus on:

1. Coverage Metrics

   1. Percentage of known fraud methods with active defenses

   2. Depth and breadth of detection capabilities (Cost of attack)

2. Discovery Rate

   1. Incremental discovery of new fraud methods over time

   2. Rate of vulnerability identification and remediation

3. Process Interception Metrics

   1. Trigger rate: Proportion of applications/transactions flagged by rules

   2. Hit rate: Proportion of triggered cases with confirmed abnormal behaviors (grey areas go to manual review)

This approach helps infer effectiveness even when direct incident data is sparse.