Better Bayesian Filtering

Paul Graham’s essay Better Bayesian Filtering describes improvements to spam filtering, achieving 99.5% filtering with <0.03% false positives.

Why Previous Research Underperformed

Insufficient training data
Ignoring message headers—”don’t ignore data”
Token stemming as premature optimization
Using all tokens rather than the most significant ones
Failure to bias against false positives

Key Improvements

Case preservation distinguishes spam patterns. Context marking changes probabilities dramatically—”free” in subject lines showed 97.82% spam probability versus 65.4% in body text.

Performance Results

Over one month: 99.75% filtering rate with 0.06% false positives.

Strategic Insights

Individual user filters outperform network-level approaches because varied probabilities across users forces spammers into costly testing cycles.

My Takeaway

False positives deserve different treatment than false negatives—they cause greater harm. Never ignore data, and bias your systems toward protecting users.