Recall and precision exist in tension. In most production systems, improving one degrades the other. Between 2021 and 2024, on a real-time security classifier, my team pushed recall from 0.50 to 0.78 while keeping the false positive rate flat. This post explains how.
The classifier was a binary model flagging events in a real-time stream. A false negative meant a missed detection with real downstream cost. A false positive meant an alert that wasted an operator's time. Both mattered, which is why "just lower the threshold" was not an acceptable answer.
Before touching any hyperparameter, we used SHAP (SHapley Additive exPlanations) to audit what the model was actually doing on the false negatives. Not just feature importances globally, but for each missed positive: which features were pulling the prediction down?
The pattern was consistent. A cluster of features related to temporal context had low SHAP values on missed events. The model had learned a shortcut: if certain temporal signals were absent, it confidently predicted negative, even when other signals were present. The root cause was a data collection artifact that made temporal features sparse for a specific event subtype.
Without the SHAP audit, we would have spent weeks tuning the wrong things.
Recall of 0.50 on the full test set masked a more nuanced picture. When we segmented by event subtype, recall on the common subtype was already 0.71. The problem was concentrated in a rarer subtype where recall was 0.18.
This changed the strategy entirely. Instead of retraining the full model, we focused on:
Segmenting first prevented us from optimizing globally in a way that would have hurt the common subtype to marginally improve the rare one.
The SHAP analysis pointed to temporal context. We engineered three new features:
None of these were clever. They were direct translations of what the error analysis showed.
The final step was calibration. A naive approach would be to lower the global threshold until recall hits 0.78, then observe what happens to precision. We did something different: we calibrated the threshold separately per subtype, targeting a fixed false positive rate budget per subtype.
This meant the rare subtype got a more aggressive threshold (lower, to catch more positives) while the common subtype kept a tighter threshold (higher, since its recall was already good). The result was that the overall false positive rate stayed flat even as overall recall improved.
The calibration was done on a held-out validation set that was not used in any previous step, to avoid optimistic estimates.
| Metric | Before | After |
|---|---|---|
| Recall (overall) | 0.50 | 0.78 |
| False positive rate | baseline | flat |
| Recall (rare subtype) | 0.18 | 0.61 |
| Recall (common subtype) | 0.71 | 0.82 |
The single biggest mistake would have been trusting the aggregate recall number. 0.50 overall looked like a model that was mediocre everywhere; it was actually a model that was fine on the common case (0.71) and badly broken on one rare subtype (0.18). Global SHAP importance wouldn't have surfaced that either. It tells you what the model generally relies on, not what it's ignoring specifically on the cases it gets wrong. Running SHAP on the false negatives themselves was what pointed at the sparse temporal features.
None of the engineered features were clever in isolation. What mattered was that they came directly out of the error analysis instead of a round of hyperparameter search. Per-segment threshold calibration gets treated as a hack in a lot of writeups; it isn't one once class distributions genuinely diverge across segments. It's just the correct tool for that situation.
Recall went from 0.50 to 0.78, a 56% relative improvement, with the false positive rate flat, because the fix targeted the actual failure mode instead of trading precision for recall globally.