How We Improved Recall from 0.50 to 0.78 Without Increasing False Positives

Recall and precision exist in tension. In most production systems, improving one degrades the other. But in a project at Intuition Machines, we pushed recall from 0.50 to 0.78 on a production classifier while keeping the false positive rate flat. This post explains how.

The classifier was a binary model flagging events in a real-time stream. A false negative meant a missed detection with real downstream cost. A false positive meant an alert that wasted an operator's time. Both mattered, which is why "just lower the threshold" was not an acceptable answer.

Step 1: Understand Where the Model Fails

Before touching any hyperparameter, we used SHAP (SHapley Additive exPlanations) to audit what the model was actually doing on the false negatives. Not just feature importances globally, but for each missed positive: which features were pulling the prediction down?

The pattern was consistent. A cluster of features related to temporal context had low SHAP values on missed events. The model had learned a shortcut: if certain temporal signals were absent, it confidently predicted negative, even when other signals were present. The root cause was a data collection artifact that made temporal features sparse for a specific event subtype.

Without the SHAP audit, we would have spent weeks tuning the wrong things.

Step 2: Segment the Problem

Recall of 0.50 on the full test set masked a more nuanced picture. When we segmented by event subtype, recall on the common subtype was already 0.71. The problem was concentrated in a rarer subtype where recall was 0.18.

This changed the strategy entirely. Instead of retraining the full model, we focused on:

• Targeted feature engineering for the low-recall subtype
• Oversampling that subtype in the training set
• A separate calibration layer for the final threshold

Segmenting first prevented us from optimizing globally in a way that would have hurt the common subtype to marginally improve the rare one.

Step 3: Feature Engineering Based on Error Analysis

The SHAP analysis pointed to temporal context. We engineered three new features:

• A rolling aggregate of activity over a longer window, to compensate for sparse short-window signals
• A binary flag indicating the sparse-data condition itself, so the model could learn to behave differently when temporal data was incomplete
• An interaction term between existing high-importance features and the data-completeness flag

None of these were clever. They were direct translations of what the error analysis showed.

Step 4: Threshold Calibration, Not Just Lowering

The final step was calibration. A naive approach would be to lower the global threshold until recall hits 0.78, then observe what happens to precision. We did something different: we calibrated the threshold separately per subtype, targeting a fixed false positive rate budget per subtype.

This meant the rare subtype got a more aggressive threshold (lower, to catch more positives) while the common subtype kept a tighter threshold (higher, since its recall was already good). The result was that the overall false positive rate stayed flat even as overall recall improved.

The calibration was done on a held-out validation set that was not used in any previous step, to avoid optimistic estimates.

Results

Lessons

• Aggregate metrics hide the real problem. Recall of 0.50 overall masked a 0.18 on the subtype that actually mattered most.
• SHAP on errors is more useful than SHAP on the full dataset. Global feature importance told us what the model liked. SHAP on false negatives told us what the model was ignoring.
• Feature engineering beats hyperparameter tuning. All three new features were constructed from domain understanding, not search.
• Per-segment calibration is not a hack. It is the correct tool when class distributions differ across segments and both precision and recall matter.

Conclusion

Improving recall without hurting precision is not a matter of luck or the right library. It requires understanding why the model fails, segmenting that failure, fixing the underlying cause through features, and then calibrating deliberately. In this project, the combination of SHAP error analysis, subtype segmentation, and per-segment threshold calibration produced a 56% relative improvement in recall with no regression in false positives.

If you are working on a similar classification problem and want to discuss the approach, feel free to reach out.