Churn prediction is one of the most common ML use cases in commercial organizations, and one of the most frequently over-engineered. This post documents the practical lessons from building and deploying a production churn model on 15 million customer records — what the data looked like, which features actually mattered, what the model got wrong, and how results were integrated into a CRM system so that sales teams could act on them.
The numbers are real. The architecture decisions reflect tradeoffs made under production constraints.
The goal was to predict which customers would cancel within 90 days, with enough lead time for account managers to intervene. The dataset covered 15M customer records across a B2B SaaS product with monthly and annual subscription tiers.
Two early requirements shaped the entire project:
At 15M records the dataset was large enough to be interesting but not large enough to require distributed computing for model training. The real complexity was in feature engineering.
import pandas as pd
import numpy as np
def build_churn_features(usage_df: pd.DataFrame, billing_df: pd.DataFrame,
support_df: pd.DataFrame, as_of_date: str) -> pd.DataFrame:
"""
Build the feature matrix for churn prediction.
as_of_date: the cutoff date — features use only data before this date,
labels are based on events 90 days after.
"""
cutoff = pd.Timestamp(as_of_date)
window_30d = cutoff - pd.Timedelta(days=30)
window_90d = cutoff - pd.Timedelta(days=90)
# --- Usage features ---
# Week-over-week decline in active users is the single strongest signal
usage_recent = usage_df[usage_df["date"].between(window_30d, cutoff)]
usage_prior = usage_df[usage_df["date"].between(window_90d, window_30d)]
dau_recent = usage_recent.groupby("account_id")["dau"].mean().rename("dau_30d_avg")
dau_prior = usage_prior.groupby("account_id")["dau"].mean().rename("dau_prior_30d_avg")
features = pd.concat([dau_recent, dau_prior], axis=1)
features["dau_trend"] = (features["dau_30d_avg"] - features["dau_prior_30d_avg"]) / (features["dau_prior_30d_avg"] + 1e-6)
# Feature adoption: % of core features used in last 30 days
feature_usage = usage_recent.groupby("account_id")["features_used"].apply(
lambda x: len(set.union(*x.apply(set))) if len(x) > 0 else 0
)
total_features = 24 # total features in the product
features["feature_adoption_pct"] = feature_usage / total_features
# --- Billing features ---
billing_recent = billing_df[billing_df["date"] <= cutoff]
payment_fails = billing_recent[billing_recent["status"] == "failed"].groupby("account_id").size()
features["payment_failures_90d"] = payment_fails.reindex(features.index, fill_value=0)
# Days to renewal — accounts close to renewal with low adoption are high risk
features["days_to_renewal"] = (
billing_recent.groupby("account_id")["renewal_date"].max() - cutoff
).dt.days
# --- Support features ---
support_recent = support_df[support_df["created_at"].between(window_90d, cutoff)]
ticket_counts = support_recent.groupby("account_id").size().rename("ticket_count_90d")
escalations = support_recent[support_recent["escalated"]].groupby("account_id").size().rename("escalations_90d")
features = features.join(ticket_counts, how="left").join(escalations, how="left")
features[["ticket_count_90d", "escalations_90d"]] = features[["ticket_count_90d", "escalations_90d"]].fillna(0)
return features.reset_index()
After extensive feature importance analysis, four signals dominated:
import lightgbm as lgb
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import roc_auc_score, precision_recall_curve, average_precision_score
import numpy as np
# Churn prediction is a temporal problem — use time-based splits
# Training on future data to predict past churn is a common leakage mistake
tscv = TimeSeriesSplit(n_splits=5, gap=30) # 30-day gap prevents leakage at boundaries
# LightGBM outperformed logistic regression, random forest, and XGBoost
# Primary reasons: handles class imbalance natively, fast on 15M rows, good calibration
params = {
"objective": "binary",
"metric": "average_precision",
"is_unbalance": True, # ~8% churn rate — handle imbalance
"learning_rate": 0.05,
"num_leaves": 63,
"min_child_samples": 100, # regularize on large dataset
"feature_fraction": 0.8,
"bagging_fraction": 0.8,
"bagging_freq": 5,
"verbose": -1,
}
auc_scores, ap_scores = [], []
for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
X_tr, X_val = X.iloc[train_idx], X.iloc[val_idx]
y_tr, y_val = y.iloc[train_idx], y.iloc[val_idx]
dtrain = lgb.Dataset(X_tr, label=y_tr)
dval = lgb.Dataset(X_val, label=y_val, reference=dtrain)
model = lgb.train(
params, dtrain,
num_boost_round=500,
valid_sets=[dval],
callbacks=[lgb.early_stopping(50), lgb.log_evaluation(100)],
)
preds = model.predict(X_val)
auc_scores.append(roc_auc_score(y_val, preds))
ap_scores.append(average_precision_score(y_val, preds))
print(f"Fold {fold+1}: AUC={auc_scores[-1]:.4f}, AP={ap_scores[-1]:.4f}")
print(f"\nMean AUC: {np.mean(auc_scores):.4f} ± {np.std(auc_scores):.4f}")
print(f"Mean AP: {np.mean(ap_scores):.4f} ± {np.std(ap_scores):.4f}")
Logistic regression was the baseline. It produced AUC of 0.73. LightGBM reached 0.84. The gap came from two factors: non-linear interactions (e.g., the effect of low feature adoption is much stronger when renewal is also near), and the ability to handle missing values natively (usage data is sparse for inactive accounts).
from sklearn.metrics import precision_recall_curve
import matplotlib.pyplot as plt
precision, recall, thresholds = precision_recall_curve(y_val, preds)
# Find the threshold that achieves at least 60% precision
# (account managers will ignore a model that's wrong >40% of the time)
target_precision = 0.60
idx = np.argmax(precision >= target_precision)
optimal_threshold = thresholds[idx]
print(f"Threshold for 60%+ precision: {optimal_threshold:.3f}")
print(f" Recall at that threshold: {recall[idx]:.3f}")
print(f" Accounts flagged per week: {(preds >= optimal_threshold).sum()}")
# Check: how many accounts can the team realistically handle?
# If the team has 20 account managers and each can reach out to 10 accounts/week,
# the model should flag ~200 accounts/week max
The threshold is a business decision, not a model decision. Setting it at 60% precision meant catching 38% of churners — not the highest possible recall, but a rate the team could actually work. Flagging 2,000 accounts/week for a team that can only handle 200 is worse than flagging 200.
The model runs weekly. Results are pushed to Salesforce via the REST API, updating a custom field on the Account object.
import requests
from typing import List
class SalesforceChurnWriter:
def __init__(self, instance_url: str, access_token: str):
self.base_url = f"{instance_url}/services/data/v57.0"
self.headers = {
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json",
}
def update_churn_scores(self, records: List[dict], batch_size: int = 200):
"""
Push churn scores to Salesforce Account objects.
records: list of {"sf_account_id": "...", "churn_score": 0.82, "churn_tier": "HIGH", "top_risk_factors": "..."}
"""
all_results = []
for i in range(0, len(records), batch_size):
batch = records[i:i + batch_size]
composite_payload = {
"allOrNone": False,
"records": [
{
"attributes": {"type": "Account"},
"Id": r["sf_account_id"],
"Churn_Score__c": r["churn_score"],
"Churn_Tier__c": r["churn_tier"],
"Churn_Risk_Factors__c": r["top_risk_factors"],
"Churn_Score_Updated__c": r["as_of_date"],
}
for r in batch
],
}
resp = requests.patch(
f"{self.base_url}/composite/sobjects",
json=composite_payload,
headers=self.headers,
timeout=30,
)
resp.raise_for_status()
results = resp.json()
errors = [r for r in results if not r.get("success")]
if errors:
print(f"Batch {i // batch_size}: {len(errors)} errors")
all_results.extend(results)
success = sum(1 for r in all_results if r.get("success"))
print(f"Updated {success}/{len(records)} Salesforce records")
return all_results
# Convert SHAP values to a plain-language risk factor string for the CRM field
def format_risk_factors(shap_explanation: list, top_n: int = 3) -> str:
items = sorted(shap_explanation, key=lambda x: abs(x["shap"]), reverse=True)[:top_n]
parts = []
for item in items:
direction = "high" if item["shap"] > 0 else "low"
parts.append(f"{item['feature']} ({direction})")
return "; ".join(parts)
The Churn_Risk_Factors__c field contains a plain-language string that account managers see directly in the CRM: "DAU decline (high); feature adoption (low); days to renewal (low)". No ML knowledge required to act on it.
After six months in production, two systematic failure modes emerged:
Some accounts had predictable low-usage periods tied to their fiscal calendar (e.g., Q4 freeze, summer holiday periods). The model incorrectly flagged these as churn risk. The fix was to add seasonality features — the account's usage relative to its own historical average for the same calendar period — rather than absolute usage levels.
def compute_seasonality_adjusted_dau(usage_df: pd.DataFrame) -> pd.DataFrame:
"""
Compute DAU relative to the account's own seasonal baseline.
"""
usage_df = usage_df.copy()
usage_df["month"] = usage_df["date"].dt.month
# Baseline: average DAU for each account × month combination, using prior years
baseline = (
usage_df[usage_df["date"].dt.year < usage_df["date"].dt.year.max()]
.groupby(["account_id", "month"])["dau"]
.mean()
.rename("baseline_dau")
)
usage_df = usage_df.join(baseline, on=["account_id", "month"])
usage_df["seasonality_adjusted_dau"] = usage_df["dau"] / (usage_df["baseline_dau"] + 1e-6)
return usage_df
New accounts have low feature adoption by definition — they have not had time to explore. The model was flagging a significant fraction of new accounts as high churn risk when the real issue was that the onboarding experience needed improvement. Segmenting the model by account age (0-90 days vs. 90+ days) resolved this.
After 12 months in production:
The most important decisions in this project were not architectural. They were: choosing average precision over AUC as the primary metric (because the positive class is rare), setting the prediction threshold based on the team's actual capacity to act, and integrating directly into Salesforce rather than building a separate tool. The model that gets used beats the model that performs better in a notebook.