CRISP-DM question¶
Which customers should receive a limited retention offer?
The target is whether a customer churns in the next 30 days. Accuracy alone is not enough: the campaign budget limits how many customers can be contacted.
Learning objective: compare a transparent baseline with a small tree model and explain why the decision threshold changes the result.
from pathlib import Path
import sys
ROOT = Path.cwd()
if not (ROOT / "src").exists():
ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import Image, display
from tutorial.data import write_datasets
plt.rcParams.update({
"figure.dpi": 150,
"font.family": "serif",
"font.serif": ["Times New Roman", "STIXGeneral", "DejaVu Serif"],
"mathtext.fontset": "stix",
"axes.grid": True,
"grid.color": "#c8c8c8",
"grid.linewidth": 0.5,
"grid.alpha": 0.7,
"axes.spines.top": False,
"axes.spines.right": False,
"axes.labelsize": 10,
"axes.titlesize": 10.5,
"lines.linewidth": 1.4,
})
paths = write_datasets(ROOT / "data")
print(f"Project root: {ROOT.name}")
df = pd.read_csv(paths["churn"])
df.head()
print(df.info())
print("\nMissing share:")
print(df.isna().mean().sort_values(ascending=False))
print("\nTarget distribution:")
print(df["churned"].value_counts(normalize=True).rename("share"))
Figure A0 | Missing values by field. Source: synthetic tutorial data; one row represents one customer. The plot makes the preparation choices visible before any model is fitted.
display(Image(filename=str(ROOT / "assets" / "img" / "missing-values.png")))
Build the model comparison directly¶
The cells below expose the full learning path: split, preprocessing, fit, predict and evaluate. Keeping preprocessing inside a Pipeline ensures that imputation and scaling are learned from the training data only.
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier
X = df.drop(columns="churned")
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.25,
random_state=42,
stratify=y,
)
numeric = ["tenure_months", "monthly_charge", "support_calls", "usage_hours"]
categorical = ["contract", "autopay", "region"]
preprocess = ColumnTransformer([
("numeric", Pipeline([
("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler()),
]), numeric),
("categorical", Pipeline([
("impute", SimpleImputer(strategy="most_frequent")),
("encode", OneHotEncoder(handle_unknown="ignore")),
]), categorical),
])
The split happens before any transformation. preprocess is then
placed inside each model Pipeline, so its medians, category levels and scaling
statistics are fitted on X_train only. The tree and logistic models therefore
share the same information boundary.
logistic = Pipeline([
("prepare", preprocess),
("model", LogisticRegression(max_iter=1_000, random_state=42)),
])
tree = Pipeline([
("prepare", preprocess),
("model", DecisionTreeClassifier(
max_depth=4,
min_samples_leaf=18,
random_state=42,
)),
])
_ = logistic.fit(X_train, y_train)
_ = tree.fit(X_train, y_train)
from sklearn.metrics import (
accuracy_score, confusion_matrix, f1_score,
precision_score, recall_score, roc_auc_score,
)
threshold = 0.40
logistic_probability = logistic.predict_proba(X_test)[:, 1]
tree_probability = tree.predict_proba(X_test)[:, 1]
logistic_prediction = (logistic_probability >= threshold).astype(int)
tree_prediction = (tree_probability >= threshold).astype(int)
def evaluate_classifier(y_true, prediction, probability):
return {
"accuracy": accuracy_score(y_true, prediction),
"precision": precision_score(y_true, prediction, zero_division=0),
"recall": recall_score(y_true, prediction, zero_division=0),
"f1": f1_score(y_true, prediction, zero_division=0),
"roc_auc": roc_auc_score(y_true, probability),
}
pd.DataFrame({
"logistic regression": evaluate_classifier(y_test, logistic_prediction, logistic_probability),
"decision tree": evaluate_classifier(y_test, tree_prediction, tree_probability),
}).T
At the selected threshold, the model with higher recall finds more churners but may also create more false alerts. F1 balances precision and recall with equal weight; that weighting is a reporting choice, not a universal business objective.
Check split sensitivity with cross-validation¶
A single holdout result can be lucky. The following cells refit the full Pipeline in five stratified folds and report the mean and standard deviation of ROC AUC. This is a stability check, not a replacement for keeping one untouched confirmation set.
from sklearn.model_selection import StratifiedKFold, cross_validate
fold_protocol = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
def cv_auc(model):
score = cross_validate(model, X, y, cv=fold_protocol, scoring="roc_auc")["test_score"]
return pd.Series({"mean_auc": score.mean(), "std_auc": score.std(), "folds": len(score)})
pd.DataFrame({
"logistic regression": cv_auc(logistic),
"decision tree": cv_auc(tree),
}).T
confusion = confusion_matrix(y_test, logistic_prediction, labels=[0, 1])
pd.DataFrame(
confusion,
index=["Actual stay", "Actual churn"],
columns=["Predicted stay", "Predicted churn"],
)
Figure A1 | Confusion matrix at the selected threshold. Source: tutorial calculations on the synthetic churn holdout set.
display(Image(filename=str(ROOT / "assets" / "img" / "churn-confusion.png")))
Figure A2 | Precision, recall, and F1 as the threshold changes. Source: tutorial calculations on the holdout set. The chosen threshold is a decision rule, not a universal constant.
display(Image(filename=str(ROOT / "assets" / "img" / "churn-thresholds.png")))
Figure A3 | Baseline and tree model compared on one holdout split. Source: tutorial calculations. The comparison is valid because both models use the same split, preprocessing protocol, and threshold.
display(Image(filename=str(ROOT / "assets" / "img" / "model-comparison.png")))
Interpretation and exercise¶
- Which model has higher Recall at the selected threshold?
- What happens if you lower the threshold from 0.40 to 0.30?
- Which error is more expensive for the business: a missed churner or an unnecessary retention offer?
The synthetic relationship is only a teaching device. It is not evidence about real customers.