Start with the question.
Not the model.

research_brief = {
    "decision": "Which customers should receive a retention offer?",
    "target": "churned within the next 30 days",
    "decision_time": "month-end, before the campaign starts",
    "baseline": "contact everyone, or target only by tenure",
    "metric": "recall at an acceptable contact volume",
    "constraint": "retention budget covers 20% of customers",
    "failure_cost": "missed churn vs unnecessary discount",
}

Why a strong prediction can still be a weak decision

Consider a churn model with excellent ranking performance. If the retention campaign only runs for one week and can contact 20 customers, the useful output is not a general probability score; it is a ranked list that fits that capacity. The decision also needs a stop rule: if the predicted benefit of the offer is lower than its cost, contacting the next customer destroys value even when the model is right about churn risk.

This distinction matters because a churn predictor estimates risk, not the causal effect of a retention action. A high-risk customer may churn even after receiving a discount, while a medium-risk customer may be highly responsive. The simplified lab ranks churn risk because it is easier to teach and audit. A real campaign evaluation would compare treatment and control groups and estimate incremental uplift.

A good objective also separates the model metric from the decision metric. ROC AUC measures ranking across all thresholds. Recall measures how many relevant cases are found at one chosen threshold. Campaign profit combines expected retained revenue, offer cost, and contact capacity. All three can be useful, but they answer different questions and should not be treated as interchangeable leaderboard scores.

import pandas as pd

df = pd.read_csv("data/customer_churn.csv")

print(df.shape)
print(df.dtypes)
print(df.isna().mean().sort_values(ascending=False))
print(df["churned"].value_counts(normalize=True))
df.describe(include="all").T

Three clocks must agree

Every predictive dataset contains at least three timelines: when an event happened, when its value became available to the organisation, and when the model must make a decision. These are not the same clock. An invoice may be dated at month-end but loaded into the warehouse three days later. A reported price may describe a completed transaction long after the property was listed. A churn label may only become known after the prediction would have been used. If the pipeline ignores availability dates, it can create a model that performs well only because it sees the future.

A data dictionary should therefore include meaning, unit, data type, source, update frequency, valid range, and availability time. “Monthly charge” is incomplete documentation; “current recurring charge in billing currency at month-end close” gives a reviewer enough information to challenge its timing. For each model input, ask whether the exact value would have been available before the prediction cutoff.

Profile before cleaning

Missing values, duplicate records and outliers are not automatically errors. A missing support-call count may mean no calls occurred, data was not captured, or the customer had no active service. An extreme value may be a legitimate VIP customer or a unit-conversion mistake. Profiling comes first because the correct preparation decision depends on the mechanism that created the value. Cleaning immediately can erase the evidence needed to understand the problem.

Sample coverage matters as much as column quality. The synthetic churn data contains four regions and a single historical snapshot; it cannot show whether relationships change through time or whether one region is systematically under-recorded. A table of clean columns can still represent the wrong people or the wrong period. That limitation belongs in the data profile and in the final memo, not in a hidden code comment.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

preprocess = ColumnTransformer([
    ("numeric", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric_columns),
    ("categorical", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("encode", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical_columns),
])

Leakage is about information, not just code order

Leakage occurs whenever training information would not be available when the prediction is made. Target leakage is the most direct form: a cancellation reason, refund note, or post-decision status predicts the target because the field is created by the target event. Preprocessing leakage is subtler: a scaler learns the mean and variance from the test set, a median imputer borrows values from the test set, or a feature-selection step chooses columns using all outcomes. The individual operation may be mathematically simple, but the reported validation score no longer represents a genuine future decision.

A Pipeline helps because it attaches transformations to a model and forces each transformation to be fitted inside every training fold. It does not block direct leakage from target-derived columns. The engineer still has to document field timing and decide whether a column could exist at prediction time. Reproducibility and validity are therefore separate requirements: a leaky workflow can run perfectly on every machine and still produce the wrong result.

Categorical encoding, missing-value imputation and outlier handling must use training-only statistics. If category levels are rare, enable `handle_unknown="ignore"` so unseen levels in test data do not crash the pipeline. If a target encoding is introduced later, it needs an internal cross-fitting scheme because the encoding otherwise carries information about each row's outcome.

Validation must mimic the deployment environment

Random splitting is reasonable for exchangeable customer records when the goal is to generalise to similar customers. Stratification preserves the target balance. A time series needs a chronological split because a future month can never train a model that is supposed to forecast the past. When the target is measured over a horizon, add a gap between training and validation if future label information overlaps the training period.

Preserve enough evidence to rebuild the prepared dataset: source version, generation seed, row counts, schema, split dates and software versions. Store transformation parameters inside the fitted pipeline rather than in ad hoc variables. This is what makes the test set reusable once rather than silently tuned across many experiments.

The churn threshold is set to 0.40. A higher threshold makes the model more conservative; a lower threshold catches more potential churners but creates more false alerts.

model = Pipeline([
    ("prepare", preprocess),
    ("model", LogisticRegression(max_iter=1_000, random_state=42)),
])

model.fit(X_train, y_train)
probability = model.predict_proba(X_test)[:, 1]
prediction = (probability >= 0.40).astype(int)

Use a model ladder, not a model zoo

Start with the weakest defensible rule, then add one capability at a time. For churn, a reasonable ladder is: rank by a simple business rule, fit logistic regression, then test a shallow tree for nonlinearities. For house prices, compare a mean prediction, linear regression, and a limited random forest. For demand, compare last month's value, the same month last year, and a simple trend-plus-seasonality model. Each rung adds one explanation for why performance might improve.

This structure protects the analysis from two failures. The first is unearned complexity: a large model wins by a tiny amount on one split, but nobody can explain why or maintain it. The second is false simplicity: a linear model may be inadequate when interactions or thresholds are central, but that should be demonstrated rather than assumed. The comparison should be narrow and interpretable enough that a reviewer can ask what changed.

Hyperparameters belong to a model family, while split rules, target definition and evaluation metrics belong to the research protocol. Changing `max_depth` is a tuning decision; changing the prediction horizon is a new research question. Keep those changes separate. Otherwise a model-selection history becomes a collection of incomparable experiments.

In the house-price case, the random forest does not beat the linear baseline. That result is intentionally retained. The synthetic price process is mostly additive, the sample is small, and the default tree uses a different inductive bias. A portfolio that shows only successful model upgrades is less credible than one that records a well-designed comparison and accepts the answer.

A metric is an argument about what matters

Accuracy asks how often the model is correct. Precision asks what share of positive predictions are correct. Recall asks what share of actual positives were found. F1 balances those two. Each answers a different question, and none contains the cost of a campaign, the value of a retained customer, or the capacity of the operations team. A complete evaluation translates the model metric into those decision consequences.

For regression, MAE is usually easier to explain because it remains in the target's units. RMSE gives larger errors more weight, so it is useful when a catastrophic miss is much worse than several small misses. For demand, MAPE can become unstable near zero, so an MAE table and absolute forecast errors should accompany it. For classification, ROC AUC can summarise ranking, but it does not choose the operating threshold. Report the threshold explicitly.

A single holdout split is itself an estimate. It can be lucky or unlucky, especially with a small sample. For exchangeable data, compare fold-to-fold variation or repeat the split with several seeds. For time series, use rolling or expanding origin evaluations. Do not repeatedly adjust the final holdout; that converts a confirmation set back into a training signal.

Inspect where errors occur before accepting an aggregate score. A model can look stable overall while failing on new customers, high-price properties, a particular month, or the exact cases that trigger an expensive action. Report the known gap between the teaching experiment and a real deployment, including sample size, synthetic data, and the absence of a separate tuning period.

python -m pytest -q
python scripts/build_examples.py
python scripts/build_case_pages.py
python -m http.server 8000

# Open: http://localhost:8000

Define what “deployed” means for this project

Monitoring is not a dashboard of decorative lines

A monitoring plan starts with the decision the model supports. Track the population and features used at scoring time, prediction volume, missing rates, score distribution and realised outcomes when they become observable. Compare those values with a documented reference period. An alert should trigger a specific action—investigate, recalculate, retrain, pause or retire—and have an owner. A chart without a trigger is only latent evidence.

A concise model card should state intended use, prohibited use, target, training period, important features, evaluation data, headline metrics, limitations and maintenance owner. It should also identify dependencies: upstream data contracts, libraries and assumptions that could break. This prevents a notebook from becoming an undocumented institutional memory.

Communication follows the same evidence chain. Start with the decision and result, then show the baseline, the uncertainty, and the conditions under which the recommendation changes. Put technical detail in an appendix. A useful memo does not narrate every cell; it helps a decision-maker distinguish evidence, assumption and recommendation.