CRISP-DM question¶

Can a model estimate residential prices closely enough to support a first-pass budget screen?

Learning objective: establish a linear baseline, compare it with a random forest, and explain why a more complex model does not automatically improve out-of-sample error.

In [1]:

from pathlib import Path
import sys

ROOT = Path.cwd()
if not (ROOT / "src").exists():
    ROOT = ROOT.parent
sys.path.insert(0, str(ROOT / "src"))

import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import Image, display

from tutorial.data import write_datasets

plt.rcParams.update({
    "figure.dpi": 150,
    "font.family": "serif",
    "font.serif": ["Times New Roman", "STIXGeneral", "DejaVu Serif"],
    "mathtext.fontset": "stix",
    "axes.grid": True,
    "grid.color": "#c8c8c8",
    "grid.linewidth": 0.5,
    "grid.alpha": 0.7,
    "axes.spines.top": False,
    "axes.spines.right": False,
    "axes.labelsize": 10,
    "axes.titlesize": 10.5,
    "lines.linewidth": 1.4,
})

paths = write_datasets(ROOT / "data")
print(f"Project root: {ROOT.name}")

Project root: data-science-crisp-dm-tutorial

In [2]:

df = pd.read_csv(paths["housing"])
df.head()

Out[2]:

	living_area_sqm	rooms	building_age	distance_to_centre_km	condition	neighborhood	price_hkd_m
0	114.1	4	20	3.89	Good	New town	6.132
1	73.8	3	31	3.51	Excellent	New town	4.075
2	127.5	4	9	12.71	Good	Harbour	8.949
3	133.2	5	16	5.84	Good	Central	10.902
4	46.5	2	20	0.56	Fair	Harbour	3.898

In [3]:

df.describe().T

Out[3]:

	count	mean	std	min	25%	50%	75%	max
living_area_sqm	900.0	103.920444	29.084014	45.000	83.40000	104.2000	122.52500	200.400
rooms	900.0	3.151111	0.865511	1.000	3.00000	3.0000	4.00000	6.000
building_age	900.0	24.378889	14.681180	0.000	12.00000	24.0000	37.00000	50.000
distance_to_centre_km	900.0	7.374767	4.678613	0.400	3.89000	6.4800	9.74500	28.000
price_hkd_m	900.0	6.560326	2.195964	1.102	4.96475	6.5755	8.08275	14.415

In [4]:

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
import numpy as np

X = df.drop(columns="price_hkd_m")
y = df["price_hkd_m"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

numeric = ["living_area_sqm", "rooms", "building_age", "distance_to_centre_km"]
categorical = ["condition", "neighborhood"]
preprocess = ColumnTransformer([
    ("numeric", StandardScaler(), numeric),
    ("categorical", OneHotEncoder(handle_unknown="ignore"), categorical),
])

The linear model is the interpretable baseline. The forest receives the identical transformed features and split; only the estimator changes.

In [5]:

linear = Pipeline([("prepare", preprocess), ("model", LinearRegression())])
forest = Pipeline([
    ("prepare", preprocess),
    ("model", RandomForestRegressor(
        n_estimators=250,
        min_samples_leaf=3,
        random_state=42,
        n_jobs=1,
    )),
])

_ = linear.fit(X_train, y_train)
_ = forest.fit(X_train, y_train)

In [6]:

def evaluate_regressor(model):
    prediction = model.predict(X_test)
    return {
        "MAE": mean_absolute_error(y_test, prediction),
        "RMSE": mean_squared_error(y_test, prediction) ** 0.5,
        "R2": r2_score(y_test, prediction),
    }

pd.DataFrame({
    "linear regression": evaluate_regressor(linear),
    "random forest": evaluate_regressor(forest),
}).T

Out[6]:

	MAE	RMSE	R2
linear regression	0.308865	0.392394	0.969046
random forest	0.470539	0.622300	0.922149

Both models see the same holdout rows. RMSE is retained next to MAE because it penalises large pricing errors more heavily. The random forest is a useful comparison, but its extra flexibility is not evidence of extra value.

Check fold-to-fold stability¶

Five-fold cross-validation on the training-plus-holdout rows shows how much the result depends on one particular holdout. Each fold refits its own preprocessing pipeline.

In [7]:

from sklearn.model_selection import KFold, cross_validate

folds = KFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
    "mae": "neg_mean_absolute_error",
    "rmse": "neg_root_mean_squared_error",
}

def cv_regression(model):
    scores = cross_validate(model, X, y, cv=folds, scoring=scoring)
    return {
        "MAE mean": -scores["test_mae"].mean(),
        "MAE sd": scores["test_mae"].std(),
        "RMSE mean": -scores["test_rmse"].mean(),
        "RMSE sd": scores["test_rmse"].std(),
    }

pd.DataFrame({
    "linear regression": cv_regression(linear),
    "random forest": cv_regression(forest),
}).T

Out[7]:

	MAE mean	MAE sd	RMSE mean	RMSE sd
linear regression	0.320344	0.011014	0.401627	0.009181
random forest	0.491742	0.022590	0.634282	0.020862

Figure B2 | Actual versus predicted prices. Source: tutorial calculations on the 20% holdout set. Points close to the reference line indicate lower prediction error.

In [8]:

display(Image(filename=str(ROOT / "assets" / "img" / "housing-predictions.png")))

Scatter plot comparing actual and predicted house prices with a reference line.

Interpretation and exercise¶

The linear baseline has lower MAE and RMSE in this controlled synthetic case. That is a useful result, not a failed demonstration:

the data-generating process is close to additive;
the sample is small;
default tree settings are not guaranteed to be optimal;
model complexity must earn its place on the holdout set.

Change one forest hyperparameter and ask whether the improvement is stable or only a single-split accident.

House Prices

CRISP-DM question¶

Check fold-to-fold stability¶

Interpretation and exercise¶