portfolio / projects / lead-income-predictor
case_study.mdMachine Learning · Forecasting
case study · supervised regression

Lead & Income Predictor
Forecasting returns before money is spent.

A lead-buying business decided its monthly spend on each partner and offer largely by gut feel. I built a model that forecasts the leads, income and scrub cost a given spend should produce. I tested seven algorithms, documented three traps that would have made the results look better than they were, and deployed the winner as a live app.

Role
Sole analyst: data → model → app
Data
9,796 rows · 53 months (2022–2026)
Stack
Python, scikit-learn, XGBoost, Streamlit
Outcome
Deployed forecasting app
0.911
Cross-validated R²
Final gradient boosting model, 5-fold CV
±0.017
CV standard deviation
The most stable of the 7 models tested
0.909
Holdout R²
Within 0.002 of CV, so no lucky split
7
Models compared
From linear baseline to stacking ensemble
Try the live app Code & notebook
01_problem.mdwhy this mattered

The business problem

The business buys leads from 77 partners across 40 offers, screens out ("scrubs") the poor-quality ones, and sells the rest to buyers. Every month someone decides how much to spend on each partner-and-offer combination.

Those decisions were based on recent memory, not on a forecast. Nobody could say in advance: "if we spend $X with this partner on this offer next month, what will we get back?"

The question I set

Given a partner, an offer, a planned spend and the month, predict three things at once:

  • Leads: how many leads the spend will buy
  • Sum Income: what the business keeps after costs
  • Sum Scrub Cost: how much will be lost on leads that fail screening
Constraint: only use information that is known before the money is spent. This rule turned out to matter more than any choice of algorithm (see lesson 1).
02_data.pyload → clean → aggregate

Data & preparation

The data is real, anonymised monthly performance from Jan 2022 to May 2026. Partner and offer names are replaced with IDs; the numbers are real.

9,796
Raw rows
77
Partners
40
Offers
53
Months
01
Select
Kept 13 of 25 columns
→
02
Clean numbers
Removed thousands separators ("1,485.33") so values weren't silently turned into missing values
→
03
Aggregate
Rolled up to monthly totals per partner and offer
→
04
Filter
Dropped 3 partners with fewer than 2 rows (too little to learn from)
→
05
Split
80/20 train/test (7,832 / 1,959 rows) plus 5-fold CV

What the exploration showed

  • Volume metrics move together strongly (r = 0.74 to 1.00). Spend and scrub cost correlate almost perfectly (r ≈ 1.00).
  • About 11% of rows are statistical outliers on every volume metric. These are real high-spend months, not errors, so I kept them.
  • Income depends on lead quality, which isn't in the data. I expected it to be the hardest target to predict, and it was.
# features known before spending
features = ["Sum Cost", "Partner", "Offer",
            "Month", "Year"]
targets  = ["Leads", "Sum Income",
            "Sum Scrub Cost"]

# one pipeline, three outputs
model = Pipeline([
    ("prep", ColumnTransformer(...)),
    ("gbm",  MultiOutputRegressor(
        GradientBoostingRegressor()))
])
03_models.ipynb7 models · 5-fold CV + holdout

Comparing seven models

Every model was tuned with grid search and judged on two things: cross-validated R² (how accurate it is) and its spread across folds (how far that accuracy can be trusted). A single holdout score wasn't enough on its own (see lesson 3).

Cross-validated R² by model
Dot = mean of 5 folds · line = ±1 standard deviation
Dot-and-whisker chart of cross-validated R squared for seven models. Gradient Boosting is highest at 0.911 with the smallest spread, 0.017. Stacking 0.907, Random Forest 0.886, Decision Tree 0.876, XGBoost 0.856, Linear Regression 0.833, Lasso 0.829.
view as table
ModelCV R²CV stdHoldout R²Verdict
Why gradient boosting won: Stacking had a slightly better holdout score (0.912 vs 0.909), but a lower CV score (0.907) and a wider spread. Combining models added complexity without adding reliable accuracy, so I deployed the simpler model and kept stacking as a backup.
04_lessons.loggrep -i "warning"

Three traps I caught

Each of these would have produced a better-looking number that the business couldn't actually rely on. All three are kept in the notebook so a reviewer can see how I caught them.

decision_tree.py
1 · data leakage

Using a number you don't have yet

An early tree used Sum Scrub Cost as an input. It took 88% of the model's feature importance. But scrub cost is only known about a month after the leads arrive, so at the moment of the spending decision the model would be relying on a number nobody has.

With leak
88%
importance on one leaked column
Fixed
0.876
honest CV R², still beats linear
lasso.py
2 · over-regularisation

A model that ignored partners

An aggressive Lasso search shrank every coefficient except spend to zero. The model gave identical predictions for every partner and offer, which is useless for this decision. I narrowed the search range and checked that partner and offer coefficients survived.

Collapsed
1
feature left in use
Fixed
α = 1
partner & offer effects kept
xgboost.py
3 · lucky split

The holdout score that lied

One XGBoost run scored 0.917 on the holdout set, the best of any model. Cross-validation on the same setup gave 0.766. The single split had simply been favourable. From then on I compared every model on CV and holdout scores side by side.

Holdout
0.917
one favourable split
5-fold CV
0.766
the real picture: gap 0.151
05_results.pyfinal model · holdout set

How well the final model forecasts

Per-target accuracy of gradient boosting on the 1,959 held-out rows. MAE is the typical size of an error, in the target's own units.

0.979
Scrub cost · R²
Typical error ±$6.40. Scrub cost tracks spend closely.
0.950
Leads · R²
Typical error ±16.9 leads, up from 0.694 with the linear baseline
0.796
Income · R²
Typical error ±$131.78. Hardest, because it depends on lead quality the data doesn't capture.

What the model found

  • Spend is the dominant driver of all three outcomes. How much you put in explains most of what comes out.
  • One partner (Partner_59) stood out: it had a strong, consistent effect on lead volume across several independent models, so it's a real pattern, not a quirk of one model.
  • Income is the weak spot: every model struggled with it, which points to a missing quality signal rather than a modelling problem.

Where not to trust it

  • Partner and offer pairs with very little history.
  • Spend levels far outside anything seen in 2022–2026. Tree models don't extrapolate beyond the range they were trained on.
  • R² and MAE describe average performance, not certainty for any single forecast. The app supports human judgement; it doesn't replace it.
06_deploy.shstreamlit run app.py

From notebook to tool

The trained pipeline is saved together with the lists of known partners and offers, and served through a Streamlit app. A planner picks a partner and offer, enters a budget and month, and gets back expected leads, income and scrub cost. It's a forecast before spending, instead of a review after.

The app only offers partners and offers it was trained on, so it can't be asked to guess for an unknown combination.

Open the app →
Concept illustration: raw leads go through a model and come out as insight
raw leads → model → forecast
07_next.mdTODO

If I took it further

  • Add a lead-quality signal (conversion history by partner) to lift income accuracy above 0.80.
  • Return prediction intervals (quantile gradient boosting) so the app shows a range, not just one number.
  • Switch to a time-based split to test how the model holds up month to month as partners change.