Case study
Calling the students most likely to leave is worse than calling at random
Every retention programme I have seen ranks customers by how likely they are to churn and works down the list. On this data that policy performs worse than picking names out of a hat, and the reason is a distinction most dashboards cannot express.
The data on this page is artificial, for privacy. The method is the one I would run against real records. Code: UPLIFT.md.
Four kinds of customer
A retention call does something different to each of them, and only one is worth the phone call.
the entire return
wasted call
worse than wasted
wasted call
A churn model scores persuadables, lost causes and sleeping dogs identically high, because all three are at risk. Only the top-left cell pays, and the bottom-left actively loses money: a call opening with "we noticed you've been thinking about stopping" can put the idea on the table for someone who wasn't holding it.
So the quantity you want is not the probability of churn. It is the difference the call makes, and that is a causal contrast rather than a prediction. No student is ever observed both called and not called, so it has no label to fit against. It needs a randomised trial.
Reading a Qini curve
The Qini curve answers one question: if I call the top k percent by this score, how many extra students do I keep, over and above what the untreated control group did on its own? The diagonal is what random targeting achieves. Anything below the diagonal is worse than not thinking at all.
Incremental students retained, by how deep you call
Randomised trial, 6,000 students, half called. Hover to read any depth.
Table view
The orange line is what almost everyone actually does. Its Qini coefficient is −17.5 against +33.2 for the uplift model. It sits below the diagonal for most of its length, which means a centre running that policy would do better firing the model and calling people alphabetically.
When is any of this worth doing?
This is the part usually left out, and it is the part an owner asks first. A phone call costs about eighteen dollars of staff time. A saved student is worth roughly $1,276 of contribution margin. At a seventy-to-one payoff the wasted calls are nearly free, so the crude policy of calling everybody is genuinely hard to beat.
Drag the cost up and watch what happens. The moment the intervention stops being a call and becomes an offer, a free month, or an hour of the director's time, blanket outreach goes underwater and targeting is worth more than the offer itself.
Net margin by policy, as the intervention gets expensive
Measured on the trial's held-out half. The dashed line is calling everyone.
What this changes
Three things, in the order an owner cares about them.
Stop ranking retention outreach by churn probability. On this data it is anticorrelated with what the intervention actually achieves, because the students most likely to leave are the ones already past reach. That is one line of code changed and it is the largest single improvement on the page.
For a cheap intervention, skip the modelling entirely and call everyone, then spend the saved effort on making the call better. Targeting earns its keep once the intervention is expensive, and the slider above tells you where your own crossover sits once you put your real numbers in.
And none of this exists without a holdout. Randomisation is what makes the effect measurable at all; without it the students a director chooses to call are exactly the students whose outcomes differ for other reasons. Withholding the call from a random third of eligible students for one term converts an entire retention budget from an article of faith into a measured quantity, and it costs a third of a term's calls.