The Fifty Thousand Coin Flips
On the response variable that does not respond, and the industry that keeps asking it questions
The Mechanism, Given Its Due
Insurance price optimisation, as practised, fails in one of two places, and the industry only ever audits one of them. It audits the right-hand side of the regression with inquisitorial zeal: the features, the interactions, the link function, the splines. The left-hand side, the thing actually being predicted, is treated as a given of nature, handed down like the boiling point of water. Yet the choice of response variable is the one decision no amount of downstream cleverness can repair. Fit the wrong Y with sufficient sophistication and you have achieved nothing except precision about the wrong question, which is worse than vagueness about the right one, because it comes with a p-value and a promotion.
I write from the broking side of the market this concerns, with my own margin on the table, so let me first do the traditional method the courtesy of stating its honest case, because it has one. The conversion-elasticity model, the standard engine of the discipline, reasons as follows: optimal margin depends on demand response; demand response must be inferred from behaviour; the purest behavioural record a firm owns is its own quotes and their fates. So one regresses acceptance on premium and risk factors, in the GLM tradition the British profession has owned since Brockman and Wright’s “Statistical Motor Rating” was presented to the Institute in 1992, reads an elasticity off the curve, and optimises. The lineage from that paper to the modern software is unusually direct: EMB, the consultancy its lead author co-founded in 1993, built the GLM pricing tool Emblem, and when Towers Watson acquired EMB in 2011 Emblem passed with it, where it beats to this day at the heart of WTW’s Radar suite. Krikler and co-authors documented the same architecture in 2004 as the method underlying Earnix. And in the habitat where the mechanism evolved, the renewal book, it is elegant and entirely defensible: no competitor stands visibly at the customer’s shoulder, the decision is a private matter between firm and policyholder, and the firm’s own labels are the only evidence in existence. Conversion is the right response variable there because conversion is where the causality terminates. One must concede all of this.
But what happens when we are not on the renewal book? What happens when we are on a price comparison website? Here the model seems to be a marksman of genuine skill who has spent twenty years perfecting his aim at the wrong target.
Derek Does Not Convert. Derek Scrolls.
Regular readers will recall Derek, an honest man with a Vauxhall and a renewal notice. Derek surrenders his registration, his occupation, his annual mileage, his nine years of no-claims history and his opinion of his own garage to a comparison site, waits a few seconds, and receives a ranked board. He clicks at or near the top of it. This is now how the great majority of new private motor business in Britain begins, and for a PCW-only distributor, which is the case before us, Derek’s scroll is the demand side, all of it.
Now watch what the traditional model does with Derek. It defines its response variable as the terminal event, the purchase, and then attempts to trace the influence of price on that event across the entire intervening machinery: the ranking algorithm, the screen, the thumb, the brand halo, the attention span of a man who has promised his wife this would take five minutes. It is measuring the cause at one end of a long corridor and the effect at the other, while declining to look at the corridor. The corridor has a name. It is called rank, it is printed on the screen in seventy-two-point type, and it mediates very nearly everything.
There must be a word for choosing your Y variable this way, and the nearest analogies are all comic. It is examining the household’s grocery receipts to determine whether the dog is fat, when the dog is present, in the room, and available for weighing. The purchase sits downstream of the rank the way the funeral sits downstream of the illness; a medical science built on funeral attendance would eventually discover antibiotics, but not soon.
For here is the second half of the joke: having chosen the response variable furthest from the causal action, the method then discovers that this variable is also the rarest. Follow the funnel of a medium-sized PCW broker writing ten thousand new policies a year (illustrative figures, but the orders of magnitude are what matter). To make those sales, the broker’s prices appear on something like one hundred and eighty million quote boards; of those, roughly half a million land in the top position; the top positions yield perhaps fifty thousand click-throughs; and one in five of those clicks becomes a policy. The traditional method takes this Niagara of one hundred and eighty million observed competitive outcomes and throws away everything except the last trickle: fifty thousand coin flips, of which ten thousand come up heads, split across segments and quarters, in a market that reprices weekly. In the old telephone-and-renewal world the same ten thousand sales would have required only fifty thousand quotes, and the conversion label was, relatively speaking, abundant; the aggregator has diluted it by a factor of several thousand while the profession’s choice of Y has not moved an inch. And since your own rating engine prices by formula, there is no experimental variation in your own price left to exploit; absent deliberate randomised testing, the elasticity is not estimated but assumed, then dressed in a standard error and carried through the pricing committee like a relic. The industry’s own vocabulary confesses as much. One says “elasticity assumption.” Nobody names their measurements after acts of faith.
The old jibe about the drunk searching for his keys under the lamppost, because that is where the light is, does not quite fit here, and the misfit is instructive. The drunk at least searched where he could see. The profession has managed the reverse: the keys lie under the lamppost, floodlit, in the form of one hundred and eighty million ranked boards a year, while the search proceeds on hands and knees in the dark shrubbery of fifty thousand conversion labels, on the grounds that the shrubbery is where keys have traditionally been sought.
The Board Is the Response Variable
So change the Y. If the customer buys from the top of the board, the object governing volume is your position, and your position is a near-deterministic function of one thing: whether your price undercuts the minimum of your competitors’ prices. That minimum is an order statistic of the market’s quote distribution, and estimating its conditional distribution is a different and vastly more tractable problem than estimating Derek’s inner elasticity through a keyhole.
Count what you now observe. Every one of those hundred and eighty million boards is an observation whether or not anyone buys; the firm starved of conversion labels is gorged on rank data. Better, each board contains its own counterfactual: no one need wonder where a price five pounds keener would have landed, because the board says. The identification problem does not ease under the new response variable. It dissolves, the exogenous variation being supplied free of charge by competitors’ weekly repricing.
The purist objects that rank is not purchase, and the purist is right; the objection has also been answered, formally, for decades. Prentice set out the criterion for a valid surrogate endpoint in 1989, and Athey, Chetty, Imbens and Kang generalised the idea as the surrogate index (NBER Working Paper 26463, 2019): where the treatment affects the outcome only through the surrogates, the effect on the surrogate carries the effect on the outcome. Price affects Derek’s purchase almost entirely through position, and the thin residual link from position to purchase, low-dimensional and slowly varying, is exactly what your fifty thousand click-throughs and ten thousand sales are sufficient to calibrate. The exclusion is not perfect: the premium sits on screen beside the rank, brand sways some clicks, funnels leak. Those caveats travel with the claim. But note what has happened to the scarce data: it has not been discarded, it has been demoted to a question small enough for it to answer. You accept a modest, measurable surrogacy bias in exchange for retiring an enormous, flattering, invisible variance, and only one of those two errors ever appears in a board pack.
Two warnings, both load-bearing. The market distribution is a moving target, competitors being optimisers too, so the fitted model is a snapshot with a shelf life of days, which argues for the high-frequency refresh the big sample permits rather than against the approach. And winning is informative: Capen, Clapp and Campbell named the winner’s curse in 1971 after watching oil firms impoverish themselves by succeeding at auctions, and topping an aggregator board is a sealed-bid auction run half a million times a year, won disproportionately where your net rates are mistakenly cheap. The conversion model suffers the identical selection and merely conceals it. In either framing, condition value on the event of winning, explicitly.
The Verdict
The charge, then, is not quackery but misdirected marksmanship: a legitimate instrument, honourably descended, whose one unexaminable choice, the choice of what to predict, was fixed in a market of renewal letters and telephone negotiations and never revisited when a ranked screen was inserted between firm and customer. For the pricing director, the residue fits on a beer mat. Make the competitor price distribution your primary target and refresh it weekly. Calibrate a thin position-to-purchase link from your own conversions. Condition lifetime value on winning, so the curse is priced rather than discovered. And if an elasticity assumption must survive somewhere, stop calling it an estimate.
A closing image, offered with due respect to the instrument. The astrolabe was one of the supreme achievements of applied mathematics: centuries of refinement, exquisite brass, and it genuinely told you where you were, provided you stood on a deck under an open sky. Bolt it to the bridge of a nuclear submarine and the mathematics remains impeccable; the brass still gleams; the readings are precise to the arc-minute. The difficulty is the several hundred feet of ocean between the instrument and every star it was built to observe. The officer polishing it is not a fool, merely a traditionalist — and the vessel, one notes, comes fitted with a periscope. Surface, man, and look at the board!
For those interested in a more mathematical argument, I have attempted the derivation in the technical appendix (PDF), it not being possible to set the formulae down here.