Our player projections say what to expect from a starter tonight. The predictability board answers a different question: how much should you trust that projection for this player? Some players produce almost the same line every night. Others swing so much that no model, ours included, can call their box score with confidence. For a start/sit decision that difference matters as much as the projection.
The benchmark: a simple average
A score needs something to beat. Ours is deliberately simple: the average of a player's season-to-date numbers and his last five games. No opponent, no lineup, no pace. Anyone could compute it in a spreadsheet.
It is harder to beat than it sounds. Many starters are very consistent, and on a stat like assists, where a point guard's role barely changes, a simple average is a strong forecast. Grading ourselves against it, player by player, shows where the model adds something and where it does not.
Why not just the average error?
The obvious measure is mean absolute error (MAE): how far off the projection was, on average. We show it for every stat on each player page, model next to benchmark. But MAE has a blind spot with small counts. For a defender who takes one three a game, predicting "about zero threes" every night gives a tiny MAE while saying nothing useful.
Our projections are also not single numbers. Before tip-off the model publishes a probability distribution for each stat: how likely 10 points is, how likely 20, how likely 35. Grading that needs a measure that looks at the whole curve.
CRPS: grading the whole distribution
CRPS, the continuous ranked probability score, is the usual tool for this in forecasting, from weather to epidemiology. It asks how much probability the forecast put close to what actually happened. A forecast that is both narrow and right scores best. One that spreads too wide is penalised, and so is a confident one that misses. Lower is better.
For each player we average the CRPS of our distributions over every game he started, and do the same for a distribution built around the simple average on the same games. The predictability score is how much lower ours is, in percent, averaged over points and rebounds. A score of 10 means our distributions were about 10% better than the simple average for that player. A score near zero means nobody can call his nights well, which is itself worth knowing.
What counts and what does not
Three rules. First, only games the player actually started count. The projections assume the starting five, so a late scratch is left out rather than counted as a miss. Second, the score uses points and rebounds only. Assists and threes appear on every page with the same comparison, but per player the margins are too thin to put in the headline number. Third, the numbers are always out of sample: the current board comes from a full-season backtest in which the model never saw the season it was predicting, and during a live season each prediction is checked against the box score the morning after, exactly as it was published.
How to use it
Start from tonight's projections, then open the player's page. A high score with a narrow range is a floor you can rely on. A score near zero with a wide range is closer to a lottery ticket, whatever the average says. The same data is available through our REST API and MCP server, so an AI assistant can run this check for a whole roster.
Euroleague players have the same kind of score, built on points, rebounds and PIR. Questions or ideas? Get in touch.