Our player projections tell you what to expect from a starter tonight. The predictability board answers a different question: how much should you trust that projection for this particular player? Some players produce on rails. Others swing wildly from night to night, and no model, ours included, can call their box score with confidence. For a fantasy start/sit decision, knowing which kind of player you are looking at matters as much as the projection itself.
The opponent: a naive forecaster
Every score needs a reference point. Ours is deliberately humble: a naive forecaster that projects the average of a player's season numbers and his last five games. No opponent adjustment, no lineup context, no pace. Just a rolling average anyone could compute in a spreadsheet.
Do not underestimate it. Elite starters are remarkably consistent, and a rolling average is genuinely hard to beat on stats like assists, where a point guard's role barely changes from week to week. Grading ourselves against this baseline, player by player, keeps us honest: a model that only beats a coin flip is not worth your attention.
Why not just average error?
The obvious metric is mean absolute error: how far off was the projection, on average? We publish it for every stat, model next to naive, on each player page. But MAE alone has a blind spot with counting stats. Take a defender who attempts one three per game. Predicting "about zero threes" every night produces a tiny MAE and looks brilliant, while saying nothing useful. On low-volume stats, MAE rewards the boring answer.
Our projections are not single numbers. Before tip-off the model publishes a full probability distribution for each stat: how likely 10 points is, how likely 20, how likely 35. That is the real product, and it needs a metric that grades the whole curve.
CRPS: grading the whole distribution
CRPS, the continuous ranked probability score, is the standard tool in forecasting science for exactly this job, from weather models to epidemiology. Intuitively, it asks: how much probability did you place close to what actually happened? A forecast that is both sharp (narrow) and right scores best. A forecast that hedges too wide gets penalized, and so does a confident forecast that misses. Lower is better, like a golf score.
For each player we compute the average CRPS of our distribution over every game he started, and the average CRPS of a distribution built around the naive forecast on the same games. The predictability score is the improvement, in percent, averaged over points and rebounds. A score of 10 means our distributions were about 10 percent better than the rolling average at calling that player's nights. Around zero means his production is hard to call for anyone: that is a real, useful fact about a player, not a failure of the chart.
What counts, and what does not
Three rules keep the number honest. First, only games a player started count. Our projections assume the starting five, so a late scratch is excluded rather than booked as a miss. Second, the score is built on points and rebounds only. Assists and threes are shown on every page with the same model-vs-naive comparison, but per player the margins there are thin enough that we prefer to display them transparently rather than fold them into the headline number. Third, a season is always evaluated out-of-sample: the current board reflects a full-season holdout backtest in which the models never saw the season they were predicting, and once a live season starts, predictions are reconciled against the box score the morning after each game, exactly as published.
How to use it
Start from tonight's projections, then check the player's page before you commit. A high predictability score with a tight probability band is a floor you can bank on. A score near zero with a wide band is a lottery ticket, whatever the point estimate says. The same data is available programmatically through our REST API and MCP server, so an AI assistant can run this exact check for your whole roster.
The methodology is league-agnostic, and EuroLeague player pages built on the same score are coming with the new season. Questions or ideas? We read everything: get in touch.