The Brier Score: How We Grade Our Own Homework

Most places that hand you a probability never tell you how they check whether that probability was any good. We do, and we grade ourselves with something called a Brier score. Don't let the name scare you off — nobody's making you do math tonight. Let's talk through what it means instead.

The report card nobody grades themselves on

Picture a hundred different forecasters, all making calls: "70% this works," "40% that works," "90% this other one works." A year later, you go back and check who was actually right, as often as they claimed to be. Most of them will quietly change the subject. We built a scoring system specifically so we can't.

What the Brier score actually measures

Here's the bar-English version. Every time we make a call, we compare our stated confidence to what actually happened, and we measure the gap. Say "green" or "red" whichever it was. If we said 90% and it came up green, that's a tiny gap — we were confident and we were right. If we said 90% and it came up red, that's a big gap — we were loud and we were wrong, and the score should punish that more than a quiet, hedgy call that missed. We do that for every call we've ever made, then average the whole pile.

The result lands somewhere between 0 and 1. Zero means every single call matched reality perfectly — which nobody actually hits, and if someone claims they did, don't believe them. One means the calls were about as wrong as they could possibly be. Lower is better. That's the whole trick: it's not just "were you right," it's "were you right in proportion to how sure you sounded."

Why this matters more than a simple win-rate

A plain win-rate can be gamed. Somebody who only calls things they're 95% sure of, and calls nothing else, can post a flashy win-rate while being useless — they've just refused to say anything interesting. A Brier score doesn't let you hide that way. It rewards being well-calibrated across the whole range of confidence — the 55% calls, the 60% calls, the 90% calls — all of them have to hold up, not just the easy layups.

Where you can actually see ours

This isn't a concept we're describing to sound rigorous. It's live, on the Truth Meter, tracking every call we've made against what actually happened. If our number's climbing toward 1, that's us telling on ourselves, publicly, before you have to find out the hard way. That's the whole point of grading your own homework where the customer can watch.

Head back to the blog, or go look at the live scorecard on the Truth Meter yourself.