Keeping it real

Today I was listening to a Freakonomics podcast about prediction and how monumentally terrible we are at it- even experts. The segment featured some small European country that had passed a law to fine or imprison psychics whose predictions were wrong. Why maintain such strict standards for their predictions when we don’t hold anyone else similarly accountable. In all honesty I’m not sure I wouldn’t be in favor of such a law because such psychics are making the claim that they are delivering a service that they are demonstrably not providing (i.e. fraud), but that’s besides the point.

The point is that it got me thinking about accountability for predictions. They are everywhere, obviously. Pundits, politicians, financial speculators (oh my)… We have fact checking organizations that (to arguable degrees of success) provide a way of holding these people to their claims about the past and present. Obviously we couldn’t rightly hold these people’s predictions to the same standard that we do (or often don’t) when they are reciting facts, but why the hell aren’t we keeping a track record?

What if every time a president, congressman, political pundit, Fed chairman, or otherwise high-profile “expert” made a prediction, we entered it into a database? What if we actually kept statistics on how often there predictions panned out? Right now all the incentives are stacked to encourage wanton prediction-making, because we only keep track of the hits, and not the misses. Why? Because right now we are effectively letting the people who are making the predictions be the ones who keep track of their success by not calling them out.

If we switched the incentives around, its a game changer. First, experts become more careful with their predictions, improving their quality overall (because face it, right now they are almost always less than worthless). Second, we the public get an idea of whose predictions are actually worth listening to. I shudder (and then smile a bit) to think about how many careers would be dashed upon the rocks if the ability to produce this metric came about.

I picture a website - freely available to all - maybe with individual profiles for all the would-be Nostradomus. Maybe a staff of statisticians would have to keep track of all the predictions made, and users could suggest new profiles they would like to see added and monitored. Picture Mint.com, but predictions instead of transactions. For example, you could search Glenn Beck’s predictions by category, by time-frame, or by outcome (yes, no, partial?), and get awesome graphs and pie charts and a big ‘ole “percentage correct” next to your name.