You can do this at the prediction aggregating website Metaculus. I used to be a top 20 forecaster for some years and found their tools quite helpful. You get your own calibration curve broken up into 21 segments of grey ranges indicating 90% credible/confidence interval of a perfect calibration (for those wondering, I have 13). This is great, but it used to actually be even better as they would give you the raw numbers instead of having to estimate it visually. There are other platforms, but they aren't as widely used.
One problem I see though, with wishes like these... :
> My wet dream would be a public database cataloging predictions made by political and financial pundits, complete with their stated confidence levels and historical track records.
...is that it's quite easy to game. If calibration became a prominent metric people would just make a lot of predictions on "easy" predictions (e.g. will the sun rise tomorrow?) and cash that in for their "hard" predictions (this meme-stock I own will rise in value, trust me I have a calibration of 99%). So we need a second metric of how hard questions are. But to calibrate that we would have to incentivize a lot of people to spend a lot of time answering a lot of questions. Some people use that as an argument for prediction markets, but given that we are terrible at solving problems with both our information landscape, *and* problems with market failures, I shudder to think what a prediction market (which is rightly also called an information market) will do to the world.
I’m not anymore—I’m not even in the top 100 anymore. To be fair, there are now a lot more people than there used to be (I can’t find an exact figure on how many accounts there are, but I’m guessing fewer than 50,000) so maybe I haven’t decreased that much percentile-wise (although I've definitely gotten worse, which might be a coincidence, or due to my depression, or medicine, or a host of other factors that might be an interesting research area).
Whether or not it’s possible to improve your own calibration, the notion helps cement why we should be wary of people who habitually express certainty and favor hyperbolic language.
You can do this at the prediction aggregating website Metaculus. I used to be a top 20 forecaster for some years and found their tools quite helpful. You get your own calibration curve broken up into 21 segments of grey ranges indicating 90% credible/confidence interval of a perfect calibration (for those wondering, I have 13). This is great, but it used to actually be even better as they would give you the raw numbers instead of having to estimate it visually. There are other platforms, but they aren't as widely used.
One problem I see though, with wishes like these... :
> My wet dream would be a public database cataloging predictions made by political and financial pundits, complete with their stated confidence levels and historical track records.
...is that it's quite easy to game. If calibration became a prominent metric people would just make a lot of predictions on "easy" predictions (e.g. will the sun rise tomorrow?) and cash that in for their "hard" predictions (this meme-stock I own will rise in value, trust me I have a calibration of 99%). So we need a second metric of how hard questions are. But to calibrate that we would have to incentivize a lot of people to spend a lot of time answering a lot of questions. Some people use that as an argument for prediction markets, but given that we are terrible at solving problems with both our information landscape, *and* problems with market failures, I shudder to think what a prediction market (which is rightly also called an information market) will do to the world.
Super congrats on being a top 20 forecaster! That’s amazing!
I’m not anymore—I’m not even in the top 100 anymore. To be fair, there are now a lot more people than there used to be (I can’t find an exact figure on how many accounts there are, but I’m guessing fewer than 50,000) so maybe I haven’t decreased that much percentile-wise (although I've definitely gotten worse, which might be a coincidence, or due to my depression, or medicine, or a host of other factors that might be an interesting research area).
This is such a handy, easy to understand concept.
Whether or not it’s possible to improve your own calibration, the notion helps cement why we should be wary of people who habitually express certainty and favor hyperbolic language.
i would pull up with this to the board game function