Loading...
 

Measure Intangibles

How Good are your Estimates

graham

Forecasting is fundamental

Forecasting (outcomes, benefits, risks) is fundamental to investing wisely.

We often focus on tight estimates of the cost of doing something, but pay less attention to the larger factors, the benefits and risks. Larger? Well, the benefit better be much larger than the cost and the risks, otherwise why take the risks?

But costs are more predictable, and it's easier to hold someone to account for costs. So we tend to obsess on what we think we can manage and pay less attention to the things we think are harder to estimate.

Some people are great estimators

A few group of people get very good at estimating (or get out). Those are people who make lots of estimates (maybe called predictions, forecasts or bets) and get very rapid feedback on whether they predicted correctly or not. They include professional weather forecasters, poker players, bookmakers and a few others.
738541main Biocular Twodisks
Their estimates include a probability of being correct, sometimes explicit ("a 20% chance of rain", or by offering odds on a horse about to race), sometimes implicit (whether they will place a bet or not).

Most of us just don't get the practice and particularly not the rapid feedback. We might regularly offer forecasts, but rarely remember our own forecasts (especially when they turn out wrong) or anyone else's. They are treated as they should be- as throw-aways, not worth the paper they are written on.

How to measure estimate quality

So how do we find out how good we are at estimating (or forecasting, if you prefer)? We could take up weather forecasting, but that would probably turn us off quickly because we would be so rarely correct. Or we could take up gambling, which would probably be much worse.
Roulette Table Us
Some 'virtual gambling' might work, like building a virtual portfolio of shares. Even that entails a lot of effort only loosely contributing to the task of measuring how good we are at estimating.

Turns out, the weather forecasting fraternity have a tool which can help, called the Brier score. They have used it for years to drive improvements.

The Brier Score of Estimate Quality

The Brier score for a single prediction is (P(T) - T)²
P(T) is the Probability you gave of being right (on a scale 0..1), and T is whether you were right (1 for right or 0 for wrong).

So the Brier score ends up as a number between 0 and 1.

A Brier score nearer to 0 represents you expressed high confidence in an forecast range that turns out to include the eventual value- you were confident and correct; and conversely 1 represents high confidence in an estimate range that does not include the actual value- you were confident but wrong.

The Brier score judges how realistic you were in expressing your confidence by the width of your forecast range: so you score better by a narrower range expressing higher confidence (if you turn out to be right) but worse by expressing high confidence (if you turn out to be wrong).

Your Brier score for one prediction or estimate will be different from the next one, because it depends both on whether you were right and how confident you were that you were right. You can get a representative personal Brier score by averaging the Brier scores you got over several predictions.

Most of us still don't do enough forecasts to build up a personal Brier score quickly, but there are some ways we can get an indication: using systematic 'general knowledge' or 'virtual gambling' to clock up a bunch of individual scores fast.

There's a saying that you can't improve what you don't measure, so does being able to measure our ability to estimate let us improve it? Turns out, yes it does. With relatively brief and simple training you can make significant improvements. This site has lots of information on that from Can you Learn to Estimate? to Estimating for Architects

Through just a few minutes of exercises in a workshop at the Project Management Institute conference recently, the average Brier score of about 50 project managers in the room improved from 0.48 to 0.37. That's something http://AcuteIP.com can facilitate for your organization.

Brier Score is just one part of a wider methodology that can improve your organization's ability to invest in the right things and get the best returns. Take a look at this printable PDF infographic depicting AcuteIP.com's approach to using the Wisdom of your Crowd

With preparation, you can do this yourself. AcuteIP.com offers a set of services to make it easy, from workshops and technical services to managing the whole exercise.
Contact Graham.Harris at AcuteIP.com to explore possibilities.

Could this be useful to someone you know? Spread the word ...

  • at LinkedIn, Like lt, and Share it with your contacts
  • at http://AcuteIP.com, Share it through the colorful social media buttons (at left on PC, or below on mobile).

Sharing posts adds to both our reputation.

© 2018 Graham.Harris at AcuteIP.com