Reducing Optimism Bias in Three Point Estimates

Reducing Optimism Bias in Three Point Estimates

Promax uses the PERT method to help with uncertainty in scoring. It uses a simple formula to give an approximation of uncertainty based on best and worst estimates.


The formula is based upon giving three estimates. The first is the best estimate “H”. The second (middle) estimate is the most likely score “M” and the third estimate is the lowest score expected “L”.

The likely score will be: S = (H + 4M + L) / 6

For example, lets assume you’re in a workshop with a number of experts discussing a score for a particular option. The general consensus is that it’s worth a 7. However, it’s possible that it could be an 8. However, some also think it’s not as good as the rest believe and think it should be a 4. So the three points of the estimate are:

M = 7
H = 8
L= 4

Then the score to be used will be (8 + 4x7 + 4) / 6 = 6.7

The standard deviation calculation is: SD = (H – L) / 6

So for our example, SD = (8-4) / 6 = 0.7

What this means is that there is a 68% chance that the score will lie between 7.4 and 6.0.

In Promax, the PERT value and the standard deviations of the individual criteria (using the mapped and weighted values) are calculated.



Criteria, Most Likely, High, Low, PERT, SD
#1, 50, 70, 40, 51.7, 5.0
#2,80, 90, 75, 80.8, 2.5
#3, 90, 100, 80, 90.0, 3.3
#4, 20, 50, 10, 23.3, 6.7
#5, 70, 80, 50, 68.3, 5.0

The PERT score is (314.1 / 5) = 62.8 and the SD is (22.5 / 5) = 4.5

Therefore, there is a 68% chance that the “score” will lie between 67.3 and 58.3. There is a 95% chance it will lie between 71.9 and 53.8.

The boxes on the Promax graphs reflect this range of 95% confidence. Therefore, rather than choosing the single point estimate to assess an option, use the range instead. If there is a major overlap you cannot be certain that the best option from a single point perspective is in fact the best one if uncertainty is taken into account.

If there is major overlap there are two generic strategies. The first is to try and reduce the uncertainty range by collecting better, more accurate data. The second is to choose the option with the smallest range of the overlapping ones (the risk averse strategy) wince there is more chance of meeting the overall score.

There are some things worth considering in the PERT approach.

1.     68.27% of all measurements fall within one standard deviation of the mean. 95.45% of all measurements fall within two standard deviations of the mean. 99.73% of all measurements fall within three standard deviations of the mean.
2.     Strictly, the probabilities above refer to repeating the scoring enough times to collect many values. Had we repeated the scoring 100 times we would get 100 values. In this example we are saying that on 68% of the occasion that we carry out the scoring the result will lie between 85.3 and 40.3 and outside of this range on the other 32% of occasions. In effect when someone estimates a score they are doing this on the basis of experience by mentally simulating what is likely to happen if they repeated it many times.

Background to PERT

PERT (program evaluation and review technique) was developed by NASA in 1962. As well as developing network / precedence diagraming to aid project planning they also developed methods to estimate project duration. Their formula (H + 4M + L) / 6 is essentially a beta distribution where the tail is longer on the pessimistic side. Fifty years later, this distribution is still very much applicable.

Techniques for Eliciting Scores

The scoring of options is a fascinating subject in its own right. It should be remembered that scores reflect the performance of an option on a particular criteria. In Promax, scores are then mapped to a value scale (0-100) in order to be able to add different criteria together (essentially to achieve the same units). Depending on how the criteria have been selected and then how the criterion is to be measured, the scores can either be real-world units or based on judgement. Real-world units are to be preferred in all cases but it may not be feasible particularly at the early stages of an evaluation.

Where expert judgement is to be used the best method to get an expert to score the option on the criteria for which they are expert on. This is often termed “Expert Elicitation (EE)”. This will provide an evidence-base which will justify their score and can be used subsequently to challenge and review.

The least successful method is to use a “decision conference” where a group of experts come together and collectively score using consensus. The problem with this technique is that the rationale for the scores is generally weak and cannot be adequately underpinned. The consequences are that in any future review or challenge the scores provided are extremely difficult to recreate with a different group of people.  

Whenever using expert judgment eliciting good three point estimates is difficult.  There are a number of biases that are introduced including:

• Availability –judgments are based on information most easily remembered
• Representativeness – judgments are based on similar yet limited data and experience. The expert does not fully consider other relevant, accessible and/or newer evidence
• Anchoring – Fixing on a particular value in the range and making insufficient adjustments away from it in constructing an uncertainty estimate
• Overconfidence (Optimism bias) – Strong tendency to be more certain about one’s judgments and conclusions than one has reason.
• Control – expert believes they can control or had control over outcomes related to an issue at hand; tendency of people to act as if they can influence a situation over which they actually have no control.

The following is a good approach to reduce these biases without being overly complex or time-consuming.

Step 1 - Framing

A - Criteria Definition
An extremely important step is to define the criteria as well as possible. In Promax, there are several ways to achieve this. First, there is a title and a description. Secondly there is the mapping stage. This is where a scale for scoring is devised. Whilst it may be easy to stick with a 1-10 scale it is much better to use a textual scale since it avoids the natural tendency to do arithmetic on the scores.

B - Resume
Understand the experts contribution in terms of their previous experiences in assessing the performance of options on the specific criteria being assessed.

C – Assumptions List
List up to 5 assumptions to be used in assessing the performance of each option on the criteria.

Step 2 – Scoring (1st Iteration)

What’s the Most Likely value, M?  
What’s the Lowest value, L
What’s the Highest value, H  

This 1st iteration tends to result in anchoring bias on M, over-confidence on L and H, and poor rationale.

Step 3 – Situational Awareness

A – Mini Risk Review
What factors contribute to the uncertainty in the scores?
What impact do these factors have on the score?
What’s the likelihood that these risks will actually happen?

List all the risks associated and estimate their impact on the score:

• Risk Factor
• Description
• Likelihood of Risk Occurring
• Impact on Score

What probability would the expert assign to “Very Unlikely”? (e.g. 10%)
What probability would the expert assign to “Extremely Unlikely”? (e.g. 1%)

Likelihood, Description, Probability
Absolutely Impossible, No chance, 0%
Extremely Unlikely, Nearly impossible to occur. Very rare, 1%
Very Unlikely, Highly unlikely to occur. Not common, 10%
Indifference 1, Indifferent between “Very Unlikely” and “Even Chance”, 30%
Even Chance, 50/50 chance of being higher or lower, 50%
Indifference 2, Indifferent between “Even Chance” and “Very Likely”, 70%
Very Likely, Highly likely to occur. Common occurrence, 90%
Extremely Likely, Nearly certain to occur. Near 100% confidence, 99%
Absolutely Certain, 100% likelihood, 100%

The “Very Likely” and “Extremely Unlikely” probabilities are in opposite of those given by the expert for “Very Unlikely” and “Extremely Unlikely”.

For the impact use the following table:
Impact on Score, Scale
Negligible, 0
Minor, 2
Notable, 4
Substantial, 8
Catastrophic, 10

List the top 5 risks. The risk score is the likelihood x impact.


B – Scenario Development
Using the information in the Assumptions List and the mini risk register, write down a scenario (chain of events) that would represent a “reasonable worst case”.

Repeat for a “reasonable best case”.

Finally repeat for a ‘reasonable most likely case”.

Step 4 – Scoring (2nd Iteration)

Based on the three scenarios developed, how would you now score the option performance?

M =
H =

L =

In Support of Objectivity

 Do you come across organisations running so called 'decision conferences". These are, in the main, a two-day workshop involving a number of people who are there to evaluate a range of options. The participants are "experts" and they score each option against a range of criteria before the option with the best score wins through.

This approach leads to a number of issues that develop later on - particularly for major decisions in large organisations.

The 2-day workshop seems ubiquitous. But why two days? Clearly because some people are travelling of course but surely this isn't the best reason. The length of the workshop needs to be as long as it needs to be and this is dependent on the objectives and hence agenda of the workshop. It's also worth bearing in mind the percentage of time for that workshop compared to the project duration itself. In many cases the actual decision is made based on spending just 0.1% to 0.3% of the available time on it and sometimes even less for mega-projects. Is that sufficient for major decisions? Analysing strengths and weakness and how robust the decision is to changing in data and stakeholder preferences is well worth investing more time.

Experts. Just how expert are they? And what are they expert about? We've all come across the loud, opinionated participant - the self confessed all-knowledgeable person. But whether these are really experts is something else. An expert should be someone who has spent many years on a particular topic and could stand their ground with other experts across the world. Just because they know a little more than the others in the room on a particular subject (maybe they're representing a department or organisation) does not make them an expert. Mark Twain defined an expert as "an ordinary fellow from another town".

Experts, by definition, know a lot about very little. They have an incredibly deep understanding of a topic that they've studied for many years. Interestingly, in a multi-criteria workshop this may mean they may know a lot about how an option performs for one or two of the criteria but clearly not for them all. So asking for their opinion on costs, for instance, may not be their strong point.

The expert offers an opinion. And opinion is subjective. So, the experts provide a subjective score and the best option wins. Unfortunately, what this means in practice is that you have no evidence to back up that opinion so that when it is reviewed or audited it fails in dramatic style. You cannot possibly recreate the decision result unless you have the same experts in the room. Clearly for decisions that involved multiple stakeholders and several hurdles of reviews this approach won't get you very far - no matter how "expert" the experts. At some point, the scores will be reviewed, different ones used and (mostly) a different option will be favoured.

A far better approach is to collect evidence beforehand on how each option performs against each criterion. For many criterion you will be able to obtain decent quantitative measures and it is well worth putting in the effort as real-world number are easy to substantiate.

For others you may need to use a rating scale. These capture subjective opinions with words or numbers; so a 0-10 scale or a High, Medium, Low scale. These sorts of scales are not designed to capture opinions but estimate the magnitude of differences. These scales are still quantitative however even if you use words to describe it - they have equal intervals between each point and represent an order from "bad" to "good" or "less" to "more".  It is vital that each point is described in as much detail as possible so that everyone knows what a "High" looks like. Part of the process should also be to write down why each option is scored that way for each criterion. This means a paragraph or two written by an "expert" ideally with references and sources to additional information to support the score. This is similar to how a "safety case" or "safety basis" may be written but obviously much shorter in length!

A very good detailed explanation on these terms can be found here.

The point of collecting the evidence is that the decision is then based on something that can be reviewed and audited by others who would themselves come up with the same decision given this information.

If someone questions the decision, new evidence needs to be provided. This reduces significantly the effort required when someone with a new opinion arrives (maybe a new Head or other change in staff or perhap a stakeholder who wasn't intimately involved). Also, if new evidence emerges during the decision process, the original decision can be assessed against this - under change control. So, you don't flip from one choice to another choice seemingly at random. If a choice needs to change there's a good reason for it backed up with evidence.

Conclusion

Resist the temptation to have a scoring workshop where experts turn up and provide the performance evaluation for each option. It is far better to collect evidence beforehand and let the experts discuss the validity of the evidence and analyse the results. This will ensure that the decision will be evidence-based and can be realistically audited/reviewed by others. It is a much more robust process to use.

Spend a little more time analysing the decision. Do not fall into holding a 2-day workshop just because that's what is always done. Spend the right amount of time with the right amount of people for the decision at hand.

Strictly Scoring System

It's that time of the year when a new season of Strictly Come Dancing has started. In the US it's called "Dancing with the Stars". It has a rather unusual method of evaluating the couples which is what this blog is about.

The format is that each couple (a professional dancer and a celebrity) do a dance to impress a panel of experts. There are four of these and they each score each couple up to a maximum of 10. So the minimum a couple could get is zero and the maximum is 40. For interest the minimum score has been 8 (Quentin Wilson & Hazel Newberry) and lots have received 40. The Wikipedia page has all the details. This then gives the Leaderboard which ranks the couples by score.

So for the first show the Leaderboard looked like this:

The second element is the Viewer's Vote. For this part, viewers phone in for their favourite couple to keep in the show. In this case the number of votes cast is not published which is difficult if you want to analyse the results. Apparently a BBC spokesman said: "We never reveal exact figures from our shows as we have a relationship of trust with our contestants and it would be unfair to disclose the exact nature of difference in their popularity." This thread  has  a good discussion on why the votes aren't published (celebrity ego, influencing voting, programme ratings and others). Many of these are applicable in real-world decision making of course!

Since they don't publish the actual numbers, the viewer's votes are put into rank order. That is, the couple with the highest number of votes is at the top, then the second highest is next and so on until the least popular couple are at the bottom. So, if there are 14 couples, then the couple who have the highest number of votes is given a score of 14.  The couple with the next highest number of votes if given 13 and so on right down to a score of 1.

In order to combine the judges score with the voters score, the judges score is also converted into a rank order.

The two couples with the lowest combined score go into a "dance off" and the judges score (based on this final dance) wins and the loser goes home.

So what issues arise with this methodology that might be applicable in business decisions?

1)  It is quite clear that the judges scores are variable. Although all experts, they clearly look for different criteria when scoring each couple and consensus is rare. The scores also change over time, so while they do improve as the couples improve there is still little to compare a score of 6 in the early stages with a score of 6 in the latter stages. This is very common when using numerical scales as it is not easy to get everyone to define a score of 6 in the same way.
2) The judges use what is called an "Interval Scale". This scale tells us the order and the size of the intervals between the scores. 
3) The Viewer's scores however use an " Ordinal Scale". This means that you can tell what the magnitude of the difference between the real scores.
4) When the judges scores are changed from an interval scale to an ordinal scale then the benefits of knowing the magnitude of the difference is lost. This means that getting a fantastic score may not be any better than getting a single point more than the next best couple.

It would be great if the BBC were to use the actual viewer votes. But this isn't likely to happen.

Conclusion

Obviously Strictly is only a television show but some of the techniques they use have issues that could easily make it into more important decisions in the workplace.

Wherever possible use interval scales rather than just putting thinks into a rank order as you can then see whether any options are much better than others. Secondly, if you have to use a numerical scale for scoring then try to calibrate the scale and explain what is meant by a 5 for instance. Make sure this recognises that a 10 is twice as good as a 5. Don't assume experts already know this!

How Many Criteria?

I'm often asked "What's the best number of criteria to use?"

As a matter of simple arithmetic, the more you have, the less impact each one will have on the decision. So if you have 4 criteria and each one is weighted equally, then each will contribute 25% to the decision. If you go up to 10 criteria then the contribution of each drops to 10% and so on. When you have over 10, even doubling the importance of a single criteria will make little difference to the final result.

However, a better way of asking that question is to turn it around and not query how many criteria should I have but what criteria should I use. The number that results is the number you need.

In determine what the criteria should be, it is never a good idea to simply brainstorm within your team to come up with ones you think are differentiators. A far better approach is to use a hierarchy of objectives. Here you consider the overall objective or mission you're trying to accomplish. Then question what lower level objective supports the main mission. Finally what sub-objectives or measures support those. It's these lower level objectives - ones that are measurable - that should be your criteria.

For most evaluations you will easily end up with more than 4 or 5! However, that isn't a problem as long as you weight them at the right level. In the diagram above, there are 4 main objectives contributing to the Mission so we weight at that level instead.

In Promax this is called "Top Down" weighting. Go to the weights tab and click the Top Down button. An arrow will appear next to the relevant weight set. This means the calculations will be based on the weights of the topics or nodes (the red boxes) rather than criteria (grey boxes).

By default, in top down weighting, the criteria below each topics are equally weighted. So if you have more criteria in one topic than in another this may lead to unforeseen results. You can easily alter them to reflect a more correct weighting but remember that it's the weights at the topic level that have the final say.

Conclusion

Try to develop criteria based on objectives rather than randomly brainstorming them. That way you can demonstrate how each option contributes to the overall mission. You can much more easily show in what areas it excels and where it is weak.

Criteria and Attributes

-->

Criteria and attribute are often used interchangeably. Often people say they do multi-criteria decision analysis (MCDA) or maybe multi-attribute decision analysis (MADA). They mean the same technique but it is worth knowing that they are, in fact, different.

Criteria are the factors which options are to be measured against. Criteria are plural and criterion is singular. Attributes are the properties of the criterion.

So for example, in looking at choosing a car, one of the factors that may be important is luggage space. 

-->

The grey boxes are criteria and the red ones are topic areas.

If you think about it, however, there are many different ways to measure luggage space; overall volume is an obvious one. However, this may not be the best choice if you want to transport specific items. You might be a golf fanatic and so it’s important that you can fit a set of golf clubs. These are typically fairly long and require quite a lot of width. The required height is small so a wide boot (trunk) is more important than a tall narrow one. 

Similarly you might want a boot (trunk) that takes rectangular items like suitcases in which case a flat load space that fits two side by side and tall enough for another layer is what’s required. The overall volume is less important.

Knowing this, it is important that when defining each criterion the question is asked "how would you measure success? You may be surprised at the different responses you get for what appears to be exactly the same criterion! This is especially true when you have different stakeholders involved.

Promax software uses the criteria properties dialogue box to set attributes. Right click on a criterion and the box pops up.

You can add a description to explain more about the criterion which is useful (essential) for explaining to others, then decide whether it's a cost, benefit or a harm (to be covered in a later blog but choose Benefit most of the time) and finally click "Edit Measurement Details". 

This comes up with the attribute details as below.

Choose your units and, if necessary, the number of decimal places to be used and that's it! You now have a properly defined criterion that can be used for evaluating options.

Promax Launching This Week

-->

This week we will be launching version 3 of Promax – our multi-criteria software. It has lots of new features from version 2 that we hope our customers will find useful and exciting.

Multi-criteria software is used for aiding decisions where you have more than one factor (criterion) to consider. It is well recognised that most people struggle to assess multiple criteria fairly and often make decisions based on their overall gut-feel; a High, Medium, Low if you like. Whilst this may be adequate for decisions that are relatively small and only involve a few people it is not a good technique for larger decisions or where there are many people involved. These sorts of decisions need better transparency and visibility of how the different factors contribute to the decision.

Although people use spread sheets to carry out their calculations it should be remembered that many (most) have had no training or experience with decision science. This can and does lead to errors in calculations and incorrect methods for valuing and weighting criteria. Additionally it is difficult to carry out adequate sensitivity analysis and view the result in anything other than a simplistic manner.

Purposely designed multi-criteria software removes these drawbacks so you can be sure the result is, in fact, valid – which for important decisions is surely extremely important. The initial cost of the software will quickly be repaid in the speed of model build and subsequent analysis. It is certainly far cheaper than using spread sheets. Simply calculate the hours spent every time someone builds the spread sheet, someone checks the calculations the the graphs are developed. Multiply these hours by the persons rate and you'll probably be surprised at the eventual cost. A few decisions will be enough to fund the software purchase and that doesn’t even include the opportunity cost of the spread sheet person (i.e. they could/should be doing something else more valuable to the business).

Promax is unique in that it has embedded well-recognised multi-attribute utility calculations into a tool that is of practical use to people who aren’t decision-science experts. It is designed around a structured decision-making methodology taking the user through defining the problem, developing criteria, weighting the criteria, identifying options and then scoring the options. The results are presented in a myriad of ways which is important to assess the robustness of the decision – the biggest overall score may not necessarily be the best option.  

Some of the great features are:

• Problem Definition – state where you are now, where you want to be and what the barriers are using a prompt-based approach.
• Creating a value tree using drag and drop – just create criteria using a mind mapping approach and drag criteria around to fit under different topics.
• Multiple ways of weighting – direct weighting, pairwise comparison and swing weighting gives plenty of choices to ensure you can weight criteria accurately as to their importance.
• Options with notes – a drag and drop approach to creating options gives great flexibility. You can add notes, images, attachments and even rate the option right on the display.
• Creativity & Triz tools / database – this functionality for coming up with ideas is much more effective than using traditional brainstorming. These exciting tools can generate many more options than you thought possible.
• Scoring with sliders and add notes – a cool new slider for scoring plus the ability to add notes to explain the rationale behind the score.
• Results – there are many ways of viewing. From a bar chart and table through to looking at strengths and weaknesses of particular options. Also, looking at value for money is a fantastic display.

Additionally there are lots of behind the scenes enhancements such as the ability to change colours, undo, import/export to/from excel and address uncertainty with three point estimates.

The professional version adds resource allocation to the software. This allows you to have combinations of options and pick the best combination for a given budget (or other resource). This is used for value engineering and for prioritisation within funding constraints.

For more information see the Promax web page.