Improving Criteria Scaling

A previous article examined how to use Promax to recreate the model for decisions involving multiple objectives as described in the book “Decision Analysis for Management Judgment” by Paul Goodwin and George Wright (1991).

In the example we used a real-world scale for some of the criteria (for example, office floor area in ft2) and a preference scale for criteria where real-world scales were difficult to define. The problem with preference scales is that they are unique to the group involved. The chances of a completely different group choosing the same preferences are remote. This means that the result cannot be adequately explained or justified and it also cannot be recreated. This may be perfectly acceptable for a decision where there is a single decision maker but it is not acceptable where there are multiple decision makers – for instance in large organisations where decisions are approved and reviewed at multiple stages.

This article looks at another type of scale that can be used for those “difficult-to-measure” criteria – namely textual scales.

Textual scales are ones where the degree of preference is explained in words rather than numbers. They are created in a similar way to preference scales except that the reasoning behind the position on the scale is explain in objective terms. See an earlier blog for the differences between objective and subjective information.

For the criteria being discussed, write, as clearly as possible, the best imaginable score. The addition of real examples that people would recognize is well worth the effort. Then consider the worst imaginable. Then imagine what lies between the best and the worst. Don’t think in terms of position (e.g. halfway between best and worst) but more in terms of discrete differences. They should be something that you can describe as being a step-change difference. Then, create as many intervals as you need that will adequately segment the scale. For example, office image could be explained thus:

T	This office compares with the best in the world. As a benchmark the TBWA Hakuhodo offices in Tokyo.
H	This office is just about the best in the area. As a benchmark the “Hatton Gardens” office block.
K	The offices in “Kent Boulevard” are the benchmark here
L	The offices along “Link Road are the benchmark here
M	The offices along “Mellor Drive” are the benchmark here
I	This office is just about the worst in the area. The offices in “India Road” and “Jason Place” are the benchmark here
P	This office compares with the worst imaginable. A dark, dingy, very run-down office which is particularly uninvited. For example you’ll smile smugly to yourself as you imagine people whose offices are much messier than yours. You’ll stretch your arms out and luxuriate in the fact that you can’t even touch both walls of your 8×8 cubicle simultaneously as you view picture of a worker toiling in a space smaller than a closet in a one bedroom apartment.

Figure 1: The Promax scorecard display once descriptions have been defined

There are some top tips in creating textual scales:

• Do not use numerical scales (e.g. 0-10) even if you explain what a 10, or a 7 or a zero is. These scales are prone to people “gaming the system”. This means that since people know that a 7 is a better score than a 5 they will persuade others that the rationale behind giving the scores makes sense. It may make sense at the time, but later on you will struggle to reconcile the scoring with the facts. So, people can quite easily generate the result they always wanted even though apparently the methodology was fair and transparent. That is why techniques using numbers such as Kepner-Tregoe, the Pugh Matrix, AHP etc. should be avoided – they appear robust and scientific but are anything of the sort.
• Do not use Very Good, Good, Average, Poor, Very Poor even if you explain what “Good” means. Again, it is too easy to game the system because you know that a Very Good is better than Good.
• Do Randomise the scale. In the same way that a well-conceived survey randomizes it’s questions in order to reduce bias so you should do this for textual scales. This means not ordering the scale from good to bad. Order them in a different way otherwise, whatever the description, you end up in the same trap as for the previous two tips – people can game the system because they know what scores will generate the result they want.

Once you have created your scale you need to value the points on it. The best imaginable is worth 100 and the worst is worth 0. The others will lie somewhere between 0 and 100. Remember, as in the mapping done for real-world scales the mapping does not need to be a linear, straight line. You may decide that anything with a better image than “Link Road” does not add any more value so you end up with more of an s-curve.

Always keep the scaling separate from the mapping. In Promax, when using textual scales you need to map the scale against the value as part of the process.

Conclusion

The use of textual scales rather than preference scales is to be preferred for decisions involving multiple decision makers since the results can be more easily reviewed and challenged. That said, there are lessons to be learnt in creating those scales so that it is difficult for participants to game the system.

How Many Criteria?

I'm often asked "What's the best number of criteria to use?"

As a matter of simple arithmetic, the more you have, the less impact each one will have on the decision. So if you have 4 criteria and each one is weighted equally, then each will contribute 25% to the decision. If you go up to 10 criteria then the contribution of each drops to 10% and so on. When you have over 10, even doubling the importance of a single criteria will make little difference to the final result.

However, a better way of asking that question is to turn it around and not query how many criteria should I have but what criteria should I use. The number that results is the number you need.

In determine what the criteria should be, it is never a good idea to simply brainstorm within your team to come up with ones you think are differentiators. A far better approach is to use a hierarchy of objectives. Here you consider the overall objective or mission you're trying to accomplish. Then question what lower level objective supports the main mission. Finally what sub-objectives or measures support those. It's these lower level objectives - ones that are measurable - that should be your criteria.

For most evaluations you will easily end up with more than 4 or 5! However, that isn't a problem as long as you weight them at the right level. In the diagram above, there are 4 main objectives contributing to the Mission so we weight at that level instead.

In Promax this is called "Top Down" weighting. Go to the weights tab and click the Top Down button. An arrow will appear next to the relevant weight set. This means the calculations will be based on the weights of the topics or nodes (the red boxes) rather than criteria (grey boxes).

By default, in top down weighting, the criteria below each topics are equally weighted. So if you have more criteria in one topic than in another this may lead to unforeseen results. You can easily alter them to reflect a more correct weighting but remember that it's the weights at the topic level that have the final say.

Conclusion

Try to develop criteria based on objectives rather than randomly brainstorming them. That way you can demonstrate how each option contributes to the overall mission. You can much more easily show in what areas it excels and where it is weak.

Criteria and Attributes

-->

Criteria and attribute are often used interchangeably. Often people say they do multi-criteria decision analysis (MCDA) or maybe multi-attribute decision analysis (MADA). They mean the same technique but it is worth knowing that they are, in fact, different.

Criteria are the factors which options are to be measured against. Criteria are plural and criterion is singular. Attributes are the properties of the criterion.

So for example, in looking at choosing a car, one of the factors that may be important is luggage space. 

-->

The grey boxes are criteria and the red ones are topic areas.

If you think about it, however, there are many different ways to measure luggage space; overall volume is an obvious one. However, this may not be the best choice if you want to transport specific items. You might be a golf fanatic and so it’s important that you can fit a set of golf clubs. These are typically fairly long and require quite a lot of width. The required height is small so a wide boot (trunk) is more important than a tall narrow one. 

Similarly you might want a boot (trunk) that takes rectangular items like suitcases in which case a flat load space that fits two side by side and tall enough for another layer is what’s required. The overall volume is less important.

Knowing this, it is important that when defining each criterion the question is asked "how would you measure success? You may be surprised at the different responses you get for what appears to be exactly the same criterion! This is especially true when you have different stakeholders involved.

Promax software uses the criteria properties dialogue box to set attributes. Right click on a criterion and the box pops up.

You can add a description to explain more about the criterion which is useful (essential) for explaining to others, then decide whether it's a cost, benefit or a harm (to be covered in a later blog but choose Benefit most of the time) and finally click "Edit Measurement Details". 

This comes up with the attribute details as below.

Choose your units and, if necessary, the number of decimal places to be used and that's it! You now have a properly defined criterion that can be used for evaluating options.