The format is that each couple (a professional dancer and a celebrity) do a dance to impress a panel of experts. There are four of these and they each score each couple up to a maximum of 10. So the minimum a couple could get is zero and the maximum is 40. For interest the minimum score has been 8 (Quentin Wilson & Hazel Newberry) and lots have received 40. The Wikipedia page has all the details. This then gives the Leaderboard which ranks the couples by score.
So for the first show the Leaderboard looked like this:
The second element is the Viewer's Vote. For this part, viewers phone in for their favourite couple to keep in the show. In this case the number of votes cast is not published which is difficult if you want to analyse the results. Apparently a BBC spokesman said: "We never reveal exact figures from our shows as we have a relationship of trust with our contestants and it would be unfair to disclose the exact nature of difference in their popularity." This thread has a good discussion on why the votes aren't published (celebrity ego, influencing voting, programme ratings and others). Many of these are applicable in real-world decision making of course!
Since they don't publish the actual numbers, the viewer's votes are put into rank order. That is, the couple with the highest number of votes is at the top, then the second highest is next and so on until the least popular couple are at the bottom. So, if there are 14 couples, then the couple who have the highest number of votes is given a score of 14. The couple with the next highest number of votes if given 13 and so on right down to a score of 1.
In order to combine the judges score with the voters score, the judges score is also converted into a rank order.
The two couples with the lowest combined score go into a "dance off" and the judges score (based on this final dance) wins and the loser goes home.
So what issues arise with this methodology that might be applicable in business decisions?
1) It is quite clear that the judges scores are variable. Although all experts, they clearly look for different criteria when scoring each couple and consensus is rare. The scores also change over time, so while they do improve as the couples improve there is still little to compare a score of 6 in the early stages with a score of 6 in the latter stages. This is very common when using numerical scales as it is not easy to get everyone to define a score of 6 in the same way.
2) The judges use what is called an "Interval Scale". This scale tells us the order and the size of the intervals between the scores.
3) The Viewer's scores however use an " Ordinal Scale". This means that you can tell what the magnitude of the difference between the real scores.
4) When the judges scores are changed from an interval scale to an ordinal scale then the benefits of knowing the magnitude of the difference is lost. This means that getting a fantastic score may not be any better than getting a single point more than the next best couple.
It would be great if the BBC were to use the actual viewer votes. But this isn't likely to happen.
Conclusion
Obviously Strictly is only a television show but some of the techniques they use have issues that could easily make it into more important decisions in the workplace.Wherever possible use interval scales rather than just putting thinks into a rank order as you can then see whether any options are much better than others. Secondly, if you have to use a numerical scale for scoring then try to calibrate the scale and explain what is meant by a 5 for instance. Make sure this recognises that a 10 is twice as good as a 5. Don't assume experts already know this!
No comments:
Post a Comment