Comparison of Jury Ranking Systems

The Professional Guidelines recommends using a numerical ranking system of 1-7. This is the best system both from a mathematical point of view and for practicality. In fact, some other systems currently used (for example 1,3,5,7 ) actually skew the results unfavorably.

Read the PDF Comparison of Jury Ranking Systems to understand the math behind several different jury ranking systems that are commonly used.

Below is an excerpt from PDF Comparison of Jury Ranking Systems

Jurors tend to use the middle range of numbers when scoring work because each piece viewed is rarely the best or worst that the juror has ever seen. In other words, most of the work is average. No matter what ranking system is used, the jurors should be encouraged to use the full range of numbers to rank the images.

Juries seldom use only the 1 – 4 range (i.e. 1, 2, 3, 4) because fewer choices usually result in too many ties.

On the other hand, a ten number range (i.e. 1 – 10) offers too many choices, which tends to slow down decision-making.

Sometimes it is suggested to use a system that removes the middle number from a 1 – 5 system (e.g. eliminating the “3”) and to use only 1, 2, 4, 5 to “force” a selection outside of average. However, using fewer score choices increases the number of possible ties (regardless of the number of jurors). Mathematically, there is no difference between 1, 2, 3, 4 and 1, 2, 4, 5 since the number of sums (outcomes) is identical.

There are two advantages of using the 1 - 7 system over the 1 – 5 system.
With just three jurors and scoring choices from 1 – 7, there are nineteen possible sums. The 1 – 5 system produces thirteen possible sums. With nineteen possible sums, it is easier to find the dividing point between accepted work (the top sums) and rejected work (the lower sums).

Secondly, difficult negotiations among jurors can be kept to a minimum when designating awards because with fewer ties, it is more evident which pieces actually receive the highest scores