Comparison of Jury Ranking Systems

The Professional Guidelines recommends using a numerical ranking system of 1-7. This is the best system both from a mathematical point of view and for practicality. In fact, some commonly used jury ranking systems actually skew the results unfavorably and produce unreliable results.

If you don't have an in depth understanding of mathematics and don't want to read this document use every number between 1-7 for jury ranking. Yes, a juror should use every number including 1 and 7.

Jurors tend to use the middle range of numbers when scoring work because each piece viewed is rarely the best or worst that the juror has ever seen. Reality is that most work submitted to a jury is "average" (i.e. 2,3,4,5,6.)

Using every number between 1-7 will produce the most reliable results for jury ranking. The additional benefit is that there will be more subtle gradations rating the work, thus you can add or delete work depending on the amount of space or entries permitted.No matter what ranking system is used, the jurors should be encouraged to use the full range of numbers to rank the images.

Read the PDF Comparison of Jury Ranking Systems to understand the math behind several different jury ranking systems that are commonly used.

Below is an excerpt:

Juries seldom use only the 1 – 4 range (i.e. 1, 2, 3, 4) because fewer choices usually result in too many ties.

On the other hand, a ten number range (i.e. 1 – 10) offers too many choices, which tends to slow down decision-making.

Sometimes it is suggested to use a system that removes the middle number from a 1 – 5 system (e.g. eliminating the “3”) and to use only 1, 2, 4, 5 to “force” a selection outside of average. However, using fewer score choices increases the number of possible ties (regardless of the number of jurors). Mathematically, there is no difference between 1, 2, 3, 4 and 1, 2, 4, 5 since the number of sums (outcomes) is identical.

There are two advantages of using the 1 - 7 system over the 1 – 5 system.
With just three jurors and scoring choices from 1 – 7, there are nineteen possible sums. The 1 – 5 system produces thirteen possible sums. With nineteen possible sums, it is easier to find the dividing point between accepted work (the top sums) and rejected work (the lower sums).

Secondly, difficult negotiations among jurors can be kept to a minimum when designating awards because with fewer ties, it is more evident which pieces actually received the highest scores.

Read the entire document to learn more about PDF PDF Comparison of Jury Ranking Systems.:

Creative Commons License
Harriete Estel Berman by Professional Guidelines for the Arts and Craft Community is licensed under a Creative Commons Attribution 4.0 International License.
Based on a work at http://www.harriete-estel-berman.info/profguidelines/profguide.html.
  ASK Harriete offers professional advice to the arts and crafts community .
  Close Window