Quality Score for Design Competition Jury

Assigning a Quality Score for Design Competitions - The Jury

Abstract: Jury is one of the core components of a design competition and a good jury means increased credibility, better publicity and added value for the award. This paper discusses, how we could calculate a quality score for the jury.

First of all, to calculate a raw quality score for the jury panel, we have check jury information such as member names, and profiles. Profiles mean the background of jury members, their professions and whether they are relevant to the competition and if they are famous etc. During this process, we refer to the public competition announcements; flyers, call for submission papers, press releases,  and the competition website to extract required information about the structure and composition of the jury panel. If we cannot find this information with aforementioned methods, we send the competition organizers an email to ask them.  We use the jury score sheet to calculate the raw store.

Jury score is calculated with the following criteria: Number of Jury Members, Jury Transparency, Homogeneous Composition of the Jury and the Profile of Jury Members.

Please note that for raw jury score, we do not consider organizational properties such as meeting and evaluation criteria here as they are discussed in other sections.

Collected Information: Jury Member Names & Profiles

Jury Score Sheet:

1. Transparency of Jury Members
1. For each three published name up to 15, Score++
   Publishing jury names increases the credibility of the design competition through transparency. Names could also be published after the competition is announced.
2. Profile of Jury Members
1. Background
1. For each different background type in jury, Score++
   For each unique different jury backgrounds included in the jury, the group intelligence increases which in return leads to a multi-disciplinary and improved judging, furthermore the multi-disciplinary jury can also advertise the competition to a broader social sphere not limited to a single domain. The backgrounds could be for example; designer, business professional, engineer, academician, traditional or new-media press member, focus group member, non-designer, company representatives, institution representatives, technical persons and craftsman etc.
2. Famous or Not
   For each jury member, up to 15, an added value is created;
1. Not Famous, Score //
   Not famous jury members usually do not create enough publicity for the award, therefore they do not create an added value in the jury in terms of equivalent advertising value generation.
2. Famous or Popular, Score ++
   Famous or popular jury members could be good references, being judged by a famous person in your field is something that you could use for greater publicity.
3. Very Famous, Score ++2
   Very Famous jury members could be great references, being judged by a very famous person in your field is something that you could use for greater publicity and credibility.
4. Extremely Famous, Score ++3
   Extremely Famous jury members in particular design fields are hard to come by and not only they create the highest publicity, but also they are a solid reference for the creating credibility and value for the overall award.
3. Number of Jury Members:
1. For each (non-public) jury member up to 10, Score ++2
   With the number of people involved, the competition become more democratic and less oligopolistic in a sense that, no jury member could particularly effect the outcome with her sole efforts, thus with more jury members, we got less-biased evaluation. One thing to consider that these people are “published” or “announced” jury members; they are not profiles of “public voting” participants.
2. For each three (non-public) jury members, after 10, up to 15 Score ++
   Decreasing returns to after a certain number of jury members is a good idea, and again after a point, it does not matter whether there are hundreds of jury members.

Maximum Possible Score (82) is reached by a jury that is composed of 15 Jury Members all Extremely Famous in Their Fields, with different backgrounds, and all announced before or after the competition.

1. Transparency of Jury Members
1. For each three published name up to 15, Score++ (Max. Score +5)
2. Profile of Jury Members
1. Background
1. For each different background type in jury, Score++ (Max. Score +12)
2. Famous or Not
1. Extremely Famous, Score ++3 (Max. Score +45)
3. Number of Jury Members:
1. For each (non-public) jury member up to 10, Score ++2(Max. Score+20)
2. For each (non-public) jury members, after 10, up to 15 Score ++ (Max. Score+5)

Short Notes: Please note that ++ means to increase by one (1), and -- means to decrease by one (1), so score ++ means, score will increase by one (1). Score ++n means to increase score by n (n) points; Score ++3 means to increase score 3 points. Score // means does not affect score.

Why a bigger award jury is more fair only when the jury does not interact?
Statistics, namely for error reduction purposes: When jury members vote on entries without interacting with each other, the jury members cannot effect each others' votes. When each jury member vote is independent, the voting biases and evaluation errors is reduced in the aggregate level; the positive and negative voting biases and evaluation errors by individual jury members averages to zero as the sample size increases, the total average decision of the jury panel when each member votes independently and without being effected by another jury member, is almost devout of errors since when many judgements are averaged, the average of all judgements tends to be more accurate than any judgement by a person alone, and as the sample size of independently voting jury members increase the error is reduced more. However this reduction in error is limited to any common biases available within the voters, thus to have even more fair voting, a jury shall also be composed of different groups, from different countries and backgrounds. In this case the jury errors are reduced in two dimensions, and this is why the grand A' Design Award and Competition jury, composed of prominent press members, established academics, entrepreneurs and design professionals are asked to cast their votes independently from each other, using a special voting platform that was scientifically developed to reduce any personal attribution and evaluation errors during design voting by combining and averaging the votes of all jury members at aggregate level. Thus the fact that jury meets or not is not counted as a part of the quality score for the competitions, but the jury size matters.

Relevant Articles

• Assigning a Quality Score for Design Competitions - What are the core qualities that makes a design competition good, fair and effective. (Theory)