Q.S. for D.C. Evaluation, Judging & Scoring

Assigning a Quality Score for Design Competitions - Evaluation, Judging and Scoring

Abstract: The evaluation of results in a design competition through judging and scoring is clearly one of the most important aspects of a design competition. If evaluation, judging and scoring is clearly explained, participants can deploy strategies to make efficient submissions that would be in line with the judging and evaluation criteria thus improving their rankings and designs.  Therefore, the evaluation, judging and scoring plays perhaps the key role in any design competition and having this information is vital, in addition the way the evaluation is done is one of the most impactful factors on the quality of a competition. In this paper we discuss how we could assign a score for the evaluation, judging and scoring process within a design competition.

First of all, to calculate a raw quality score for the evaluation, judging and scoring criteria we usually have to contact the competition organizers as this information is rarely available on public competition announcements made to public such as email communications, flyers, call for submission papers, press releases. In some cases we could reach this information from the competition website but usually we cannot find this information with aforementioned methods, and so we send the competition organizers an email to ask them.  We use the evaluation, judging and scoring score sheet to calculate the raw store.

Evaluation, Judging and Scoring score is calculated with the following criteria: Does Jury Meet Physically, Selection System, Voting System (Open or Public), Score Sheet and Jury Feedbacks, Numerical Scores, Competition Statistics, Judging Criteria and Weights.

Please note that for raw evaluation, judging and scoring score, we do not consider other details such as the jury details, the competition timeline etc. as they are discussed separately.

Collected Information: Scoring and Judging Details, Evaluation and Judging Criteria

Evaluation, Judging and Scoring Score Sheet:

1. Judging Criteria
   Judging criteria are like rules of a game, they define how the entries are judged in a design competition, and it indicates what the jury members focus on while voting on the designs. This information is usually not given for several reasons such as to disguise inner mechanics or for not to provide any opportunity to strategically design designs based on the criteria, but unfortunately one of the main reasons is that the judging criteria is not given because there is not such criteria present; there are no rules.
1. For each defined Criterion up to 10, Score++10
   A Criterion is a little piece of information or an inherent value of a design that the jury checks. It is important for designers to know what the criterion is so that they could develop better designs that match the evaluation criteria. Providing this information publicly is important.
2. For each defined Weight of Criterion up to 10, Score++5
   This is the effect of a single criterion on the overall score. The weight of a criterion is a strong indication for a designer, so that the participant could give relevant importance to a given criterion. Providing this information to public not only results in better entries, but also creates a transparency and eliminates any insider effects.
2. Scoring and Judging Details
1. Visible/Hidden Submission Processes
   In visible submission process, the designs submitted are visible to all or some even before results are announced, and in hidden submission process, the submitted designs are only visible when the results are announced.
1. Hidden Submission Process, Score++10
   Submitted designs are not visible to anyone except jury until competition ends. This is the best practice for respected design competitions. By hiding submitted designs you are also protecting them from being copied during the competition process.
2. Visible Process, Score++1
   Submitted designs are visible to all (or other participants) during the course of the competition. This is a terrible way to run a design competition for the designer; you could see what other people did and in some cases also see comments on them and do something better, this particular system is usually employed by innovation-contests where a company asks designers to provide a solution under the name of a design competition. In any case, letting this information in advance is a plus to avoid such competitions.
3. Semi-Visible Process, Score++2
   Submitted designs are visible to all only after the second step of the competition. In this case, usually the second step is after submissions are closed, the designers are asked to invite their friends for voting and therefore the process is semi-visible, this is definetely not a preferred process, however it is better than visible process.
4. Quasi-Visible Process, Score++5
   Submitted designs are visible to all at the final public voting step of the competition. In this case, usually a jury panel eliminates the results to a limited number such as ten, and again designs are asked to be voted by the public, this is better than the semi-visible process, but still not as good as the hidden process.
5. Top-Secret Process, Score++5
   Submitted designs are not visible to anyone (except jury) even when the competition ends. This is one of the most ridiculous systems that you could use unless you are running a high-tech military design competition, the truth is, for anything else than the high-tech military design competitions, the top-secret process makes the whole thing appear as shadowy and obscure, participants ask why they cannot see other submissions and there is a reason; they would like to compare their designs with that of others. In any case, this option could sometimes be positive, but in most cases it is negative and not preferred by the entrants.
6. Opted-In Process, Score++7
   Submitted designs are visible at the end of competition only if the designers agree. The result of this option is simple; only the winners list will prevail; as most non-winners will not want to show that they have lost, and also to protect their designs,  they will not share them after the competition.  For some designers, this might be preferred and it is better than visible, semi-visible or quasi-visible processes.
2. Evaluation Criteria and Judging URL, Score++
   This is a webpage address where the evaluation criteria and judging are clearly explained, and it clearly helps to have this information easily accessible, as it creates added transparency and credibility for the competition.
3. Physical Jury Meeting, Score++
   This tells us whether the jury physically meets or not. The physical meeting of the jury could be considered as preferable as it takes additional resources and organizational power to arrange such a meeting, however on the other hand jury meeting physically could also lead to biases, in any case providing this information is important as some participants might give a decision to join or not depending on this.
4. Selection System, Score++
   The selection system is how jury makes the voting; such as step-by-step elimination, majority voting or by following a scoring sheet / checklist. It is a great debate which selection system is better, therefore instead of assigning different scores for selection systems such as step-by-step elimination, majority voting or score-sheet based, instead we give a positive score for indicating the system used.
5. Open/Close Voting System
   An Open voting system defines a design competition where public is given a voice, on the other hand in a closed voting system the decision is made by the jury.
1. No Public Voting, Score++10
   This is the most preferable way as not only designs are protected by not letting others view it during the competition, but also it is much more respectful than making designers to invite their friends to vote for the designs.
2. Public Voting, Score++
   Public voting method is usually employed in low quality design competitions where the aim is to advertise; designers are asked to invite their friends for voting and the designer with the most friends have a greater chance of winning, this method, is simply an ugly, non-elegant way of involving designers especially if the public voting is open in all stages. Sharing this information ahead is a plus as designers could avoid such competitions.
3. Hybrid: Public Voting Only After Submissions are Closed, Score++3
   Better than public voting; instead of allowing voting during submission process, at the end of the competition some select entries are asked to be voted. This is also not the most preferred way but somehow better than purely public voting.
6. Score Sheet
   In case there are clear evaluation criteria, there also exists a scoring-sheet / checklist where jury members vote on the specific details of a design, in this case if these votes are scored we have a score sheet indicating how a design scored on different evaluation criteria that could be beneficial for participants to see where they lost or gained the most points, their strengths and weaknesses etc.
1. Score Sheets Available Online to Only to Participants, Score++7
   By accessing the score sheets that shows points received from different voting criteria, participants can improve their designs over the years by working on their weak points. This is a great plus for entrants.
2. Score Sheets Available Online to All, Score++2
   This is not a preferred way to share information, most designers would find that it is disturbing for them for others to see their evaluation, however it could also be a way to understand the judging and evaluation system when all considered. In any ways, this is slightly better than not providing score sheets at all.
3. Hard-Copy Score Sheets Will Be Sent To Participants, Score++7
   This is a good way to share information with designers, they can see how their design has performed on the different criteria and they can use the hard-copy score sheets also as future reference as online versions could be lost over time.
4. Score Sheets Are Available Online to Participants and As Hard Copy, Score++10
   This is the best way to share the information with designers by providing them also ease of access using internet.
5. Score Sheets Not Available, Score++
   This is the least preferred choice, by not providing the score sheet, in a sense you are asking designers to shoot in pitch-black; they do not see where they hit. However sharing this information in advance is always a plus.
7. Jury Notes and Feedbacks
   Jury notes and feedbacks are of great importance for the added value of self-assessment for designers, by thinking on the jury notes and feedbacks participants can improve their work over time. It is better to provide jury notes and feedbacks; however it is not always easy especially when the number of submissions is high.
1. Jury Notes Are Available Online to Only to Participants, Score++7
   It is easy to share jury notes with the participants through an online platform, and the best practice is that only the designer of a specific design sees what the jury thinks of their design.
2. Jury Notes Are Available Online to Everyone, Score++3
   Many designers would find it undesirable to see some negative comments on their design, especially it is a nightmare if everyone can see them so this method is not preferred, on the other hand, by being able to see the jury notes on others designs, it might also be a way for entrants to understand and comprehend the competition judging criteria which could lead to better designs for the next editions.
3. Hard-Copy Jury Notes Send to Participants After Competition, Score++10
   It is best to send hard copy jury notes to participants one by one so that only the designer of a specific design sees what the jury thinks of their design.
4. Jury Notes or Feedbacks are not available, Score++
   This is not preferred, but stating this information might be important, especially if a designer aims to improve his skills through a design competition, by knowing that there will not be any feedbacks, the entrant can make a choice not to join.
8. Numerical Score
1. There is not a Numeric Score , Score++
   For some participants, having a numeric score is preferred, and for some others it might not be the case, in any condition, it is better to state this information a head so that the entrants could make a join not join decision based on this input.
2. There is a Numeric Score, Score++5
   If criteria based voting is followed, there should be a numeric score, it might be a good idea to share this score with the participants so that they could improve their designs through self-assessments.
9. Competition Statistics
   In most cases not only participants but also the press and spectators would like to learn how many people joined a particular design competition, how much of the nominations were winners, the composition of winners, demographics etc. to name a few. Providing this information is clearly a plus.
1. Competition Statistics Are Available Online Only to Participants, Score++3
   Anonymous competition statistics could be made public, sharing this information only with the participants do not create added value for the design domain in general, however it is still better than not sharing them at all.
2. Competition Statistics Are Available Online to All, Score++10
   This is one of the best options, so that not only participants but also spectators and press members or anyone who would like to know could freely access to this information. Definitely increases the credibility of the competition.
3. Competition Statistics Are Hard-Copy Sent to Participants, Score++4
   This is a great way to provide extra information, and as it is hard-copy, it could be used as a reference easier, but still why send these valuable information only to participants.
4. Competition Statistics Are Hard-Copy Sent to Participants and Available Online, Score++10
   This is the ideal case where the statistics are shared online and furthermore hard-copy versions are sent to participants to have hardcopy reference.
5. Competition Statistics Are Not Available, Score++
   This is the least preferred choice as it creates an obscurity, when people do not know what is going on, it feels fishy and shadowy.

Maximum Possible Score (208) is achieved with the following configuration:

1. Judging Criteria
1. For each defined Criterion up to 10, Score++10, (Max. Score +100)
2. For each defined Weight of Criterion up to 10, Score++5 (Max. Score +50)
2. Scoring and Judging Details
1. Visible/Hidden Submission Processes
1. Hidden Submission Process, Score++10
2. Evaluation Criteria and Judging URL, Score++
3. Physical Jury Meeting, Score++
4. Selection System, Score++
5. Open/Close Voting System
1. No Public Voting, Score++10
6. Score Sheet
1. Score Sheets Are Available Online to Participants and As Hard Copy, Score++10
7. Jury Notes and Feedbacks
1. Hard-Copy Jury Notes Send to Participants After Competition, Score++10
8. Numerical Score
1. There is a Numeric Score, Score++5
9. Competition Statistics
1. Competition Statistics Are Available Online to All or Competition Statistics Are Hard-Copy Sent to Participants and Available Online, Score++10

Short Notes: Please note that ++ means to increase by one (1), and -- means to decrease by one (1), so score ++ means, score will increase by one (1). Score ++n means to increase score by n (n) points; Score ++3 means to increase score 3 points. Score // means does not affect score.

Relevant Articles

• Assigning a Quality Score for Design Competitions - What are the core qualities that makes a design competition good, fair and effective. (Theory)