The Department of Art and Design at Utah State University is a dynamic community dedicated to excellence in visual art, design, and scholarship. Recognised for its nationally accredited programs, the department combines rigorous foundational study, creative innovation, and professional skill development within a supportive environment.
The Interior Architecture and Design program at Utah State University (USU) is an acclaimed and competitive four-year Bachelor’s degree, recognised as the first in Utah to earn Council for Interior Design Accreditation (CIDA). Students complete 120 semester credits and undergo rigorous portfolio reviews in their first and second years to progress in the program.
Recruiting fairly for this highly competitive course through traditional assessment techniques presents significant challenges related to bias, reliability, and validity. Not getting this could produce several negative consequences that can undermine both fairness and institutional quality.
Sticking with traditional assessment methods creates risk of bias, reduced creativity, inefficiency, and reputational harm-whereas Comparative Judgement with RM Compare delivers greater fairness, validity, and alignment with the values of art and design education.
Comparative Judgement using RM Compare is a superior approach for allocating places in competitive courses like USU’s Interior Architecture and Design program, especially when assessing creative portfolios. Unlike traditional assessment methods that rely on marking schemes or numeric grades-often ill-suited for subjective, open-ended work-Comparative Judgement harnesses the collective expertise of multiple assessors to make direct side-by-side comparisons between candidate submissions. This results in several key benefits:
In summary, Comparative Judgement via RM Compare delivers a more fair, valid, reliable, and efficient way to allocate places in creative, competitive programmes by focusing on authentic, expert-led comparison rather than rigid marking schemes.
Candidates were asked to submit multi-page design portfolio including a series of projects inspired by Eero Saarinen’s TWA flight centre at New York’s JFK airport, widely regarded as a masterpiece of modern architecture.
The design brief sought to encourage diverse responses from candidates to allow them to fully demonstrate their approach, design skills and potential. In total, 67 portfolios were submitted which were anonymised and added to an RM Compare session.
A panel of 18 (four faculty and invited alumni) judges was added to the session. The alumni group was spread across multiple states nationwide. Each judge was asked to complete around 37 judgments - you can see the judge’s view below.
When making their judgements they were asked to consider the holistic statement only, which in this case was - “Choose the portfolio that shows the best overall qualifications into the USU Interior Architecture & Design program.”
In total 646 judgements were made with each portfolio being seen 19 times.
The average decision time was 76 seconds with all judges completing their 37 allocated judgements in under an hour (so 18 hours total work).
We can see from The rank order (Figure 2) how the session achieved very high levels of reliability (0.83). Interestingly, we can see that in this case a stable, reliable rank was achieved after each portfolio had been seen around 12 times (round 12 = reliability 0.83) which could further reduce workload in future sessions.
In the Judge misfit chart (Figure 3), we can see how there was a very high level of agreement between the judges on the quality of the portfolios presented. That is to say, there was no judge who had a radically different perception of quality.
In the Item misfit chart (Figure 4), we can see that all of the items were presented in a way that made them comparable by the judges – that is to say there were none that were ambiguous or presented in a way that made them hard to compare.
Increased reliability and objectivity |
The Session reliability of 0.87 was extremely high, and much better than could be expected from traditional assessment methods. Noise and bias were dramatically reduced. | ||
Greater validity and breadth |
Candidates were able to express themselves with a high degree of freedom and were able to submit more authentic portfolios as a result, which better illustrated their strengths. | ||
Efficiency and scalability |
The total time of 18 hours for 67 portfolios (16 minutes per portfolio), with no need for additional standardisation meetings, was considerably more efficient that traditional approaches. |
||
Enhanced feedback and engagement |
Judges values to opportunity to look across the breadth of submissions. The process itself helped to develop tacit knowledge around quality (what good looks like), increasing confidence. |
The Department of Art and Design uses comparative assessment because it is a fairer and more trustworthy way to judge creative portfolios. Instead of relying on a single marker or rigid criteria, multiple expert assessors compare each candidate’s work directly with others, anonymously and several times. This means decisions reflect a broad expert consensus rather than one person’s opinion, reducing bias and giving every applicant an equal chance whatever their style or background. Comparative assessment ensures true creative talent stands out, so admissions decisions can be made confidently and transparently.
Adaptive Comparative Judgement with RM Compare has firmly established itself as the best method to fairly assess and select applicants into the Interior Design and Architecture course at the Utah State University. Such has been its success that Professor Darrin Brooks, the course leader, is now sharing the benefits of the tool deeper into the university community and beyond.