Fairness issues in a computer-based architectural licensure examination

摘要

A series of field trials were carried out to address test design and fairness issues related to the development of a computer-based performance assessment of problem solving in an architectural context. Examinees were given design problems consisting of a set of specifications and code requirements and were then expected to construct solutions on the computer using a set of provided computer tools. For the field trial, these solutions were evaluated by expert architects, but in the operational system solutions are automatically scored by the computer. Although a number of different issues were addressed in the field trials, this report focuses on fairness issues. One aspect of fairness, whether examinees with relatively little computer experience are disadvantaged on the computer-delivered examination, does not appear to be a major problem. A more complex fairness issue relates to the need to create tasks that are sufficiently comparable so that it is a matter of indifference to examinees which particular set of tasks they receive. Tasks intended to measure the same skills should be of comparable difficulty and highly correlated. For some types of design problems, both standards could be met; for other problem types, tasks were of comparable difficulty, but not highly correlated; and for still other problem types, tasks were neither of comparable difficulty nor highly correlated. Uncovering the possible reasons for these inconsistencies was a crucial step in setting the final test specifications for the operational examination.