Measuring relevance judgments

摘要

Accurate, reliable measurement of relevance is fundamental to research in information science and to the design, development, and evaluation of information systems. This article describes a study focusing on the measurement of relevance and the application of magnitude estimation, an open-ended scaling technique developed in the field of sensory psychophysics, to the task of measuring relevance judgments. The study found that magnitude estimation is highly appropriate for the measurement of relevance judgments and less influenced by potential biases than commonly used category rating scaling procedures. One biasing factor—a range context effect—was found for both magnitude estimation and category rating scales.