Human subjectivity and performance limits in document retrieval

摘要

Test sets for the document-retrieval task composed of human relevance judgments have been constructed that allow one to compare human performance directly with that of automatic methods and that place absolute limits on performance by any method. Current retrieval systems are found to generate only about half of the information allowed by these absolute limits. The data presented here strongly suggest that most of the improvement that could be achieved consistent with these limits can only be achieved by incorporating specific subject information into retrieval systems.