An analytic measure predicting information retrieval system performance

摘要

The performance of information retrieval, hypertext linkage, and text filtering systems may be measured by using historical data or by estimating performance using Bayesian probabilistic or artificial intelligence methods. The measurement of performance is necessary to evaluate document retrieval systems, electronic mail filters, office information systems, and, in general, retrieval from databases when the searcher has incomplete information about the characteristics of the records to be retrieved. We provide a method for estimating precision or retrieval quality without examining individual database documents. This method requires knowledge of only the query or expressed information need and a set of database parameters constant for all queries. The concepts of historic and expected precision are examined. The analytic expected precision measure is used to examine the performance of a system using relevance feedback to increase the accuracy of parameter estimates. Use of a precision-document graph instead of the commonly used precision-recall graph is examined, and several uses of the precision-document graph for a computer-human interface are suggested, including its use as a graphic aid assisting users in deciding when to stop searching. An economic model of information retrieval and the stopping problem is provided, and the conditions are examined under which the user should stop searching when either using or not using relevance feedback.