On sample sizes for non-matched-pair IR experiments

摘要

The problem of determining an adequate sample size, for an information retrieval experiment comparing two systems on separate samples of requests, is discussed. Minimal sample sizes under different assumptions are derived. It is concluded that such experiments are very likely to require samples of considerable size.