Statistical comparisons of non-deterministic IR systems using two dimensional variance

作者：

Highlights：

• We propose methods to compare non-deterministic IR systems.

• We show pitfalls in using standard significance tests to compare such systems.

• We verify the applicability of proposed methods using simulations and a case study.

• We show how to compare a non-deterministic IR system for equivalent effectiveness.

摘要

•We propose methods to compare non-deterministic IR systems.•We show pitfalls in using standard significance tests to compare such systems.•We verify the applicability of proposed methods using simulations and a case study.•We show how to compare a non-deterministic IR system for equivalent effectiveness.

论文关键词：Information retrieval evaluation,Non-determinism,Randomization,Statistical analysis,Distributed IR,Personalised web search

论文评审过程：Received 3 November 2014, Revised 1 June 2015, Accepted 8 June 2015, Available online 25 June 2015, Version of Record 25 June 2015.

论文官网地址：https://doi.org/10.1016/j.ipm.2015.06.005