A multinomial selection procedure for evaluating pattern recognition algorithms

摘要

This paper introduces a multinomial selection problem (MSP) procedure as an alternative to classification accuracy and receiver operating characteristic analysis for evaluating competing pattern recognition algorithms. This new application of MSP demonstrates increased differentiation power over traditional classifier evaluation methods when applied to three “toy” problems of varying difficulty. The MSP procedure is also used to compare the performance of statistical classifiers and artificial neural networks on three real-world classification problems. The results provide confidence in the MSP procedure as a useful tool in distinguishing between competing classifiers and providing insights on the strength of conviction of a classifier.