An empirical margin explanation for the effectiveness of DECORATE ensemble learning algorithm

摘要

In the past two decades, some successful ensemble learning algorithms have been proposed, typically as Bagging, AdaBoost, DECORATE, etc. Although all adopting diversity-based combination, the former two algorithms are generated by manipulating the original training set, while the latter one by augmenting the original training set using artificial training examples and has exhibited both stronger robustness to noise than AdaBoost and better resilience to missing features than Bagging and AdaBoost. To better understand the effectiveness of DECORATE, a study has already been conducted from the perspective of the bias–variance theory. However, this theory fails in explaining the effectiveness of AdaBoost which constantly improves the generalization performance by adding more base classifiers but it increases the variance. As known, the margin theory is another important means that can be used to explain the effectiveness of ensemble learning. A recent study has manifested its success in explaining the effectiveness of AdaBoost. Motivated by the margin theory, in this paper, we try to empirically analyze the effectiveness of DECORATE from this perspective by conducting experiments on 15 standard data sets, we find that the margin theory can also well explain the effectiveness of DECORATE. The consistency with the bias–variance based analysis states that the margin theory could be used not only as a more general analysis tool for the effectiveness of ensemble learning to a large extent but also as an optimization objective to design effective ensemblers.