Classifier learning with a new locality regularization method

摘要

It is well known that the generalization capability is one of the most important criterions to develop and evaluate a classifier for a given pattern classification problem. The localized generalization error model (RSM) recently proposed by Ng et al. [Localized generalization error and its application to RBFNN training, in: Proceedings of the International Conference on Machine Learning and Cybernetics, China, 2005; Image classification with the use of radial basis function neural networks and the minimization of the localized generalization error, Pattern Recognition 40(1) (2007) 4–18] provides a more intuitive look at the generalization error. Although RSM gives a brand-new method to promote the generalization performance, it is in nature equivalent to another type of regularization. In this paper, we first prove the essential relationship between RSM and regularization, and demonstrate that the stochastic sensitivity measure in RSM exactly corresponds to a regularizing term. Then, we develop a new generalization error bound from the regularization viewpoint, which is inspired by the proved relationship between RSM and regularization. Moreover, we derive a new regularization method, called as locality regularization (LR), from the bound. Different from the existing regularization methods which artificially and externally append the regularizing term in order to smooth the solution, LR is naturally and internally deduced from the defined expected risk functional and calculated by employing locality information. Through combining with spectral graph theory, LR introduces the local structure information of the samples into the regularizing term and further improves the generalization capability. In contrast with RSM, which is relatively sensitive to the different sampling of the samples, LR uses the discrete k-neighborhood rather than the common continuous Q-neighborhood in RSM to differentiate the relative position of different training samples automatically and avoid the complex computation of Q for various classifiers. Furthermore, LR uses the regularization parameter to control the trade-off between the training accuracy and the classifier stability. Experimental results on artificial and real world problems show that LR yields better generalization capability than both RSM and some traditional regularization methods.