Kernelized support vector machine with deep learning: An efficient approach for extreme multiclass dataset

Classification with thousands of classes and a large number of features is often computationally intractable. The presence of irrelevant features can decrease the classification performance and increase the computational complexity of classification. Moreover, classification with many classes (thousands or more) often leads to class-confusability, and the more confusable classes increase the training error. A Robust classification model for use in high-dimensional data with a large number of classes (e.g. k ≥ 10⁴) requires a prudent combination of a feature extractor and a classifier. While support vector machines with the appropriate kernel is promising regarding producing decision from the well-behaved features, but often present a negative repercussion at modeling the large scale data and more especially, the ultra-large number of classes. Architectures such as deep belief networks exhibit an impressive power to learn and collect robust features. In this paper, we present a hybrid system where a supervised deep belief network is trained to select generic features, and a kernel-based SVM is trained from the features that learned by the DBN. In our hybrid model, we substituted linear kernel for nonlinear ones (due to a large number of classes) without loss of accuracy, and gives significant gains on real world dataset involving 20,000 to 65,000 classes compared to state of the art approaches.