Regression Algorithm Based on Self-Distillation and Ensemble Learning

Low-dimensional feature regression is a common problem in various disciplines, such as chemistry, kinetics, and medicine, etc. Most common solutions are based on machine learning, but as deep learning evolves, there is room for performance improvements. A few researchers have proposed deep learning-based solutions such as ResidualNet, GrowNet and EnsembleNet. The latter two methods are both boost methods, which are more suitable for shallow network, and the model performance is basically determined by the first model, with limited effect of subsequent boosting steps. We propose a method based on self-distillation and bagging, which selects the well-performing base model and distills several student models by appropriate regression distillation algorithm. Finally, the output of these student models is averaged as the final result. This integration method can be applied to any form of network. The method achieves good results in the CASP dataset, and the R2(Coefficient of Determination) of the model is improved from (0.65) to (0.70) in comparison with the best base model ResidualNet.