When Handcrafted Filter Meets CNN: A Lightweight Conv-Filter Mixer Network for Efficient Image Super-Resolution

Due to their powerful representational ability, convolutional neural networks (CNN) have achieved great success in image super-resolution (SR). In the trained SR models such as EDSR, we observe that partial convolutions exhibit analogous characteristics compared to handcrafted filters which avoid parameters with much less computational cost. This inspires us to substitute the handcrafted filters for the learnable convolutions in the SR models, such that the network complexity and the computational overhead are significantly reduced. In this study, we propose a novel lightweight SR network dubbed as Conv-Filter Mixer (CFM). Specifically, our CFM encapsulates three kinds of computations: learnable convolution, integrated filter unit (IFU), and identity mapping. Among them, IFU consists of diverse handcrafted filters to efficiently extract primitive representations in a non-parametric manner, making the limited parameterized components of lightweight networks focus on learning abstract and intricate features. To further improve efficiency, we introduce channel splitting and shuffling structures to mix the features produced by heterogeneous components efficiently. Extensive experiments demonstrate that our CFM achieves state-of-the-art performance with fewer parameters and computational costs.