基于 OpenGL ES 的移动端实时视频超分辨率显示

Current mainstream video super-resolution algorithms are primarily applied in business scenarios such as server-side or offline video conversion. When deployed on mobile devices, challenges such as complex computations and slow inference speeds are observed. Although these mainstream super-resolution algorithms can satisfy the accuracy requirements of image quality, satisfying the performance requirements in terms of processing time is challenging, which affects the practical application of the algorithms, particularly in Real-Time audio and video Communication (RTC) business scenarios. This paper proposes a real-time video Super-Resolution technology based on OpenGL ES (OGSR) and Convolutional Neural Network (CNN) improvement and optimization. First, by using grouped convolution and channel obfuscation, the neural network model is optimized without significantly reducing the quality of the super-resolution image, which exponentially reduces the computational cost of forward inference. Subsequently, the OpenGL ES graphics acceleration interface is used to outline the model parameters and channel data into the fastest sampled texture data, which is uploaded to the graphics memory for parallel computing on the GPU. Finally, using the Shader of the GPU, the channel index and model parameter index are calculated in reverse by rendering pixel coordinates to achieve the core module of the super-resolution algorithm, thereby achieving fine-grained concurrency at the pixel level. The experimental results show that the triple super-resolution amplification of QVGA (320×240 pixels) and nHD (640X360 pixels) resolution video frames can achieve a frame rate of 15-30 frame/s on mobile phones of various models. Moreover, the quality error of the enlarged image is within 2% of that of the standard CNN model, which meets the requirements of real-time business scenarios and significantly improves performance.