Multimodal fusion-based high-fidelity compressed ultrafast photography

Featuring high frame rate and large sequence depth in a single shot, compressed ultrafast photography (CUP) has emerged as an outstanding tool for observing ultrafast phenomena, especially those unrepeatable or irreversible ones. However, the lower image quality in CUP due to high data compressive ratio has always been a tough issue, hampering its further applications in capturing the transient scenes with fine structural information. To overcome this disadvantage in CUP, here we report a multimodal fusion-based compressed ultrafast photography to achieve high-fidelity ultrafast imaging, termed MF-CUP. MF-CUP simultaneously records the dynamic scenes with three different imaging models, involving CUP, transient imaging and spatiotemporal integration imaging. Attributed to the joint acquisition of the dynamic scenes from different imaging models and the multimodal fusion image reconstruction algorithm enabled by untrained neural network, MF-CUP acquires the higher fidelity in both spatial and temporal domains compared with traditional CUP. Both the simulation and experimental results demonstrate that MF-CUP can effectively enhance the accuracy and quality of reconstructed images. Given this high-fidelity imaging ability of MF-CUP, it will provide a powerful tool for the detection of ultrafast dynamics with fine details.