Collaborative use of RGB and thermal imaging for remote breathing rate measurement under realistic conditions

Recent studies in computer vision show that, due to inhalation and exhalation, pixel variations can be captured by RGB-thermal videos to estimate the breathing rate (BR). In the last few years, considerable progress has been made to BR contactless monitoring, however, still many issues remain open. The performances of those methods significantly degrade under challenging conditions, specifically when subjects’ spontaneous movements, facial expressions and illumination variations are involved. In this paper, we propose a dynamic time warping-based optimization framework to automatically and accurately select regions that are most useful for robust BR estimation irrespective of these interfering factors. A signal quality index called Respiratory Signals Quality Index based on dynamic time warping (RSQI_dtw) is empirically developed. In order to address the undetected problems of facial areas due to large movements, two different methods based on face tracking or motion detection by RGB-thermal images are used to acquire the respiratory signals, and BRs are measured by the re-concatenated signal. In our framework, a time-domain processing procedure is further proposed, which contains de-trend by recursive least squares (RLS) algorithm, normalization and de-noise by band-pass filter for extracting mainstream signals. The results of validation experiments conducted with 36 subjects suggest that the proposed approach outperforms state-of-the-art BR estimation methods significantly under real-world conditions. Our study can be utilized to assist medical staff in diagnosis and treatment by remote and accurate respiratory rate detection and reduce close contact between medical staff and patients to a certain extent.