Arrow's Impossibility Theorem inspired subjective image quality assessment approach

A large number of subjective image quality assessment databases have been constructed in the last decade, in which the Mean Opinion Score (MOS) (with single or double stimulus), and Paired Comparison (PC) are two dominant approaches for collecting the ground truth quality ratings and usually up to 15 or more subjects are needed for each image. In this paper, we show the fact that there is a potential “dictatorship” risk of using such averaging-over-multiple-rating type of method. Using Arrow's Impossibility Theorem (AIT), we prove that meeting of the unanimity and independence of irrelevant alternatives (IIA) will generate a “pivotal subject”, who in fact determines the final rank of image quality. We also prove that no an ideal democratic approach to synthesize the whole opinions of subjects. Therefore, we advocate to recruit a small number of experts (a.k.a the “golden eyes”) for subjective viewing tests. In order to verify the reliability of our proposal, experiments on two different databases conducting on the general distorted images and professional images (here is Terahertz security image) are performed. In each experiment, the raw scores of images are subjectively assigned by at least 15 inexperienced viewers and 3 experts, and meanwhile the MOS or difference mean opinion score (DMOS) are obtained. Afterwards, the correlation of the scores rated by naive subjects and experts is analyzed. For general image experiment, it is revealed that DMOS of inexperience viewers are highly related to DMOS of experts based on six effective evaluation metrics. In professional image experiment, the preferences of experts also maintain favourable relevance with the opinions of inexperienced viewers in overall quality of THz image. Moreover, considering the quality assessments of illegal substance regions in THz images, the experts have higher accuracy than the inexperienced observers. In conclusion, the results of two validation experiments verify that a small number of experts are more suitable for assessing the perceptual quality of images, which can reduce cost and simplify procedure of creating databases.