Evaluating quality-in-use of FLOSS through analyzing user reviews

Quality-in-use (QU) is an important measure for evaluating the quality of a software system from user respective. Several approaches have been proposed to evaluate QU of FLOSS. Meanwhile, they are usually less effective, as they usually assume that sufficient, precise usage statistics can be collected, which may be impractical for evaluating many real-world FLOSS systems. This paper presents QUIndicator, a novel, fine-grained approach to evaluating QU of FLOSS using user reviews. The key idea of QUIndicator is to, for a specific FLOSS system, (1) use a topic model to cluster its user reviews into different topics, transform topics into characteristics of a QU model and compute the weight of each characteristic, (2) take review aspect as the minimum analysis unit, and also apply sentiment analysis to analyze the sentiment strength of each review aspect, and (3) match review aspects with their corresponding characteristics in QU model and evaluate the QU of the system. Wilson interval is adopted to keep fairness by punishing FLOSS systems with insufficient reviews. We have evaluated QUIndicator on ten FLOSS genre datasets. The evaluation results show that when a FLOSS system has sufficient reviews, QUIndicator can achieve a p@3 value up to 30% higher than those of baselines, and also achieves over 75% Spearman Coefficient with the ground truth. When the system has insufficient reviews, QUIndicator can achieve a p@3 value up to 25% higher than those of baselines and over 55% Spearman Coefficient with the ground truth.