Characterizing and Finding System Setting-Related Defects in Android Apps

Android, the most popular mobile system, offers a number of user-configurable system settings (e.g., network, location, and permission) for controlling devices and apps. Even popular, well-tested apps may fail to properly adapt their behaviors to diverse setting changes, thus frustrating their users. However, there exists no effort to systematically investigate such defects. To this end, we conduct the first large-scale empirical study to understand and characterize these system setting-related defects (in short as 'setting defects'), which reside in apps and are triggered by system setting changes. We devote substantial manual effort (over four person-months) to analyze 1,074 setting defects from 180 popular apps on GitHub. We investigate the impact, root causes, and consequences of these setting defects and their correlations. We find that (1) setting defects have a wide impact on apps' correctness with diverse root causes, (2) the majority of these defects (≈70.7%) cause non-crashing (logic) failures, and (3) some correlations exist between the setting categories, root causes, and consequences. Motivated and informed by these findings, we propose two bug-finding techniques that can synergistically detect setting defects from both the GUI and code levels. Specifically, at the GUI level, we design and introduce setting-wise metamorphic fuzzing, the first automated dynamic testing technique to detect setting defects (causing crash and non-crashing failures, respectively) for Android apps. We implement this technique as an end-to-end, automated GUI testing tool named SetDroid. At the code level, we distill two major fault patterns and implement a static analysis tool named SetChecker to identify potential setting defects. We evaluate SetDroid and SetChecker on 26 popular, open-source Android apps, and they find 48 unique, previously-unknown setting defects. To date, 35 have been confirmed and 21 have been fixed by app developers. We also apply SetDroid and SetChecker on five highly popular industrial apps, namely WeChat, QQMail, TikTok, CapCut, and AlipayHK, all of which each have billions of monthly active users. SetDroid successfully detects 17 previously unknown setting defects in these apps' latest releases, and all defects have been confirmed and fixed by the app vendors. After that, we collaborate with ByteDance and deploy these two bug-finding techniques internally to stress-test TikTok, one of its major app products. Within a two-month testing campaign, SetDroid successfully finds 53 setting defects, and SetChecker finds 22 ones. So far, 59 have been confirmed and 31 have been fixed. All these defects escaped from prior developer testing. By now, SetDroid has been integrated into ByteDance's official app testing infrastructure named FastBot for daily testing. These results demonstrate the strong effectiveness and practicality of our proposed techniques.