The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing Tasks

Rule-verification is essential for the construction of high-quality knowledge bases. Compared with the traditional approach of inviting experts with domain knowledge to accomplish the rule-verification task, crowdsourcing is a more efficient way, which could save time and effort. However, the current presentation mode of rules relies primarily on predicate logic. On one hand, predicate logic is rather abstract and complex, making it difficult for ordinary users without domain knowledge to comprehend. On the other hand, the singular presentation mode lacks personalization, failing to provide targeted assistance to different users. In this work, to facilitate users’ understanding of rules, we are motivated to design three explainable interfaces by incorporating the information presentation modes including instance (INS), natural language (NL), and knowledge graph (KG). Through our first-round quasi-experimental study, we find that NL and KG interfaces are preferred by users in general for the rule-verification crowdsourcing tasks. More notably, we identify the important role of users’ personality, cognitive style, and tasks’ difficulty level in influencing users’ perception of the interface design. For instance, among those who score low on (Formula presented.) verbalizers perceive NL interface more useful than visualizers. In addition, users are more likely to use KG interface when dealing with some complicated tasks. Inspired by our findings, we further develop a personalized interface recommendation model with the consideration of individual-level factors and tasks’ features. According to the results of both modeling comparison and the second-round user evaluation, we observe that recommending personalized interface based on our designed model could not only enhance users’ subjective perception like perceived usefulness and satisfaction but also improve their task performance.