Generating tasks for study of struggling search

Interactive information-seeking systems are designed to help users with their struggling during the searching for complex fact checking tasks, where a searcher may have clear information needs but experience difficulty in collecting required information. However, evaluation and comparison of such systems requires a large number of such tasks, which are difficult to collect or make up. To the best of our knowledge, there has not been a commonly used task set for evaluating struggling search of this kind. This paper proposes a convenient method to generate complex fact checking tasks. Each of the generated task has a clearly defined goal, which however takes an average searcher a significant amount of effort to reach. We conducted lab user studies to verify the feasibility of this method. The results confirmed its feasibility and efficiency.