Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework

Crowdsourcing is quite cheap and effective to get a data set labeled by multiple annotators in a short amount of time. Although there are many traditional methods focusing on quality control of crowdsourcing, little pays close attention to the intrinsic ambiguity in dataset, which is hard for workers to make the right decision. In this paper, we introduce DCRB, a collaborative crowdsourcing framework that can obtain high quality output even for ambiguous tasks. We enable users' disagreement and encourage them to provide in-depth explanations when the disagreement occurs. We use these explanations to analyze the ambiguous tasks and design collaborative crowdsourcing to improve the output. In addition, we also design 'reward brave' incentive mechanism to encourage users' valuable explanations. The experimental results show that our method significantly improves the accuracy of crowdsourcing, especially for those ambiguous crowdsourcing tasks.