Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework

Meihong Wang; Yuling Sun; Jing Yang; Liang He

doi:10.1109/CSCWD.2018.8465368

Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework

Meihong Wang
, Yuling Sun
, Jing Yang
, Liang He

School of Computer Science and Technology

East China Normal University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Crowdsourcing is quite cheap and effective to get a data set labeled by multiple annotators in a short amount of time. Although there are many traditional methods focusing on quality control of crowdsourcing, little pays close attention to the intrinsic ambiguity in dataset, which is hard for workers to make the right decision. In this paper, we introduce DCRB, a collaborative crowdsourcing framework that can obtain high quality output even for ambiguous tasks. We enable users' disagreement and encourage them to provide in-depth explanations when the disagreement occurs. We use these explanations to analyze the ambiguous tasks and design collaborative crowdsourcing to improve the output. In addition, we also design 'reward brave' incentive mechanism to encourage users' valuable explanations. The experimental results show that our method significantly improves the accuracy of crowdsourcing, especially for those ambiguous crowdsourcing tasks.

Original language	English
Title of host publication	Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018
Editors	Jean-Paul Barthes, Haibin Zhu, Junzhou Luo, Weiming Shen, Jinghui Zhang, Fang Dong
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	683-688
Number of pages	6
ISBN (Print)	9781538614822
DOIs	https://doi.org/10.1109/CSCWD.2018.8465368
State	Published - 13 Sep 2018
Event	22nd IEEE International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018 - Nanjing, China Duration: 9 May 2018 → 11 May 2018

Publication series

Name	Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018

Conference

Conference	22nd IEEE International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018
Country/Territory	China
City	Nanjing
Period	9/05/18 → 11/05/18

Keywords

ambiguous tasks
crowdsourcing
disagreement
incentive mechanism
quality control

Access to Document

10.1109/CSCWD.2018.8465368

Cite this

Wang, M., Sun, Y., Yang, J., & He, L. (2018). Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework. In J.-P. Barthes, H. Zhu, J. Luo, W. Shen, J. Zhang, & F. Dong (Eds.), Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018 (pp. 683-688). Article 8465368 (Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CSCWD.2018.8465368

Wang, Meihong ; Sun, Yuling ; Yang, Jing et al. / Enabling the Disagreement among Crowds : A Collaborative Crowdsourcing Framework. Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018. editor / Jean-Paul Barthes ; Haibin Zhu ; Junzhou Luo ; Weiming Shen ; Jinghui Zhang ; Fang Dong. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 683-688 (Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018).

@inproceedings{ef35da816b474876b059f9fd1af04103,

title = "Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework",

abstract = "Crowdsourcing is quite cheap and effective to get a data set labeled by multiple annotators in a short amount of time. Although there are many traditional methods focusing on quality control of crowdsourcing, little pays close attention to the intrinsic ambiguity in dataset, which is hard for workers to make the right decision. In this paper, we introduce DCRB, a collaborative crowdsourcing framework that can obtain high quality output even for ambiguous tasks. We enable users' disagreement and encourage them to provide in-depth explanations when the disagreement occurs. We use these explanations to analyze the ambiguous tasks and design collaborative crowdsourcing to improve the output. In addition, we also design 'reward brave' incentive mechanism to encourage users' valuable explanations. The experimental results show that our method significantly improves the accuracy of crowdsourcing, especially for those ambiguous crowdsourcing tasks.",

keywords = "ambiguous tasks, crowdsourcing, disagreement, incentive mechanism, quality control",

author = "Meihong Wang and Yuling Sun and Jing Yang and Liang He",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 22nd IEEE International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018 ; Conference date: 09-05-2018 Through 11-05-2018",

year = "2018",

month = sep,

day = "13",

doi = "10.1109/CSCWD.2018.8465368",

language = "英语",

isbn = "9781538614822",

series = "Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "683--688",

editor = "Jean-Paul Barthes and Haibin Zhu and Junzhou Luo and Weiming Shen and Jinghui Zhang and Fang Dong",

booktitle = "Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018",

address = "美国",

}

Wang, M, Sun, Y, Yang, J & He, L 2018, Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework. in J-P Barthes, H Zhu, J Luo, W Shen, J Zhang & F Dong (eds), Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018., 8465368, Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018, Institute of Electrical and Electronics Engineers Inc., pp. 683-688, 22nd IEEE International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018, Nanjing, China, 9/05/18. https://doi.org/10.1109/CSCWD.2018.8465368

Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework. / Wang, Meihong; Sun, Yuling; Yang, Jing et al.
Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018. ed. / Jean-Paul Barthes; Haibin Zhu; Junzhou Luo; Weiming Shen; Jinghui Zhang; Fang Dong. Institute of Electrical and Electronics Engineers Inc., 2018. p. 683-688 8465368 (Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Enabling the Disagreement among Crowds

T2 - 22nd IEEE International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018

AU - Wang, Meihong

AU - Sun, Yuling

AU - Yang, Jing

AU - He, Liang

PY - 2018/9/13

Y1 - 2018/9/13

N2 - Crowdsourcing is quite cheap and effective to get a data set labeled by multiple annotators in a short amount of time. Although there are many traditional methods focusing on quality control of crowdsourcing, little pays close attention to the intrinsic ambiguity in dataset, which is hard for workers to make the right decision. In this paper, we introduce DCRB, a collaborative crowdsourcing framework that can obtain high quality output even for ambiguous tasks. We enable users' disagreement and encourage them to provide in-depth explanations when the disagreement occurs. We use these explanations to analyze the ambiguous tasks and design collaborative crowdsourcing to improve the output. In addition, we also design 'reward brave' incentive mechanism to encourage users' valuable explanations. The experimental results show that our method significantly improves the accuracy of crowdsourcing, especially for those ambiguous crowdsourcing tasks.

AB - Crowdsourcing is quite cheap and effective to get a data set labeled by multiple annotators in a short amount of time. Although there are many traditional methods focusing on quality control of crowdsourcing, little pays close attention to the intrinsic ambiguity in dataset, which is hard for workers to make the right decision. In this paper, we introduce DCRB, a collaborative crowdsourcing framework that can obtain high quality output even for ambiguous tasks. We enable users' disagreement and encourage them to provide in-depth explanations when the disagreement occurs. We use these explanations to analyze the ambiguous tasks and design collaborative crowdsourcing to improve the output. In addition, we also design 'reward brave' incentive mechanism to encourage users' valuable explanations. The experimental results show that our method significantly improves the accuracy of crowdsourcing, especially for those ambiguous crowdsourcing tasks.

KW - ambiguous tasks

KW - crowdsourcing

KW - disagreement

KW - incentive mechanism

KW - quality control

UR - https://www.scopus.com/pages/publications/85054401075

U2 - 10.1109/CSCWD.2018.8465368

DO - 10.1109/CSCWD.2018.8465368

M3 - 会议稿件

AN - SCOPUS:85054401075

SN - 9781538614822

T3 - Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018

SP - 683

EP - 688

BT - Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018

A2 - Barthes, Jean-Paul

A2 - Zhu, Haibin

A2 - Luo, Junzhou

A2 - Shen, Weiming

A2 - Zhang, Jinghui

A2 - Dong, Fang

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 9 May 2018 through 11 May 2018

ER -

Wang M, Sun Y, Yang J, He L. Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework. In Barthes JP, Zhu H, Luo J, Shen W, Zhang J, Dong F, editors, Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 683-688. 8465368. (Proceedings of the 2018 IEEE 22nd International Conference on Computer Supported Cooperative Work in Design, CSCWD 2018). doi: 10.1109/CSCWD.2018.8465368

Enabling the Disagreement among Crowds: A Collaborative Crowdsourcing Framework

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this