Theoretical guarantee for crowdsourcing learning with unsure option

Pan，Yigong1; Tang，Ke2; Sun，Guangzhong1

2023-05-01

10.1016/j.patcog.2023.109316

PATTERN RECOGNITION   Impact Factor & Quartile

0031-3203

Crowdsourcing learning, in which labels are collected from multiple workers through crowdsourcing platforms, has attracted much attention during the past decade. This learning paradigm would reduce the labeling cost since crowdsourcing workers may be non-expert and hence less costly. On the other hand, crowdsourcing learning algorithms also suffer from being misled by incorrect labels introduced by imperfect workers. To control such risks, recently, it has been suggested to provide workers an additional unsure option during the labeling process. Although the benefits of the unsure option have been empirically demonstrated, theoretical analysis is still limited. In this article, a theoretical analysis of crowdsourcing learning with the unsure option is presented. Specifically, an upper bound of minimally sufficient number of crowd labels required for learning a probably approximately correct (PAC) classification model with and without the unsure option are given respectively. Next, a condition under which providing (or not providing) an unsure option to workers is derived. Then, the theoretical results are extended to guide non-identical label options (with or without unsure options) to different workers. Last, several useful applications are proposed based on theoretical results.

Crowdsourcing learning Labeling Machine learning

SCI

English

Corresponding

WOS:000963387200001

ENGINEERING

2-s2.0-85146716208

Scopus

1.State Key Laboratory of Processors (SKLP),School of Computer Science and Technology,University of Science and Technology of China (USTC),Hefei,Anhui,230026,China
2.Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation,Department of Computer Science and Engineering,Southern University of Science and Technology (SUSTech),Shenzhen,Guangdong,518055,China

Pan，Yigong,Tang，Ke,Sun，Guangzhong. Theoretical guarantee for crowdsourcing learning with unsure option[J]. PATTERN RECOGNITION,2023,137.

Pan，Yigong,Tang，Ke,&Sun，Guangzhong.(2023).Theoretical guarantee for crowdsourcing learning with unsure option.PATTERN RECOGNITION,137.

Pan，Yigong,et al."Theoretical guarantee for crowdsourcing learning with unsure option".PATTERN RECOGNITION 137(2023).