Deakin University

File(s) under permanent embargo

Bee colony based worker reliability estimation algorithm in microtask crowdsourcing

conference contribution
posted on 2017-01-31, 00:00 authored by Alireza Moayedikia, K L Ong, Y L Boo, William YeohWilliam Yeoh
© 2016 IEEE. Estimation of worker reliability on microtask crowdsourcing platforms has gained attention from many researchers. On microtask platforms no worker is fully reliable for a task and it is likely that some workers are spammers, in the sense that they provide a random answer to collect the financial reward. Existence of spammers is harmful as they increase the cost of microtasking and will negatively affect the answer aggregation process. Hence, to discriminate spammers and non-spammers one needs to measure worker reliability to predict how likely that a worker put an effort in solving a task. In this paper we introduce a new reliability estimation algorithm works based on bee colony algorithm called REBECO. This algorithm relies on Gaussian process model to estimate reliability of workers dynamically. With bees that go in search of pollen, some are more successful than the others. This maps well to our problem, where some workers (i.e., bees) are more successful than other workers for a given task thus, giving rise to a reliability measure. Answer aggregation with respect to worker reliability rates has been considered as a suitable replacement for conventional majority voting. We compared REBECO with majority voting using two real world datasets. The results indicate that REBECO is able to outperform MV significantly.



Machine Learning and Applications. Conference (2016 : 15th : Anaheim, California)


713 - 717




Anaheim, California

Place of publication

Piscataway, N.J.

Start date


End date






Publication classification

E Conference publication; E1 Full written paper - refereed

Copyright notice

2016, IEEE

Title of proceedings

Proceedings - 2016 15th IEEE International Conference on Machine Learning and Applications, ICMLA 2016