A Weighted Rank aggregation approach towards crowd opinion analysis

摘要

In crowd opinion aggregation models, the expertise of annotators plays an important role to derive the appropriate judgment. It is seen that in most of the aggregation methods annotators’ accuracy and bias are considered as two important features and based on it the priority of annotators is assigned. But instead of relying upon these limited features, the quality of annotators can be suitably exploited using rank-based features to further improve the prediction. Basically, the annotators are ranked according to various features and therefrom multiple separate rankings are produced. These rankings, if properly weighted, can lead to obtain the final aggregated ranking in a better way. In this paper, we have developed a novel weighted rank aggregation approach and applied the same on three artificially generated ranking datasets with varying noise. Moreover, the comparative effectiveness of the proposed method is demonstrated by applying it on three Amazon Mechanical Turk datasets.