Agree to disagree: on labelling helpful app reviews

Mobile apps designers seek to prioritise and refine app features so as to optimise user experience across the ensemble of possible situations and contexts in which the app is used. App reviews---some helpful, others irrelevant---can be analysed for feedback on this user experience. However, few studies have specifically examined the helpfulness of app reviews. In this paper, we surveyed users and developers to rate 167 reviews for helpfulness, obtaining a total of 2,558 helpfulness ratings captured on a 5 point Likert scale. We found only slight agreement (nominal Krippendorff's alpha = 0.039) between participants on the helpfulness of reviews. Differences between reviews become evident when we summarise all the helpfulness ratings per review. We conclude that the disagreement among users limits the potential of mobile app review recommender systems.