Evaluating non-deterministic retrieval systems

Jayasinghe,GK, Webber,W, Sanderson,M, Dharmasena,LS and Culpepper,JS 2014, Evaluating non-deterministic retrieval systems, in Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval; SIGIR 2014, ACM, New York, NY, pp. 911-914, doi: 10.1145/2600428.2609472.

Attached Files
Name Description MIMEType Size Downloads

Title Evaluating non-deterministic retrieval systems
Author(s) Jayasinghe,GK
Webber,W
Sanderson,M
Dharmasena,LSORCID iD for Dharmasena,LS orcid.org/0000-0002-3362-4819
Culpepper,JS
Conference name ACM SIGIR Research and Development in Information Retrieval Conference (37th : 2014 : Gold Coast, Qld.)
Conference location Gold Coast, Qld.
Conference dates 6-11 Jul. 2014
Title of proceedings Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval; SIGIR 2014
Publication date 2014
Start page 911
End page 914
Publisher ACM
Place of publication New York, NY
Keyword(s) Effectiveness evaluation
Experimental design
Experimentation
Information retrieval
Measurement
Statistical analysis
Summary The use of sampling, randomized algorithms, or training based on the unpredictable inputs of users in Information Retrieval often leads to non-deterministic outputs. Evaluating the effectiveness of systems incorporating these methods can be challenging since each run may produce different effectiveness scores. Current IR evaluation techniques do not address this problem. Using the context of distributed information retrieval as a case study for our investigation, we propose a solution based on multivariate linear modeling. We show that the approach provides a consistent and reliable method to compare the effectiveness of non-deterministic IR algorithms, and explain how statistics can safely be used to show that two IR algorithms have equivalent effectiveness. Copyright 2014 ACM.
ISBN 9781450322591
Language eng
DOI 10.1145/2600428.2609472
Field of Research 080608 Information Systems Development Methodologies
Socio Economic Objective 970108 Expanding Knowledge in the Information and Computing Sciences
HERDC Research category E3 Extract of paper
ERA Research output type X Not reportable
Copyright notice ©2014, Association for Computing Machinery
Persistent URL http://hdl.handle.net/10536/DRO/DU:30070462

Document type: Conference Paper
Collection: School of Information and Business Analytics
Connect to link resolver
 
Unless expressly stated otherwise, the copyright for items in DRO is owned by the author, with all rights reserved.

Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 0 times in TR Web of Science
Scopus Citation Count Cited 3 times in Scopus
Google Scholar Search Google Scholar
Access Statistics: 58 Abstract Views, 0 File Downloads  -  Detailed Statistics
Created: Tue, 10 Mar 2015, 13:39:11 EST

Every reasonable effort has been made to ensure that permission has been obtained for items included in DRO. If you believe that your rights have been infringed by this repository, please contact drosupport@deakin.edu.au.