Deakin University
Browse

Evaluating non-deterministic retrieval systems

Version 2 2024-06-03, 18:32
Version 1 2015-03-10, 14:39
conference contribution
posted on 2014-01-01, 00:00 authored by G K Jayasinghe, W Webber, M Sanderson, Lasitha DharmasenaLasitha Dharmasena, J S Culpepper
The use of sampling, randomized algorithms, or training based on the unpredictable inputs of users in Information Retrieval often leads to non-deterministic outputs. Evaluating the effectiveness of systems incorporating these methods can be challenging since each run may produce different effectiveness scores. Current IR evaluation techniques do not address this problem. Using the context of distributed information retrieval as a case study for our investigation, we propose a solution based on multivariate linear modeling. We show that the approach provides a consistent and reliable method to compare the effectiveness of non-deterministic IR algorithms, and explain how statistics can safely be used to show that two IR algorithms have equivalent effectiveness. Copyright 2014 ACM.

History

Event

ACM SIGIR Research and Development in Information Retrieval Conference (37th : 2014 : Gold Coast, Qld.)

Pagination

911 - 914

Publisher

ACM

Location

Gold Coast, Qld.

Place of publication

New York, NY

Start date

2014-07-06

End date

2014-07-11

ISBN-13

9781450322591

Language

eng

Publication classification

E3 Extract of paper

Copyright notice

2014, Association for Computing Machinery

Title of proceedings

Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval; SIGIR 2014

Usage metrics

    Research Publications

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC