RMIT University
Browse

Does selective search benefit from WAND optimization?

conference contribution
posted on 2024-10-31, 19:25 authored by Yubin Kim, Jamie Callan, Shane CulpepperShane Culpepper, Alistair Moffat
Selective search is a distributed retrieval technique that reduces the computational cost of large-scale information retrieval. By partitioning the collection into topical shards, and using a resource selection algorithm to identify a subset of shards to search, selective search allows retrieval effectiveness to be maintained while evaluating fewer postings, often resulting in 90+% reductions in querying cost. However, there has been only limited attention given to the interaction between dynamic pruning algorithms and topical index shards. We demonstrate that the WAND dynamic pruning algorithm is more effective on topical index shards than it is on randomly-organized index shards, and that the savings generated by selective search and WAND are additive. We also compare two methods for applying WAND to topical shards: searching each shard with a separate top-k heap and threshold; and sequentially passing a shared top-k heap and threshold from one shard to the next, in the order established by a resource selection mechanism. Separate top-k heaps provide low query latency, whereas a shared top-k heap provides higher throughput.

Funding

Beyond keyword search for ranked document retrieval

Australian Research Council

Find out more...

Efficient and effective ad-hoc search using structured and unstructured geospatial information

Australian Research Council

Find out more...

History

Related Materials

  1. 1.
    DOI - Is published in 10.1007/978-3-319-30671-1_11
  2. 2.
    ISBN - Is published in 9783319306704 (urn:isbn:9783319306704)

Start page

145

End page

158

Total pages

14

Outlet

Proceedings of the 38th European Conference on IR Research, ECIR 2016

Editors

N. Ferro, F. Crestani, M. -F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff and G. Silvello

Name of conference

Advances in Information Retrieval

Publisher

Springer

Place published

Switzerland

Start date

2016-03-20

End date

2016-03-23

Language

English

Copyright

© Springer 2016

Former Identifier

2006062217

Esploro creation date

2020-06-22

Fedora creation date

2016-06-07

Usage metrics

    Scholarly Works

    Keywords

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC