https://github.com/tonellotto/terrier-micro
An efficient layer to perform query processing on top of Terrier
https://github.com/tonellotto/terrier-micro
efficiency information-retrieval query-processing terrier
Last synced: 1 day ago
JSON representation
An efficient layer to perform query processing on top of Terrier
- Host: GitHub
- URL: https://github.com/tonellotto/terrier-micro
- Owner: tonellotto
- License: lgpl-3.0
- Created: 2019-04-10T14:13:13.000Z (about 7 years ago)
- Default Branch: 1.5.3
- Last Pushed: 2023-06-14T22:29:32.000Z (about 3 years ago)
- Last Synced: 2025-10-29T11:43:13.449Z (8 months ago)
- Topics: efficiency, information-retrieval, query-processing, terrier
- Language: Java
- Size: 2.46 MB
- Stars: 2
- Watchers: 3
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: COPYING
Awesome Lists containing this project
README
# Terrier Micro
[](https://travis-ci.org/tonellotto/terrier-micro)
[](https://www.codacy.com?utm_source=github.com&utm_medium=referral&utm_content=tonellotto/terrier-micro&utm_campaign=Badge_Grade)
[](https://www.gnu.org/licenses/lgpl-3.0)
This project provides a lightweight implementation of some query processing strategies built on top of Terrier 5. It re-implements the query processing pipeline of the Terrier search engine, removing all unnecessary features such as document score modifiers, multiple weighting models, etc.
If you use this package to conduct search or experimentation, whether be it a research paper, dissertation, article, poster, presentation, or documentation, please cite the following paper:
@article{fnt,
author = {Tonellotto, Nicola and Macdonald, Craig and Ounis, Iadh},
issn = {1554-0669},
journal = {Foundations and Trends in Information Retrieval},
number = {4--5},
pages = {319--492},
title = {Efficient Query Processing for Scalable Web Search},
volume = {12},
year = {2018}
}
This package is [free software](http://www.gnu.org/philosophy/free-sw.html) distributed under the [GNU Lesser General Public License](http://www.gnu.org/copyleft/lesser.html).
## Pre-requisites
[Elias-Fano compression for Terrier](https://github.com/tonellotto/terrier-ef) is required for testing purposes or if you plan to use it in your experiments, but it is not explicitly required for using the Terrier Micro package.
To install the Elias-Fano compression for Terrier package (version 1.5.1) on your local machine, please run the following commands.
```bash
git clone https://github.com/tonellotto/terrier-ef
cd terrier-ef
git checkout 1.5.1
mvn install appassembler:assemble
```
## Usage
If not already available, e.g. from Maven Central, you should git clone and install Terrier Micro (version 1.5.1):
```bash
git clone https://github.com/tonellotto/terrier-micro
cd terrier-micro
git checkout 1.5.1
mvn install appassembler:assemble
```
The main script to perform batch query processing is the [retrieve](./docs/retrieve.md) tool.
If you want to use all available processors on your machine to perform batch query processing, use the [parallel retrieve](./docs/parallel_retrieve.md) tool.
Two other scripts are provided, to support advanced query processing strategies: the [ms-generate](./docs/ms-gen.md) and [bmw-generate](./docs/bmw-gen.md) tools.
## Python
The [python](./python) folder repo holds static copies of notebooks for learning to use Terrier Micro (Java). The notebooks in this repo are sync'ed (by hand) with notebooks in Colab. For convenience, there is a small pre-built index, available to download [here](https://drive.google.com/open?id=1si4B1McN4u7a_kF8fTexsnStc7qWrlBb).
* _Terrier Micro demo on Robust 2004_: [local](./python/terrier_robust04_demo.ipynb) and [colab](https://colab.research.google.com/drive/1M2UkPA2dWrpFx4zZO7beNl5bG6BA9-mR) notebooks.
## Credits
Developed by Nicola Tonellotto, ISTI-CNR. Contributions by Craig Macdonald, University of Glasgow, and Matteo Catena, ISTI-CNR.