https://github.com/philipphager/ultr-reproducibility
SIGIR 2024 - Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
https://github.com/philipphager/ultr-reproducibility
click-model learning-to-rank mono-bert unbiased-learning-to-rank
Last synced: 11 months ago
JSON representation
SIGIR 2024 - Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
- Host: GitHub
- URL: https://github.com/philipphager/ultr-reproducibility
- Owner: philipphager
- License: mit
- Created: 2023-10-30T15:50:13.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-01T14:06:58.000Z (about 2 years ago)
- Last Synced: 2025-04-04T01:32:08.691Z (over 1 year ago)
- Topics: click-model, learning-to-rank, mono-bert, unbiased-learning-to-rank
- Language: Python
- Homepage: https://dl.acm.org/doi/10.1145/3626772.3657892
- Size: 613 KB
- Stars: 7
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Baidu-ULTR Reproducibility
Main repository for the paper `Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset` published at the SIGIR 2024 reproducibility track.
## Overview
### Repositories
| Dataset | Description |
|---------------------------|-------------------------|
| [Baidu-ULTR reprocibility](https://github.com/philipphager/ultr-reproducibility/edit/main/README.md) | **This repository**, containing the code to tune, train, and evaluate all reranking methods including reference implementations of [ULTR methods in Jax and Rax](https://github.com/philipphager/ultr-reproducibility/blob/main/src/loss.py). |
| [Baidu-ULTR MonoBERT models](https://github.com/philipphager/baidu-bert-model) | Repository containing the code to train flax-based MonoBERT models from scratch (optionally with ULTR). |
| [Reranking datasets](https://github.com/philipphager/baidu-ultr) | Code to preprocess and publish the two reranking datasets to Huggingface (see below). |
| [ULTR bias toolkit](https://github.com/philipphager/ultr-bias-toolkit) | Reference implementation of Intervention Harvesting methods. RegressionEM as used in our work was implemented in this repository. |
### Datasets
| Dataset | Description |
|---------------------------|-------------------------|
| [Language modeling dataset](https://huggingface.co/datasets/philipphager/baidu-ultr-pretrain/tree/main) | Subset of the original Baidu-ULTR used in our work to train and evaluate MonoBERT cross-encoders. |
| [Reranking dataset (Baidu BERT)](https://huggingface.co/datasets/philipphager/baidu-ultr_baidu-mlm-ctr) | The first four partition of Baidu-ULTR with query-document embeddings produced by the official MonoBERT cross-encoder released by Baidu plus additional LTR features computed by us. |
| [Reranking dataset (our BERT)](https://huggingface.co/datasets/philipphager/baidu-ultr_uva-mlm-ctr) | The first four partition of Baidu-ULTR with query-document embeddings produced by our naive MonoBERT cross-encoder and additional LTR features. |
### Hyperparameters
#### Base Language Models
We list all hyperparameters used to train our models [here](https://github.com/philipphager/ultr-reproducibility/tree/main/config/hyperparameter).
#### Reranking models
We train small feed-forward networks with ReLU activation on fixed query-document embeddings and LTR vectors to compare ULTR objectives. We tune the model architecture per dataset and lr and dropout regularization per method/dataset combination. We list all final hyperparameters of the reranking models under [`config/hyperparameters/`](https://github.com/philipphager/ultr-reproducibility/tree/main/config/hyperparameter).
### Position Bias
Position bias as estimated with the [ULTR Bias Toolkit](https://github.com/philipphager/ultr-bias-toolkit) on partitions 1-3 of Baidu-ULTR.
### Installation
* If [Poetry](https://python-poetry.org/docs/cli/) is available, you can install all dependencies by running: `poetry install`.
* If [Mamba](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) is available, you can use `mamba env create --file environment.yaml` which supports CUDA 11.8.
### Usage
Select a dataset `["baidu", "uva", "ltr"]` and model/loss combination, e.g.,: `["naive-pointwise", "regression-em", "dla", "pairwise-debias"]` and run:
```bash
python main.py data=baidu model=naive-pointwise
```
### Reference
```
@inproceedings{Hager2024BaiduULTR,
author = {Philipp Hager and Romain Deffayet and Jean-Michel Renders and Onno Zoeter and Maarten de Rijke},
title = {Unbiased Learning to Rank Meets Reality: Lessons from Baidu’s Large-Scale Search Dataset},
booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`24)},
organization = {ACM},
year = {2024},
}
```
### License
This repository uses the [MIT License](https://github.com/philipphager/ultr-reproducibility/blob/main/LICENSE).