{"id":17874833,"url":"https://github.com/philipphager/ultr-reproducibility","last_synced_at":"2025-08-20T00:37:23.726Z","repository":{"id":221564586,"uuid":"712004663","full_name":"philipphager/ultr-reproducibility","owner":"philipphager","description":"SIGIR 2024 - Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset","archived":false,"fork":false,"pushed_at":"2024-05-01T14:06:58.000Z","size":628,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-04T01:32:08.691Z","etag":null,"topics":["click-model","learning-to-rank","mono-bert","unbiased-learning-to-rank"],"latest_commit_sha":null,"homepage":"https://dl.acm.org/doi/10.1145/3626772.3657892","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipphager.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-30T15:50:13.000Z","updated_at":"2024-09-13T19:02:08.000Z","dependencies_parsed_at":"2024-05-01T15:45:27.848Z","dependency_job_id":null,"html_url":"https://github.com/philipphager/ultr-reproducibility","commit_stats":null,"previous_names":["philipphager/ultr-reproducibility"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/philipphager/ultr-reproducibility","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fultr-reproducibility","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fultr-reproducibility/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fultr-reproducibility/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fultr-reproducibility/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipphager","download_url":"https://codeload.github.com/philipphager/ultr-reproducibility/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fultr-reproducibility/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271246965,"owners_count":24725901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-19T02:00:09.176Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["click-model","learning-to-rank","mono-bert","unbiased-learning-to-rank"],"created_at":"2024-10-28T11:12:06.944Z","updated_at":"2025-08-20T00:37:23.693Z","avatar_url":"https://github.com/philipphager.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Baidu-ULTR Reproducibility\nMain repository for the paper `Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset` published at the SIGIR 2024 reproducibility track.\n\n## Overview\n### Repositories\n| Dataset                   | Description |\n|---------------------------|-------------------------|\n| [Baidu-ULTR reprocibility](https://github.com/philipphager/ultr-reproducibility/edit/main/README.md) | **This repository**, containing the code to tune, train, and evaluate all reranking methods including reference implementations of [ULTR methods in Jax and Rax](https://github.com/philipphager/ultr-reproducibility/blob/main/src/loss.py). |\n| [Baidu-ULTR MonoBERT models](https://github.com/philipphager/baidu-bert-model) | Repository containing the code to train flax-based MonoBERT models from scratch (optionally with ULTR). |\n| [Reranking datasets](https://github.com/philipphager/baidu-ultr) | Code to preprocess and publish the two reranking datasets to Huggingface (see below). |\n| [ULTR bias toolkit](https://github.com/philipphager/ultr-bias-toolkit) | Reference implementation of Intervention Harvesting methods. RegressionEM as used in our work was implemented in this repository. |\n\n### Datasets\n| Dataset                   | Description |\n|---------------------------|-------------------------|\n| [Language modeling dataset](https://huggingface.co/datasets/philipphager/baidu-ultr-pretrain/tree/main) | Subset of the original Baidu-ULTR used in our work to train and evaluate MonoBERT cross-encoders. |\n| [Reranking dataset (Baidu BERT)](https://huggingface.co/datasets/philipphager/baidu-ultr_baidu-mlm-ctr) | The first four partition of Baidu-ULTR with query-document embeddings produced by the official MonoBERT cross-encoder released by Baidu plus additional LTR features computed by us. |\n| [Reranking dataset (our BERT)](https://huggingface.co/datasets/philipphager/baidu-ultr_uva-mlm-ctr) | The first four partition of Baidu-ULTR with query-document embeddings produced by our naive MonoBERT cross-encoder and additional LTR features. |\n\n### Hyperparameters\n#### Base Language Models\nWe list all hyperparameters used to train our models [here](https://github.com/philipphager/ultr-reproducibility/tree/main/config/hyperparameter).\n\n#### Reranking models\nWe train small feed-forward networks with ReLU activation on fixed query-document embeddings and LTR vectors to compare ULTR objectives. We tune the model architecture per dataset and lr and dropout regularization per method/dataset combination. We list all final hyperparameters of the reranking models under [`config/hyperparameters/`](https://github.com/philipphager/ultr-reproducibility/tree/main/config/hyperparameter).\n\n### Position Bias\n\u003cp align=\"center\"\u003e\n \u003cimg src='https://github.com/philipphager/ultr-reproducibility/assets/9155371/c1bb9d2d-9c82-4c3f-a09d-dae7ce10c8f4' width='600'\u003e\n\u003c/p\u003e\n\nPosition bias as estimated with the [ULTR Bias Toolkit](https://github.com/philipphager/ultr-bias-toolkit) on partitions 1-3 of Baidu-ULTR.\n\n### Installation\n* If [Poetry](https://python-poetry.org/docs/cli/) is available, you can install all dependencies by running: `poetry install`.\n* If [Mamba](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html) is available, you can use `mamba env create --file environment.yaml` which supports CUDA 11.8.\n\n### Usage\nSelect a dataset `[\"baidu\", \"uva\", \"ltr\"]` and model/loss combination, e.g.,: `[\"naive-pointwise\", \"regression-em\", \"dla\", \"pairwise-debias\"]` and run:\n```bash\n python main.py data=baidu model=naive-pointwise\n```\n\n### Reference\n```\n@inproceedings{Hager2024BaiduULTR,\n  author = {Philipp Hager and Romain Deffayet and Jean-Michel Renders and Onno Zoeter and Maarten de Rijke},\n  title = {Unbiased Learning to Rank Meets Reality: Lessons from Baidu’s Large-Scale Search Dataset},\n  booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`24)},\n  organization = {ACM},\n  year = {2024},\n}\n```\n\n### License\nThis repository uses the [MIT License](https://github.com/philipphager/ultr-reproducibility/blob/main/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fultr-reproducibility","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipphager%2Fultr-reproducibility","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fultr-reproducibility/lists"}