{"id":17874845,"url":"https://github.com/philipphager/sigir-cmip","last_synced_at":"2025-08-16T19:06:15.235Z","repository":{"id":156129536,"uuid":"527740017","full_name":"philipphager/sigir-cmip","owner":"philipphager","description":" SIGIR 2023 - An Offline Metric for the Debiasedness of Click Models","archived":false,"fork":false,"pushed_at":"2023-04-19T10:21:02.000Z","size":291,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-08T14:23:36.649Z","etag":null,"topics":["click-model","conditional-mu","learning-to-rank","unbiased-learning-to-rank"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2304.09560","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipphager.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-22T21:37:50.000Z","updated_at":"2024-05-01T13:25:50.000Z","dependencies_parsed_at":"2023-07-15T17:18:04.703Z","dependency_job_id":null,"html_url":"https://github.com/philipphager/sigir-cmip","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fsigir-cmip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fsigir-cmip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fsigir-cmip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fsigir-cmip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipphager","download_url":"https://codeload.github.com/philipphager/sigir-cmip/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246911467,"owners_count":20853658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["click-model","conditional-mu","learning-to-rank","unbiased-learning-to-rank"],"created_at":"2024-10-28T11:12:09.960Z","updated_at":"2025-04-03T00:14:38.403Z","avatar_url":"https://github.com/philipphager.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# An Offline Metric for the Debiasedness of Click Models\nSource code for the SIGIR 2023 paper `An Offline Metric for the Debiasedness of Click Models`. For a standalone implementation of the proposed CMIP metric, [see this repository](https://github.com/philipphager/CMIP).\n\n## Setup\n### 1. Virtual environment with Conda\n\nDependency management\n\n1. Setup [conda](https://www.anaconda.com/)\n   / [miniconda](https://docs.conda.io/en/latest/miniconda.html) on your device.\n2. Create environment and install dependencies: `conda env create -f environment.yaml`\n3. Activating environment: `conda activate sigir-cmip`\n\n### 2. Experiments\n\nAll experimental runs are documented inside the `scripts/` directory. To execute an experiment: \n\n1. Make the scripts executable: `chmod +x ./scripts/*`\n2. Run a script locally use, e.g.: `./scripts/graded-pbm.sh`\n3. To execute a script on a [SLURM cluster](https://slurm.schedmd.com/documentation.html) add: `./scripts/graded-pbm.sh +launcher=slurm`\n4. You can configure the SLURM resources in: `config/launcher/slurm.yaml`\n\nDocumentation of each experiment can be found inside the scripts.\n\n### 3. Pre-commit\n\nAutomatically format and lint modified files in commit.\n\n1. Make sure you activate your environment\n2. Initialize pre-commit: `pre-commit install`\n3. (Optional) Run on checks against all files (not just\n   changed): `pre-commit run --all-files`\n\n### 4. Datasets\n\nThe project automatically downloads the dataset used in this work to: `~/.ltr_datasets`.\n\n1. You can change the directory by modifying the `base_dir` variable in: `config/env.yaml`\n2. To avoid downloading datasets, you can directly place the original .zip file into\n   the `download` subdirectory, e.g.:\n   `~/.ltr_datasets/download/MSLR-WEB30K.zip`\n\n### 5. Logging\n\nLog metrics with [Weights \u0026 Biases](https://github.com/wandb/wandb).\n\n1. Make sure you activate your environment\n2. Log into Weights \u0026 Biases before your first run: `wandb login`\n3. Add your wandb entity and project name inside `config/config.yaml`\n\n### 6. Visualizations\n\nAll code for plotting is in the `notebooks/` directory. The code requires the results to be logged to Weights \u0026 Biases (Section 5).\n\n1. Make sure you activate your environment\n2. Start a jupyterlab server: `python -m jupyterlab`\n3. Add wandb parameters in notebook header and run all cells\n\n## Hyperparameters and configuration\nYou can find a list of model parameters and training configurations under `config/`.\n\n## Reference\n```\n@inproceedings{Deffayet2023Debiasedness,\n  author = {Romain Deffayet and Philipp Hager and Jean-Michel Renders and Maarten de Rijke},\n  title = {An Offline Metric for the Debiasedness of Click Models},\n  booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`23)},\n  organization = {ACM},\n  year = {2023},\n}\n```\n\n## License\nThis project uses the [MIT license](https://github.com/philipphager/sigir-cmip/blob/main/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fsigir-cmip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipphager%2Fsigir-cmip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fsigir-cmip/lists"}