{"id":17874830,"url":"https://github.com/philipphager/cmip","last_synced_at":"2025-07-06T00:34:14.534Z","repository":{"id":156129503,"uuid":"594742019","full_name":"philipphager/cmip","owner":"philipphager","description":"An Offline Metric for the Debiasedness of Click Models","archived":false,"fork":false,"pushed_at":"2023-04-17T14:30:43.000Z","size":13,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-10T23:39:40.198Z","etag":null,"topics":["click-model","conditional-mutual-information","learning-to-rank","unbiased-learning-to-rank"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/philipphager.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-29T14:05:18.000Z","updated_at":"2023-04-20T14:03:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"dfdc9d25-d8a3-4e4d-80cb-6ccf18f9d291","html_url":"https://github.com/philipphager/cmip","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fcmip","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fcmip/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fcmip/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/philipphager%2Fcmip/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/philipphager","download_url":"https://codeload.github.com/philipphager/cmip/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246911467,"owners_count":20853658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["click-model","conditional-mutual-information","learning-to-rank","unbiased-learning-to-rank"],"created_at":"2024-10-28T11:12:05.970Z","updated_at":"2025-04-03T00:14:35.504Z","avatar_url":"https://github.com/philipphager.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CMIP - Conditional Mutual Information with the logging Policy\nCMIP implementation from the 2023 SIGIR paper: `An Offline Metric for the Debiasedness of Click Models`.\n\nThe metric quantifies the mutual information between a new click model policy and the production system that collected the train dataset (logging policy), conditional on human relevance judgments. CMIP quantifies the degree of debiasedness (see paper for details). A policy is said to be debiased w.r.t. its logging policy with a `cmip \u003c= 0`.  \n\n## Example\n```Python\nimport numpy as np\n\nn_queries = 1_000\nn_results = 25\n\n# Human relevance annotations per query-document pair\ny_true = np.random.randint(5, size=(n_queries, n_results))\n# Relevance scores of the logging policy\ny_logging_policy = y_true + np.random.randn(n_queries, n_results)\n# Relevance scores of a new policy (in this case, strongly dependent on logging policy) \ny_predict = y_logging_policy + np.random.randn(n_queries, n_results)\n# Number of documents per query, used for masking\nn = np.full(n_queries, n_results)\n```\n\n```Python\nfrom cmip_metric import CMIP\n\nmetric = CMIP()\nmetric(y_predict, y_logging_policy, y_true, n)\n\u003e 0.2687 # The policy predicting y_predict is not debiased w.r.t. the logging policy.\n```\n## Installation\n```\npip install cmip-metric\n```\n\n## Reference\n\n**Note: To be published at:**\n\n```\n@inproceedings{Deffayet2023Debiasedness,\n  author = {Romain Deffayet and Philipp Hager and Jean-Michel Renders and Maarten de Rijke},\n  title = {An Offline Metric for the Debiasedness of Click Models},\n  booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`23)},\n  organization = {ACM},\n  year = {2023},\n}\n```\n\n## License\nThis project uses the [MIT license](https://github.com/philipphager/CMIP/blob/main/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fcmip","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fphilipphager%2Fcmip","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fphilipphager%2Fcmip/lists"}