{"id":26696721,"url":"https://github.com/cleanlab/multiannotator-benchmarks","last_synced_at":"2025-04-13T03:28:18.663Z","repository":{"id":67076432,"uuid":"508884707","full_name":"cleanlab/multiannotator-benchmarks","owner":"cleanlab","description":"Benchmarking algorithms for assessing quality of data labeled by multiple annotators","archived":false,"fork":false,"pushed_at":"2023-02-07T05:47:50.000Z","size":18860,"stargazers_count":32,"open_issues_count":1,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T20:34:54.623Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cleanlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-30T00:32:31.000Z","updated_at":"2025-01-21T23:34:29.000Z","dependencies_parsed_at":"2023-06-09T12:45:40.006Z","dependency_job_id":null,"html_url":"https://github.com/cleanlab/multiannotator-benchmarks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Fmultiannotator-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Fmultiannotator-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Fmultiannotator-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cleanlab%2Fmultiannotator-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cleanlab","download_url":"https://codeload.github.com/cleanlab/multiannotator-benchmarks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248659182,"owners_count":21141091,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-26T20:27:31.658Z","updated_at":"2025-04-13T03:28:18.648Z","avatar_url":"https://github.com/cleanlab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking methods for classification data labeled by multiple annotators\n\nCode to reproduce results from the paper:\n\n**[CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://arxiv.org/abs/2210.06812)**  \n*NeurIPS 2022 Human in the Loop Learning Workshop*\n\nThis repository benchmarks algorithms that estimate:\n1. A consensus label for each example that aggregates the individual annotations.\n2. A confidence score for the correctness of each consensus label.\n3. A rating for each annotator which estimates the overall correctness of their labels.\n\nThis repository is only for intended for scientific purposes. \nTo apply the CROWDLAB algorithm to your own multi-annotator data, you should instead use [the implementation](https://docs.cleanlab.ai/stable/tutorials/multiannotator.html) from the official [cleanlab](https://github.com/cleanlab/cleanlab) library.\n\nCode to benchmark methods for active learning with multiple data annotators can be found in the [active_learning_benchmarks](active_learning_benchmarks) folder.\n\n## Install Dependencies\n\nTo run the model training and benchmark, you need to install the following dependencies:\n```\npip install ./cleanlab\npip install ./crowd-kit\npip install -r requirements.txt\n```\n\nNote that our `cleanlab/` and `crowd-kit/` folders here contain forks of the [cleanlab](https://github.com/cleanlab/cleanlab) and [crowd-kit](https://github.com/Toloka/crowd-kit) libraries. These forks differ from the main libraries as follows:\n\n- The `cleanlab` fork contains various multi-annotator algorithms studied in the benchmark (to obtain consensus labels and compute consensus and annotator quality scores) that are not present in the main library.\n- The `crowd-kit` fork addresses some numeric underflow issues in the original library (needed for properly ranking examples by their quality). Instead of operating directly on probabilities, our fork does calculations on log-probabilities with the log-sum-exp trick.\n\n## Run Benchmarks\n\nTo benchmark various multi-annotator algorithms using given predictions from already trained classifier models, run the following notebooks:\n\n1. [benchmark.ipynb](2_benchmark.ipynb) - runs the benchmarks and saves results to csv\n2. benchmark_results_[...].ipynb - visualize benchmark results in plots\n\n## Generate Data and Train Classfier Model\n\nTo generate the multi-annotator datasets and train the image classifier considered in our benchmarks, run the following notebooks:\n\n1. [preprocess_data.ipynb](0_preprocess_data.ipynb) - preprocesses the dataset\n2. [create_labels_df.ipynb](0_create_labels_df.ipynb) - generates correct absolute label paths for images in preprocessed data\n3. [xval_model_train.ipynb](1_xval_model_train.ipynb) /  [xval_model_train_perfect_model.ipynb](1_xval_model_train_perfect_model.ipynb) - trains a model and obtains predicted class probabilities for each image\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleanlab%2Fmultiannotator-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcleanlab%2Fmultiannotator-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcleanlab%2Fmultiannotator-benchmarks/lists"}