{"id":13577481,"url":"https://github.com/jfc43/self-training-ensembles","last_synced_at":"2025-04-05T12:30:21.917Z","repository":{"id":106673903,"uuid":"417252068","full_name":"jfc43/self-training-ensembles","owner":"jfc43","description":"Propose a principled and practically effective framework for unsupervised accuracy estimation and error detection tasks with theoretical analysis and state-of-the-art performance.","archived":false,"fork":false,"pushed_at":"2022-02-17T02:32:04.000Z","size":243,"stargazers_count":14,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-11-05T14:45:28.340Z","etag":null,"topics":["deep-learning","error-detection","machine-learning","pytorch","self-training-ensembles","unsupervised-accuracy-estimation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jfc43.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-10-14T19:07:03.000Z","updated_at":"2024-03-19T07:08:21.000Z","dependencies_parsed_at":"2024-01-16T20:39:21.637Z","dependency_job_id":null,"html_url":"https://github.com/jfc43/self-training-ensembles","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfc43%2Fself-training-ensembles","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfc43%2Fself-training-ensembles/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfc43%2Fself-training-ensembles/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jfc43%2Fself-training-ensembles/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jfc43","download_url":"https://codeload.github.com/jfc43/self-training-ensembles/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247338455,"owners_count":20922985,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","error-detection","machine-learning","pytorch","self-training-ensembles","unsupervised-accuracy-estimation"],"created_at":"2024-08-01T15:01:21.842Z","updated_at":"2025-04-05T12:30:21.474Z","avatar_url":"https://github.com/jfc43.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles\nThis repository is the official implementation of [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://arxiv.org/abs/2106.15728). \n\n## Requirements\n* It is tested under Ubuntu Linux 16.04.1 and Python 3.6 environment, and requires some packages to be installed: [PyTorch](https://pytorch.org/), [numpy](http://www.numpy.org/) and [scikit-learn](https://scikit-learn.org/). \n* To install requirements: `pip install -r requirements.txt`\n\n## Downloading Datasets\n* [MNIST-M](http://bit.ly/2fNqL6N): download it from the Google drive. Extract the files and place them in `./dataset/mnist_m/`. \n* [SVHN](http://ufldl.stanford.edu/housenumbers/): need to download Format 2 data (`*.mat`). Place the files in `./dataset/svhn/`.\n* [USPS](https://www.kaggle.com/bistaumanga/usps-dataset): download the usps.h5 file. Place the file in `./dataset/usps/`.\n\n## Overview of the Code\n* `train_model.py`: train standard models via supervised learning. \n* `train_dann.py`: train domain adaptive (DANN) models. \n* `eval_pipeline.py`: evaluate various methods on all tasks. \n\n## Running Experiments\n\n### Examples\n\n* To train a standard model via supervised learning, you can use the following command:\n\n`python train_model.py --source-dataset {source dataset} --model-type {model type} --base-dir {directory to save the model}`\n\n`{source dataset}` can be `mnist`, `mnist-m`, `svhn` or `usps`.\n\n`{model type}` can be `typical_dnn` or `dann_arch`. \n\n* To train a domain adaptive (DANN) model, you can use the following command:\n\n`python train_dann.py --source-dataset {source dataset} --target-dataset {target dataset} --base-dir {directory to save the model} [--test-time]`\n\n`{source dataset}` (or `{target dataset}`) can be `mnist`, `mnist-m`, `svhn` or `usps`.\n\nThe argument `--test-time` is to indicate whether to replace the target training dataset with the target test dataset. \n\n* To evaluate a method on all training-test dataset pairs, you can use the following command:\n\n`python eval_pipeline.py --model-type {model type} --method {method}`\n\n`{model type}` can be `typical_dnn` or `dann_arch`. \n\n`{method}` can be `conf_avg`, `ensemble_conf_avg`, `conf`, `trust_score`, `proxy_risk`, `our_ri` or `our_rm`.\n\n### Training\n\nYou can run the following scrips to pre-train all models needed for the experiments. \n* `run_all_model_training.sh`: train all supervised learning models. \n* `run_all_dann_training.sh`: train all DANN models. \n* `run_all_ensemble_training.sh`: train all ensemble models.\n\n### Evaluation\n\nYou can run the following script to get the results reported in the paper.\n* `run_all_evaluation.sh`: evaluate all methods on all tasks. \n\n### Pre-trained Models\n\nWe provide pre-trained models produced by our training scripts: `run_all_model_training.sh`, `run_all_dann_training.sh` and `run_all_ensemble_training.sh`. \n\nYou can download the pre-trained models from [Google Drive](https://drive.google.com/drive/folders/1PCUVBW1Wf1JqyN_goC1GiAi-sYFtO26e?usp=sharing). \n\n## Experimental Results\n![Main Results](results.png)\n\n## Acknowledgements\nPart of this code is inspired by [estimating-generalization](https://github.com/chingyaoc/estimating-generalization) and [TrustScore](https://github.com/google/TrustScore). \n\n## Citation \nPlease cite our work if you use the codebase: \n```\n@article{chen2021detecting,\n  title={Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles},\n  author={Chen, Jiefeng and Liu, Frederick and Avci, Besim and Wu, Xi and Liang, Yingyu and Jha, Somesh},\n  journal={Advances in Neural Information Processing Systems},\n  volume={34},\n  year={2021}\n}\n```\n\n## License\nPlease refer to the [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjfc43%2Fself-training-ensembles","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjfc43%2Fself-training-ensembles","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjfc43%2Fself-training-ensembles/lists"}