{"id":27251630,"url":"https://github.com/voxel51/zcore","last_synced_at":"2025-08-23T16:08:27.248Z","repository":{"id":266122425,"uuid":"889150826","full_name":"voxel51/zcore","owner":"voxel51","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-01T13:32:10.000Z","size":4101,"stargazers_count":74,"open_issues_count":1,"forks_count":8,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-07-01T14:40:48.174Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/voxel51.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-15T17:54:48.000Z","updated_at":"2025-06-25T20:49:19.000Z","dependencies_parsed_at":"2025-07-01T14:31:18.652Z","dependency_job_id":"c602999d-cc4a-47d1-bedd-1e4dc013951b","html_url":"https://github.com/voxel51/zcore","commit_stats":null,"previous_names":["voxel51/zcore"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/voxel51/zcore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxel51%2Fzcore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxel51%2Fzcore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxel51%2Fzcore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxel51%2Fzcore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/voxel51","download_url":"https://codeload.github.com/voxel51/zcore/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/voxel51%2Fzcore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271755406,"owners_count":24815398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-11T01:11:22.380Z","updated_at":"2025-08-23T16:08:27.192Z","avatar_url":"https://github.com/voxel51.png","language":"Shell","funding_links":[],"categories":["public repositories"],"sub_categories":[],"readme":"# Zero-Shot Coreset Selection\n\n([Brent A. Griffin](https://github.com/griffbr)\\*, [Jacob Marks](https://github.com/jacobmarks), [Jason J. Corso](https://github.com/jasoncorso)) @ [Voxel51](https://voxel51.com)\n\n\\* Corresponding author\n\n**Z**ero-Shot **Core**set Selection ([ZCore](https://arxiv.org/pdf/2411.15349 \"ZCore Paper\")) is a method of coreset selection for unlabeled data. Deep learning methods rely on massive data, resulting in substantial costs for storage, annotation, and model training. Coreset selection aims to select a subset of the data to train models with lower cost while ideally performing on par with the full data training. Although the majority of real-world data are unlabeled, previous state-of-the-art coreset methods cannot select data that are unlabeled. As a solution, ZCore addresses the problem of coreset selection without labels _or_ training on candidate data. Instead, ZCore uses existing foundation models to generate a zero-shot embedding space for unlabeled data, then quantifies the relative importance of each example based on overall coverage and redundancy within the embedding distribution. On ImageNet, the ZCore coreset achieves a higher accuracy than previous label-based coresets at a 90% prune rate, while removing annotation requirements for 1.15 million images.\n\n__Zero-Shot Coreset Selection Overview__\n![alt text](./figure/main_figure.jpg?raw=true \"ZCore Overview\")\n\n## Using ZCore\nWe provide example ZCore commands for coreset selection and subsequent model training for the EuroSAT10 dataset from our paper. See instructions in **Repeat Trials** to repeat experiment trials and **Dataset Setup** for full ImageNet, CIFAR, or EuroSAT setup.\n\nStep 1. **Dataset**. [Download](https://www.dropbox.com/scl/fo/1mhwsunssr6g2v1wio0vq/AEI2cx3aZ2vWvFmSLDfUHtQ?rlkey=kbxo4uae43tnzvk6k7x5hk28u\u0026st=8tkh3oyl\u0026dl=0 \"EuroSAT split download\") and unzip ``eurosat10.zip`` in ``./data``.\n\nStep 2. **Zero-Shot Coreset Selection**\n```bash\npython zeroshot_coreset_selection.py --dataset eurosat10 --data_dir ./data --results_dir ./results --embedding clip resnet18 --num_workers 10\n```\n[FiftyOne](https://github.com/voxel51/fiftyone) dependency to generate embeddings (``pip install fiftyone``).\n\nStep 3. **Train Coreset Model**\n```bash\npython train_coreset_model.py --prune_rate 0.7 --dataset eurosat10 --data_dir ./data --score_file ./results/eurosat10/zcore-eurosat10-clip-resnet18-1000Ks-2sd-ri-1000nn-4ex-0/score.npy\n```\n\n## Repeat Trials\nWe provide examples scripts to repeat ZCore experiments over multiple trials in `./repeat-trial-scripts`.\n\nRepeat **ZCore Selections** for EuroSAT10\n```bash\nchmod +x ./repeat-trial-scripts/eurosat10-score-x5.sh\n./repeat-trial-scripts/eurosat10-score-x5.sh\n```\n\nRepeat **Coreset Model Training** for EuroSAT10\n```bash\nchmod +x ./repeat-trial-scripts/eurosat10-train-x5.sh\n./repeat-trial-scripts/eurosat10-train-x5.sh\n```\n\nWe provide example repeat trial results in `./results/example/eurosat10`. To tabulate these repeat trials run:\n```bash\npython process_repeat_trials.py --base_score_dir ./results/example/eurosat10/zcore-eurosat10-clip-resnet18-1000Ks-2sd-ri-1000nn-4ex\n```\nto generate the following table:\n```\nSetting p30-s51 p50-s51 p70-s51 p80-s51 p90-s51 \n\nTrial Results\n0       93.80   91.93   86.10   80.98   63.63   \n1       93.39   91.26   85.74   78.88   65.58   \n2       93.63   91.21   87.91   79.84   66.70   \n3       93.90   92.38   86.91   79.86   65.16   \n4       94.06   92.26   86.47   80.20   67.75   \n\nAggregate Results\nMean    93.76   91.81   86.63   79.95   65.76   \nStdDev  0.230   0.491   0.750   0.677   1.398   \nOverall Mean: 83.58 \n```\n\n## Datasets\n\n**ImageNet** can be downloaded [here](https://www.kaggle.com/c/imagenet-object-localization-challenge/data \"ImageNet download\") and subsequently reformatted using:\n```bash\ncd ./ILSVRC/Data/CLS-LOC/val/                                                               \nwget -qO- https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh | bash\n```\n\n**CIFAR10** and **CIFAR100** can be downloaded [here](https://www.cs.toronto.edu/~kriz/cifar.html \"CIFAR download\").\n\n**EuroSAT80**, **EuroSAT40**, **EuroSAT20**, and **EuroSAT10** can be downloaded [here](https://www.dropbox.com/scl/fo/1mhwsunssr6g2v1wio0vq/AEI2cx3aZ2vWvFmSLDfUHtQ?rlkey=kbxo4uae43tnzvk6k7x5hk28u\u0026st=8tkh3oyl\u0026dl=0 \"EuroSAT split download\").\n\n## Citation\n\nIf you find this code useful, please consider citing our [paper](https://arxiv.org/pdf/2411.15349):\n\n```bibtex\n@article{griffin24zcore,\n  title={Zero-Shot Coreset Selection: Efficient Pruning for Unlabeled Data},\n  author={Griffin, Brent A and Marks, Jacob and Corso, Jason J},\n  journal={arXiv preprint arXiv:2411.15349},\n  year={2024}\n}\n```\n\nYou may also want to check out our open-source toolkit, [FiftyOne](https://voxel51.com/fiftyone), which provides a powerful interface for exploring, analyzing, and visualizing datasets for computer vision and machine learning.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoxel51%2Fzcore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvoxel51%2Fzcore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvoxel51%2Fzcore/lists"}