{"id":20663744,"url":"https://github.com/vita-group/smc-bench","last_synced_at":"2025-04-19T15:56:16.098Z","repository":{"id":79903226,"uuid":"551114987","full_name":"VITA-Group/SMC-Bench","owner":"VITA-Group","description":"[ICLR 2023] \"Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!\" Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, AJAY KUMAR JAISWAL, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2023-08-29T14:07:34.000Z","size":36081,"stargazers_count":28,"open_issues_count":0,"forks_count":2,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-18T07:36:50.139Z","etag":null,"topics":["benchmark","deep-learning","dynamic-sparse-training","pruning","sparse-neural-networks","sparsity"],"latest_commit_sha":null,"homepage":"https://openreview.net/forum?id=J6F3lLg4Kdp","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-13T21:35:17.000Z","updated_at":"2025-04-10T20:23:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"a4d4aabc-3495-45aa-92d8-1087b775ef4f","html_url":"https://github.com/VITA-Group/SMC-Bench","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSMC-Bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSMC-Bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSMC-Bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSMC-Bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/SMC-Bench/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249731407,"owners_count":21317342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","deep-learning","dynamic-sparse-training","pruning","sparse-neural-networks","sparsity"],"created_at":"2024-11-16T19:19:37.103Z","updated_at":"2025-04-19T15:56:16.093Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [ICLR 2023] [Sparsity May Cry Benchmark (SMC-Bench)](https://openreview.net/forum?id=J6F3lLg4Kdp)\n\nOfficial PyTorch implementation of **SMC-Bench** - Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!\n\n[Shiwei Liu](https://shiweiliuiiiiiii.github.io/), [Tianlong Chen](https://tianlong-chen.github.io/about/), [Zhenyu Zhang](https://scholar.google.com/citations?user=ZLyJRxoAAAAJ\u0026hl=zh-CN), [Xuxi Chen](http://xxchen.site/), [Tianjin Huang](https://research.tue.nl/en/persons/tianjin-huang), [Ajay Jaiswal](https://ajay1994.github.io/), [Zhangyang Wang](https://vita-group.github.io/)\n\nUniversity of Texas at Austin, Eindhoven University of Technology\n\nThe \"Sparsity May Cry\" Benchmark (SMC-Bench) is a collection of benchmark in pursuit of a more general evaluation and unveiling the true potential of sparse algorithms. SMC-Bench contains carefully curated 4 diverse tasks with 10 datasets, that accounts for capturing a wide-range of domain-specific knowledge. \n\n\nThe benchmark organizers can be contacted at s.liu@tue.nl.\n\nTable of contents\n* [Installation](#installation-of-smc-bench)\n* [Training](#training-of-smc-bench)\n* [Evaluated Sparse Algorithms](#sparse-algorithms)\n* [Tasks, Datasets, and Models](#tasks-models-and-datasets)\n* [Results](#results)\n--- \n\n## Installation of SMC-Bench   \nPlease check [INSTALL.md](INSTALL.md) for installation instructinos.\n\n## Training of SMC-Bench   \nPlease check [TRAINING.md](TRAINING.md) for installation instructinos.\n\n## Tasks Models and Datasets\nSpecifically, we consider a broad set of tasks including *commonsense reasoning, arithmatic reasoning, multilingual translation, and protein prediction*, whose content spans multiple domains, requiring a vast amount of commonsense knowledge, solid mathematical and scientific background to solve. Note that none of the datasets in SMC-Bench has been created from scratch for the benchmark, we rely on pre-existing datasets as they have been implicitly agreed by researchers as challenging, interesting, and of high practical value.  The models and datasets that we used for SMC-Bench are summarized below. \n\n--- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/VITA-Group/SMC-Bench/blob/main/Images/Summary.png\" width=\"800\" height=\"350\"\u003e\n\u003c/p\u003e\n\n## Sparse Algorithms\n*After Taining*: [Lottery Ticket Hypothesis](https://arxiv.org/abs/1803.03635), [Magnitude After Training](https://proceedings.neurips.cc/paper/2015/file/ae0eb3eed39d2bcef4622b2499a05fe6-Paper.pdf), [Random After Training](https://arxiv.org/abs/1812.10240), [oBERT](https://arxiv.org/abs/2203.07259).\n\n*During Taining*: [Gradual Magnitude Pruning](https://arxiv.org/abs/1902.09574a).\n\n*Before Training*: [Magnitude Before Training](https://arxiv.org/abs/2009.08576), [SNIP](https://arxiv.org/abs/1810.02340), [Rigging the Lottery](https://arxiv.org/abs/1911.11134), [Random Before Training](https://arxiv.org/abs/2202.02643).\n\n## Results\n\nCommonsense Reasoning\n--- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/VITA-Group/SMC-Bench/blob/main/Images/Commonsense_reasoning.png\" width=\"800\" height=\"250\"\u003e\n\u003c/p\u003e\n\nArithmatic Reasoning\n--- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/VITA-Group/SMC-Bench/blob/main/Images/Arithmatic_reasoning.png\" width=\"800\" height=\"500\"\u003e\n\u003c/p\u003e\n\nProtein Property Prediction\n--- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/VITA-Group/SMC-Bench/blob/main/Images/Protain_thermal_stability_prediction.png\" width=\"800\" height=\"250\"\u003e\n\u003c/p\u003e\n\nMultilingual Translation\n--- \n\u003cp align=\"center\"\u003e\n\u003cimg src=\"https://github.com/VITA-Group/SMC-Bench/blob/main/Images/Multilingual_translation.png\" width=\"800\" height=\"250\"\u003e\n\u003c/p\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsmc-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fsmc-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsmc-bench/lists"}