{"id":22066380,"url":"https://github.com/jaketae/nn-svd","last_synced_at":"2026-05-08T06:04:47.657Z","repository":{"id":235983115,"uuid":"787566085","full_name":"jaketae/nn-svd","owner":"jaketae","description":"Neural network compression with SVD","archived":false,"fork":false,"pushed_at":"2024-04-21T00:24:35.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T00:43:58.399Z","etag":null,"topics":["model-compression","neural-network","neural-network-compression","singular-value-decomposition","svd"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaketae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-04-16T19:07:34.000Z","updated_at":"2024-04-25T05:15:18.000Z","dependencies_parsed_at":"2024-04-25T07:39:14.249Z","dependency_job_id":null,"html_url":"https://github.com/jaketae/nn-svd","commit_stats":null,"previous_names":["jaketae/nn-svd"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fnn-svd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fnn-svd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fnn-svd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaketae%2Fnn-svd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaketae","download_url":"https://codeload.github.com/jaketae/nn-svd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245146367,"owners_count":20568310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["model-compression","neural-network","neural-network-compression","singular-value-decomposition","svd"],"created_at":"2024-11-30T19:27:55.393Z","updated_at":"2026-05-08T06:04:42.619Z","avatar_url":"https://github.com/jaketae.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## NN-SVD\n\nThis repository contains code for [NN-SVD](https://drive.google.com/file/d/1eEMpPgskY99Imvdi9LH7lt10nr38iBdV/view?usp=share_link). This work was completed as part of MATH 232: Advanced Linear Algebra.\n\n## Abstract\n\n\u003e Neural networks have become the standard architecture in machine learning. A typical deep learning solution involves an over-parameterized neural network, which is trained on large amounts of data. However, the recent exponential growth trend in model size highlights the need for effective compression methods that reduce the model footprint. In this work, we propose NN-SVD, a simple yet effective method that employs singular value decomposition to neural network layer weights to compress the model. We show that NN-SVD is theoretically robust under mild Lipschitz assumptions and demonstrates strong empirical performance on a small fully-connected model trained on the MNIST dataset.\n\n## Quickstart\n\nThis project was developed and tested on an Ubuntu 20.04.5 LTS server with NVIDIA RTX 3090 GPUs on CUDA 11.7, using Python 3.8.17.\n\n1. Clone this repository.\n\n```\n$ git clone https://github.com/jaketae/nn-svd.git\n```\n\n2. Create a Python virtual environment and install package requirements. Depending on the local platform, you may have to adjust package versions (e.g., PyTorch) as appropriate.\n\n```\n$ cd nn-svd\n$ python -m venv venv\n$ pip install -U pip wheel # update pip\n$ pip install -r requirements.txt\n```\n\n3. Train and test the model.\n\n```\n$ CUDA_VISIBLE_DEVICES=0 python -m src.main\n```\n\nExperiment logs and checkpoints are saved in `logs`.\n\n## Training\n\nTo train a model, run [`src/main.py`](src/main.py). The full list of supported arguments are shown below.\n\n```\n$ python -m src.main --help\nusage: main.py [-h] [--seed SEED] [--max_epochs MAX_EPOCHS] [--batch_size BATCH_SIZE] [--num_workers NUM_WORKERS]\n               [--hidden_size HIDDEN_SIZE] [--lr LR] [--rank RANK] [--symmetric] [--no-symmetric] [--orthonormal]\n               [--no-orthonormal] [--alpha ALPHA] [--checkpoint_path CHECKPOINT_PATH]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --seed SEED\n  --max_epochs MAX_EPOCHS\n  --batch_size BATCH_SIZE\n  --num_workers NUM_WORKERS\n  --hidden_size HIDDEN_SIZE\n  --lr LR\n  --rank RANK\n  --symmetric\n  --no-symmetric\n  --orthonormal\n  --no-orthonormal\n  --alpha ALPHA\n  --checkpoint_path CHECKPOINT_PATH\n```\n\nFor instance, to train a model with a rank constraint of 8 and symmetric regularization, run\n\n```\n$ CUDA_VISIBLE_DEVICES=0 python -m src.main --symmetric --rank 8\n```\n\nThe script will report the hyperparameters and training log with the final test results under [`logs`](./logs/).\n\n## NN-SVD\n\nTo perform NN-SVD on a trained checkpoint, run [`src/main.py`](src/main.py) with the `--checkpoint_path` flag.\n\nFor instance, to perform NN-SVD with rank parameter 16, run\n\n```\n$ CUDA_VISIBLE_DEVICES=0 python -m src.main --rank 16 --checkpoint_path src/logs/full_rank/lightning_logs/version_0/checkpoints/epoch\\=9-step\\=1080.ckpt\n```\n\nThis command will load the checkpoint at the specified location, perform NN-SVD, then run evaluation on the compressed model (without any training). The final result will be saved under [`logs`](./logs/).\n\n## License\n\nReleased under the [MIT License](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Fnn-svd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaketae%2Fnn-svd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaketae%2Fnn-svd/lists"}