{"id":13738143,"url":"https://github.com/wy1iu/OPT","last_synced_at":"2025-05-08T15:32:37.020Z","repository":{"id":112529499,"uuid":"378521440","full_name":"wy1iu/OPT","owner":"wy1iu","description":"Implementation for \u003cOrthogonal Over-Parameterized Training\u003e in CVPR'21.","archived":false,"fork":false,"pushed_at":"2021-07-16T06:19:34.000Z","size":955,"stargazers_count":19,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-15T06:33:26.627Z","etag":null,"topics":["energy","hypersphere","hypersphereical-learning","neural-network","orthogonal","over-parameterization","training","visual-recognition"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wy1iu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-06-19T23:21:07.000Z","updated_at":"2024-09-04T08:08:00.000Z","dependencies_parsed_at":"2023-05-15T21:45:31.532Z","dependency_job_id":null,"html_url":"https://github.com/wy1iu/OPT","commit_stats":{"total_commits":6,"total_committers":2,"mean_commits":3.0,"dds":0.5,"last_synced_commit":"ed64a2342d8e3fee4b3c7670b15c9cce9cda8765"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wy1iu%2FOPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wy1iu%2FOPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wy1iu%2FOPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wy1iu%2FOPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wy1iu","download_url":"https://codeload.github.com/wy1iu/OPT/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253096471,"owners_count":21853606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["energy","hypersphere","hypersphereical-learning","neural-network","orthogonal","over-parameterization","training","visual-recognition"],"created_at":"2024-08-03T03:02:12.316Z","updated_at":"2025-05-08T15:32:32.504Z","avatar_url":"https://github.com/wy1iu.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Orthogonal Over-Parameterized Training\n\nBy Weiyang Liu, Rongmei Lin, Zhen Liu, James Rehg, Liam Paull, Li Xiong, Le Song, Adrian Weller\n\n### License \n*OPT* is released under the MIT License (refer to the LICENSE file for details).\n\n### Contents\n0. [Introduction](#introduction)\n0. [Citation](#citation)\n0. [Short Video Introduction](#short-video-introduction)\n0. [Requirements](#requirements)\n0. [Usage](#usage)\n\n### Introduction\nThe inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. See our previous work -- [MHE](https://wyliu.com/papers/LiuNIPS18_MHE.pdf) for an in-depth introduction.\n\nBy maintaining the minimum hyperspherical energy during training, OPT can greatly improve the empirical generalization. Specifically, OPT fixes the randomly initialized weights of the neurons and learns an orthogonal transformation that applies to these neurons. We consider multiple ways to learn such an orthogonal transformation, including unrolling orthogonalization algorithms, applying orthogonal parameterization, and designing orthogonality-preserving gradient descent. For better scalability, we propose the stochastic OPT which performs orthogonal transformation stochastically for partial dimensions of neurons. \n\n**Our OPT is accepted to [CVPR 2021](http://cvpr2021.thecvf.com/) as oral presentation and the full paper is available on [arXiv](https://arxiv.org/abs/2004.04690) and [here](https://wyliu.com/papers/LiuOPT2021_v6.pdf).**\n\n\u003cimg src=\"assets/opt_paradigm.png\" width=\"60%\" height=\"60%\"\u003e\n\n### Citation\nIf you find our work useful in your research, please consider to cite:\n\n    @InProceedings{Liu2021OPT,\n        title={Orthogonal Over-Parameterized Training},\n        author={Liu, Weiyang and Lin, Rongmei and Liu, Zhen and Rehg, James M. and Paull, Liam \n         and Xiong, Li and Song, Le and Weller, Adrian},\n        booktitle={CVPR},\n        year={2021}\n    }\n\n### Short Video Introduction\nWe also provide a short video introduction to help interested readers quickly go over our work and understand the essence of OPT. Please click the following figure to watch the Youtube video.\n\n[![OPT_talk](https://img.youtube.com/vi/Oy8c1t1nonU/0.jpg)](https://youtu.be/Oy8c1t1nonU)\n\n\n### Requirements\n1. `Python 3.7` \n2. `TensorFlow 1.14.0`\n\n### Usage\nThis repository provides both OPT and S-OPT implementations on CIFAR-100 as a demostration.\n\n#### Part 1: Clone the repositary\n```Shell  \ngit clone https://github.com/wy1iu/OPT.git\n```\n#### Part 2: Download the official CIFAR-100 training and testing data (python version)\n```Shell  \nwget https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz\n```\n\n#### Part 3: Train and test with the following code in different folder. \n```Shell\n# Run Cayley Parameterization OPT\ncd opt_cp\npython train.py\n```\n```Shell\n# Run Gram-Schmidt OPT\ncd opt_gs\npython train.py\n```\n\n```Shell\n# Run Householder Reflection OPT\ncd opt_hr\npython train.py\n```\n\n```Shell\n# Run Lowdin’s Symmetric OPT\ncd opt_ls\npython train.py\n```\n\n```Shell\n# Run Orthogonality-Preserving Gradient Descent OPT\ncd opt_ogd\npython train.py\n```\n\n```Shell\n# Run Orthogonality Regularization OPT\ncd opt_or\npython train.py\n```\n\n```Shell\n# Run Stochastic OPT (Gram-Schmidt)\ncd sopt_gs\npython train.py\n```\n### Contact\n\n- [Weiyang Liu](https://wyliu.com)\n- [Rongmei Lin](https://rmlin.github.io/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwy1iu%2FOPT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwy1iu%2FOPT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwy1iu%2FOPT/lists"}