{"id":15601013,"url":"https://github.com/lucidrains/deep-linear-network","last_synced_at":"2025-06-14T06:02:24.711Z","repository":{"id":62567414,"uuid":"304546238","full_name":"lucidrains/deep-linear-network","owner":"lucidrains","description":"A simple implementation of a deep linear Pytorch module","archived":false,"fork":false,"pushed_at":"2020-10-16T20:02:06.000Z","size":61,"stargazers_count":21,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-14T06:01:49.599Z","etag":null,"topics":["artificial-intelligence","deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucidrains.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-10-16T07:01:39.000Z","updated_at":"2025-05-13T10:45:40.000Z","dependencies_parsed_at":"2022-11-03T16:30:32.604Z","dependency_job_id":null,"html_url":"https://github.com/lucidrains/deep-linear-network","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/lucidrains/deep-linear-network","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeep-linear-network","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeep-linear-network/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeep-linear-network/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeep-linear-network/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucidrains","download_url":"https://codeload.github.com/lucidrains/deep-linear-network/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucidrains%2Fdeep-linear-network/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259768515,"owners_count":22908228,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","deep-learning","pytorch"],"created_at":"2024-10-03T02:11:40.661Z","updated_at":"2025-06-14T06:02:24.657Z","avatar_url":"https://github.com/lucidrains.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"./diagram.png\" width=\"500px\"\u003e\u003c/img\u003e\n\n## Deep Linear Network - Pytorch\n\nA simple to use deep linear network module. Useful for matrix factorization or for passing an input tensor through a series of square weight matrices, where it was discovered that gradient descent implicitly regularizes the output to low-rank solutions.\n\nLeCun's paper uses this unique property to optimize the latent of an autoencoder to be low-rank.\n\nThe module will take care of collapsing the linear weight matrices into one weight matrix, caching it across evaluation calls (but expired on training).\n\n## Install\n\n```bash\n$ pip install deep-linear-network\n```\n\n## Usage\n\nMatrix factorization\n\n```python\nimport torch\nfrom deep_linear_network import DeepLinear\n\nx = torch.randn(1, 1024, 256)\nlinear = DeepLinear(256, 10, 512)  # w1 (256 x 10) @ w2 (10 x 512)\nlinear(x) # (1, 1024, 512)\n```\n\nDeep Linear Network\n\n```python\nimport torch\nfrom deep_linear_network import DeepLinear\n\nx = torch.randn(1, 1024, 256)\nlinear = DeepLinear(256, 256, 256, 256, 128) # w1-w3 (256 x 256) w4 (256 x 128)\nlinear(x) # (1, 1024, 128)\n```\n\n## Citations\n\n```bibtex\n@misc{arora2019implicit,\n    title={Implicit Regularization in Deep Matrix Factorization}, \n    author={Sanjeev Arora and Nadav Cohen and Wei Hu and Yuping Luo},\n    year={2019},\n    eprint={1905.13655},\n    archivePrefix={arXiv},\n    primaryClass={cs.LG}\n}\n```\n\n```bibtex\n@misc{jing2020implicit,\n    title={Implicit Rank-Minimizing Autoencoder}, \n    author={Li Jing and Jure Zbontar and Yann LeCun},\n    year={2020},\n    eprint={2010.00679},\n    archivePrefix={arXiv},\n    primaryClass={cs.LG}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fdeep-linear-network","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucidrains%2Fdeep-linear-network","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucidrains%2Fdeep-linear-network/lists"}