{"id":16512962,"url":"https://github.com/daskol/lotr","last_synced_at":"2025-10-28T04:32:01.359Z","repository":{"id":221253721,"uuid":"753864271","full_name":"daskol/lotr","owner":"daskol","description":"Low Tensor Rank adaptation of large language models","archived":false,"fork":false,"pushed_at":"2024-06-09T16:58:19.000Z","size":135,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-09T18:20:35.554Z","etag":null,"topics":["fine-tuning","llm","lora","lotr","parameter-efficient-tuning","peft"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2402.01376","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daskol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-06T23:40:08.000Z","updated_at":"2024-06-09T16:58:19.000Z","dependencies_parsed_at":"2024-02-07T00:46:13.997Z","dependency_job_id":null,"html_url":"https://github.com/daskol/lotr","commit_stats":null,"previous_names":["daskol/lotr"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daskol%2Flotr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daskol%2Flotr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daskol%2Flotr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daskol%2Flotr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daskol","download_url":"https://codeload.github.com/daskol/lotr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219860179,"owners_count":16556026,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fine-tuning","llm","lora","lotr","parameter-efficient-tuning","peft"],"created_at":"2024-10-11T16:06:46.087Z","updated_at":"2025-10-28T04:31:56.007Z","avatar_url":"https://github.com/daskol.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LoTR: Low Tensor Rank Adaptation of Large Language Models\n\n*Low Tensor Rank adaptation of large language models*\n\n## Overview\n\nThis repository is the original implementation of LoTR ([arXiv:2402.01376][4]),\na novel approach for parameter-efficient fine-tuning of LLMs which represents a\ngradient update to parameters in a form of tensor decomposition. Low-rank\nadapter for each layer is constructed as a product of three matrices, and\ntensor structure arises from sharing left and right multipliers of this product\namong layers. Simultaneous compression of a sequence of layers with low-rank\ntensor representation allows LoTR to archive even better parameter efficiency\nthen LoRA especially for deep models. Moreover, the core tensor does not depend\non original weight dimension and can be made arbitrary small, which allows for\nextremely cheap and fast downstream fine-tuning.\n\n```bibtex\n@misc{bershatsky2024lotr,\n  title         = {{LoTR}: Low Tensor Rank Weight Adaptation},\n  author        = {Daniel Bershatsky and Daria Cherniuk and Talgat Daulbaev and Aleksandr Mikhalev and Ivan Oseledets},\n  year          = {2024},\n  eprint        = {2402.01376},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.CL}\n}\n```\n\n[1]: https://arxiv.org/abs/2106.09685\n[2]: https://arxiv.org/abs/2205.13571\n[3]: ./doc/iclr2024/fig/parameter-efficiency.png\n[4]: https://arxiv.org/abs/2402.01376\n\n## Experiments\n\n### Logging Files\n\nWe assume that all raw experiment results (i.e. logging files, first of all)\nare located in `log` directory. This directory's high-level structure should\nreflect experimental setup. So the path relative to this directory should have\nstructure as follows.\n\n```\n\u003cdataset\u003e/\u003cmodel\u003e/\u003cmethod\u003e/\u003cparam1\u003e/\u003cparam2\u003e/.../\u003cseed\u003e/\u003ctfevents-file\u003e\n```\n\nThe model segment preceeds the method path segment since number of differnt\nmodels usually are smaller that number of methods and training pipeline usually\nparameterized by model and then by method. All floating point parameters should\nbe used in scientific notation to ensure that no significant digits are lost.\nThe lat directory is random seed used to run an experiment.\n\nNote that the requirements above are involuntary since there is no\nfull-featured machine learning experiment management software.\n\n### Convertion to Arrow Parquet\n\nTensorBoard `tfvents`-file are quite large files which take noticably long time\nto read and load. So we convert `tfevents`-files to `parquet`-files with the\nfollowing command.\n\n```shell\npython -m lotr.tb2parquet log/glue data/glue.parquet \\\n    --names model method task lr rank seed \\\n```\n\nNow, one can read a single `parquet`-file with all time series as follows.\n\n```python\nimport pandas as pd\ndf = pd.read_parquet('data/glue.parquet')\n```\n\nTo be more specific, 20Mb of `tfevents`-file are converted to 200Kb of\n`parquet`-file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaskol%2Flotr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaskol%2Flotr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaskol%2Flotr/lists"}