{"id":28232136,"url":"https://github.com/daizedong/llama-dlo","last_synced_at":"2025-09-23T17:14:43.785Z","repository":{"id":289792661,"uuid":"841365237","full_name":"DaizeDong/LLaMA-DLO","owner":"DaizeDong","description":"The official implementation of ICLR 2025 Workshop paper \"DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs\".","archived":false,"fork":false,"pushed_at":"2025-04-25T04:03:51.000Z","size":271,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-06T21:09:51.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DaizeDong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-08-12T09:05:26.000Z","updated_at":"2025-04-27T20:57:32.000Z","dependencies_parsed_at":"2025-04-25T04:33:10.741Z","dependency_job_id":"70c4a2ad-7d34-42af-b924-de39192b36d6","html_url":"https://github.com/DaizeDong/LLaMA-DLO","commit_stats":null,"previous_names":["daizedong/llama-dlo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DaizeDong/LLaMA-DLO","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DaizeDong%2FLLaMA-DLO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DaizeDong%2FLLaMA-DLO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DaizeDong%2FLLaMA-DLO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DaizeDong%2FLLaMA-DLO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DaizeDong","download_url":"https://codeload.github.com/DaizeDong/LLaMA-DLO/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DaizeDong%2FLLaMA-DLO/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276613376,"owners_count":25673420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-23T02:00:09.130Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-18T20:08:32.670Z","updated_at":"2025-09-23T17:14:43.773Z","avatar_url":"https://github.com/DaizeDong.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs\n\n**Zhen Tan\\*, Daize Dong\\*, Xinyu Zhao, Jianing Cai, Jie Peng, Yu Cheng, Tianlong Chen**\n\nPublished on the *First Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models (SCOPE - ICLR 2025 Workshop)*.\n\n[![OpenReview](https://img.shields.io/badge/OpenReview-PDF-b31b1b.svg?style=plastic)](https://openreview.net/forum?id=E9Jw3IHuDH)\n\n## Introduction\n\nIn this paper, we introduce Dynamic Layer Operations (DLO), a novel approach for vertically scaling transformer-based Large Language Models (LLMs) by dynamically expanding, activating, or skipping layers using a sophisticated routing policy based on layerwise feature similarity. Unlike traditional Mixture-of-Experts (MoE) methods that focus on extending the model width, our approach targets model depth, addressing the redundancy observed across layer representations for various input samples. Our framework is integrated with the Supervised Fine-Tuning (SFT) stage, eliminating the need for resource-intensive Continual Pre-Training (CPT). Experimental results demonstrate that DLO not only outperforms the original unscaled models but also achieves comparable results to densely expanded models with significantly improved efficiency. Our work offers a promising direction for building efficient yet powerful LLMs.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"dlo.svg\" alt=\"dlo\" width=\"600px\"\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\nconda create --name llama-dlo python=3.11\nconda activate llama-dlo\npip install -e .\npip install -r requirements.txt\npip install flash-attn --no-build-isolation\n```\n\n## Prepare Datasets\n\nFirst change the `SAVE_PATH` in `entrypoints/data/download_dataset.py` and run the following command to download the dataset:\n\n```bash\npython entrypoints/data/download_dataset.py\n```\n\nThen you should change the `DATASET_PATH` in `entrypoints/data/reformat_datasets.py` and `entrypoints/data/mix_datasets.py`, and run the following command to reformat and mix the dataset:\n\n```bash\nbash scripts/data/reformat_datasets.sh\nbash scripts/data/mix_datasets.sh\n```\n\n## Prepare Models\n\nTo convert a LLaMA model into LLaMA-DLO / LLaMA-Pro, please change the `model_path` and `output_path` first, and run:\n\n```bash\nbash scripts/convert/convert_llama_dlo.sh\nbash scripts/convert/convert_llama_pro.sh\n```\n\n## Run Finetuning\n\nTo finetune a model w/o DLO implementation, please run:\n\n```bash\nbash scripts/finetune/finetune_normal.sh\n```\n\nTo finetune a DLO model, please run:\n\n```bash\nbash scripts/finetune/finetune_dlo.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaizedong%2Fllama-dlo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaizedong%2Fllama-dlo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaizedong%2Fllama-dlo/lists"}