{"id":13491686,"url":"https://github.com/arcee-ai/mergekit","last_synced_at":"2025-05-13T17:06:40.183Z","repository":{"id":189647784,"uuid":"681002458","full_name":"arcee-ai/mergekit","owner":"arcee-ai","description":"Tools for merging pretrained large language models.","archived":false,"fork":false,"pushed_at":"2025-05-10T08:22:22.000Z","size":899,"stargazers_count":5645,"open_issues_count":217,"forks_count":539,"subscribers_count":57,"default_branch":"main","last_synced_at":"2025-05-12T05:08:22.698Z","etag":null,"topics":["llama","llm","model-merging"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arcee-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-08-21T03:50:04.000Z","updated_at":"2025-05-12T03:39:38.000Z","dependencies_parsed_at":"2023-12-22T12:03:32.437Z","dependency_job_id":"47810459-b03d-453d-b945-ed66bf839b5f","html_url":"https://github.com/arcee-ai/mergekit","commit_stats":{"total_commits":209,"total_committers":23,"mean_commits":9.08695652173913,"dds":"0.14832535885167464","last_synced_commit":"93ace70a2df88a51eac2a3deb26862955ab4c229"},"previous_names":["cg123/ties-merge","cg123/mergekit"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arcee-ai%2Fmergekit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arcee-ai%2Fmergekit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arcee-ai%2Fmergekit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arcee-ai%2Fmergekit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arcee-ai","download_url":"https://codeload.github.com/arcee-ai/mergekit/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253990466,"owners_count":21995774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llama","llm","model-merging"],"created_at":"2024-07-31T19:00:59.367Z","updated_at":"2025-05-13T17:06:40.171Z","avatar_url":"https://github.com/arcee-ai.png","language":"Python","funding_links":[],"categories":["Python","llm","Autonomous Research \u0026 Content Generation","Model Merging \u0026 Quantization","A01_文本生成_文本对话","NLP","Building","7. Training \u0026 Fine-tuning Ecosystem","10. Model Preparation \u0026 Quantization","🤖 AI \u0026 Machine Learning","Machine Learning"],"sub_categories":["Prompt Libraries","Model Merging","大语言对话模型及数据","3. Pretraining","Tools","Data \u0026 Alignment Tools"],"readme":"# mergekit\n\n`mergekit` is a toolkit for merging pre-trained language models. `mergekit` uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.\n\n## Contents\n\n- [Why Merge Models?](#why-merge-models)\n- [Features](#features)\n- [Installation](#installation)\n- [Contributing](#contributing)\n- [Usage](#usage)\n- [Merge Configuration](#merge-configuration)\n  - [Parameter Specification](#parameter-specification)\n  - [Tokenizer Configuration](#tokenizer-configuration)\n  - [Chat Template Configuration](#chat-template-configuration)\n  - [Examples](#examples)\n- [Merge Methods](#merge-methods)\n- [LoRA Extraction](#lora-extraction)\n- [Mixture of Experts Merging](#mixture-of-experts-merging)\n- [Evolutionary Merge Methods](#evolutionary-merge-methods)\n- [Multi-Stage Merging (`mergekit-multi`)](#multi-stage-merging-mergekit-multi)\n- [Raw PyTorch Model Merging (`mergekit-pytorch`)](#raw-pytorch-model-merging-mergekit-pytorch)\n- [Merge in the Cloud](#-merge-in-the-cloud-)\n- [Citation](#citation)\n\n## Why Merge Models?\n\nModel merging is a powerful technique that allows combining the strengths of different models without the computational overhead of ensembling or the need for additional training. By operating directly in the weight space of models, merging can:\n\n- Combine multiple specialized models into a single versatile model\n- Transfer capabilities between models without access to training data\n- Find optimal trade-offs between different model behaviors\n- Improve performance while maintaining inference costs\n- Create new capabilities through creative model combinations\n\nUnlike traditional ensembling which requires running multiple models, merged models maintain the same inference cost as a single model while often achieving comparable or superior performance.\n\n## Features\n\nKey features of `mergekit` include:\n\n- Supports Llama, Mistral, GPT-NeoX, StableLM, and more\n- Many [merge methods](#merge-methods)\n- GPU or CPU execution\n- Lazy loading of tensors for low memory use\n- Interpolated gradients for parameter values (inspired by Gryphe's [BlockMerge_Gradient](https://github.com/Gryphe/BlockMerge_Gradient) script)\n- Piecewise assembly of language models from layers (\"Frankenmerging\")\n- [Mixture of Experts merging](#mixture-of-experts-merging)\n- [LORA extraction](#lora-extraction)\n- [Evolutionary merge methods](#evolutionary-merge-methods)\n- [Multi-stage merging](#multi-stage-merging-mergekit-multi) for complex workflows.\n- [Merging of raw PyTorch models (`mergekit-pytorch`)](#raw-pytorch-model-merging-mergekit-pytorch).\n\n🌐 GUI Launch Alert 🤗 - We are excited to announce the launch of a mega-GPU backed graphical user interface for mergekit in Arcee! This GUI simplifies the merging process, making it more accessible to a broader audience. Check it out and contribute at the [Arcee App](https://app.arcee.ai). There is also a [Hugging Face Space](https://huggingface.co/mergekit-community) with limited amounts of GPUs.\n\n## Installation\n\n```sh\ngit clone https://github.com/arcee-ai/mergekit.git\ncd mergekit\n\npip install -e .  # install the package and make scripts available\n```\n\nIf the above fails with the error of:\n\n```\nERROR: File \"setup.py\" or \"setup.cfg\" not found. Directory cannot be installed in editable mode:\n(A \"pyproject.toml\" file was found, but editable mode currently requires a setuptools-based build.)\n```\n\nYou may need to upgrade pip to \u003e 21.3 with the command `python3 -m pip install --upgrade pip`\n\n## Contributing\n\nWe welcome contributions to `mergekit`! If you have ideas for new merge methods, features, or other improvements, please check out our [contributing guide](CONTRIBUTING.md) for details on how to get started.\n\n## Usage\n\nThe script `mergekit-yaml` is the main entry point for `mergekit`. It takes a YAML configuration file and an output path, like so:\n\n```sh\nmergekit-yaml path/to/your/config.yml ./output-model-directory [--cuda] [--lazy-unpickle] [--allow-crimes] [... other options]\n```\n\nThis will run the merge and write your merged model to `./output-model-directory`.\n\nFor more information on the arguments accepted by `mergekit-yaml` run the command `mergekit-yaml --help`.\n\n### Uploading to Huggingface\n\nWhen you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. `mergekit` generates a `README.md` for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely; or use the generated `README.md` as-is. It is also possible to edit your `README.md` online once it has been uploaded to the Hub.\n\nOnce you're happy with your model card and merged model, you can upload it to the Hugging Face Hub using the [huggingface_hub](https://huggingface.co/docs/huggingface_hub/index) Python library.\n\n```sh\n# log in to huggingface with an access token (must have write permission)\nhuggingface-cli login\n# upload your model\nhuggingface-cli upload your_hf_username/my-cool-model ./output-model-directory .\n```\n\nThe [documentation](https://huggingface.co/docs/huggingface_hub/guides/cli#huggingface-cli-upload) for `huggingface_hub` goes into more detail about other options for uploading.\n\n## Merge Configuration\n\nMerge configurations are YAML documents specifying the operations to perform in order to produce your merged model.\nBelow are the primary elements of a configuration file:\n\n- `merge_method`: Specifies the method to use for merging models. See [Merge Methods](#merge-methods) for a list.\n- `slices`: Defines slices of layers from different models to be used. This field is mutually exclusive with `models`.\n- `models`: Defines entire models to be used for merging. This field is mutually exclusive with `slices`.\n- `base_model`: Specifies the base model used in some merging methods.\n- `parameters`: Holds various parameters such as weights and densities, which can also be specified at different levels of the configuration.\n- `dtype`: Specifies the data type used for the merging operation.\n- `tokenizer` or `tokenizer_source`: Determines how to construct a tokenizer for the merged model.\n- `chat_template`: Specifies a chat template for the merged model.\n\n### Parameter Specification\n\nParameters are flexible and can be set with varying precedence. They can be specified conditionally using tensor name filters, which allows finer control such as differentiating between attention heads and fully connected layers.\n\nParameters can be specified as:\n\n- **Scalars**: Single floating-point values.\n- **Gradients**: List of floating-point values, specifying an interpolated gradient.\n\nThe parameters can be set at different levels, with decreasing precedence as follows:\n\n1. `slices.*.sources.parameters` - applying to a specific input slice\n2. `slices.*.parameters` - applying to a specific output slice\n3. `models.*.parameters` or `input_model_parameters` - applying to any tensors coming from specific input models\n4. `parameters` - catchall\n\n### Tokenizer Configuration\n\nThe tokenizer behavior can be configured in two ways: using the new `tokenizer` field (recommended) or the legacy `tokenizer_source` field (maintained for backward compatibility). These fields are mutually exclusive - you should use one or the other, not both.\n\n#### Modern Configuration (tokenizer)\n\nThe `tokenizer` field provides fine-grained control over vocabulary and embeddings:\n\n```yaml\ntokenizer:\n  source: \"union\"  # or \"base\" or a specific model path\n  tokens:          # Optional: configure specific tokens\n    \u003ctoken_name\u003e:\n      source: ...  # Specify embedding source\n      force: false # Optional: force this embedding for all models\n  pad_to_multiple_of: null  # Optional: pad vocabulary size\n```\n\n##### Tokenizer Source\n\nThe `source` field determines the vocabulary of the output model:\n\n- `union`: Combine vocabularies from all input models (default)\n- `base`: Use vocabulary from the base model\n- `\"path/to/model\"`: Use vocabulary from a specific model\n\n##### Token Embedding Handling\n\nWhen merging models with different vocabularies, mergekit uses smart defaults to handle token embeddings:\n\n- If a token exists in the base model, its embedding is used as the default\n- If only one model has the token, that model's embedding is used\n- Otherwise, an average of all available embeddings is used\n\nYou can override these defaults for specific tokens:\n\n```yaml\ntokenizer:\n  source: union\n  tokens:\n    # Use embedding from a specific model\n    \u003c|im_start|\u003e:\n      source: \"path/to/chatml/model\"\n\n    # Force a specific embedding for all models\n    \u003c|special|\u003e:\n      source: \"path/to/model\"\n      force: true\n\n    # Map a token to another model's token embedding\n    \u003c|renamed_token|\u003e:\n      source:\n        kind: \"model_token\"\n        model: \"path/to/model\"\n        token: \"\u003c|original_token|\u003e\"  # or use token_id: 1234\n```\n\n##### Practical Example\n\nHere's how you might preserve both Llama 3 Instruct and ChatML prompt formats when merging models:\n\n```yaml\ntokenizer:\n  source: union\n  tokens:\n    # ChatML tokens\n    \u003c|im_start|\u003e:\n      source: \"chatml_model\"\n    \u003c|im_end|\u003e:\n      source: \"chatml_model\"\n\n    # Llama 3 tokens - force original embeddings\n    \u003c|start_header_id|\u003e:\n      source: \"llama3_model\"\n      force: true\n    \u003c|end_header_id|\u003e:\n      source: \"llama3_model\"\n      force: true\n    \u003c|eot_id|\u003e:\n      source: \"llama3_model\"\n      force: true\n```\n\n#### Legacy Configuration (tokenizer_source)\n\nFor backward compatibility, the `tokenizer_source` field is still supported:\n\n```yaml\ntokenizer_source: \"union\"  # or \"base\" or a model path\n```\n\nThis provides basic tokenizer selection but lacks the fine-grained control of the modern `tokenizer` field.\n\n### Chat Template Configuration\n\nThe optional `chat_template` field allows overriding the chat template used for the merged model.\n\n```yaml\nchat_template: \"auto\"  # or a template name or Jinja2 template\n```\n\nOptions include:\n\n- `\"auto\"`: Automatically select the most common template among input models\n- Built-in templates: `\"alpaca\"`, `\"chatml\"`, `\"llama3\"`, `\"mistral\"`, `\"exaone\"`\n- A Jinja2 template string for custom formatting\n\n### Examples\n\nSeveral examples of merge configurations are available in [`examples/`](examples/).\n\n## Merge Methods\n\n`mergekit` offers many methods for merging models, each with its own strengths and weaknesses. Choosing the right method depends on your specific goals, the relationship between the models you're merging, and the desired characteristics of the final model.\n\nFor detailed explanations, parameter descriptions, and use cases for each method, please see our [**Merge Method Guide**](docs/merge_methods.md).\n\n### Method Overview\n\n| Method (`value`)                                                                                                      | Core Idea                                                            | # Models | Base Model | Key Strengths / Use Cases                                       |\n|:----------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------|:--------:|:----:|:---------------------------------------------------------------|\n| [**Linear** (`linear`)](docs/merge_methods.md#linear-linear)                                                          | Simple weighted average of model parameters.                         |    ≥2    |  -   | Averaging similar checkpoints, model soups.                     |\n| [**SLERP** (`slerp`)](docs/merge_methods.md#slerp-slerp)                                                              | Spherical linear interpolation between two models.                   |     2    |  ✓   | Smoothly transitioning between two models.                      |\n| [**NuSLERP** (`nuslerp`)](docs/merge_methods.md#nuslerp-nuslerp)                                                        | Enhanced SLERP with flexible weighting.                              |     2    |  *   | More intuitive SLERP; task vector SLERP.                        |\n| [**Multi-SLERP** (`multislerp`)](docs/merge_methods.md#multi-slerp-multislerp)                                          | Barycentric SLERP for multiple models.                               |    ≥2    |  *   | Spherical interpolation for \u003e2 models.                          |\n| [**Karcher Mean** (`karcher`)](docs/merge_methods.md#karcher-mean-karcher)                                              | Riemannian barycenter of model parameters.                           |    ≥2    |  -   | Geometrically sound averaging on manifolds.                     |\n| [**Task Arithmetic** (`task_arithmetic`)](docs/merge_methods.md#task-arithmetic-task_arithmetic)                      | Linearly combine \"task vectors\" (differences from a base).           |    ≥2    |  ✓   | Transferring/combining fine-tuned skills.                       |\n| [**TIES** (`ties`)](docs/merge_methods.md#ties-merging-ties)                                                          | Task arithmetic + sparsification \u0026 sign consensus.                   |    ≥2    |  ✓   | Merging many models, reducing interference.                     |\n| [**DARE** (`dare_linear`, `dare_ties`)](docs/merge_methods.md#dare-dare_linear-dare_ties)                               | Task arithmetic + random pruning \u0026 rescaling.                        |    ≥2    |  ✓   | Robust skill retention, similar to TIES.                        |\n| [**DELLA** (`della`, `della_linear`)](docs/merge_methods.md#della-della-della_linear)                                   | Task arithmetic + adaptive magnitude-based pruning.                  |    ≥2    |  ✓   | Prioritizing important changes, reducing interference.          |\n| [**Model Breadcrumbs** (`breadcrumbs`, `breadcrumbs_ties`)](docs/merge_methods.md#model-breadcrumbs-breadcrumbs_ties)   | Task arithmetic + outlier removal (small \u0026 large diffs).             |    ≥2    |  ✓   | Refining task vectors by removing extreme changes.              |\n| [**SCE** (`sce`)](docs/merge_methods.md#sce-sce)                                                                      | Task arithmetic + adaptive matrix-level weighting based on variance. |    ≥2    |  ✓   | Dynamically weighting models based on parameter variance.       |\n| [**Model Stock** (`model_stock`)](docs/merge_methods.md#model-stock-model_stock)                                        | Geometric weight calculation for linear interpolation.               |    ≥3    |  ✓   | Finding good linear interpolation weights for many checkpoints. |\n| [**Nearswap** (`nearswap`)](docs/merge_methods.md#nearswap-nearswap)                                                    | Interpolate where parameters are similar.                            |     2    |  ✓   | Selective merging based on parameter similarity.                |\n| [**Arcee Fusion** (`arcee_fusion`)](docs/merge_methods.md#arcee-fusion-arcee_fusion)                                    | Dynamic thresholding for fusing important changes.                   |     2    |  ✓   | Identifying and merging salient features.                       |\n| [**Passthrough** (`passthrough`)](docs/merge_methods.md#passthrough-passthrough)                                        | Directly copies tensors from a single input model.                      |     1    |  -   | Frankenmerging, layer stacking, model surgery.                  |\n\n**Key for `Base Model` Column:**\n\n- ✓: **Required** - One of the input models *must* be designated as the `base_model`.\n- *: **Optional** - One of the input models *can* be designated as the `base_model`.\n- -: **Not Applicable** - `base_model` has no effect on this method.\n\n## LoRA Extraction\n\nMergekit allows extracting PEFT-compatible low-rank approximations of finetuned models.\n\n### Usage\n\n```sh\nmergekit-extract-lora --model finetuned_model_id_or_path --base-model base_model_id_or_path --out-path output_path [--no-lazy-unpickle] [--cuda] [--max-rank=desired_rank] [--sv-epsilon=tol]\n```\n\n## Mixture of Experts Merging\n\nThe `mergekit-moe` script supports merging multiple dense models into a mixture of experts, either for direct use or for further training. For more details see the [`mergekit-moe` documentation](docs/moe.md).\n\n## Evolutionary Merge Methods\n\nSee [`docs/evolve.md`](docs/evolve.md) for details.\n\n## Multi-Stage Merging (`mergekit-multi`)\n\n`mergekit-multi` enables the execution of complex, multi-stage model merging workflows. You can define multiple merge configurations in a single YAML file, where later merges can use the outputs of earlier ones as inputs. This is useful for building up sophisticated models through a series of targeted merges.\n\nSee the [`mergekit-multi` documentation](docs/multimerge.md) for usage details and examples.\n\n## Raw PyTorch Model Merging (`mergekit-pytorch`)\n\nFor merging arbitrary PyTorch models (not necessarily Hugging Face Transformers), `mergekit-pytorch` provides a way to apply mergekit's algorithms directly to `.pt` or `.safetensors` checkpoints. The configuration is similar to the YAML format used in `mergekit-yaml`, but does not support layer slicing or tokenizer configuration.\n\n### Usage\n\n```sh\nmergekit-pytorch path/to/your/raw_config.yml ./output_pytorch_model_directory [options]\n```\n\nUse `mergekit-pytorch --help` for detailed options.\n\n## ✨ Merge in the Cloud ✨\n\nWe host merging on Arcee's cloud GPUs - you can launch a cloud merge in the [Arcee App](https://app.arcee.ai). Or through python - grab an ARCEE_API_KEY:\n\n`export ARCEE_API_KEY=\u003cyour-api-key\u003e`\n`pip install -q arcee-py`\n\n```python\nimport arcee\narcee.merge_yaml(\"bio-merge\",\"./examples/bio-merge.yml\")\n```\n\nCheck your merge status at the [Arcee App](https://app.arcee.ai)\n\nWhen complete, either deploy your merge:\n\n```python\narcee.start_deployment(\"bio-merge\", merging=\"bio-merge\")\n```\n\nOr download your merge:\n\n`!arcee merging download bio-merge`\n\n## Citation\n\nIf you find `mergekit` useful in your research, please consider citing the [paper](https://aclanthology.org/2024.emnlp-industry.36/):\n\n```bibtex\n@inproceedings{goddard-etal-2024-arcees,\n    title = \"Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models\",\n    author = \"Goddard, Charles  and\n      Siriwardhana, Shamane  and\n      Ehghaghi, Malikeh  and\n      Meyers, Luke  and\n      Karpukhin, Vladimir  and\n      Benedict, Brian  and\n      McQuade, Mark  and\n      Solawetz, Jacob\",\n    editor = \"Dernoncourt, Franck  and\n      Preo{\\c{t}}iuc-Pietro, Daniel  and\n      Shimorina, Anastasia\",\n    booktitle = \"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track\",\n    month = nov,\n    year = \"2024\",\n    address = \"Miami, Florida, US\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.emnlp-industry.36\",\n    doi = \"10.18653/v1/2024.emnlp-industry.36\",\n    pages = \"477--485\",\n    abstract = \"The rapid growth of open-source language models provides the opportunity to merge model checkpoints, combining their parameters to improve performance and versatility. Advances in transfer learning have led to numerous task-specific models, which model merging can integrate into powerful multitask models without additional training. MergeKit is an open-source library designed to support this process with an efficient and extensible framework suitable for any hardware. It has facilitated the merging of thousands of models, contributing to some of the world{'}s most powerful open-source model checkpoints. The library is accessible at: https://github.com/arcee-ai/mergekit.\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farcee-ai%2Fmergekit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farcee-ai%2Fmergekit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farcee-ai%2Fmergekit/lists"}