https://github.com/louisbrulenaudet/mergekit

Tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models.
https://github.com/louisbrulenaudet/mergekit

dare-ties huggingface large-language-models leaderboard llm merge-llm mergekit mixture-of-experts moe slerp ties transformer

Last synced: 3 months ago
JSON representation

Tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models.

• Host: GitHub
• URL: https://github.com/louisbrulenaudet/mergekit
• Owner: louisbrulenaudet
• License: apache-2.0
• Created: 2024-02-11T21:28:28.000Z (over 2 years ago)
• Default Branch: main
• Last Pushed: 2025-09-18T19:03:58.000Z (9 months ago)
• Last Synced: 2026-01-12T15:16:16.979Z (5 months ago)
• Topics: dare-ties, huggingface, large-language-models, leaderboard, llm, merge-llm, mergekit, mixture-of-experts, moe, slerp, ties, transformer
• Language: Jupyter Notebook
• Homepage: https://louisbrulenaudet.com
• Size: 14.6 KB
• Stars: 8
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE
  • Citation: CITATION.cff

Awesome Lists containing this project

README

          
# MergeKit, a tools for merging pretrained LLM and create Mixture of Experts (MoE) from open-source models.

[![Python](https://img.shields.io/pypi/pyversions/tensorflow.svg)](https://badge.fury.io/py/tensorflow) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![Maintainer](https://img.shields.io/badge/maintainer-@louisbrulenaudet-blue)

Mergekit uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as little as 8 GB of VRAM. Many merging algorithms are supported, with more coming as they catch my attention.

When you have a merged model you're happy with, you may want to share it on the Hugging Face Hub. mergekit generates a README.md for your merge with some basic information for a model card. You can edit it to include more details about your merge, like giving it a good name or explaining what it's good at; rewrite it entirely ; or use the generated README.md as-is. It is also possible to edit your README.md online once it has been uploaded to the Hub.

## Citing this project

If you use this code in your research, please use the following BibTeX entry.

```BibTeX

@misc{louisbrulenaudet2023,

	author = {Louis Brulé Naudet},

	title = {MergeKit, a tools for merging pretrained Large Language Models and create Mixture of Experts (MoE) from open-source models},

	howpublished = {\url{https://github.com/louisbrulenaudet/mergeKit}},

	year = {2024}

}

```

## Feedback

If you have any feedback, please reach out at [louisbrulenaudet@icloud.com](mailto:louisbrulenaudet@icloud.com).