{"id":28676516,"url":"https://github.com/zjunlp/modelkinship","last_synced_at":"2025-06-13T23:04:59.490Z","repository":{"id":258531669,"uuid":"802794392","full_name":"zjunlp/ModelKinship","owner":"zjunlp","description":"Exploring Model Kinship for Merging Large Language Models","archived":false,"fork":false,"pushed_at":"2025-03-28T10:30:37.000Z","size":3219,"stargazers_count":23,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-28T10:41:27.354Z","etag":null,"topics":["artificial-intelligence","kinship","knowledge-fusion","large-language-models","merging","model-kinship","model-merging","natural-language-processing"],"latest_commit_sha":null,"homepage":"https://potatobearp.github.io/publication/modelkinship/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zjunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-19T09:36:28.000Z","updated_at":"2025-01-18T10:11:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"3e07bb28-505e-4ca7-9289-f2c9ffc17b41","html_url":"https://github.com/zjunlp/ModelKinship","commit_stats":null,"previous_names":["zjunlp/modelkinship"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zjunlp/ModelKinship","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FModelKinship","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FModelKinship/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FModelKinship/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FModelKinship/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zjunlp","download_url":"https://codeload.github.com/zjunlp/ModelKinship/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zjunlp%2FModelKinship/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259732771,"owners_count":22903087,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","kinship","knowledge-fusion","large-language-models","merging","model-kinship","model-merging","natural-language-processing"],"created_at":"2025-06-13T23:04:58.554Z","updated_at":"2025-06-13T23:04:59.471Z","avatar_url":"https://github.com/zjunlp.png","language":"Jupyter Notebook","readme":"\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"pics/logo.jpg\" width=\"400\"/\u003e\n\u003ch1 align=\"center\"\u003e Exploring Model Kinship for Merging LLMs \u003c/h1\u003e\n\u003cb align=\"center\"\u003eThe degree of\nsimilarity or relatedness between LLMs, analogous to biological evolution\u003c/b\u003e\n\n\n  \u003ca href=\"https://arxiv.org/abs/2410.12613\"\u003e📄arXiv\u003c/a\u003e •\n  \u003ca href=\"https://potatobearp.github.io/publication/modelkinship/\"\u003e📒 Blog\u003c/a\u003e•\n  \u003ca href=\"https://huggingface.co/collections/zjunlp/model-kinship-670d59c69bf3598b0bed1cbd\"\u003e🤗 HF\u003c/a\u003e •\n  \u003ca href=\"https://notebooklm.google.com/notebook/720d2f64-fdd2-47e2-a086-7870b8db23e5/audio\"\u003e🎧NotebookLM Audio\u003c/a\u003e\n\n\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)\n![](https://img.shields.io/github/last-commit/zjunlp/ModelKinship?color=green)\n\u003ca href=\"https://colab.research.google.com/drive/141VAI89emgSIcwkswATEXSEENoAMywTO?usp=sharing\"\u003e\n        \u003cimg alt=\"Open In Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\nWe introduce [Model Kinship](https://arxiv.org/pdf/2410.12613), the metric for degree of similarity or relatedness between LLMs for continual model merging, analogous to biological evolution. \n\nCurrently, we support **Model Kinship** with 3 Similarity Metrics, others will be supported in the future. \n\n---\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Reproduction](#reproduction)\n- [Supported Metrics](#supported-metrics)\n- [Notebook](#notebook)\n- [Acknowledgement](#acknowledgement)\n- [Citation](#citation)\n\n## Overview\n\nModel merging provides a novel paradigm to leverage information from multiple models without the need of additional training.  Recently, the development of a model merging toolkit has enabled non-experts to conduct merging experiments, spurring a trend of model merging on the Hugging Face Open LLM Leaderboard.\n\nCurrently, the model merging community has built powerful models through iterative merging steps. This process resembles artificial selection—a concept from biology in which humans deliberately select for or against specific traits in organisms.\n\n![](pics/evolution.jpg)\n\nHowever, the reasons behind the success of this process remain unknown, resulting in numerous trial-and-error attempts for slight performance improvements.\nDrawing inspiration from evolutionary biology, our project examines the weight changes that occur during post pre-training stages (e.g., fine-tuning, merging). We propose **Model Kinship**, a metric that evaluates the relatedness between two models by calculating the similarity of their weight changes, analogous to genetic variance in inheritance. In our paper we show that **Model Kinship** can be used for optimising the merging strategy.\n\nThis toolkit provides a simple way to calculate **Model Kinship** for model merging.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/zjunlp/ModelKinship.git\npip install -e ./ModelKinship\n```\n\n---\n\n## Usage\n\n```bash\n# Input Format\nmerge_cal model-1 model-2 model-base metrics [options]\n\n# Calculate Model Kinship based on Euclidean Distance (CPU)\nmerge_cal OpenPipe/mistral-ft-optimized-1218 \\\nmlabonne/NeuralHermes-2.5-Mistral-7B \\\nmistralai/Mistral-7B-v0.1 \\\ned\n\n# Multiple Calculation (CPU)\nmerge_cal OpenPipe/mistral-ft-optimized-1218 \\\nmlabonne/NeuralHermes-2.5-Mistral-7B \\\nmistralai/Mistral-7B-v0.1 \\\ncs,pcc,ed\n\n```\n\n### Optional Arguments\n\n- `--low-precision`: Enable 8-bit quantization for parameter extraction. Reduces memory usage but may slightly affect accuracy.\n  ```bash\n  merge_cal model-1 model-2 model-base metrics --low-precision\n  ```\n\n- `--split-calculation`: Calculate metrics per layer/key instead of full vector to reduce RAM usage.\n  ```bash\n  merge_cal model-1 model-2 model-base metrics --split-calculation\n  ```\n\nExample with multiple options:\n```bash\nmerge_cal OpenPipe/mistral-ft-optimized-1218 \\\nmlabonne/NeuralHermes-2.5-Mistral-7B \\\nmistralai/Mistral-7B-v0.1 \\\npcc cs --low-precision --split-calculation\n```\n\n---\n\n## Reproduction\nTo reproduce our experiments, both an evaluation toolkit and a merging toolkit for large language models are required. We recommend using the following tools:\n\n- [lm-evaluation-harness by EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness)\n- [mergekit by arcee-ai](https://github.com/arcee-ai/mergekit)\n\nMerged Models in Our Experiments are Open Access:\n- [Merged Models Repository](https://huggingface.co/PotatoB)\n\n---\n\n## Supported Metrics:\n- Cosine Similarity - cs\n- Pearson Correlation Coefficient - pcc\n- Euclidean Distance - ed\n\n---\n\n## Notebook:\n\nTo conduct iterative merging experiments, you can use following notebook for a quick start.\n\n- [Notebook for Iterative Merging](https://colab.research.google.com/drive/141VAI89emgSIcwkswATEXSEENoAMywTO?usp=sharing)\n\nThis notebook includes 3 main functions:\n- Selection - calculate the model kinship to predict the potential benefit of providing merge.\n- Merging - merge the providing models.\n- Recycling - upload the merged model (evaluation is optional).\n---\n\n## Acknowledgement\n\nWe would like to express our gratitude to the developers and contributors of the following external toolkits, which were instrumental in the success of our research on model merging and kinship analysis:\n\n- [lm-evaluation-harness by EleutherAI](https://github.com/EleutherAI/lm-evaluation-harness) for providing a comprehensive evaluation framework for large language models.\n- [mergekit by arcee-ai](https://github.com/arcee-ai/mergekit) for offering an essential toolkit for model merging experiments.\n\nThese toolkits have significantly contributed to our ability to conduct and reproduce large-scale experiments, and their open-source availability has been invaluable to the broader research community.\n\n---\n\n## Citation:\n\n```bash\n@misc{hu2024exploringmodelkinshipmerging,\n      title={Exploring Model Kinship for Merging Large Language Models}, \n      author={Yedi Hu and Yunzhi Yao and Ningyu Zhang and Shumin Deng and Huajun Chen},\n      year={2024},\n      eprint={2410.12613},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL},\n      url={https://arxiv.org/abs/2410.12613}, \n}\n```","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmodelkinship","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzjunlp%2Fmodelkinship","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzjunlp%2Fmodelkinship/lists"}