{"id":13605701,"url":"https://github.com/SkunkworksAI/hydra-moe","last_synced_at":"2025-04-12T05:34:29.920Z","repository":{"id":191748752,"uuid":"685314572","full_name":"SkunkworksAI/hydra-moe","owner":"SkunkworksAI","description":null,"archived":false,"fork":false,"pushed_at":"2023-11-02T22:53:15.000Z","size":36023,"stargazers_count":409,"open_issues_count":14,"forks_count":15,"subscribers_count":23,"default_branch":"alpha_v1_merged","last_synced_at":"2024-08-02T19:38:21.291Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SkunkworksAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-08-31T00:54:13.000Z","updated_at":"2024-07-31T07:58:47.000Z","dependencies_parsed_at":null,"dependency_job_id":"b127c9f8-ae44-4520-82de-8a0ca7ccc47f","html_url":"https://github.com/SkunkworksAI/hydra-moe","commit_stats":null,"previous_names":["skunkworksai/hydra-moe"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkunkworksAI%2Fhydra-moe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkunkworksAI%2Fhydra-moe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkunkworksAI%2Fhydra-moe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SkunkworksAI%2Fhydra-moe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SkunkworksAI","download_url":"https://codeload.github.com/SkunkworksAI/hydra-moe/tar.gz/refs/heads/alpha_v1_merged","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223498002,"owners_count":17155238,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:01.749Z","updated_at":"2024-11-07T10:31:07.192Z","avatar_url":"https://github.com/SkunkworksAI.png","language":"Python","funding_links":[],"categories":["Mixture of Experts (Sparse MoE)","Python"],"sub_categories":[],"readme":"## Hydra-MoE: A new class of Open-Source Mixture of Experts\n\n\u003cp align=\"center\"\u003e\n🤗 \u003ca href=\"https://huggingface.co/HydraLM\" target=\"_blank\"\u003eHF Repo\u003c/a\u003e • 🐦 \u003ca href=\"https://twitter.com/skunkworks_ai\" target=\"_blank\"\u003eTwitter\u003c/a\u003e • ⚡ \u003ca href=\"https://github.com/hydrallm\" target=\"_blank\"\u003eGithub\u003c/a\u003e • 👋 Join our \u003ca href=\"https://discord.gg/bNKsa8gE2y\" target=\"_blank\"\u003eDiscord\u003c/a\u003e \u003cbr\u003e\n\u003c/p\u003e\n\n\u003cimg src='imgs/llamas.png' width=300 height=300 \u003e\n\n## Mission\nA scalable, efficient and Open Source Mixture of Experts (MoE) architecture to enable OSS AI to achieve SOTA (GPT-4 level) performance.\n\n\n## Quickstart\nSetup:\n```\nsh setup_moe.sh\n```\n\nRun Inference:\n```\npython main.py --inference\n```\n\nRun with Docker:\n\nWebui:\n```\ncp .envtemplate .env # Add HUGGINGFACE_API_TOKEN to .env or \ndocker compose up webui\n```\n\nCLI :\n```\ndocker compose up cli\n```\n\n## Description\n\nSkunkworks OSS team introduces Hydra-MoE, an innovative Mixture of Experts (MoE) architecture that leverages LoRA/QLoRA experts to scale and augment the performance of base language models. The central aim of this research is to transmute any base language model into an advanced, lightweight, and efficient MoE framework, employing swappable QLoRA Expert Adapters, with the objective of achieving performance levels that rival state-of-the-art models and can run on commodity/consumer hardware. \n\n## Project Status\n\nThis project has completed a PoC stage and is currently training, optimizing, evaluating and scaling several MoE architectures developed by our team.\n\n## Goals\n\nOur objectives for this project are:\n- Enhance Llama-2 (or any base model) capabilities to be closer to GPT-4's level of performance, through MoE architecture.\n- Serve as an exhaustive open source attempt to validate MoE techniques for large language models.\n- Release open source and effective MoE models, framework and infrastructure for scaling capabilities of any LLMs.\n\n## Our Architectures\n\nWe have designed and developed several MoE architectures which have achieved promising early results.\n\n1. Hydra-α: Swappable-QLoRA experts MoE\n   - \u003ca href=\"https://arxiv.org/abs/2303.14177\" target=\"_blank\"\u003ec-BTM Inspired\u003c/a\u003e.\n   - K-means clustering for unsupervised domain discovery from instruct dataset.\n   - \u003ca href=\"https://arxiv.org/abs/2305.14314\" target=\"_blank\"\u003eQLoRA\u003c/a\u003e finetuned experts on discovered clusters.\n   - Dynamic swapping of QLoRA Experts at inference through similarity and classifier based methods.\n   - Weighted-average merging of experts with base model at inference.\n     \n2. Hydra-β: End-to-End (E2E) MoE-QLoRA\n   - An Extension to Hydra-α with better gating, merging and an E2E training method.\n   - Expert merging through \u003ca href=\"https://arxiv.org/abs/2306.01708\" target=\"_blank\"\u003eTIES merging\u003c/a\u003e and other methods.\n   - E2E Training of gating/routing functions with exotic loss/reward functions.\n\nThese architectures are currently being trained on further data and scaled to larger sizes. More architectures are being experimented with and will be revealed when validated.\nEarly results indicate our MoE architectures outperform base models across several benchmarks. These results are currently being validated and replicated before being published.\n\n## Evaluation Criteria\n\nWe evaluate Hydra-MoEs using the LM-Evaluation-Harness, Fast-Eval and human evaluations in order to compare capabilities to the base model leveraged by the MoE. Key evaluation metrics are perplexity, accuracy on downstream tasks, and human evaluations.\n\n## Initial Expert Domains\n\nWe are training experts for the following domains:\n\n- Math \n- Science (Physics, Chemistry, Biology...etc)\n- Reasoning\n- Coding\n- Writing\n- Truthfulness\n- Structured data handling\n- Tool integration\n- Role-Playing\n\n## Datasets\nThe Skunkworks community is collecting, consolidating and standardizing high quality public and private Instruct data sources to craft a unified source of data for our MoE initiative and any open-source project. The collective size of the datasets exceeds 90 separate sources covering all domains listed above.\n\nWe are collaborating with several academic and open source groups (including \u003ca href=\"https://twitter.com/alignment_lab\" target=\"_blank\"\u003eAlignment Lab\u003c/a\u003e and members of Nous) to expand datasets further. We are also crafting new instruct datasets from high quality raw data sources.\n\nFull list of (public) datasets: https://docs.google.com/spreadsheets/d/1tkRZ8T7LZ0ojDUzNFWMKcrPiQJ8Glh-GIFfB7Bksb60/edit?usp=sharing\n\n## Roadmap\nIn less than 2 weeks (in July), our community came together to build a viable OSS MoE. Since then, we've been expanding our experiments, improving designs, implementing new archs, performing sweeps, optimizing code and evaluating models. \n\nFor the next few weeks, we will be exploring new architectures as well as optimizing, training and scaling existing performant architectures. Our objective is to achieve SOTA performance within the coming months by scaling a 70B base model with our most performant MoE arch.\n\n## Release Plan\nWe open-source everything. Currently, all of our datasets and trained experts are open-sourced on the HydraLM HF repo. In the next month(s), we aim to open-source the training/inference infrastructure for anyone to easily and rapidly build their own Hydra-MoE with custom data. We aim to continuously open-source more advanced architectures as we scale and validate them.\n\n## Compute Requirements\n\nThe Skunkworks team is currently seeking supporters/sponsors to provide us access to reliable continuous compute for the next 3-6 months. Our target compute objective is 64x GPUs (preferably H100s/A100s). \nThis compute will help us scale our MoE experiments and the most performant MoE architecture to GPT-4 level performance.\n\n## Contributors and Collaborations\n\nThe Skunkworks community is comprised of hundreds of the top minds in OSS AI. The core team collaborates directly with individuals, open-source groups, academics and companies to accelerate the development of our data pipeline and MoE architectures.\nWe welcome open source contributions to HydraLM! Please join our Discord for details.\n\nContributors:\n- \u003ca href=\"https://twitter.com/far__el\" target=\"_blank\"\u003eFarouk E.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/prateeky2806\" target=\"_blank\"\u003ePrateek Y.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/sumo_43\" target=\"_blank\"\u003eArtem\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/propback_\" target=\"_blank\"\u003eAlpay A.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/Sentdex\" target=\"_blank\"\u003eHarrison K.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/Yampeleg\" target=\"_blank\"\u003eYam P.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/alignment_lab\" target=\"_blank\"\u003eAlignment Lab\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/Teknium1\" target=\"_blank\"\u003eTeknium\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/intrstllrninja\" target=\"_blank\"\u003einterstellarninja\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/jeremyphoward\" target=\"_blank\"\u003eJeremy H.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/TheSlavant\" target=\"_blank\"\u003eYaroslav\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/winglian\" target=\"_blank\"\u003eCaseus\u003c/a\u003e\n- \u003ca href=\"https://github.com/thennal10\" target=\"_blank\"\u003ePremeditator\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/ldjconfirmed\" target=\"_blank\"\u003eLuigi D.\u003c/a\u003e\n- \u003ca href=\"https://twitter.com/kranthigv\" target=\"_blank\"\u003eKranthi\u003c/a\u003e\n- \u003ca href=\"https://huggingface.co/Neilbert\" target=\"_blank\"\u003eNeilbert\u003c/a\u003e\n\nAnd many more. Full list of contributors will be posted soon.\n\n\n## Background on MoE\nThere is a tradeoff in dense language models between capabilities and inference time.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003ca \u003e\u003cimg src=\"imgs/triangle_of_success.png\" alt=\"WizardLM\" style=\"width: 50%; min-width: 50%; display: block; margin: auto;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nTo speed up inference, there is data or model parallelism. Alternatively, if we want to solve this problem without decreasing “efficiency”, i.e. how much GPU time you’re spending per inference, we would need a way to decouple inference FLOPs from model capabilities. Mixture of Experts are a class of sparse language models designed to perform this tradeoff at the cost of additional memory. In an MoE model, the linear layers of the model are replaced by a Gating Mechanism that routes to N experts.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003ca \u003e\u003cimg src=\"imgs/moe.png\" alt=\"WizardLM\" style=\"width: 50%; min-width: 50%; display: block; margin: auto;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nThe Gating network’s job is to compute probabilities for each expert, per token. Then, at each token, k=1, 2, or 3 experts are selected and used for the MoE layer, with the results concatenated in the end. In practice, the tokens are split up (almost) evenly between the experts, and processed in sequence during the forward pass. By increasing the number of experts, the amount of memory that the model uses goes up, but the cost of inference stays exactly the same, since we haven’t changed the amount of computation done by the model during each foward pass. MoE is an architecture that allows us to make the tradeoff we wanted to make. But do the results of MoE models stack up to dense transformers? Given that they still have more parameters in total, can we expect better performance from MoE mdoels with a similar number of inference FLOPs? In the original MoE paper from 2017, the authors use RNNs. But since RNNs are old, we will compare the results from the Switch Transformer paper from 2021, where the Google authors use the T5 architecture.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003ca \u003e\u003cimg src=\"imgs/graph.png\" alt=\"WizardLM\" style=\"width: 50%; min-width: 50%; display: block; margin: auto;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nBased on the results, MoE models not only outperform dense models in terms of inference FLOPs, they also punch above their weight class: Switch-Base achieves lower perplexity on C4 than T5-Large while using 30% the FLOPs and 10x the parameters, while Switch-Large is competitive with T5-XXL with 6% the FLOPs and 2.5x the parameters.\n\nThere are several other MoE architectures that achieve varying results but consistently display the ability to scale LMs including Branch-Train-Merge (BTM), Clustered-BTM (c-BTM), GLaM... All papers are listed below in related works.\n\nGPT-4 is rumored to be a MoE architecture with 8x200B experts.\n\n## Related works\n- \u003ca href=\"https://arxiv.org/pdf/2112.06905.pdf\" target=\"_blank\"\u003eGLaM: Efficient Scaling of Language Models with Mixture-of-Experts\u003c/a\u003e\n- \u003ca href=\"https://arxiv.org/pdf/2108.05036.pdf\" target=\"_blank\"\u003eDEMix Layers: Disentangling Domains for Modular Language Modeling\u003c/a\u003e\n- \u003ca href=\"https://arxiv.org/pdf/2208.03306.pdf\" target=\"_blank\"\u003eBranch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models\u003c/a\u003e\n- \u003ca href=\"https://arxiv.org/abs/2303.14177\" target=\"_blank\"\u003eScaling Expert Language Models with Unsupervised Domain Discovery\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/pdf/2306.03745.pdf\" target=\"_blank\"\u003eSoft Merging of Experts with Adaptive Routing\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/abs/2101.03961\" target=\"_blank\"\u003eSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/pdf/2005.00247.pdf\" target=\"_blank\"\u003eAdapterFusion: Non-Destructive Task Composition for Transfer Learning\u003c/a\u003e \n- \u003ca href=\"https://suchin.io/assets/knnprompt.pdf\" target=\"_blank\"\u003eNearest Neighbor Zero-Shot Inference\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/abs/2205.12701\" target=\"_blank\"\u003eEliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/pdf/2306.04845.pdf\" target=\"_blank\"\u003eMixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts\u003c/a\u003e \n- \u003ca href=\"https://arxiv.org/abs/2212.05055\" target=\"_blank\"\u003eSparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints\u003c/a\u003e \n- \u003ca href=\"https://www.microsoft.com/en-us/research/uploads/prod/2022/05/Mixture_of_Adaptations_EMNLP_2022-2.pdf\" target=\"_blank\"\u003eAdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning\u003c/a\u003e \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSkunkworksAI%2Fhydra-moe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSkunkworksAI%2Fhydra-moe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSkunkworksAI%2Fhydra-moe/lists"}