{"id":13613132,"url":"https://github.com/databricks/dbrx","last_synced_at":"2025-10-25T00:10:27.424Z","repository":{"id":230030027,"uuid":"777959247","full_name":"databricks/dbrx","owner":"databricks","description":"Code examples and resources for DBRX, a large language model developed by Databricks","archived":false,"fork":false,"pushed_at":"2024-05-01T21:06:57.000Z","size":65,"stargazers_count":2555,"open_issues_count":14,"forks_count":240,"subscribers_count":40,"default_branch":"main","last_synced_at":"2025-05-14T01:22:48.414Z","etag":null,"topics":["databricks","gen-ai","generative-ai","llm","llm-inference","llm-training","mosaic-ai"],"latest_commit_sha":null,"homepage":"https://www.databricks.com/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-26T20:21:43.000Z","updated_at":"2025-05-10T12:12:56.000Z","dependencies_parsed_at":"2024-04-19T16:41:34.792Z","dependency_job_id":"2a458223-b3c4-47b4-ae8f-6b29cb212a25","html_url":"https://github.com/databricks/dbrx","commit_stats":null,"previous_names":["databricks/dbrx"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fdbrx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fdbrx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fdbrx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fdbrx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/dbrx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254276447,"owners_count":22043867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks","gen-ai","generative-ai","llm","llm-inference","llm-training","mosaic-ai"],"created_at":"2024-08-01T20:00:40.363Z","updated_at":"2025-10-25T00:10:27.363Z","avatar_url":"https://github.com/databricks.png","language":"Python","readme":"# DBRX\n\nDBRX is a large language model trained by Databricks, and made available under an open license. This repository contains the minimal code and examples to run inference, as well as a collection of resources and links for using DBRX.\n\n* [Founder's Blog](https://www.databricks.com/blog/announcing-dbrx-new-standard-efficient-open-source-customizable-llms), [DBRX Technical Blog](https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm)\n* Hugging Face: https://huggingface.co/collections/databricks/\n* LLM Foundry: https://github.com/mosaicml/llm-foundry\n\nA reference model code can be found in this repository at [modeling_dbrx.py](model/modeling_dbrx.py). \n\n**Note:** this model code is supplied for references purposes only, please see the [Hugging Face](https://huggingface.co/collections/databricks/) repository for the official supported version.\n\n## Model details\n\nDBRX is a Mixture-of-Experts (MoE) model with 132B total parameters and 36B live parameters. We use 16 experts, of which 4 are active during training or inference. DBRX was pre-trained for 12T tokens of text. DBRX has a context length of 32K tokens.\n\nThe following models are open-sourced:\n\n| Model                                                            | Description                               |\n|------------------------------------------------------------------|-------------------------------------------|\n| [DBRX Base](https://huggingface.co/databricks/dbrx-base)         | Pre-trained base model                    |\n| [DBRX Instruct](https://huggingface.co/databricks/dbrx-instruct) | Finetuned model for instruction following |\n\nThe model was trained using optimized versions of our open source libraries [Composer](https://www.github.com/mosaicml/composer), [LLM Foundry](https://www.github.com/mosaicml/llm-foundry), [MegaBlocks](https://github.com/databricks/megablocks) and [Streaming](https://github.com/mosaicml/streaming).\n\nFor the instruct model, we used the ChatML format. Please see the [DBRX Instruct model card](./MODEL_CARD_dbrx_instruct.md) for more information on this.\n\n\n## Quick start\n\nTo download the weights and tokenizer, please first visit the DBRX Hugging Face page and accept the license. Note: access to the Base model requires manual approval. \n\nWe recommend having at least 320GB of memory to run the model.\n\nThen, run:\n\n```\npip install -r requirements.txt # Or requirements-gpu.txt to use flash attention on GPU(s)\nhuggingface-cli login           # Add your Hugging Face token in order to access the model\npython generate.py              # See generate.py to change the prompt and other settings\n```\n\nFor more advanced usage, please see LLM Foundry ([chat script](https://github.com/mosaicml/llm-foundry/blob/main/scripts/inference/hf_chat.py), [batch generation script](https://github.com/mosaicml/llm-foundry/blob/main/scripts/inference/hf_generate.py))\n\nIf you have any package installation issues, we recommend using our Docker image: [`mosaicml/llm-foundry:2.2.1_cu121_flash2-latest`](https://github.com/mosaicml/llm-foundry?tab=readme-ov-file#mosaicml-docker-images)\n\n## Inference\n\nBoth TensorRT-LLM and vLLM can be used to run optimized inference with DBRX. We have tested both libraries on NVIDIA A100 and H100 systems. To run inference with 16-bit precision, a minimum of 4 x 80GB multi-GPU system is required.\n\n### TensorRT-LLM\n\nDBRX support is being added to TensorRT-LLM library: [Pending PR](https://github.com/NVIDIA/TensorRT-LLM/pull/1363)\n\nAfter merging, instructions to build and run DBRX TensorRT engines will be found at: [README](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/dbrx/README.md)\n\n### vLLM\n\nPlease see the [vLLM docs](https://docs.vllm.ai/en/latest/) for instructions on how to run DBRX with the vLLM engine.\n\n### MLX\n\nIf you have an Apple laptop with a sufficiently powerful M-series chip, quantized version of DBRX can be run with MLX. See instructions for running DBRX on MLX [here](https://huggingface.co/mlx-community/dbrx-instruct-4bit).\n\n\n### LLama.cpp\n\nIf you have an Apple M-series chip laptop with atleast 64GB RAM, you can run a quantized version of DBRX using [llama.cpp](https://github.com/ggerganov/llama.cpp).\n1. Compile llama.cpp \n1. Download a quantized ggml version of dbrx-instruct such as [dranger003/dbrx-instruct-iMat.GGUF](https://huggingface.co/dranger003/dbrx-instruct-iMat.GGUF)\n1. From llama.cpp folder, run:\n```\n./main -ngl 41 -m ./models/ggml-dbrx-instruct-16x12b-iq1_s.gguf -n 256 --repeat_penalty 1.0 --color -i -r \"User:\" -f prompts/chat-with-bob.txt\n```\n\n## Finetune\n\nTo finetune DBRX with our open source library [LLM Foundry](https://www.github.com/mosaicml/llm-foundry), please see the instructions in our training script (found [here](https://github.com/mosaicml/llm-foundry/tree/main/scripts/train)). We have finetuning support for both:\n* Full parameter finetuning, see the yaml config [dbrx-full-ft.yaml](https://github.com/mosaicml/llm-foundry/blob/main/scripts/train/yamls/finetune/dbrx-full-ft.yaml)\n* LoRA finetuning, see the yaml config [dbrx-lora-ft.yaml](https://github.com/mosaicml/llm-foundry/blob/main/scripts/train/yamls/finetune/dbrx-lora-ft.yaml)\n\nNote: LoRA support currently cannot finetune the experts, since the experts are fused. Stay tuned for more.\n\n## Model card\n\nThe model cards can be found at:\n* [DBRX Base](MODEL_CARD_dbrx_base.md)\n* [DBRX Instruct](MODEL_CARD_dbrx_instruct.md)\n\n## Integrations\n\nDBRX is available on the Databricks platform through:\n* [Mosaic AI Model Serving](https://docs.databricks.com/machine-learning/foundation-models/supported-models.html#dbrx-instruct)\n* [Mosaic AI Playground](https://docs.databricks.com/en/large-language-models/ai-playground.html)\n\nOther providers have recently added support for DBRX:\n* [You.com](https://you.com/)\n* [Perplexity Labs](https://labs.perplexity.ai/)\n* [LlamaIndex](https://docs.llamaindex.ai/en/stable/api_reference/llms/databricks/) ([starter example gist](https://gist.github.com/dennyglee/3a4fd9042c283549b727e081397842da))\n\nThe same tools used to train high quality MoE models such as DBRX are available for Databricks customers. Please reach out to us at https://www.databricks.com/company/contact if you are interested in pre-training, finetuning, or deploying your own DBRX models!\n\n## Issues\nFor issues with model output, or community discussion, please use the Hugging Face community forum ([instruct](https://huggingface.co/databricks/dbrx-instruct), [base](https://huggingface.co/databricks/dbrx-base))\n\nFor issues with LLM Foundry, or any of the underlying training libraries, please open an issue on the relevant GitHub repository.\n\n## License\n\nOur model weights and code are licensed for both researchers and commercial entities. The [Databricks Open Source License](https://www.databricks.com/legal/open-model-license) can be found at [LICENSE](LICENSE), and our Acceptable Use Policy can be found [here](https://www.databricks.com/legal/acceptable-use-policy-open-model).\n","funding_links":[],"categories":["Repos","Python","Project List","A01_文本生成_文本对话"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e","大语言对话模型及数据"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fdbrx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fdbrx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fdbrx/lists"}