{"id":16241076,"url":"https://github.com/UbiquitousLearning/mllm","last_synced_at":"2025-10-25T02:31:01.910Z","repository":{"id":218860437,"uuid":"685159924","full_name":"UbiquitousLearning/mllm","owner":"UbiquitousLearning","description":"Fast Multimodal LLM on Mobile Devices","archived":false,"fork":false,"pushed_at":"2025-10-18T13:14:31.000Z","size":48463,"stargazers_count":1121,"open_issues_count":39,"forks_count":136,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-10-21T20:44:03.530Z","etag":null,"topics":["large-language-models","llama","multimodal"],"latest_commit_sha":null,"homepage":"https://ubiquitouslearning.github.io/mllm_website","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/UbiquitousLearning.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-08-30T16:33:36.000Z","updated_at":"2025-10-21T16:47:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"294224ef-c014-4ce2-a56c-352d53bc8cd2","html_url":"https://github.com/UbiquitousLearning/mllm","commit_stats":{"total_commits":1252,"total_committers":21,"mean_commits":59.61904761904762,"dds":0.6166134185303515,"last_synced_commit":"ea17acffc39d05ce03ec6c9384bf2c4428cc4d99"},"previous_names":["ubiquitouslearning/mllm"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/UbiquitousLearning/mllm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UbiquitousLearning%2Fmllm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UbiquitousLearning%2Fmllm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UbiquitousLearning%2Fmllm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UbiquitousLearning%2Fmllm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/UbiquitousLearning","download_url":"https://codeload.github.com/UbiquitousLearning/mllm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/UbiquitousLearning%2Fmllm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280871982,"owners_count":26405605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models","llama","multimodal"],"created_at":"2024-10-10T14:03:53.248Z","updated_at":"2025-10-25T02:30:56.902Z","avatar_url":"https://github.com/UbiquitousLearning.png","language":"C++","funding_links":[],"categories":["Hardware Acceleration and Deployment Strategies","Multimodal","Open Source Projects","3. Inference Engines \u0026 Serving","🚀 Inference Engines"],"sub_categories":["Popular On-Device LLMs Framework","5. Multimodal LLM","1. DL Framework on Mobile","LLM \u0026 GenAI Specialized"],"readme":"\u003ch1 align=\"center\"\u003e\nmllm\n\u003c/h1\u003e\n\n\u003ch3 align=\"center\"\u003e\nfast and lightweight \u003cins\u003emultimodal LLM\u003c/ins\u003e inference engine for mobile and edge devices\n\u003c/h3\u003e\n\n\u003ch4 align=\"center\"\u003e\n| Arm CPU | X86 CPU | Qualcomm NPU(QNN) |\n\u003c/h4\u003e\n\n\u003ch4 align=\"center\"\u003e\n\n[![Website](https://img.shields.io/badge/website-visit-green)](https://ubiquitouslearning.github.io/mllm_website/)\n[![Documentation](https://img.shields.io/badge/view-docs-blue)](https://ubiquitouslearning.github.io/mllm_website/introduction/getstarted/)\n[![Android App](https://img.shields.io/badge/android-app-pink)](https://github.com/lx200916/ChatBotApp/)\n[![Actions Status](https://github.com/UbiquitousLearning/mllm/workflows/Tests/badge.svg)](https://github.com/UbiquitousLearning/mllm/actions)\n\u003c/h4\u003e\n\n- Plain C/C++ implementation without dependencies\n- Optimized for multimodal LLMs like fuyu-8B and LLaVA\n- Supported: ARM NEON, x86 AVX2, Qualcomm NPU (QNN), etc\n- Various quantization schemes\n- End-to-end Android app demo\n\nWhy on-device multimodal LLM? - It's a key building block for [intelligent personal agent](https://arxiv.org/pdf/2401.05459.pdf), text-based image searching/retrieval, screen VQA, and many more exciting mobile apps, without giving away your private data (chat history, screenshots, taken photos, etc).\n\n## Recent update\n- [2024 November 21] Support new model: Phi 3 Vision https://github.com/UbiquitousLearning/mllm/pull/186\n- [2024 August 30] Support new model: MiniCPM 2B https://github.com/UbiquitousLearning/mllm/pull/132\n- [2024 August 15] Support new model: Phi 3 mini https://github.com/UbiquitousLearning/mllm/pull/119\n- [2024 Aug 10] Supporting Qualcomm NPU: https://github.com/UbiquitousLearning/mllm/pull/112 | [try it out](https://github.com/UbiquitousLearning/mllm/tree/main/src/backends/qnn) | [paper](https://arxiv.org/pdf/2407.05858v1)\n\n\n### Contents\n- [Android Demo](#android-demo)\n- [Support models](#support-models)\n- [Quick Start](#quick-start)\n    - [Get the Code](#get-the-code)\n    - [Check prerequisites](#check-prerequisites)\n    - [Run Qwen with Hexagon NPU accelerating using QNN](#run-qwen-with-hexagon-npu-accelerating-using-qnn)\n    - [Run with the CPU of Android](#run-with-the-cpu-of-android)\n    - [Run for Linux](#run-for-linux)\n- [Customization](#customization)\n    - [Convert models](#convert-models)\n    - [Convert vocabulary](#convert-vocabulary)\n    - [Quantize models](#quantize-models)\n- [Roadmap](#roadmap)\n- [Documentation](#documentation)\n- [Contribution](#contribution)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n\n\n## Android Demo\n\n\u003ctable\u003e\n    \u003ctr\u003e\n\u003c!--         \u003ctd\u003eChatting\u003c/td\u003e --\u003e\n        \u003ctd\u003eAndroid Intent Invocation\u003c/td\u003e\n        \u003ctd\u003eImage Understanding\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n\u003c!--         \u003ctd\u003e  \u003cvideo src=\"https://github.com/user-attachments/assets/972b3bad-d659-4d76-9141-64ad0ad34d64\"\u003e \u003c/td\u003e --\u003e\n        \u003ctd\u003e  \u003cvideo src=\"https://github.com/user-attachments/assets/deb99f8d-9727-4519-9ca7-c39deb7c5b47\"\u003e \u003c/td\u003e\n        \u003ctd\u003e  \u003cvideo src=\"https://github.com/user-attachments/assets/55321a43-8484-4f74-b7b2-d4495f3626d9\"\u003e \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003eChat CPU\u003c/td\u003e\n        \u003ctd\u003eChat NPU\u003c/td\u003e\n    \u003c/tr\u003e    \n    \u003ctr\u003e\n        \u003ctd\u003e  \u003cvideo src=\"https://github.com/user-attachments/assets/2b0ab0d6-6727-4b85-9ee3-b39d23de5dde\"\u003e \u003c/td\u003e\n        \u003ctd\u003e  \u003cvideo src=\"https://github.com/user-attachments/assets/395f8e6e-2ab9-40bc-bf26-164ba5695c64\"\u003e \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\n## Support models\n\n### Language models\n\n| Model                                                                       | CPU \u003cbr\u003e FP32 | CPU \u003cbr\u003e INT4  | Hexagon NPU \u003cbr\u003e INT8 |\n|-----------------------------------------------------------------------------|------|-----|----------------------------|\n| [LLaMA 2 7B](https://github.com/facebookresearch/llama)                   | [✔️](https://huggingface.co/mllmTeam/llama-2-7b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/llama-2-7b-mllm/tree/main)   |  |\n| [LLaMA 3 1B](https://github.com/meta-llama/llama3)                   | [✔️](https://huggingface.co/mllmTeam/llama-3.2-1b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/llama-3.2-1b-mllm/tree/main)   |  |\n| [LLaMA 3 3B](https://github.com/meta-llama/llama3)                   | [✔️](https://huggingface.co/mllmTeam/llama-3.2-3b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/llama-3.2-3b-mllm/tree/main)   |  |\n| [Alpaca 7B](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2)                | [✔️](https://huggingface.co/mllmTeam/chinese-alpaca-7b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/chinese-alpaca-7b-mllm/tree/main)   |  |\n| [TinyLLaMA 1.1B](https://github.com/jzhang38/TinyLlama)                     | [✔️](https://huggingface.co/mllmTeam/tinyllama-1.1b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/tinyllama-1.1b-mllm/tree/main)   |  |\n| [LLaVA 7B](https://github.com/haotian-liu/LLaVA)                            | [✔️](https://huggingface.co/mllmTeam/llava-1.5-7b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/llava-1.5-7b-mllm/tree/main)   |  |\n| [Gemma 2B](https://github.com/google/gemma_pytorch)                         | [✔️](https://huggingface.co/mllmTeam/gemma-2b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/gemma-2b-mllm/tree/main)   |  |\n| [Gemma 2 2B](https://github.com/google/gemma_pytorch)                         | [✔️](https://huggingface.co/mllmTeam/gemma-2-2b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/gemma-2-2b-mllm/tree/main)   |  |\n| [Qwen 1.5 0.5B](https://github.com/QwenLM/Qwen)                                 | [✔️](https://huggingface.co/mllmTeam/qwen-1.5-0.5b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/qwen-1.5-0.5b-mllm/tree/main)   |  |\n| [Qwen 1.5 1.8B](https://github.com/QwenLM/Qwen)                            | [✔️](https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm)  | [✔️](https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm)   | [✔️](https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm) |\n| [Qwen 2.5 1.5B](https://github.com/QwenLM/Qwen2.5) | [✔️](https://huggingface.co/mllmTeam/qwen-2.5-1.5b-mllm/tree/main) | [✔️](https://huggingface.co/mllmTeam/qwen-2.5-1.5b-mllm/tree/main) | |\n| [Mistral 7B](https://github.com/mistralai/mistral-src)                      | [✔️](https://huggingface.co/mllmTeam/mistral-7b-instruct-v0.2-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/mistral-7b-instruct-v0.2-mllm/tree/main)   |  |\n| [Yi 6B](https://huggingface.co/01-ai/Yi-1.5-6B)                             | [✔️](https://huggingface.co/mllmTeam/yi-1.5-6b-chat-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/yi-1.5-6b-chat-mllm/tree/main)   |  |\n| [StableLM 2 1.6B](https://github.com/Stability-AI/StableLM)                     | [✔️](https://huggingface.co/mllmTeam/stablelm-2-1.6b-chat-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/stablelm-2-1.6b-chat-mllm/tree/main)   |  |\n| [OPT 1.3B](https://github.com/facebookresearch/metaseq/tree/main/projects/OPT)                     | [✔️](https://huggingface.co/mllmTeam/opt-1.3b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/opt-1.3b-mllm/tree/main)   |  |\n| [Phi 3 mini 3.8B](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct)                     |  [✔️](https://huggingface.co/mllmTeam/phi-3-mini-instruct-mllm/tree/main)   | [✔️](https://huggingface.co/mllmTeam/phi-3-mini-instruct-mllm/tree/main)   |  |\n| [MiniCPM 2B](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp32)                     |  [✔️](https://huggingface.co/mllmTeam/minicpm-2b-dpo-mllm/tree/main)   | [✔️](https://huggingface.co/mllmTeam/minicpm-2b-dpo-mllm/tree/main)   |  |\n| [MiniCPM 3 4B](https://huggingface.co/openbmb/MiniCPM3-4B)                     |  [✔️](https://huggingface.co/mllmTeam/minicpm3-4b-mllm/tree/main)   | [✔️](https://huggingface.co/mllmTeam/minicpm3-4b-mllm/tree/main)   |  |\n| [MiniCPM MoE 8x2B](https://huggingface.co/openbmb/MiniCPM-MoE-8x2B)                     |  [✔️](https://huggingface.co/mllmTeam/minicpm-moe-8x2b-mllm/tree/main)   | [✔️](https://huggingface.co/mllmTeam/minicpm-moe-8x2b-mllm/tree/main)   |  |\n| [SmolLM 1.7B](https://huggingface.co/HuggingFaceTB/SmolLM-1.7B-Instruct)                     |  [✔️](https://huggingface.co/mllmTeam/smollm-1.7b-instruct-mllm/tree/main)   | [✔️](https://huggingface.co/mllmTeam/smollm-1.7b-instruct-mllm/tree/main)   |  |\n| [DCLM 1B](https://huggingface.co/TRI-ML/DCLM-1B) | [✔️](https://huggingface.co/mllmTeam/dclm-1b-mllm/tree/main)| [✔️](https://huggingface.co/mllmTeam/dclm-1b-mllm/tree/main)| |\n| [OpenELM 1.1B](https://github.com/apple/corenet/tree/main/projects/openelm) | [✔️](https://huggingface.co/mllmTeam/openelm-1.1b-mllm/tree/main)| [✔️](https://huggingface.co/mllmTeam/openelm-1.1b-mllm/tree/main)| |\n[PhoneLM 1.5B](https://github.com/UbiquitousLearning/PhoneLM) | [✔️](https://huggingface.co/mllmTeam/phonelm-1.5b-mllm/tree/main)| [✔️](https://huggingface.co/mllmTeam/phonelm-1.5b-mllm/tree/main)| [✔️](https://huggingface.co/mllmTeam/phonelm-1.5b-mllm/tree/main)|\n\n### Multimodal models\n\n| Model                                                                       | CPU \u003cbr\u003e FP32 | CPU \u003cbr\u003e INT4  | \n|-----------------------------------------------------------------------------|------|-----|\n| [Fuyu 8B](https://www.adept.ai/blog/fuyu-8b)                                | [✔️](https://huggingface.co/mllmTeam/fuyu-8b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/fuyu-8b-mllm/tree/main)   |  \n| [Vision Transformer](https://github.com/google-research/vision_transformer) | [✔️](https://huggingface.co/mllmTeam/vit-base-patch16-224-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/vit-base-patch16-224-mllm/tree/main)   | \n| [CLIP](https://github.com/openai/CLIP)                                      | [✔️](https://huggingface.co/mllmTeam/clip-vit-base-patch32-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/clip-vit-base-patch32-mllm/tree/main)   |\n| [ImageBind](https://github.com/facebookresearch/ImageBind) (3 modalities)   | [✔️](https://huggingface.co/mllmTeam/imagebind_huge-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/imagebind_huge-mllm/tree/main)   | \n| [LLaVA 7B](https://github.com/haotian-liu/LLaVA)                            | [✔️](https://huggingface.co/mllmTeam/llava-1.5-7b-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/llava-1.5-7b-mllm/tree/main)   |\n| [Phi 3 Vision](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct)                            | [✔️](https://huggingface.co/mllmTeam/Phi-3-vision-instruct-mllm/tree/main)  | [✔️](https://huggingface.co/mllmTeam/Phi-3-vision-instruct-mllm/tree/main)   |\n\n\n## Quick Start\n\n### Get the Code\n\n```bash\ngit clone https://github.com/UbiquitousLearning/mllm\ncd mllm\n```\n\n### Check prerequisites\n\nBuilding mllm requires following tools:\n\n- gcc(11.4+) / clang (11.0+)\n- CMake \u003e= 3.18\n- Android NDK Toolchains \u003e= 26\n\n\n\u003e Note that building OpenMP libs on macOS may fail due to Apple LLVM compiler, so we disable OpenMP on macOS by default, you may experience slower performance on macOS. Build mllm is more recommended on Linux.\n\n### Run Qwen with Hexagon NPU accelerating using QNN\n\n*`NOTE:` The QNN backend is preliminary version which can do end-to-end inference. It is still under active development for better performance and more supported models.*\n\nWe support running Qwen-1.5-1.8B-Chat using [Qualcomm QNN](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk) to get Hexagon NPU acceleration on devices with Snapdragon 8 Gen3. The details of QNN environment set up and design is [here](./src/backends/qnn/README.md). The prefilling stage is performered by QNN \u0026 CPU, and the inference stage is performed by CPU.\n\nBuild the target with QNN backend.\n\n```bash\ncd ../script\n./build_qnn_android.sh\n```\n\nDownload the model from [here](https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm/blob/main/), or using the following instructions\n\n```bash\nmkdir ../models \u0026\u0026 cd ../models\n# Download int8 model used by npu \u0026 q4k model used by cpu\nwget https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm/resolve/main/qwen-1.5-1.8b-chat-int8.mllm?download=true  -O qwen-1.5-1.8b-chat-int8.mllm\nwget https://huggingface.co/mllmTeam/qwen-1.5-1.8b-chat-mllm/resolve/main/qwen-1.5-1.8b-chat-q4k.mllm?download=true  -O qwen-1.5-1.8b-chat-q4k.mllm\n```\n\nRun on an android phone with at least 16GB of memory.\n\n```bash\ncd ../script\n./run_qwen_npu.sh\n```\n\nThere are two arguments in the executable. `-s` is for the sequence length of prefilling, the default value is 64 in the demo we provided. `-c` for type of QNN prefilling options, when it is set to 1, the input will be splited into many chunks of sequence 32 and be executed in a pipeline. When it is set to 0, the input will be executed in one chunk.\n\nResult are as followed:\n\n```\n\u003e ./main_qwen_npu -s 64 -c 1\n[Q] \u003c|im_start|\u003esystem\nYou are a helpful assistant.\u003c|im_end|\u003e\n\u003c|im_start|\u003euser\nGive me a short introduction to large language model.\u003c|im_end|\u003e\n\u003c|im_start|\u003eassistant\n\n[A] A short introduction to a large language model is a type of artificial intelligence language model that is designed to understand and generate human language text. These models are typically trained on large amounts of text data, such as books, articles, and other written materials, to learn the patterns and structures of human language. They use a combination of natural language processing (NLP)\n```\n\n### Run with the CPU of Android\n\n#### Build\n\n  ```bash\n  export ANDROID_NDK=/path/to/your/ndk\n  cd scripts\n  ./build_android.sh\n  ```\n\n#### Run Fuyu-8B\n\nDownload the model from [here](https://huggingface.co/mllmTeam/fuyu-8b-mllm/tree/main/), or using the following instructions\n\n```bash\nmkdir ../models \u0026\u0026 cd ../models\n# Download fuyu-8b-q4_k.mllm\nwget https://huggingface.co/mllmTeam/fuyu-8b-mllm/resolve/main/fuyu-8b-q4_k.mllm?download=true  -O fuyu-8b-q4_k.mllm\n```\n\nRun on an android phone with at least 12GB of memory.\n\n```bash\ncd ../script\n./run_fuyu.sh\n```\n\nResult are as followed:\n\n```\n\u003e ./demo_fuyu\n[Q] [../assets/bus.png]Generate a coco-style caption.\n\n[A]  A blue bus driving down▁the▁road next▁to a sidewalk.\n[Q] [../assets/two_cats.jpg]What's this?\n\n[A]  Two cats laying on▁top▁of a pink blanket.\n```\n\n#### Run LLaMA-2-7B\n\nDownload model from [here](https://huggingface.co/mllmTeam/llama-2-7b-mllm/tree/main/), or using the following instructions\n\n```bash\nmkdir ../models \u0026\u0026 cd ../models\n# Dowload llama-2-7b-chat-q4_k.mllm\nwget https://huggingface.co/mllmTeam/llama-2-7b-mllm/resolve/main/llama-2-7b-chat-q4_k.mllm?download=true  -O llama-2-7b-chat-q4_k.mllm\n```\n\nRun on an android phone with at least 12GB of memory.\n\n```bash\ncd ../script\n./run_llama.sh\n```\n\nResult are as followed:\n\n```\n\u003e ./demo_llama\n[Q]  Hello, who are you?\n[A]\n\nI am a bot, I'm here to help answer your questions and provide information. I'm just an AI, I don't have personal experiences or feelings like humans do, but I'm here to help you with any questions you may have. Is there something specific you would like to know or talk about?\n[Q]  What can you do?\n[A]\nI can help with a wide range of tasks, including:\n  \n* Answering questions on various topics, such as science, history, technology, and more.\n* Providing information on different subjects, such as culture, entertainment, and lifestyle.\n* Helping with language-related tasks, such as translating text or improving grammar and syntax.\n* Generating text, such as articles, stories, or even entire books.\n* Assisting with coding\n[Q]  Please introduce Beijing University of Posts and Telecommunications.\n[A]\nBeijing University of Posts and Telecommunications (BUPT) is a public research university located in Beijing, China. It was established in 1955 and is one of the top universities in China, with a strong focus on engineering, technology, and telecommunications.\nBUPT offers a wide range of undergraduate and graduate programs in fields such as computer science, electrical engineering, software engineering, and communication engineering. The university\n```\n\n#### Run ImageBind\n\nDownload model from [here](https://huggingface.co/mllmTeam/imagebind_huge-mllm/tree/main), or using the following instructions\n\n```bash\nmkdir ../models \u0026\u0026 cd ../models\n# Download imagebind_huge-q4_k.mllm\nwget https://huggingface.co/mllmTeam/imagebind_huge-mllm/resolve/main/imagebind_huge-q4_k.mllm?download=true -O imagebind_huge-q4_k.mllm \n```\n\nRun on an android phone with at least 4GB of memory.\n\n```bash\ncd ../script\n./run_imagebind.sh\n```\n\nResult are as followed:\n```\n\u003e ./demo_imagebind \nvision X text :\n0.9985647 0.0013827 0.0000526 \n0.0000365 0.9998636 0.0000999 \n0.0000115 0.0083149 0.9916736 \nvision X audio :\n0.8054272 0.1228001 0.0717727 \n0.0673458 0.8429284 0.0897258 \n0.0021967 0.0015335 0.9962698 \n```\n\n\n### Run for Linux\n\n#### Build\n```bash\ncd scripts\n./build.sh\n ```\n\n#### Run Fuyu-8B\n\n```bash\ncd ./bin\n./demo_fuyu -m ../models/fuyu-8b-q4_k.mllm -v ../vocab/fuyu_vocab.mllm\n ```\n\n#### Run LLaMA-2-7B\n\n```bash\ncd ./bin\n./demo_llama -m ../models/llama-2-7b-chat-q4_k.mllm -v ../vocab/llama2_vocab.mllm\n```\n\n\n#### Run ImageBind\n\n```bash\ncd ./bin\n./demo_imagebind -m ../models/imagebind_huge-q4_k.mllm -v ../vocab/clip_vocab.mllm\n```\n\n\n## Customization\n\n### Convert models\n\nYou can download models from [here](https://huggingface.co/mllmTeam), or you can convert a pytorch/safetensor model to\nmllm model by yourself.\n\n```bash\ncd tools/convertor\npip install -r ./requirements.txt\n\n# for one file pytorch model\npython converter.py --input_model=model.pth --output_model=model.mllm --type=torch\n\n# for multi-file pytorch model\npython converter.py --input_model=pytorch_model.bin.index.json --output_model=model.mllm --type=torch\n\n# for one file safetensor model\npython converter.py --input_model=model.bin --output_model=model.mllm --type=safetensor\n\n# for multi-file safetensor model\npython converter.py --input_model=model.safetensors.index.json --output_model=model.mllm --type=safetensor\n``` \n\n### Convert vocabulary\n\nYou can convert vocabulary to mllm vocabulary as followed.\n\n```bash\ncd tools/convertor\npython vocab.py --input_file=tokenizer.json --output_file=vocab.mllm --type=Unigram\n```\n\n### Quantize models\n\nYou can quantize mllm model to int4 model by yourself.\nmllm only support two quantize modes: Q4_0 and Q4_K.\n\n```bash\ncd bin\n./quantize model.mllm model_q4_k.mllm Q4_K\n```\n\n## Roadmap\n\n- More backends like QNN\n- More models like PandaGPT\n- More optimizations like LUT-GEMM\n- [More..](https://ubiquitouslearning.github.io/mllm_website/roadmap/roadmap/)\n\n## Documentation\n\nSee the [documentation](https://ubiquitouslearning.github.io/mllm_website/introduction/getstarted/) here for more\ninformation\n\n## Contribution\n\nRead the [contribution](https://ubiquitouslearning.github.io/mllm_website/contributing/contributing/) before you\ncontribute.\n\n## Acknowledgments\n\nmllm reuses many low-level kernel implementation from [ggml](https://github.com/ggerganov/ggml) on ARM CPU.\nIt also utilizes [stb](https://github.com/nothings/stb) and [wenet](https://github.com/wenet-e2e/wenet) for\npre-processing images and audios.\nmllm also has benefitted from following projects: [llama.cpp](https://github.com/ggerganov/llama.cpp)\nand [MNN](https://github.com/alibaba/MNN).\n\n## License\n\n### Overall Project License\n\nThis project is licensed under the terms of the MIT License. Please see the [LICENSE](LICENSE) file in the root\ndirectory for the full text of the MIT License.\n\n### Apache 2.0 Licensed Components\n\nCertain component([wenet](https://github.com/wenet-e2e/wenet)) of this project is licensed under the Apache License 2.0.\nThese component is clearly identified in their respective subdirectories along with a copy of the Apache License 2.0.\nFor the full text of the Apache License 2.0, please refer to the [LICENSE-APACHE](third_party/wenet_audio/LICENSE) file\nlocated in the relevant subdirectories.\n\n## Citation\n```\n@article{xu2025fast,\n  title={Fast On-device LLM Inference with NPUs},\n  author={Xu, Daliang and Zhang, Hao and Yang, Liming and Liu, Ruiqi and Huang, Gang and Xu, Mengwei and Liu, Xuanzhe},\n  booktitle={International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)},\n  year={2025}\n}\n@misc{yi2023mllm,\n  title = {mllm: fast and lightweight multimodal LLM inference engine for mobile and edge devices},\n  author = {Rongjie Yi and Xiang Li and Zhenyan Lu and Hao Zhang and Daliang Xu and Liming Yang and Weikai Xie and Chenghua Wang and Xuanzhe Liu and Mengwei Xu},\n  year = {2023},\n  publisher = {mllm Team},\n  url = {https://github.com/UbiquitousLearning/mllm}\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUbiquitousLearning%2Fmllm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FUbiquitousLearning%2Fmllm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FUbiquitousLearning%2Fmllm/lists"}