{"id":22275603,"url":"https://github.com/shubham0204/smolchat-android","last_synced_at":"2025-04-13T00:45:46.943Z","repository":{"id":266223668,"uuid":"886113190","full_name":"shubham0204/SmolChat-Android","owner":"shubham0204","description":"Running any GGUF SLMs/LLMs locally, on-device in Android","archived":false,"fork":false,"pushed_at":"2025-04-12T01:41:38.000Z","size":13904,"stargazers_count":251,"open_issues_count":14,"forks_count":29,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-13T00:45:36.463Z","etag":null,"topics":["android","cpp","ggml","kotlin","llamacpp","small-language-models"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shubham0204.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"shubham0204","custom":["https://www.paypal.me/ShubhamPanchal0204/"]}},"created_at":"2024-11-10T08:26:09.000Z","updated_at":"2025-04-12T15:19:46.000Z","dependencies_parsed_at":"2024-12-31T03:22:18.313Z","dependency_job_id":"8be057ea-9b31-4069-bc14-48a0f0e48e54","html_url":"https://github.com/shubham0204/SmolChat-Android","commit_stats":null,"previous_names":["shubham0204/smolchat-android"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FSmolChat-Android","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FSmolChat-Android/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FSmolChat-Android/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FSmolChat-Android/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shubham0204","download_url":"https://codeload.github.com/shubham0204/SmolChat-Android/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248650420,"owners_count":21139672,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","cpp","ggml","kotlin","llamacpp","small-language-models"],"created_at":"2024-12-03T14:10:10.326Z","updated_at":"2025-04-13T00:45:46.937Z","avatar_url":"https://github.com/shubham0204.png","language":"Kotlin","funding_links":["https://github.com/sponsors/shubham0204","https://www.paypal.me/ShubhamPanchal0204/"],"categories":[],"sub_categories":[],"readme":"# SmolChat - On-Device Inference of SLMs in Android\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/1.jpg\" alt=\"app_img_01\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/2.jpg\" alt=\"app_img_02\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/3.jpg\" alt=\"app_img_03\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/4.jpg\" alt=\"app_img_04\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/5.jpg\" alt=\"app_img_05\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/6.jpg\" alt=\"app_img_06\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/7.jpg\" alt=\"app_img_07\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/8.jpg\" alt=\"app_img_08\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/9.jpg\" alt=\"app_img_09\"\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cimg src=\"resources/app_screenshots/10.jpg\" alt=\"app_img_10\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Installation\n\n### GitHub\n\n1. Download the latest APK from [GitHub Releases](https://github.com/shubham0204/SmolChat-Android/releases/) and transfer it to your Android device.\n2. If your device does not downloading APKs from untrusted sources, search for **how to allow downloading APKs from unknown sources** for your device.\n\n### Obtainium\n\n[Obtainium](https://obtainium.imranr.dev/) allows users to update/download apps directly from their sources, like GitHub or FDroid. \n\n1. [Download the Obtainium app](https://obtainium.imranr.dev/) by choosing your device architecture or 'Download Universal APK'.\n2. From the bottom menu, select '➕Add App'\n3. In the text field labelled 'App source URL *', enter the following URL and click 'Add' besides the text field: `https://github.com/shubham0204/SmolChat-Android`\n4. SmolChat should now be visible in the 'Apps' screen. You can get notifications about newer releases and download them directly without going to the GitHub repo.\n\n## Project Goals\n\n- Provide a usable user interface to interact with local SLMs (small language models) locally, on-device\n- Allow users to add/remove SLMs (GGUF models) and modify their system prompts or inference parameters (temperature, \n  min-p)\n- Allow users to create specific-downstream tasks quickly and use SLMs to generate responses\n- Simple, easy to understand, extensible codebase\n\n## Setup\n\n1. Clone the repository with its submodule originating from llama.cpp,\n\n```commandline\ngit clone --depth=1 https://github.com/shubham0204/SmolChat-Android\ncd SmolChat-Android\ngit submodule update --init --recursive\n```\n\n2. Android Studio starts building the project automatically. If not, select **Build \u003e Rebuild Project** to start a project build.\n\n3. After a successful project build, [connect an Android device](https://developer.android.com/studio/run/device) to your system. Once connected, the name of the device must be visible in top menu-bar in Android Studio.\n\n## Working\n\n1. The application uses llama.cpp to load and execute GGUF models. As llama.cpp is written in pure C/C++, it is easy \n   to compile on Android-based targets using the [NDK](https://developer.android.com/ndk). \n\n2. The `smollm` module uses a `llm_inference.cpp` class which interacts with llama.cpp's C-style API to execute the \n   GGUF model and a JNI binding `smollm.cpp`. Check the [C++ source files here](https://github.com/shubham0204/SmolChat-Android/tree/main/smollm/src/main/cpp). On the Kotlin side, the [`SmolLM`](https://github.com/shubham0204/SmolChat-Android/blob/main/smollm/src/main/java/io/shubham0204/smollm/SmolLM.kt) class provides \n   the required methods to interact with the JNI (C++ side) bindings.\n\n3. The `app` module contains the application logic and UI code. Whenever a new chat is opened, the app instantiates \n   the `SmolLM` class and provides it the model file-path which is stored by the [`LLMModel`](https://github.com/shubham0204/SmolChat-Android/blob/main/app/src/main/java/io/shubham0204/smollmandroid/data/DataModels.kt) entity.\n   Next, the app adds messages with role `user` and `system` to the chat by retrieving them from the database and\n   using `LLMInference::addChatMessage`.\n\n4. For tasks, the messages are not persisted, and we inform to `LLMInference` by passing `_storeChats=false` to\n   `LLMInference::loadModel`.\n\n## Technologies\n\n* [ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) is a pure C/C++ framework to execute machine learning \n  models on multiple execution backends. It provides a primitive C-style API to interact with LLMs \n  converted to the [GGUF format](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) native to [ggml](https://github.com/ggerganov/ggml)/llama.cpp. The app uses JNI bindings to interact with a small class `smollm.\n  cpp` which uses llama.cpp to load and execute GGUF models.\n\n* [noties/Markwon](https://github.com/noties/Markwon) is a markdown rendering library for Android. The app uses \n  Markwon and [Prism4j](https://github.com/noties/Prism4j) (for code syntax highlighting) to render Markdown responses \n  from the SLMs.\n\n## More On-Device ML Projects\n\n- [shubham0204/Android-Doc-QA](https://github.com/shubham0204/Android-Document-QA): On-device RAG-based question \n  answering from documents\n- [shubham0204/OnDevice-Face-Recognition-Android](https://github.com/shubham0204/OnDevice-Face-Recognition-Android): \n  Realtime face recognition with FaceNet, Mediapipe and ObjectBox's vector database\n- [shubham0204/FaceRecognition_With_FaceNet_Android](https://github.com/shubham0204/OnDevice-Face-Recognition-Android):\n  Realtime face recognition with FaceNet, MLKit\n- [shubham0204/CLIP-Android](https://github.com/shubham0204/CLIP-Android): On-device CLIP inference in Android \n  (search images with textual queries)\n- [shubham0204/Segment-Anything-Android](https://github.com/shubham0204/Segment-Anything-Android): Execute Meta's \n  SAM model in Android with onnxruntime\n- [shubham0204/Depth-Anything-Android](https://github.com/shubham0204/Depth-Anything-Android): Execute the \n  Depth-Anything model in Android with onnxruntime for monocular depth estimation\n- [shubham0204/Sentence-Embeddings-Android](https://github.com/shubham0204/Sentence-Embeddings-Android): Generate \n  sentence-embeddings (from models like `all-MiniLM-L6-V2`) in Android\n\n## Future\n\nThe following features/tasks are planned for the future releases of the app:\n\n- Assign names to chats automatically (just like ChatGPT and Claude)\n- Add a search bar to the navigation drawer to search for messages within chats\n- Add a background service which uses BlueTooth/HTTP/WiFi to communicate with a desktop application to send queries \n  from the desktop to the mobile device for inference\n- Enable auto-scroll when generating partial response in `ChatActivity`\n- Measure RAM consumption\n- Integrate [Android-Doc-QA](https://github.com/shubham0204/Android-Document-QA) for on-device RAG-based question answering from documents\n- Check if llama.cpp can be compiled to use Vulkan for inference on Android devices (and use the mobile GPU)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2Fsmolchat-android","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshubham0204%2Fsmolchat-android","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2Fsmolchat-android/lists"}