{"id":51620572,"url":"https://github.com/loomax-labs/InferrLM","last_synced_at":"2026-07-16T17:01:20.870Z","repository":{"id":284925351,"uuid":"929978830","full_name":"loomax-labs/InferrLM","owner":"loomax-labs","description":"InferrLM - On-device AI for iOS \u0026 Android","archived":false,"fork":false,"pushed_at":"2026-06-29T23:06:59.000Z","size":224260,"stargazers_count":102,"open_issues_count":0,"forks_count":24,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-07-12T21:07:59.110Z","etag":null,"topics":["anthropic","document-processing","edge-ai","embeddings","gemini","gguf","http-server","llama-cpp","llamacpp","local-inference","local-llm","multimodal-ai","on-device-ai","openai","rag"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loomax-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"ko_fi":"subhajitgorai"}},"created_at":"2025-02-09T20:38:33.000Z","updated_at":"2026-07-12T16:09:32.000Z","dependencies_parsed_at":"2025-04-11T20:31:56.466Z","dependency_job_id":"ec8ca20a-e8db-4eb1-b8dc-d22085586b71","html_url":"https://github.com/loomax-labs/InferrLM","commit_stats":null,"previous_names":["sbhjt-gr/ragionare-llm-runner","sbhjt-gr/inferra","loomax-labs/inferrlm"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/loomax-labs/InferrLM","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomax-labs%2FInferrLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomax-labs%2FInferrLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomax-labs%2FInferrLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomax-labs%2FInferrLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loomax-labs","download_url":"https://codeload.github.com/loomax-labs/InferrLM/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loomax-labs%2FInferrLM/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35551282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-16T02:00:06.687Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anthropic","document-processing","edge-ai","embeddings","gemini","gguf","http-server","llama-cpp","llamacpp","local-inference","local-llm","multimodal-ai","on-device-ai","openai","rag"],"created_at":"2026-07-12T19:00:22.313Z","updated_at":"2026-07-16T17:01:20.854Z","avatar_url":"https://github.com/loomax-labs.png","language":"TypeScript","funding_links":["https://ko-fi.com/subhajitgorai"],"categories":["Misc"],"sub_categories":["Notes"],"readme":"\n## InferrLM (Previously Inferra)\n\u003cp\u003e\n  \u003ca href=\"\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/App_Version-0.8.7-6a1b9a\" alt=\"App Version 0.8.7\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/AGPL-3.0\" target=\"_blank\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-AGPL--3.0-orange\" alt=\"License: AGPL-3.0\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003cp\u003e\n  \u003cimg src=\"assets/source/InferrLM-header.jpg\" alt=\"InferrLM Header\" width=\"600\"\u003e\n\u003c/p\u003e\n\nInferrLM is a mobile application that brings LLMs \u0026 SLMs directly to your Android \u0026 iOS device and lets your device act as a local server. Cloud-based models like Claude, Gemini and ChatGPT are also supported. File attachments with RAG are also well-supported for local models.\n\n\u003cp\u003e\n  \u003ca href=\"https://play.google.com/store/apps/details?id=com.gorai.ragionare\"\u003e\u003cimg src=\"https://upload.wikimedia.org/wikipedia/commons/7/78/Google_Play_Store_badge_EN.svg\" alt=\"Get it on Google Play\" width=\"178\" style=\"vertical-align:middle\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://apps.apple.com/us/app/inferra/id6754396856\"\u003e\u003cimg src=\"https://developer.apple.com/assets/elements/badges/download-on-the-app-store.svg\" alt=\"Download on the App Store\" width=\"164\" style=\"vertical-align:middle\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\nIf you want to support me and the development of this project, you can donate to me through [Ko-fi](https://ko-fi.com/subhajitgorai).\n\n## Demo\n\nDemos of using the Apple Foundation Model and downloading MLX models from HuggingFace and running them on-device.\n\n\u003cp\u003e\n  \u003cimg src=\"assets/demo_1.gif\" alt=\"Demo 1\" height=\"350\"\u003e\n  \u003cimg src=\"assets/demo_2.gif\" alt=\"Demo 2\" height=\"350\"\u003e\n\u003c/p\u003e\n\n## Features\n\n### Core Inference\n- Core local inference through llama.cpp with support for GGUF models on both Android and iOS.\n- Local MLX inference for Apple Silicon devices.\n- Seamless integration with cloud-based models from OpenAI, Gemini, and Anthropic. You need your own API keys and an InferrLM registered account for remote models. Using remote models is optional.\n- Customizable base URLs for OpenAI-compatible providers like OpenRouter, Groq, Ollama, LM Studio, Together AI. This allows you to access alternative API endpoints within the app.\n- Apple Foundation model support for Apple Intelligence supported devices.\n\n### Vision and Multimodal\n- Vision support through multimodal models with their corresponding projector (mmproj) files. You can read more about them [here](https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md).\n- Built-in camera lets you capture pictures directly within the app and send them to models.\n\n### Document Processing and RAG\n- RAG (Retrieval-Augmented Generation) support for enhanced document understanding and context-aware responses.\n- File attachment support with a built-in document extractor that performs OCR locally on all pages of your documents and extracts text content to send to the local models.\n- Document ingestion system that processes and indexes your files for efficient retrieval during conversations.\n- Native file upload support for the remote models.\n\n### Local Server\n- Built-in HTTP server that exposes REST APIs for accessing your models from any device on your local network. The server can be started from the Server tab. Share your InferrLM chat interface with computers, tablets, or other devices through a URL.\n- Full API documentation is available [HERE](docs/REST_APIs.md) and at the server homepage.\n- A command-line interface tool is available at [github.com/sbhjt-gr/InferrLM-CLI](https://github.com/sbhjt-gr/InferrLM-CLI) that demonstrates how to build applications using its API.\n\n### Model Management\n- Download manager that fetches models directly from HuggingFace. Cherry-picked model list optimized for running on edge devices is available in Models -\u003e \"Download Models\" tab.\n- Downloaded models appear in the chat screen model selector and in the \"Stored Models\" tab inside the \"Models\" tab.\n- Import models from local storage or download directly from URLs.\n\n### Chat Experience\n- Messages support editing, regeneration, copy functionality and markdown rendering.\n- Fast native markdown rendering with math rendering support powered by `react-native-nitro-markdown`, a C++ based renderer built on the Nitro Modules bridge.\n- Dedicated branching support on each chat bubble lets you fork the conversation from any message, preserving the original thread so you can explore alternate directions without losing prior context.\n- Code generated by the models is rendered inside codeblocks with clipboard functionality.\n- Chat history management with the ability to pin conversations.\n\nIf you want to contribute or just try to run it locally, follow the guide below. Your work/modifications should adhere to our \u003ca href=\"https://github.com/sbhjt-gr/InferrLM/blob/main/LICENSE\"\u003eLICENSE\u003c/a\u003e.\n\n### Prerequisites\n\n- Node.js (\u003e= 16.0.0, \u003c 23.0.0)\n- npm or yarn\n- Expo CLI\n- Android Studio (for Android development)\n- Xcode (for iOS development)\n\n### Installation\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/sbhjt-gr/InferrLM\n   cd InferrLM\n   ```\n\n2. **Install dependencies**\n   ```bash\n   yarn install\n   ```\n\n3. **Set up environment variables**\n  Configure your API keys and backend settings. The list of variables is available in the [app.config.json](app.config.js)\n\n4. **Run on device or emulator**\n   ```bash\n   # For Android\n   npx expo run:android\n   \n   # For iOS\n   npx expo run:ios\n   ```\n\n## REST API\n\nInferrLM includes a built-in HTTP server that exposes the models using the OpenAI API for accessing your local models from any device on your local network. This allows you to integrate InferrLM with other applications, scripts, or services.\n\n### Starting the Server\n\n1. Open the InferrLM app\n2. Navigate to the Server tab\n3. Toggle the server switch to start it\n4. The server URL will be displayed (typically `http://YOUR_DEVICE_IP:8889`)\n\n## Command Line Interface\n\nThe InferrLM-CLI tool is a terminal-based client that connects to your InferrLM server and provides an interactive chat interface directly from your command line. This serves as both a functional tool and a reference implementation for developers who want to build applications using the InferrLM REST API.\n\nThe CLI is built with React and Ink to provide a basic terminal UI with features like streaming responses, conversation history, and an interactive setup flow. You can find the complete source code and installation instructions at [github.com/sbhjt-gr/InferrLM-CLI](https://github.com/sbhjt-gr/InferrLM-CLI).\n\nTo get started with the CLI, make sure your InferrLM server is running on your mobile device, then install the CLI tool and follow the setup instructions provided in its repository.\n\n\n### API Documentation\n\nOnce the server is running, you can access the complete API documentation by opening the server URL in any web browser. The documentation includes:\n\n- Chat and completion endpoints\n- Model management operations\n- RAG and embeddings APIs\n- Server configuration and status\n\nFor detailed API reference, see the [REST API Documentation](docs/REST_APIs.md).\n\n## License\n\nThis project is distributed under the AGPL-3.0 License. Please read it [here](https://github.com/sbhjt-gr/InferrLM/blob/main/LICENSE). Any modifications must adhere to the rules of this LICENSE.\n\n## Contributing\n\nContributions are welcome! You can find issues in the [issues](https://github.com/sbhjt-gr/InferrLM/issues) tab or raise new ones and start your work.\n\nRead our [Contributing Guide](docs/CONTRIBUTING.md) for detailed contribution guidelines, code standards, and best practices. \n\n## Tech Stack\n\n- **Framework**: React Native 0.81 with Expo 54 (New Architecture)\n- **App language**: TypeScript, JavaScript\n- **iOS native modules**: Swift\n- **Android native modules**: Kotlin\n- **Inference engine**: C, C++\n- **Navigation**: React Navigation\n- **Database**: OP-SQLite, Expo SQLite\n\n## Acknowledgments\n\n- [llama.cpp](https://github.com/ggerganov/llama.cpp) - The underlying engine for running local GGUF models on both Android and iOS.\n- [mlx-swift-lm](https://github.com/ml-explore/mlx-swift-lm) - Swift library for running MLX language models on Apple Silicon, powering the MLX inference backend on iOS.\n- [inferrlm-llama.rn](https://github.com/sbhjt-gr/inferra-llama.rn) - The customized React Native adapter which provides the bridge for llama.cpp. Originally forked and self-hosted from [llama.rn](https://github.com/mybigday/llama.rn) for updating llama.cpp more frequently.\n- [@inferrlm/react-native-mlx](https://github.com/sbhjt-gr/react-native-nitro-mlx) - Apple Silicon MLX inference engine for iOS, providing optimized on-device performance via the Nitro Modules bridge forked and maintained from [react-native-nitro-mlx](https://github.com/corasan/react-native-nitro-mlx)\n- [react-native-nitro-markdown](https://github.com/sbhjt-gr/react-native-nitro-markdown) - Native C++ markdown renderer for React Native, used for fast chat message rendering.\n- [react-native-rag](https://github.com/software-mansion-labs/react-native-rag) + [@langchain/textsplitters](https://github.com/langchain-ai/langchainjs) - RAG implementation for React Native that powers the document retrieval and ingestion features using LangChain.\n- [react-native-ai](https://github.com/callstackincubator/ai) - The adaptor that provides access to the Apple Foundation model with its Swift API.\n- If someone thinks they also need to be mentioned here, please let me know.\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=sbhjt-gr/InferrLM\u0026type=Date)](https://star-history.com/#sbhjt-gr/InferrLM\u0026Date)\n\n---\n\n\u003cp align=\"center\"\u003e\n  \u003csub\u003eStar this repository if you find it useful!\u003c/sub\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomax-labs%2FInferrLM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Floomax-labs%2FInferrLM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Floomax-labs%2FInferrLM/lists"}