{"id":21458079,"url":"https://github.com/shubham0204/OnDevice-RAG-Android","last_synced_at":"2025-10-05T07:31:44.004Z","repository":{"id":241596798,"uuid":"803570959","full_name":"shubham0204/Android-Document-QA","owner":"shubham0204","description":"A custom RAG pipeline for multi-document QA from PDF/DOCX documents, in Android","archived":false,"fork":false,"pushed_at":"2024-09-07T07:21:14.000Z","size":99571,"stargazers_count":35,"open_issues_count":6,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-07T08:35:18.555Z","etag":null,"topics":["android","large-language-models","on-device-ml","question-answering","retrieval-augmented-generation"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shubham0204.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"shubham0204"}},"created_at":"2024-05-21T01:31:39.000Z","updated_at":"2024-09-07T07:21:18.000Z","dependencies_parsed_at":"2024-10-20T11:48:51.846Z","dependency_job_id":null,"html_url":"https://github.com/shubham0204/Android-Document-QA","commit_stats":null,"previous_names":["shubham0204/android-document-qa"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FAndroid-Document-QA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FAndroid-Document-QA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FAndroid-Document-QA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shubham0204%2FAndroid-Document-QA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shubham0204","download_url":"https://codeload.github.com/shubham0204/Android-Document-QA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235377126,"owners_count":18980187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["android","large-language-models","on-device-ml","question-answering","retrieval-augmented-generation"],"created_at":"2024-11-23T06:17:36.656Z","updated_at":"2025-10-05T07:31:33.981Z","avatar_url":"https://github.com/shubham0204.png","language":"Kotlin","funding_links":["https://github.com/sponsors/shubham0204"],"categories":[],"sub_categories":[],"readme":"# Document Question-Answering with Local RAG in Android\n\n\u003e A simple Android app that allows the user to add a PDF/DOCX document and ask natural-language questions whose \n\u003e answers are generated by the means of an LLM\n\nhttps://github.com/shubham0204/Android-Document-QA/assets/41076823/d4d6a21c-a29e-4429-9d43-ca38785c5b50\n\n(The PDF used in the demo can be found in [resources](https://github.com/shubham0204/Android-Document-QA/tree/main/resources) directory)\n\n[YT Video](https://youtu.be/Av0N40Weu0M?feature=shared)\n\n## Goals\n\n- Demonstrate the collective use of an on-device vector database, embeddings model and a custom text-splitter to build a retrieval-augmented generation (RAG) based pipeline for simple document question-answering\n- Use modern Android development practices and recommended architecture guidelines\n- Explore and suggest better tools/alternatives for building fully offline, on-device RAG pipeline for Android with minimum compute and storage requirements\n\n| Feature            | On-Device | Remote |\n|--------------------|-----------|--------|\n| Sentence Embedding | ✅         |        |\n| Text Splitter      | ✅         |        |\n| Vector Database    | ✅         |        |\n| LLM                |           | ✅      |\n\n## Setup\n\n1. Clone the `main` branch, \n\n```bash\n$\u003e git clone --depth=1 https://github.com/shubham0204/Android-Document-QA\n```\n\n2. [Get an API key from Google AI Studio](https://ai.google.dev/gemini-api/docs/api-key) to use the Gemini API. Copy \nthe key and paste it in `local.properties` present in the root directory of the project,\n\n```\ngeminiKey=\"AIza[API_KEY_HERE]\"\n```\n\nPerform a Gradle sync, and run the application. \n\n## Tools\n\n1. [Apache POI](https://poi.apache.org/) and [iTextPDF](https://github.com/itext/itextpdf) for parsing DOCX and PDF documents\n2. [ObjectBox](https://objectbox.io/) for on-device vector-store and NoSQL database\n3. [Sentence Embeddings (`all-MiniLM-L6-V2`)](https://github.com/shubham0204/Sentence-Embeddings-Android) for generating on-device text/sentence embeddings\n4. [Gemini Android SDK](https://developer.android.com/ai/google-ai-client-sdk) as a hosted large-language model (Uses Gemini-1.5-Flash)\n\n## Working \n\nThe basic working flow on the app is as follows:\n\n1. When the user selects a PDF/DOCX document (the only ones which can be imported for now), the text is parsed with \nthe libraries mentioned in (1) of [Tools](#tools). See [PDFReader.kt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/java/com/ml/shubham0204/docqa/domain/readers/PDFReader.kt) and [DOCXReader.kt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/java/com/ml/shubham0204/docqa/domain/readers/DOCXReader.kt) for reference.\n2. Chunks or overlapping sub-sequences are produced from the text, given the size of sequence (`chunkSize`) and \nthe extent of overlap between two sequences (`chunkOverlap`). See [WhiteSpaceSplitter.kt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/java/com/ml/shubham0204/docqa/domain/splitters/WhiteSpaceSplitter.kt) for reference.\n3. Each chunk is encoded into a fixed-size vector i.e. a text embedding. The embeddings are inserted in the vector database, with each chunk/embedding having a distinct `chunkId`. See [SentenceEmbeddingProvider.kt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/java/com/ml/shubham0204/docqa/domain/embeddings/SentenceEmbeddingProvider.kt) for reference.\n4. When the user submits a query, we find the top-K most similar chunks from the database by comparing their embeddings.\n5. The chunks corresponding to the nearest embeddings are injected into a pre-built prompt along with the query, which is provided to the LLM. The LLM generates a well-formed natural language answer to the user's query. See [GeminiRemoteAPI.kt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/java/com/ml/shubham0204/docqa/domain/llm/GeminiRemoteAPI.kt) for reference.\n\nSee the [prompt](https://github.com/shubham0204/Android-Document-QA/blob/main/app/src/main/res/values/strings.xml),\n\n```text\nYou are an intelligent search engine. You will be provided with some retrieved context, as well as the users query.\nYour job is to understand the request, and answer based on the retrieved context.\nStrictly Use ONLY the following pieces of context to answer the question at the end.\nProvide only the answer as a response\n\nHere is the retrieved context:\n    $CONTEXT\n\nHere is the users query:\n    $QUERY\n```\n\n## Discussion\n\n### Why not use on-device LLMs instead of the Gemini's Cloud SDK?\n\nUsing an on-device LLM is possible in Android, but at the expense of a large app size (\u003e1GB) and compute requirements. Google's Edge AI SDK has some options where models like Gemma, MS Phi-2, Falcon can be used completely on-device and accessed via Mediapipe's Android/iOS/Web APIs. See the [official documentation for Mediapipe LLM Inference](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference), it also includes instructions ofr LoRA fine-tuning.\n\nMoreover, the same docs [specific for Android](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android) mention the fact,\n\n\u003e During development, you can use adb to push the model to your test device for a simpler workflow. For deployment, host the model on a server and download it at runtime. The model is too large to be bundled in an APK.\n\nThe integration using Mediapipe LLM inference API is easy. Due to the absence of a good Android device, I went ahead with the Cloud API, but it would be great to have an on-device option. [Gemini Nano](https://ai.google.dev/gemini-api/docs/get-started/android_aicore) currently available on limited devices is also an on-device solution.\n\nOther tools for using LLMs on Android:\n\n1. [mlc](https://llm.mlc.ai/docs/deploy/android.html) (Also see [Llama3 on Android](https://github.com/NSTiwari/Llama3-on-Mobile))\n2. [llama.cpp for Android](https://github.com/JackZeng0208/llama.cpp-android-tutorial)\n\n### (Solved) Better alternatives for the Universal Sentence Encoder (embedding model)\n\n#### Problem: \nThe app currently uses the [Universal Sentence Encoder](https://www.kaggle.com/models/google/universal-sentence-encoder) model from Google, as it was the only possible way to generate text/sentence embeddings on an Android device, with a builtin API and tokenizer. It generates an embedding of size 100. \n\nAfter checking the retrieved context (similar chunks) for a few questions, I recognized that the embedding model was not able to understand the context of the sentence to a significant extent. I couldn't find a metric to validate this point on [HuggingFace's MTEB](https://huggingface.co/spaces/mteb/leaderboard). Models such as [sentence-transformers](https://huggingface.co/sentence-transformers) were particular great at understanding context, but their integration in an Android app remains an open problem.\n\n#### Solution\n\nThe `all-MiniLM-L2-V6` model from [sentence-transformers](https://huggingface.co/sentence-transformers) has been ported to Android with the help of ONNX/onnxruntime and the Rust-implementation of [huggingface/tokenziers](https://github.com/huggingface/tokenizers). See the app's [assets](https://github.com/shubham0204/Android-Document-QA/tree/main/app/src/main/assets) folder to find the ONNX model `tokenizer.json`\n\nSee the main repository [shubham0204/Sentence-Embeddings-Android](https://github.com/shubham0204/Sentence-Embeddings-Android) for more details.\n\n## Contributions and Open Problems\n\nFeel free to [raise an issue](https://github.com/shubham0204/Android-Document-QA/issues/new) or [open a PR](https://github.com/shubham0204/Android-Document-QA/pulls). The following can be improved in the app:\n\n1. Add on-device LLM capabilities\n2. Build a new text-splitter, taking inspiration from Langchain or LlamaIndex\n3. Port a good, small-in-size embedding model from HuggingFace to work in the app\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2FOnDevice-RAG-Android","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshubham0204%2FOnDevice-RAG-Android","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshubham0204%2FOnDevice-RAG-Android/lists"}