Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/priyanshuchawda/gemma_control

GemmaControl — A private, on-device Android AI agent for WhatsApp productivity. Captures and organises WhatsApp notifications into an actionable inbox, leverages Google's FunctionGemma 270M (via LiteRT-LM) to propose tool calls, and enables safe user-confirmed replies — all running fully offline with AES-GCM 256-bit encrypted local storage.
https://github.com/priyanshuchawda/gemma_control

aes-gcm ai-agent android android-16 android-keystore edge-ai function-calling gemma jetpack-compose kotlin litertlm llm mvvm notification-listener on-device-ai privacy productivity room-database whatsapp workmanager

Last synced: 1 day ago
JSON representation

GemmaControl — A private, on-device Android AI agent for WhatsApp productivity. Captures and organises WhatsApp notifications into an actionable inbox, leverages Google's FunctionGemma 270M (via LiteRT-LM) to propose tool calls, and enables safe user-confirmed replies — all running fully offline with AES-GCM 256-bit encrypted local storage.

Awesome Lists containing this project

README 

          
# GemmaControl: Private On-Device WhatsApp AI Productivity Agent



GemmaControl is an English-only, private, on-device Android productivity agent for WhatsApp notification workflows. The app captures new WhatsApp notifications after user permission, organises them into a local actionable inbox, lets FunctionGemma propose approved tool calls, saves local follow-ups, priority flags, encrypted local reminders, and supports safe user-confirmed WhatsApp replies.



Built entirely as a native application for **Android 16 (API Level 36)**, GemmaControl keeps WhatsApp capture, storage, voice handling, and model inference local on a **Xiaomi Redmi 13 5G** handset, utilizing Google's on-device **LiteRT-LM SDK** and the FunctionGemma MobileActions 270M model.



---



## 🚀 Key Functional Scope



### 1. Notification Ingestion & Actionable Inbox

- **MessagingStyle Parsing**: Extracts chat history lists and sender authors dynamically from bundled notification structures.

- **Group Separation**: Differentiates group conversation headers from the individual message authors.

- **Content Classification**: Tags captured notifications as text, media placeholders, missed calls, hidden content, system notifications, or unknown content before local reads and storage.

- **Event Deduplication**: Uses deterministic parser candidates that are transformed into keyed HMAC `dedupeToken` values before Room storage.

- **Privacy Constraints**: Captured alerts are parsed in memory, and stored locally *only* under explicit storage consent toggles.



### 2. Local Task Management & Productivity

- **Follow-Ups**: Save notification events as unresolved local tasks (`create_follow_up_from_message`), list pending tasks, and mark follow-ups complete.

- **Prioritization**: Mark captured WhatsApp messages as high or normal priority inside the local inbox.

- **Reminders**: Schedule encrypted local reminder rows and WorkManager notification delivery (`schedule_reminder_for_message`).

- **Priorities**: Flag and pin important messages inside the local Compose inbox.

- **Inbox Cleanup**: Dismiss and hide noise from the local inbox without clearing notifications inside WhatsApp.



### 3. Safe Execution Boundaries (Android 16 Target)

- **Adaptive Read-Aloud**: `Read latest WhatsApp messages` targets currently active notification rows, while `Read stored messages` uses the local history index. Reads 1-3 messages directly, summarizes larger sets by chat, and supports continue, chat-specific reads, summarize, and important-only voice phrases.

- **Edge-to-Edge Jetpack Compose**: Mandatory edge-to-edge UI display with full support for window insets and IME (keyboard) overlays.

- **Predictive Back Navigation**: Built using predictive-back-compatible AndroidX navigation frameworks and custom `BackHandler` hooks.

- **Verified Click-to-Chat Intents**: Securely launches click-to-chat `ACTION_VIEW` intents utilizing E.164 phone formats.

- **Live RemoteInput Reply**: Fetches active system notifications by key in real-time, verifying active status and reply parameters before execution.

- **UI Confirmation Sheets**: Requires a physical user tap on Compose modals before firing direct reply intents. Auto-sending is structurally blocked.



---



## 🛠️ Architecture & Security Design



- **Clean MVVM Architecture**: Separates data structures (Room SQLite database), notification handlers, LiteRT engines, and Jetpack Compose views.

- **Encryption at Rest**: Encrypts sensitive message texts and drafted responses inside Room SQLite utilizing **AES-GCM 256-bit encryption** keys protected by the **Android Keystore**.

- **Truthful Media Handling**: WhatsApp photo/video/audio/document notifications are treated as placeholders only. The app never claims to inspect media contents unless actual media bytes become a future explicit feature.

- **Scoped Network Access**: `android.permission.INTERNET` is declared only for explicit FunctionGemma `.litertlm` model binary downloads. WhatsApp notification data, prompts, tool calls, and replies stay local.

- **Local Reminder Notifications**: `android.permission.POST_NOTIFICATIONS` is declared only for user-confirmed local reminder alerts on Android 13+.

- **LiteRT-LM Native Tool Boundary**: Registers a deliberately small native `ToolSet` with automatic LiteRT tool callbacks enabled. Tool callbacks only capture typed proposals/results; Kotlin still owns permission checks, safety routing, and all user-confirmed execution.



---



## 📂 Project Structure



```text

├── docs/                     # Comprehensive technical documentation

│   ├── ARCHITECTURE.md       # On-device data flow, boundaries, and OS limits

│   ├── CAPABILITY_MATRIX.md  # Android permissions, capability sources, and tool mapping

│   ├── GENERIC_NOTIFICATION_SOURCE_ABSTRACTION.md # WhatsApp-first source model for future SMS/Gmail/Phone/Calendar

│   ├── XIAOMI_HYPEROS_RELIABILITY_CHECKLIST.md # Physical reliability checklist for HyperOS capture

│   ├── GEMMA_MODEL_REFERENCE.md # Gemma model/runtime selection reference

│   ├── GEMMA_ASSISTANT_USAGE_PLAN.md # App-specific model usage roadmap

│   ├── OFFLINE_MODEL_STRATEGY_DECISION.md # No-download model architecture decision for #109

│   ├── EMBEDDING_GEMMA_SEMANTIC_MEMORY_PLAN.md # No-download semantic retrieval spike for #113

│   ├── LOCAL_SUMMARIZATION_MODEL_DECISION.md # No-download Gemma 4/Gemma 3n decision for #117

│   ├── FUNCTION_GEMMA_FINE_TUNING_DECISION.md # No-training FunctionGemma routing decision for #116

│   ├── MEDIA_UNDERSTANDING_BOUNDARY.md # No-model media placeholder/user-selected file boundary for #118

│   ├── ASSISTANT_SAFETY_POLICY.md # Deterministic safety and ShieldGemma gate decision for #119

│   ├── ACCESSIBILITY_SERVICE_EVALUATION.md # V1/V2 decision and safety policy for #122

│   ├── DEVICE_INFO.md        # Current connected handset capability snapshot

│   ├── PHONE_TESTING_TOOLKIT.md # Local laptop and phone testing tools, commands, and setup

│   ├── PRODUCT_SCOPE.md      # V1 functional capabilities and non-goals

│   ├── REAL_DEVICE_ASSISTANT_TEST_MATRIX.md # Manual assistant/device validation matrix

│   ├── TOOL_REGISTRY.md      # Parameter type schemas for the 16 registry tools

│   ├── NOTIFICATION_PARSING.md # MessagingStyle parsers and keyed dedupe tokens

│   ├── SECURITY_AND_PRIVACY.md # Key cryptography and search memory trade-offs

│   └── DEVICE_VALIDATION.md  # Physical telemetry parameters and test milestones

├── GemmaControl/             # Native Android 16 Kotlin Application

│   ├── app/                  # Main application module

│   │   ├── build.gradle.kts  # Target and compile configurations (API 36)

│   │   └── src/main/         # Kotlin Compose sources, listener services, and manifest

│   └── build.gradle.kts      # Project root Gradle declaration

├── website/                  # Static TypeScript download website

├── plan.md                   # High-level product roadmap

├── detailed_plan.md          # Actionable phase-by-phase implementation blueprints

└── README.md                 # Project cockpit reference

```



---



## 🌐 Download Website



The `website/` directory contains a static TypeScript site for distributing GemmaControl without committing app binaries.



```powershell

cd website

npm run verify

npm run serve

```



The primary APK button points to the canonical GitHub Releases artifact configured in `website/src/release-links.ts`. APK/AAB/EXE/MSI files remain ignored and must not be committed.



---



## AI Contributor Instructions



This repository is developed against one known physical device profile unless a later issue explicitly expands the matrix:



- Phone: Xiaomi Redmi 13 5G / `2406ERN9CI`

- OS: HyperOS `OS3.0`, Android `16`, API `36`

- ABI/RAM: `arm64-v8a`, approximately 6 GB physical RAM

- ADB serial used in current validation docs: `1431df87`

- Android app package: `com.example.gemmacontrol`

- Android project root: `GemmaControl/`



Before changing behavior, read:



- `plan.md` for the product roadmap and phase boundaries.

- `detailed_plan.md` for implementation sequencing and safety constraints.

- `docs/ARCHITECTURE.md` for runtime boundaries.

- `docs/GEMMA_ASSISTANT_USAGE_PLAN.md` and `docs/GEMMA_MODEL_REFERENCE.md` for model roles.

- `docs/TOOL_REGISTRY.md` for the allowed assistant tool surface.

- `docs/PHONE_TESTING_TOOLKIT.md` for laptop, ADB, Appium, Maestro, scrcpy, and manual phone testing setup.

- `docs/REAL_DEVICE_ASSISTANT_TEST_MATRIX.md` for real-device validation scenarios.



Development rules:



- Keep WhatsApp data local. Do not log, commit, or summarize private message contents in issues, PRs, docs, or final reports.

- Do not make direct WhatsApp replies automatic. Reply sending must remain behind a visible user confirmation step.

- Treat FunctionGemma as the local tool-routing model, not as the source of unrestricted app control.

- Kotlin validation owns permissions, safety checks, schema validation, and execution decisions even when LiteRT native callbacks propose tool calls.

- Current media notifications are placeholders only. Do not claim image/audio/video understanding unless a future explicit media feature handles actual user-selected bytes.

- Use the issue -> `feat/` branch -> PR -> merge -> delete branch -> clean `main` workflow for implementation work.

- Do not commit APK/AAB/EXE/MSI/model binaries. Large model files belong outside git-tracked source unless a release process explicitly says otherwise.



Minimum verification for code changes:



```powershell

cd C:\Users\Admin\Desktop\gemma_control\GemmaControl

.\gradlew.bat :app:testDebugUnitTest

.\gradlew.bat :app:assembleDebug

```



Real-device validation, when the phone is connected:



```powershell

adb devices -l

adb install -r -d .\app\build\outputs\apk\debug\app-debug.apk

adb shell am start -n com.example.gemmacontrol/.MainActivity

scrcpy --serial 1431df87 --stay-awake --turn-screen-on

```



Use Appium or Maestro for repeatable UI flows, and use manual testing for real WhatsApp incoming messages, voice prompts, Xiaomi permission prompts, and confirmed replies. Full setup and command examples are documented in `docs/PHONE_TESTING_TOOLKIT.md`.



---



## 📱 Hardware Telemetry (Discovered via ADB)

- **Manufacturer/Model**: `Xiaomi 2406ERN9CI` (Redmi 13 5G)

- **OS Environment**: `Android 16` (API Level 36)

- **Processor (ABI)**: `arm64-v8a`

- **Total Physical RAM**: approximately 6 GB (`5,531,208 kB`)

- **Connection Port**: Connected via USB (Serial: `1431df87`)

- **Verified Package**: `package:com.whatsapp` is confirmed present on the profile.