https://github.com/rbayuokt/expo-mlkit-ocr
Production-ready Expo Module for on-device text recognition (OCR) using Google ML Kit Text Recognition v2 + OCRTextOverlay
https://github.com/rbayuokt/expo-mlkit-ocr
expo expo-mlkit expo-mlkit-ocr expo-ocr expo-text-recognition mlkit ocr react-native react-native-ocr text-recognition
Last synced: 9 days ago
JSON representation
Production-ready Expo Module for on-device text recognition (OCR) using Google ML Kit Text Recognition v2 + OCRTextOverlay
- Host: GitHub
- URL: https://github.com/rbayuokt/expo-mlkit-ocr
- Owner: rbayuokt
- Created: 2026-05-01T11:47:20.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2026-05-01T17:04:48.000Z (2 months ago)
- Last Synced: 2026-05-01T18:15:37.160Z (2 months ago)
- Topics: expo, expo-mlkit, expo-mlkit-ocr, expo-ocr, expo-text-recognition, mlkit, ocr, react-native, react-native-ocr, text-recognition
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/expo-mlkit-ocr
- Size: 3.55 MB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# expo-mlkit-ocr
Production-ready Expo Module for on-device text recognition (OCR) using **Google ML Kit Text Recognition v2** for both iOS and Android.
## Preview
## Features
- ✅ **On-device OCR** — no network requests required
- ✅ **ML Kit v2** — official Google ML Kit standalone (not Firebase ML Kit)
- ✅ **Structured output** — blocks, lines, and elements with bounding boxes
- ✅ **iOS & Android** — native implementations using Expo Modules API
- ✅ **TypeScript support** — fully typed API
- ✅ **Expo Config Plugin** — automatic native setup
- ✅ **Device support check** — `isSupported()` to detect device compatibility before OCR
- ✅ **Interactive overlay** — `` component for visualizing & selecting recognized text
## Installation
```bash
npx expo install expo-mlkit-ocr expo-image-picker expo-build-properties
```
> ⚠️ **`expo-build-properties` is REQUIRED** to set the iOS deployment target to `16.0` (needed by Google ML Kit on iOS).
## Setup
### 1. Configure `app.json` / `app.config.js`
Add **both plugins** to your config:
```json
{
"expo": {
"plugins": [
[
"expo-mlkit-ocr",
{
"iosEngine": "auto"
}
],
[
"expo-build-properties",
{
"ios": {
"deploymentTarget": "16.0"
}
}
]
]
}
}
```
> ⚠️ **Important:** Without `deploymentTarget: "16.0"` and `useFrameworks: "static"`, you will get:
> ```
> ERROR [runtime not ready]: Error: Cannot find native module 'ExpoMlkitOcr'
> ```
### 2. Build Your App
**With EAS Build:**
```bash
eas build --platform ios
eas build --platform android
```
**Or with local prebuild:**
```bash
npx expo prebuild --clean
npx expo run:ios
# or
npx expo run:android
```
> ❌ **Will NOT work in Expo Go** — this is a custom native module. You need a development client or EAS Build.
### Troubleshooting
**`Cannot find native module 'ExpoMlkitOcr'`**
- Did you add `expo-build-properties` plugin with `deploymentTarget: "16.0"`? ⚠️ Most common cause
- Did you run `npx expo prebuild --clean` after installing?
- Are you running on a development client (not Expo Go)?
**Build fails on iOS Simulator (arm64)**
- Use `iosEngine: "auto"` or `"vision"` in plugin config (uses Apple Vision instead of ML Kit)
## Usage
### Basic Example
```typescript
import { recognizeText } from 'expo-mlkit-ocr';
import * as ImagePicker from 'expo-image-picker';
async function pickAndRecognize() {
const result = await ImagePicker.launchImageLibraryAsync({
mediaTypes: ['images'],
});
if (result.canceled || !result.assets[0]) return;
try {
const recognition = await recognizeText(result.assets[0].uri);
console.log('Recognized text:', recognition.text);
console.log('Blocks:', recognition.blocks);
} catch (error) {
console.error('Recognition failed:', error);
}
}
```
### Output Format
The function returns a `RecognitionResult` object with this structure:
```typescript
export type RecognitionResult = {
text: string; // Full recognized text
blocks: TextBlock[];
};
export type TextBlock = {
text: string;
boundingBox: {
x: number;
y: number;
width: number;
height: number;
};
lines: TextLine[];
};
export type TextLine = {
text: string;
boundingBox: {
x: number;
y: number;
width: number;
height: number;
};
elements: TextElement[];
};
export type TextElement = {
text: string;
boundingBox: {
x: number;
y: number;
width: number;
height: number;
};
};
```
**Bounding box coordinates** are in the image's native coordinate system (top-left origin):
- `x`, `y` — top-left corner
- `width`, `height` — dimensions in pixels
### Example Output
```json
{
"text": "Hello World\nExample Text",
"blocks": [
{
"text": "Hello World",
"boundingBox": { "x": 100, "y": 50, "width": 200, "height": 50 },
"lines": [
{
"text": "Hello World",
"boundingBox": { "x": 100, "y": 50, "width": 200, "height": 50 },
"elements": [
{
"text": "Hello",
"boundingBox": { "x": 100, "y": 50, "width": 80, "height": 50 }
},
{
"text": "World",
"boundingBox": { "x": 190, "y": 50, "width": 110, "height": 50 }
}
]
}
]
}
]
}
```
## API Reference
### `recognizeText(uri: string): Promise`
Recognizes text from an image at the provided URI.
**Parameters:**
- `uri` (string) — file URI or content URI to the image (e.g., from `expo-image-picker`)
**Returns:**
- `Promise` — structured text recognition result
**Errors:**
- `INVALID_URI` — provided URI is not valid
- `IMAGE_LOAD_FAILED` — image could not be loaded from the URI
- `RECOGNITION_FAILED` — text recognition failed (rare)
### `isSupported(): boolean`
Checks if the device supports ML Kit text recognition (OCR).
Returns `true` if the device meets the minimum OS requirements; `false` otherwise. Call this before attempting `recognizeText()` to provide graceful fallback UI for unsupported devices.
**Minimum OS versions:**
- **iOS**: 16.0+
- **Android**: API 21+ (Android 5.0)
**Example:**
```typescript
import { isSupported, recognizeText } from 'expo-mlkit-ocr';
export default function App() {
if (!isSupported()) {
return OCR is not supported on this device.;
}
return (
{
const result = await recognizeText(imageUri);
console.log(result.text);
}}
/>
);
}
```
### `` — Interactive Bounding Box Overlay
Renders interactive bounding boxes over an image to visualize OCR results. Tap boxes to select text and trigger a callback (e.g., copy to clipboard).
**Usage:**
```typescript
import { recognizeText, OCRTextOverlay } from 'expo-mlkit-ocr';
import { Image, Clipboard } from 'react-native';
import { useState } from 'react';
export default function App() {
const [result, setResult] = useState(null);
async function pickAndRecognize() {
const picked = await ImagePicker.launchImageLibraryAsync({ mediaTypes: ['images'] });
if (!picked.assets[0]) return;
const asset = picked.assets[0];
const ocrResult = await recognizeText(asset.uri);
setResult(ocrResult);
}
return (
<>
{result && (
Clipboard.setString(item.text)}
>
)}
>
);
}
```
**Props:**
| Prop | Type | Default | Description |
|------|------|---------|-------------|
| `result` | `RecognitionResult` | — | OCR result from `recognizeText()` |
| `imageWidth` | `number` | — | Native image width (pixels) |
| `imageHeight` | `number` | — | Native image height (pixels) |
| `children` | `ReactNode` | — | Image component to wrap |
| `highlightLevel` | `'block' \| 'line' \| 'element'` | `'line'` | Which level to highlight |
| `resizeMode` | `'contain' \| 'cover'` | `'contain'` | Image resize behavior |
| `boxColor` | `string` | `'#00BFFF'` | Box border & fill color (hex) |
| `selectedBoxColor` | `string` | `'#FF6347'` | Color when a box is selected |
| `boxOpacity` | `number` | `0.25` | Fill opacity (0–1) |
| `strokeWidth` | `number` | `2` | Border width (pixels) |
| `cornerRadius` | `number` | `4` | Rounded corner radius (pixels) |
| `multiSelect` | `boolean` | `true` | Allow selecting multiple boxes |
| `onSelect` | `(item) => void` | — | Callback when box(es) are tapped (single item or array) |
| `style` | `ViewStyle` | — | Optional wrapper style |
**Multi-Select Example:**
```typescript
// Single selection (one box at a time)
console.log('Selected:', item.text)}
>
// Multiple selection (tap multiple boxes)
{
if (Array.isArray(items)) {
console.log('Selected items:', items.map(i => i.text).join(', '));
} else {
console.log('Single item:', items.text);
}
}}
>
```
**Highlights:**
- ✅ Pure React Native (no external canvas library)
- ✅ Tap to toggle selection + visual highlight
- ✅ Multi-select mode: tap multiple boxes, all stay highlighted
- ✅ Single-select mode: only one box highlighted at a time
- ✅ Works with any `` component (react-native, expo-image, etc.)
- ✅ Automatic coordinate scaling for `contain` and `cover` resize modes
- ✅ Full TypeScript support
## Common Errors
### `Error: expo-mlkit-ocr is not supported on web.`
The module only works on iOS and Android. For web support, use a third-party OCR service (e.g., Tesseract.js).
```typescript
import { Platform } from 'react-native';
import { recognizeText } from 'expo-mlkit-ocr';
if (Platform.OS !== 'web') {
const result = await recognizeText(uri);
} else {
// Use a web-based OCR service
}
```
### `IMAGE_LOAD_FAILED`
- Ensure the URI is valid and the file exists
- Use URIs from `expo-image-picker` or `expo-camera` which are guaranteed to work
- On Android, both `file://` and `content://` URIs are supported
### iOS Simulator (arm64) + iOS 26.x
Google ML Kit CocoaPods binaries exclude the `ios-arm64-simulator` slice, so arm64-only simulator runtimes (for example iOS 26.x on Apple Silicon) can’t link ML Kit frameworks and will fail at build/link time.
To keep development on simulator unblocked, `expo-mlkit-ocr` supports **Apple Vision** as a fallback OCR engine on iOS. The **JavaScript response shape stays the same** (`text`, `blocks`, `lines`, `elements`, `boundingBox`); only the underlying OCR engine changes.
```json
{
"expo": {
"plugins": [
["expo-mlkit-ocr", { "iosEngine": "vision" }]
]
}
}
```
Supported `iosEngine` values:
- `"auto"` (default): use Apple Vision (simulator-friendly default)
- `"mlkit"`: use Google ML Kit (won’t build on arm64-only iOS Simulator runtimes)
- `"vision"`: always use Apple Vision (disables ML Kit pods)
Legacy option (equivalent to `iosEngine: "vision"`):
- `disableMlkitOnSimulator: true`
## Development
### Project Structure
```
expo-mlkit-ocr/
├── src/ # TypeScript source
│ ├── index.ts
│ ├── ExpoMlkitOcr.types.ts
│ ├── ExpoMlkitOcrModule.ts
│ ├── ExpoMlkitOcrModule.web.ts
│ └── OCRTextOverlay.tsx # Interactive overlay component
├── ios/
│ ├── ExpoMlkitOcrModule.swift # ML Kit integration
│ └── ExpoMlkitOcr.podspec
├── android/
│ ├── build.gradle
│ └── src/main/java/.../ExpoMlkitOcrModule.kt
├── app.plugin.js # Expo config plugin entry
├── plugins/
│ └── withMlkitSimulatorArm64Fix.js
├── example/ # Example app
│ ├── App.tsx
│ └── app.json
└── expo-module.config.json
```
### Building from Source
```bash
# Install dependencies
npm install
# Build the module (src/ → build/)
npm run prepare
# Open the example app (iOS)
npm run open:ios
# Or run the example app with Expo CLI
cd example
npx expo start
# Scan QR code with Expo Go or run on device/simulator
```
### Running the Example App
The example app at `example/` demonstrates:
1. Picking an image from the device library
2. Running OCR with `recognizeText()`
3. Displaying the results (full text, blocks, lines, elements)
## License
MIT
## Contributing
Contributions are welcome! Please ensure:
- TypeScript code compiles without errors
- Native code follows platform conventions
- Example app works on both iOS and Android
## References
- [ML Kit Text Recognition (v2) — iOS](https://developers.google.com/ml-kit/vision/text-recognition/v2/ios)
- [ML Kit Text Recognition (v2) — Android](https://developers.google.com/ml-kit/vision/text-recognition/v2/android)
- [Expo Modules API](https://docs.expo.dev/modules/get-started/)
Made with ❤️ by [rbayuokt](https://github.com/rbayuokt)