{"id":48380504,"url":"https://github.com/karthik-saiharsh/distributed-ocr","last_synced_at":"2026-04-05T19:31:34.027Z","repository":{"id":342524037,"uuid":"1156800252","full_name":"karthik-saiharsh/distributed-ocr","owner":"karthik-saiharsh","description":"DOCR: A local-first, distributed Optical Character Recognition (OCR) platform built with Go, React, and Wails. ","archived":false,"fork":false,"pushed_at":"2026-03-06T08:49:52.000Z","size":1225,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-06T12:44:21.064Z","etag":null,"topics":["distributed-systems","go","ocr","queuing","tesseract","tesseract-ocr","text-recognition","wails","work-stealing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karthik-saiharsh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-13T04:11:47.000Z","updated_at":"2026-03-06T08:49:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/karthik-saiharsh/distributed-ocr","commit_stats":null,"previous_names":["karthik-saiharsh/distributed-ocr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/karthik-saiharsh/distributed-ocr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthik-saiharsh%2Fdistributed-ocr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthik-saiharsh%2Fdistributed-ocr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthik-saiharsh%2Fdistributed-ocr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthik-saiharsh%2Fdistributed-ocr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karthik-saiharsh","download_url":"https://codeload.github.com/karthik-saiharsh/distributed-ocr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karthik-saiharsh%2Fdistributed-ocr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31448215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-05T15:22:31.103Z","status":"ssl_error","status_checked_at":"2026-04-05T15:22:00.205Z","response_time":75,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","go","ocr","queuing","tesseract","tesseract-ocr","text-recognition","wails","work-stealing"],"created_at":"2026-04-05T19:31:33.419Z","updated_at":"2026-04-05T19:31:34.022Z","avatar_url":"https://github.com/karthik-saiharsh.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DOCR: Privacy-First Distributed OCR Grid\n\n\u003cp align=\"center\"\u003e\n  A local-first, distributed Optical Character Recognition (OCR) platform built with Go, React, and Wails.\n\u003c/p\u003e\n\n## Overview\n\nIn the modern digital infrastructure, organizations face a massive bottleneck: the digitization of physical archives. Processing tens of thousands of high-resolution pages is computationally expensive and slow on a single machine. Cloud solutions (like AWS Textract) introduce severe data privacy concerns (HIPAA, GDPR) and require constant internet connectivity.\n\n**Swarm** solves this by creating a **decentralized compute grid** out of ad-hoc local devices (laptops, desktops) sitting around in your office. It uses a **Master-Worker topology** over a Local Area Network (LAN) to securely and privately distribute OCR tasks using advanced distributed systems techniques.\n\n---\n\n## Key Features\n\n*   **Privacy-First \u0026 Local:** Zero cloud dependency. Sensitive documents (medical records, legal contracts) never leave your local network. Air-gap friendly.\n*   **Dynamic Load Balancing (Work Stealing):** Idle worker nodes proactively \"steal\" tasks from busy nodes via direct P2P RPC, ensuring maximum CPU utilization across the cluster.\n*   **Autonomic Peer Discovery (SWIM Gossip):** Nodes dynamically form a mesh network via UDP multicast gossip. If a laptop is closed or disconnects, the cluster self-heals without task loss.\n*   **Result Verification (Consensus):** Implements redundant execution. Multiple workers process the same chunk and the Master verifies consensus to defend against malicious nodes or corrupted processing.\n*   **Cross-Platform GUI:** A sleek interface built with React, Vite, and Wails, giving a native desktop feel on Mac, Windows, and Linux.\n\n---\n\n## System Architecture\n\n\u003cimg src=\"https://github.com/karthik-saiharsh/distributed-ocr/blob/main/Architechture.jpeg?raw=true\"\u003eSystem Architechture\u003c/img\u003e\n\nThe architecture relies on high-performance concurrent processing in Go and robust networking protocols:\n\n1.  **Master Node (Orchestrator):** Runs the Wails GUI, manages the Global Job Queue, parses multi-page PDFs locally, and validates the integrity of returned OCR data.\n2.  **Worker Nodes:** Stateless compute units running the Tesseract CGO wrapper. Features a local Double-Ended Queue (Deque) optimized for both LIFO local processing (cache locality) and FIFO work stealing.\n\n*See [`explanation.md`](explanation.md) for a deep dive into the network topologies and data flow.*\n\n---\n\n## Prerequisites\n\nTo run or develop Swarm, ensure you have the following installed:\n\n1.  **[Go](https://go.dev/doc/install)** (1.20+)\n2.  **[Node.js \u0026 npm](https://nodejs.org/en/)**\n3.  **[Wails Setup](https://wails.io/docs/gettingstarted/installation)**\n4.  **Tesseract OCR:** Required on each machine for the core engine:\n    *   **macOS:** `brew install tesseract`\n    *   **Linux (Ubuntu):** `sudo apt-get install tesseract-ocr libtesseract-dev`\n    *   **Windows:** UB-Mannheim Tesseract installer\n\n*(Note: We use `github.com/gen2brain/go-fitz` for cross-platform PDF handling).*\n\n---\n\n## Development Setup\n\n1.  **Clone the Repository:**\n    ```bash\n    git clone https://github.com/your-org/distributed-ocr.git\n    cd distributed-ocr\n    ```\n\n2.  **Frontend Setup:**\n    The project uses a React/Vite frontend located in `/frontend`.\n    ```bash\n    cd frontend\n    npm install\n    npm run dev\n    ```\n\n3.  **Backend/App Setup:**\n    The main Wails application is bound in `app.go`. To run the application in development mode with hot-reloading:\n    ```bash\n    # From the project root\n    wails dev\n    ```\n    *Linux Users:* Run with `wails dev -tags webkit2_41` to support specific webkit dependencies.\n\n---\n\n## Running the Distributed Cluster\n\nTo see the distributed work stealing and gossip protocols in action across physical machines:\n\n1.  **LAN Connection:** Connect multiple computers (e.g., Laptop A and Laptop B) to the exact same local Wi-Fi or router.\n2.  **Build the Release:**\n    Compile the app for production on both machines:\n    ```bash\n    wails build\n    ```\n3.  **Launch Nodes:** Open the compiled app executable (found in `build/bin/`) on both computers.\n4.  **Discover Peers:** On Laptop A (your designated Master), click **Scan For Nodes**. The UDP gossip protocol will automatically discover Laptop B's IP address.\n5.  **Distribute Work:** Click **Upload Document** on Laptop A and select a large PDF. \n6.  **Watch the Magic:** Laptop A chunks the PDF and distributes it via RPC. Laptop B will compute the OCR using its local Tesseract instance and return strings back to Laptop A to be verified and stitched back together!\n\n---\n\n## Repository Structure\n\n*   `/frontend` - React, TypeScript, Vite frontend source.\n*   `/master` - Orchestrator logic, consensus verification, and job queuing.\n*   `/worker` - Node executor, Task Deque, and Tesseract C/Go bindings.\n*   `/swim` - Custom UDP Gossip and discovery protocol implementation.\n*   `/rpc` - Protobuf/TCP communication interfaces for task assignments and work stealing.\n\n---\n\n## Contributing\n\nWe welcome pull requests! \n1. Create a new branch for your feature (`git checkout -b feature/nice-feature`).\n2. Make your backend changes in Go or frontend changes inside `/frontend`.\n3. Please do **not** commit to `main` directly.\n4. Submit a PR!\n\n## License\nGNU GPL V3\n\n## Created By\n[@karthik-saiharsh](https://www.github.com/karthik-saiharsh), [@Adith1207](https://www.github.com/Adith1207), [@Dharsh045](https://www.github.com/Dharsh045), [@RoshJ-17](https://www.github.com/RoshJ-17)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarthik-saiharsh%2Fdistributed-ocr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarthik-saiharsh%2Fdistributed-ocr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarthik-saiharsh%2Fdistributed-ocr/lists"}