https://github.com/robocorp/example-desktop-image-ocr

Example robot for automating GnuCash with image templates and OCR
https://github.com/robocorp/example-desktop-image-ocr

Last synced: about 1 year ago
JSON representation

Example robot for automating GnuCash with image templates and OCR

• Host: GitHub
• URL: https://github.com/robocorp/example-desktop-image-ocr
• Owner: robocorp
• License: apache-2.0
• Created: 2020-12-11T08:42:59.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2024-01-23T09:10:56.000Z (over 2 years ago)
• Last Synced: 2025-04-27T19:46:19.124Z (about 1 year ago)
• Language: Python
• Size: 30.3 KB
• Stars: 11
• Watchers: 17
• Forks: 4
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Desktop automation using images templates and OCR

![GnuCash Logo](https://www.gnucash.org/images/gnc-download.png)

This robot demonstrates automating a desktop application with image recognition and OCR.

The system being automated is a cross-platform free accounting software called [GnuCash](https://www.gnucash.org/).

This demo uses a mock account.

Most of the interaction happens by locating a desired text by OCR. Regions and offsets are involved too,

e.g. when you want to type text into an input field with a text label next to it,

you first find the label, then get a region or offset relative to that text and click there.

## Note about OCR

OCR refers to recognizing characters, words, and text in general. Image recognition is more

generic (and simpler) and might refer to the recognition of, e.g. colors and shapes.

OCR or image recognition based automation is usually the last resort in automation.

When thinking whether you should use OCR to automate your application, first investigate if you could:

- Read the raw data programmatically

- Use native UI selectors

- Use UI hotkeys

- Use the clipboard

Usually image recognition must be used when dealing with Citrix (or similar).

Even then hotkeys and the clipboard are oftentimes more reliable than pure OCR.

Another use case for OCR is reading text from documents (PDF, JPEG),

this demo does not consider that use case at all.

When utilizing OCR or image-based locators, it is imperative to execute the process within a consistent environment. This entails maintaining uniform resolution and scaling settings across all executions, as the locators are predominantly calibrated for the specific display configurations for which they were originally designed.

## Requirements

This robot assumes the running environment has a functional installation

of [GnuCash](https://www.gnucash.org/). All other dependencies are

handled by RCC and conda.yaml.