{"id":19607280,"url":"https://github.com/robocorp/example-desktop-image-template-matching","last_synced_at":"2025-07-20T06:06:13.431Z","repository":{"id":103903108,"uuid":"329839188","full_name":"robocorp/example-desktop-image-template-matching","owner":"robocorp","description":"Cross-platform desktop automation using image template matching and keyboard shortcuts.","archived":false,"fork":false,"pushed_at":"2022-07-07T12:49:26.000Z","size":11198,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-02-26T16:50:32.102Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"RobotFramework","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robocorp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-15T07:25:17.000Z","updated_at":"2023-06-16T20:17:15.000Z","dependencies_parsed_at":"2023-07-26T03:46:38.973Z","dependency_job_id":null,"html_url":"https://github.com/robocorp/example-desktop-image-template-matching","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/robocorp/example-desktop-image-template-matching","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robocorp%2Fexample-desktop-image-template-matching","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robocorp%2Fexample-desktop-image-template-matching/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robocorp%2Fexample-desktop-image-template-matching/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robocorp%2Fexample-desktop-image-template-matching/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robocorp","download_url":"https://codeload.github.com/robocorp/example-desktop-image-template-matching/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robocorp%2Fexample-desktop-image-template-matching/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266076095,"owners_count":23872730,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T10:09:47.187Z","updated_at":"2025-07-20T06:06:13.402Z","avatar_url":"https://github.com/robocorp.png","language":"RobotFramework","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Travel directions desktop automation robot on macOS\n\n\u003e In image template-based [desktop automation](https://robocorp.com/docs/development-guide/desktop), you provide the robot with screenshots of the parts of the interface that it needs to interact with, like a button or input field. The images are saved together with your automation code. The robot will compare the image to what is currently displayed on the screen and find its target.\n\n## Cross-platform desktop automation library\n\nRobocorp provides cross-platform desktop automation support with the [RPA.Desktop library](https://robocorp.com/docs/libraries/rpa-framework/rpa-desktop). It works on Windows, Linux, and macOS.\n\n## Travel directions robot\n\nThis example robot demonstrates the use of image templates and keyboard shortcuts to find travel directions between two random locations on Earth.\n\nThe robot:\n\n- Interacts with a web browser to select two random locations on Earth (from https://www.randomlists.com/random-location).\n- Tries to find the directions using the Maps desktop app on macOS (Monterey), using image templates and keyboard shortcuts.\n- Falls back on the web version of Google Maps if Maps fails to find directions.\n\n\u003e **Note:** This robot requires **macOS Monterey**. (Tested on version 12.4.) The layout and the behavior of the Maps app vary between macOS releases. macOS will ask for permissions the first time you run the robot. Go to `System Preferences` -\u003e `Security \u0026 Privacy` and check `Robocorp Lab`, `Code`, `VS Code` or `Terminal` (depending on where you run the robot from) in the `Accessibility` and `Screen Recording` sections.\n\nAnother important topic:\n\n\u003e System settings can impact the recognition of the images: How the interface elements look on a screen depends on system settings like color schemes, transparency, and system fonts. Images taken on a system might end up looking different than the target system, and the robot might not recognize them, stopping the process.\n\nIn this case, macOS should use the \"Dark\" appearance under `System Preferences` -\u003e `General`. See our [Desktop automation](https://robocorp.com/docs/development-guide/desktop) page for more information.\n\n### The settings\n\n```robot\n*** Settings ***\nDocumentation     Finds travel directions between two random locations.\n...               Selects two random locations on Earth.\n...               Finds the directions using the Maps app on macOS (Big Sur).\n...               Falls back on Google Maps, if Maps fails to find directions.\nLibrary           Process\nLibrary           RPA.Browser\nLibrary           RPA.Desktop\nTask Teardown     Close All Browsers\n```\n\nThe robot uses three [libraries](https://robocorp.com/docs/languages-and-frameworks/robot-framework/basics#what-are-libraries) to automate the task. Finally, it will close all the browsers it happened to open.\n\n### The task: Find travel directions between two random locations\n\n```robot\n*** Tasks ***\nFind travel directions between two random locations\n    @{locations}=    Get random locations\n    Open the Maps app\n    Maximize the window\n    View directions    ${locations}[0]    ${locations}[1]\n```\n\n### Variables\n\n```robot\n*** Variables ***\n${RANDOM_LOCATION_WEBSITE}=    https://www.randomlists.com/random-location\n${DIRECTIONS_SCREENSHOT}=    ${CURDIR}${/}output${/}directions.png\n```\n\n### Keyword: Get random locations\n\n```robot\n*** Keywords ***\nGet random locations\n    Open Available Browser    ${RANDOM_LOCATION_WEBSITE}    headless=True\n    Set Window Size    1600    1200\n    @{location_elements}=    Get WebElements    css:.rand_medium\n    ${location_1}=    Get Text    ${location_elements}[0]\n    ${location_2}=    Get Text    ${location_elements}[1]\n    [Return]    ${location_1}    ${location_2}\n```\n\nThe robot uses a web browser to scrape and return two random locations from a suitable website.\n\n### Keyword: Open the Maps app\n\n```robot\n*** Keywords ***\nOpen the Maps app\n    Run Process    open    -a    Maps\n    Wait For Element    alias:Maps.MapMode\n```\n\nThe robot opens the Maps app using the `Run Process` keyword from the `Process` library. It executes the `open -a Maps` command. You can run the same command in your terminal to see what happens!\n\nThe robot knows when the Maps app is open by waiting for the `Maps.MapMode` [image template](https://robocorp.com/docs/developer-tools/robocorp-lab/locating-and-targeting-UI-elements#image-template-matching-paint-windows-10) to return a match.\n\n### Keyword: Maximize the window\n\n```robot\n*** Keywords ***\nMaximize the window\n    ${not_maximized}=\n    ...    Run Keyword And Return Status\n    ...    Find Element    alias:Desktop.WindowControls\n    Run Keyword If\n    ...    ${not_maximized}\n    ...    RPA.Desktop.Press Keys    ctrl    cmd    f\n    Wait For Element    not alias:Desktop.WindowControls\n```\n\nThe robot maximizes the Maps app window using a keyboard shortcut unless the app is already maximized. The `Run Keyword If` is used for [conditional execution](https://robocorp.com/docs/languages-and-frameworks/robot-framework/conditional-execution).\n\nThe robot knows the Maps app is maximized when the `Desktop.WindowControls` image template does **not** return a match (when the close/minimize/maximize icons are not anywhere on the screen).\n\n### Keyword: Open and reset the directions view\n\n```robot\n*** Keywords ***\nOpen and reset the directions view\n    ${directions_open}=\n    ...    Run Keyword And Return Status\n    ...    Find Element    alias:Maps.SwapLocations\n    Run Keyword Unless\n    ...    ${directions_open}\n    ...    RPA.Desktop.Press Keys    cmd    r\n    Wait For Element    alias:Maps.SwapLocations\n    Click    alias:Maps.ResetFromAndToLocationsIcon\n    RPA.Desktop.Press Keys    cmd    r\n    Wait For Element    alias:Maps.SwapLocations\n```\n\nThe robot sets the directions view in the Maps app to a known starting state (empty from and to locations).\n\n- Conditional execution is used to handle the possible states for the view (it might or might not be open already).\n- Image templates are used to wait for specific app states so that the robot knows when something has been completed.\n- Keyboard shortcuts are used to toggle the directions view.\n\n### Keyword: Accept Google consent\n\n```robot\n*** Keywords ***\nAccept Google consent\n    Click Button When Visible    xpath://form//button\n```\n\n### Keyword: View directions using Google Maps\n\n```robot\n*** Keywords ***\nView directions using Google Maps\n    [Arguments]    ${location_1}    ${location_2}\n    Go To    https://www.google.com/maps/dir/${location_1}/${location_2}/\n    Accept Google consent\n    Wait Until Element Is Visible    css:.section-directions-options\n    Screenshot    filename=${DIRECTIONS_SCREENSHOT}\n```\n\nThe robot waits until Google Maps has loaded the directions and takes a full web page screenshot.\n\n### Keyword: Enter location\n\n```robot\n*** Keywords ***\nEnter location\n    [Arguments]    ${locator}    ${location}\n    Wait For Element    ${locator}\n    Click    ${locator}\n    Type Text    ${location}    enter=True\n```\n\nThe robot needs to input the from and to locations. This keyword provides a generic way to target those elements on the UI.\n\n### Keyword: View directions\n\n```robot\n*** Keywords ***\nView directions\n    [Arguments]    ${location_1}    ${location_2}\n    Open and reset the directions view\n    Enter location    alias:Maps.FromLocation    ${location_1}\n    Enter location    alias:Maps.ToLocation    ${location_2}\n    ${directions_found}=\n    ...    Run Keyword And Return Status\n    ...    Wait For Element    alias:Maps.RouteIcon    timeout=20.0\n    Run Keyword Unless\n    ...    ${directions_found}\n    ...    View directions using Google Maps    ${location_1}    ${location_2}\n    Run Keyword If\n    ...    ${directions_found}\n    ...    Take Screenshot    ${DIRECTIONS_SCREENSHOT}\n```\n\nThe robot tries to find the directions using the Maps app. If that fails, the robot gets the directions from Google Maps.\n\n## Summary\n\n- Image template matching is a cross-platform way to find and target UI elements.\n- Keyboard shortcuts are the preferred way to interact with desktop applications (the shortcuts are usually more stable and predictable than the UI).\n- Conditional logic can be used to select different actions based on the state of the application.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobocorp%2Fexample-desktop-image-template-matching","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobocorp%2Fexample-desktop-image-template-matching","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobocorp%2Fexample-desktop-image-template-matching/lists"}