{"id":18499636,"url":"https://github.com/asimmon/marionette","last_synced_at":"2025-08-21T23:32:17.587Z","repository":{"id":40614411,"uuid":"382698205","full_name":"asimmon/marionette","owner":"asimmon","description":"Marionette is a test automation framework based on image and text recognition for .NET.","archived":false,"fork":false,"pushed_at":"2024-07-09T21:30:29.000Z","size":19243,"stargazers_count":49,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-02T12:54:14.765Z","etag":null,"topics":["automation-framework","automation-testing","csharp","ui-test-automation","ui-testing"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/asimmon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-03T19:49:12.000Z","updated_at":"2024-11-24T17:35:49.000Z","dependencies_parsed_at":"2023-02-01T05:15:44.141Z","dependency_job_id":null,"html_url":"https://github.com/asimmon/marionette","commit_stats":null,"previous_names":["asimmon/askaiser-marionette"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimmon%2Fmarionette","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimmon%2Fmarionette/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimmon%2Fmarionette/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/asimmon%2Fmarionette/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/asimmon","download_url":"https://codeload.github.com/asimmon/marionette/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230542287,"owners_count":18242332,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation-framework","automation-testing","csharp","ui-test-automation","ui-testing"],"created_at":"2024-11-06T13:46:42.400Z","updated_at":"2024-12-20T06:05:47.975Z","avatar_url":"https://github.com/asimmon.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- omit in toc --\u003e\n# Marionette\n\n[![nuget](https://img.shields.io/nuget/v/Askaiser.Marionette.svg?logo=nuget)](https://www.nuget.org/packages/Askaiser.Marionette/)\n[![build](https://img.shields.io/github/actions/workflow/status/asimmon/askaiser-marionette/ci.yml?branch=master\u0026logo=github)](https://github.com/asimmon/askaiser-marionette/actions/workflows/ci.yml)\n\n**Marionette is a test automation framework based on image and text recognition**. It includes a C# source generator that allows you to quickly interact with C# properties generated from images in your project or elsewhere. The framework is built on top of **OpenCV** and **Tesseract OCR** and only supports Windows for now.\n\n**Table of contents**\n\n- [🔥 Motivation and features](#-motivation-and-features)\n- [🎥 Demo video of Windows Calculator automation](#-demo-video-of-windows-calculator-automation)\n- [🚀 Getting started](#-getting-started)\n- [🏭 Change the C# source generator behavior](#-change-the-c-source-generator-behavior)\n- [✍ Manually creating image and text elements](#-manually-creating-image-and-text-elements)\n- [📕 API reference](#-api-reference)\n- [🤖 Running tests in a CI environment](#-running-tests-in-a-ci-environment)\n\n## 🔥 Motivation and features\n\n* Unlike other test automation frameworks, Marionette **does not rely on hardcoded identifiers, CSS or XPath selectors**. It uses image and text recognition to ensure that you interact with elements that are **actually visible** on the screen.\n* Maintaining identifiers, CSS and XPath selectors over time can be hard. Capturing small screenshots and finding text with an OCR is not.\n* With the built-in C# source generator, you can start **writing the test code right away**.\n* You can interact with the whole operating system, instead of a single application.\n* You can interact with any kind of desktop applications.\n* It works well with [BDD](https://en.wikipedia.org/wiki/Behavior-driven_development) and [SpecFlow](https://specflow.org/).\n\n\n## 🎥 Demo video of Windows Calculator automation\n\n\u003e This short video (1m40s) shows how to interact with an existing application with no prior knowledge of its source code, in a matter of minutes.\n\n* `00:00` : Quickly capture screenshots of the app you're testing,\n* `00:08` : Rename and organize your screenshots in a meaningful way,\n* `00:22` : Drop your screenshots in your C# project,\n* `00:30` : Use `ImageLibraryAttribute` to **automatically generate C# properties** from your screenshots,\n* `01:06` : Use `MarionetteDriver` to interact with the generated properties (or text recognized by the OCR)!\n\nhttps://user-images.githubusercontent.com/14242083/126416123-aebd0fce-825f-4ece-90e9-762503dc4cab.mp4\n\n\n## 🚀 Getting started\n\n```\ndotnet add package Askaiser.Marionette\n```\n\nIt supports **.NET Standard 2.0**, **.NET Standard 2.1** an **.NET 6**, but only on Windows for now.\n\n```csharp\n[ImageLibrary(\"path/to/your/images\")]\npublic partial class MyLibrary\n{\n    // In the \"path/to/your/images\" directory, we assume here that there are multiple small screenshots of UI elements\n    // with these relative paths: \"pages/login/title.png\", \"pages/login/email.png\", \"pages/login/password.png\" and \"pages/login/submit.png\".\n    // These file names actually control the behavior of the C# source generator.\n    // This behavior is explained in the next section.\n}\n\nusing (var driver = MarionetteDriver.Create(/* optional DriverOptions */))\n{\n    // in this exemple, we enter a username and password in a login page\n    await driver.WaitForAsync(MyLibrary.Instance.Pages.Login.Title, waitFor: TimeSpan.FromSeconds(5));\n\n    await driver.SingleClickAsync(MyLibrary.Instance.Pages.Login.Email);\n    await driver.TypeTextAsync(\"much@automated.foo\", sleepAfter: TimeSpan.FromSeconds(0.5));\n    await driver.SingleClickAsync(MyLibrary.Instance.Pages.Login.Password);\n    await driver.TypeTextAsync(\"V3ry5ecre7!\", sleepAfter: TimeSpan.FromSeconds(0.5));\n\n    await driver.SingleClickAsync(MyLibrary.Instance.Pages.Login.Submit);\n    \n    // insert more magic here\n}\n```\n\nThe [sample project](https://github.com/asimmon/askaiser-marionette/tree/master/samples/Askaiser.Marionette.ConsoleApp) shows the basics of using this library.\n\n\n## 🏭 Change the C# source generator behavior\n\nGiven the following partial image library class:\n\n```csharp\n[ImageLibrary(\"path/to/your/images/directory\")]\npublic partial class MyLibrary\n{\n}\n\nvar library = new MyLibrary(); // or use MyLibrary.Instance generated singleton\n```\n\nThe source generator will behave differently depending on the name of the screenshots/images added to the `ImageLibraryAttribute`'s directory path.\n\n\u003e **Use lowercase characters**. Dashes (`-`) and underscores (`_`) are considered as special characters that will change the generated C# code as shown below.\n\n* Adding an image `okbutton.png` will create an image property `library.Okbutton`,\n* Adding an image `ok-button.png` will create an image property `library.OkButton` (here the **dash** is a word separator),\n* Adding an image in a **subdirectory** `my-feature\\ok-button.png` will create an image property `library.MyFeature.OkButton`,\n* Adding an image with a **double dash** `my-feature--ok-button.png` simulates a subdirectory and will create an image property `library.MyFeature.OkButton`.\n\nAdding multiple images **suffixed with a zero-based incremental number** such as `ok-button_0.png`, `ok-button_1.png`, `ok-button_2.png`, etc. will create a single array property `library.OkButton` that can be interacted with methods that accept an array of elements (`MoveToAnyAsync`, `WaitForAnyAsync`, `SingleClickAnyAsync`, etc.).\nIt is very convenient when you have screenshots of an element in multiple states, for instance a button in its normal, pressed, focus state and you want to click on the button no matter what its current state is.\n\n**Underscores** `_` can be also used to specify image recognition options:\n\n* Adding an image `foo_gs.png` will create an image property `library.Foo` which will be grayscaled during image recognition,\n* Adding an image `foo_0.8` will create an image property `library.Foo` that will use a search threshold of `0.8` instead of the default `0.95`,\n\nYou can mix these modifiers. Here we will create an single array property `library.Header` with these images:\n* `header_gs_0.85_0.png` (first item of the array, grayscaled with a 0.85 threshold),\n* `header_gs_0.9_1.png` (second item of the array, grayscaled with a 0.9 threshold),\n* `header_2.png` (third and last item of the array, keep original colors with and use default threshold).\n\n\n## ✍ Manually creating image and text elements\n\n### Image search\n\n```csharp\n// Instead of relying on the source generator that works with image files, you can create an ImageElement manually\nvar bytes = await File.ReadAllBytesAsync(\"path/to/your/image.png\");\nvar image = new ImageElement(name: \"sidebar-close-button\", content: bytes, threshold: 0.95m, grayscale: false);\n```\n\n* `ImageElement.Threshold` is a floating number between 0 and 1. It defines the accuracy of the image search process. `0.95` is the default value.\n* `ImageElement.Grayscale` defines whether or not the engine will apply grayscaling preprocessing. Image search is faster with grayscaling.\n\n**Image recognition works best with PNG images.**\n\n### Text search\n\n```csharp\n// Although many methods accept a simple string as an element, you can manually create a TextElement\nvar text = new TextElement(\"Save changes\", options: TextOptions.BlackAndWhite | TextOptions.Negative);\n```\n\n**Text options** are flags that define the preprocessing behavior of your monitor's screenshots before executing the OCR.\n* `TextOptions.None` : do not use preprocessing,\n* `TextOptions.Grayscale` : Use grayscaling,\n* `TextOptions.BlackAndWhite` : Use grayscaling and binarization (this is the default value),\n* `TextOptions.Negative` : Use negative preprocessing, very helpful with white text on dark background.\n\n\n## 📕 API reference\n\nMany parameters are optional. Most methods that look for an element (image or text) expect to find **only one occurrence** of this element. `ElementNotFoundException` and `MultipleElementFoundException` can be thrown.\n\nYou can use `DriverOptions.FailureScreenshotPath` to automatically save screenshots when these exceptions occur.\n\n### Configuration and utilities\n\n```csharp\nstatic Create()\nstatic Create(DriverOptions options)\n\nGetScreenshotAsync()\nSaveScreenshotAsync(Stream destinationStream)\nSaveScreenshotAsync(string destinationPath)\nGetCurrentMonitorAsync()\nGetMonitorsAsync()\nSetCurrentMonitor(int monitorIndex)\nSetCurrentMonitor(MonitorDescription monitor)\nSetMouseSpeed(MouseSpeed speed)\nSleepAsync(int millisecondsDelay)\nSleepAsync(TimeSpan delay)\n```\n\n### Basic methods\n\n```csharp\nWaitForAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nWaitForAllAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nWaitForAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nSingleClickAsync(int x, int y)\nDoubleClickAsync(int x, int y)\nTripleClickAsync(int x, int y)\nRightClickAsync(int x, int y)\nMoveToAsync(int x, int y)\nDragFromAsync(int x, int y)\nDropToAsync(int x, int y)\nTypeTextAsync(string text, TimeSpan? sleepAfter)\nKeyPressAsync(VirtualKeyCode[] keyCodes, TimeSpan? sleepAfter)\nKeyDownAsync(VirtualKeyCode[] keyCodes, TimeSpan? sleepAfter)\nKeyUpAsync(VirtualKeyCode[] keyCodes, TimeSpan? sleepAfter)\nScrollDownAsync(int scrollTicks)\nScrollUpAsync(int scrollTicks)\n```\n\n### Mouse interaction with an element\n\n```csharp\nMoveToAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nSingleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nDoubleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nTripleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nRightClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nDragFromAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nDropToAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nScrollDownUntilVisibleAsync(IElement element, TimeSpan totalDuration, int scrollTicks, Rectangle? searchRect)\nScrollUpUntilVisibleAsync(IElement element, TimeSpan totalDuration, int scrollTicks, Rectangle? searchRect)\n```\n\n### Check for element visibility\n\n```csharp\nIsVisibleAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nIsAnyVisibleAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nAreAllVisibleAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\n```\n\n### Mouse interaction with the first available element of a collection\n\n```csharp\nMoveToAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nSingleClickAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nDoubleClickAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nTripleClickAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nRightClickAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nDragFromAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\nDropToAnyAsync(IEnumerable\u003cIElement\u003e elements, TimeSpan? waitFor, Rectangle? searchRect)\n```\n\n### Text-based interaction\n\n```csharp\nWaitForAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nMoveToAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nSingleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nDoubleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nTripleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nRightClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nDragFromAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nDropToAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nIsVisibleAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\n```\n\n### Mouse interaction with points\n\n```csharp\nMoveToAsync(Point coordinates)\nSingleClickAsync(Point coordinates)\nDoubleClickAsync(Point coordinates)\nTripleClickAsync(Point coordinates)\nRightClickAsync(Point coordinates)\nDragFromAsync(Point coordinates)\nDropToAsync(Point coordinates)\n```\n\n### Mouse interaction with `WaitFor` search result\n\n```csharp\nMoveToAsync(SearchResult searchResult)\nSingleClickAsync(SearchResult searchResult)\nDoubleClickAsync(SearchResult searchResult)\nTripleClickAsync(SearchResult searchResult)\nRightClickAsync(SearchResult searchResult)\nDragFromAsync(SearchResult searchResult)\nDropToAsync(SearchResult searchResult)\n```\n\n### Key press with single key code\n\n```csharp\nKeyPressAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)\nKeyDownAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)\nKeyUpAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)\n```\n\n### `System.Drawing.Image`-based interaction\n\n```csharp\nWaitForAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nMoveToAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nSingleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nDoubleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nTripleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nRightClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nDragFromAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nDropToAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\nIsVisibleAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\n```\n\n### Finding elements locations without throwing not found exceptions or multiple element found exceptions\n\n```csharp\nFindLocationsAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)\nFindLocationsAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)\nFindLocationsAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)\n```\n\n\n## 🤖 Running tests in a CI environment\n\n\u003e 👷‍♂️ Work in progress\n\nThis section will soon show how Marionette can be used in an [Azure Pipelines](https://azure.microsoft.com/en-us/products/devops/pipelines/) continuous integration environment:\n\n1. Setup a dedicated Windows agent, or use the [built-in Windows agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops\u0026tabs=browser#microsoft-hosted-agents),\n2. Use Microsoft's [Screen Resolution Utility task](https://marketplace.visualstudio.com/items?itemName=ms-autotest.screen-resolution-utility-task) to setup a virtual monitor and change its resolution.\n\nThere seems to be an [equivalent for GitHub actions](https://github.com/actions/runner-images/issues/2935). If you use another CI environment, please search its documentation for an equivalent behavior.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasimmon%2Fmarionette","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fasimmon%2Fmarionette","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fasimmon%2Fmarionette/lists"}