{"id":19226000,"url":"https://github.com/developer239/automation-engine","last_synced_at":"2025-02-23T10:19:05.716Z","repository":{"id":155404730,"uuid":"580826194","full_name":"developer239/automation-engine","owner":"developer239","description":"Automate repetitive visual tasks by writing custom Lua/TypeScript scripts. 🤖 Debug, test and run them in the engine.","archived":false,"fork":false,"pushed_at":"2023-09-17T23:05:53.000Z","size":183831,"stargazers_count":1,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-04T21:33:09.062Z","etag":null,"topics":["automation","cpp","display","instance-segmentation","keyboard","mouse","object-detection","yolo"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/developer239.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-21T14:47:55.000Z","updated_at":"2024-10-02T11:02:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"58792f1a-990b-4430-b113-e99a9f1a290b","html_url":"https://github.com/developer239/automation-engine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer239%2Fautomation-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer239%2Fautomation-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer239%2Fautomation-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/developer239%2Fautomation-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/developer239","download_url":"https://codeload.github.com/developer239/automation-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240298873,"owners_count":19779353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","cpp","display","instance-segmentation","keyboard","mouse","object-detection","yolo"],"created_at":"2024-11-09T15:17:01.992Z","updated_at":"2025-02-23T10:19:05.693Z","avatar_url":"https://github.com/developer239.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🤖 Automation Engine\n\n![master](https://github.com/developer239/automation-engine/actions/workflows/ci.yml/badge.svg)\n[![macOS](https://svgshare.com/i/ZjP.svg)](https://svgshare.com/i/ZjP.svg)\n\n## Prologue\n\nSeven years ago (in 2016), I watched [AI play Google's Downosaur](https://youtu.be/P7XHzqZjXQs). Although I don't speak\nPortuguese, I was amazed that AI could play the game. After watching similar videos\nlike [MarI/O](https://youtu.be/qv6UVOQ0F44), I decided to learn how to do something similar.\n\nOther than AI, I was mostly interested in being able to automate simple arcade games just to see how difficult it would\nbe to write a script that could beat human players or just last longer playing the game and be more efficient\nin that way.\n\nI didn't know any programming languages except for PHP and JavaScript, so naturally, I decided to create a JavaScript\napplication that could do just that. Couple of weekends and a couple\nof [hacks](https://stackoverflow.com/questions/48245948/electron-desktopcapturer-to-cv-videocapture-electron-nodejs-opencv4nodejs)\nlater, I put together Electron, OpenCV4NodeJS, RobotJS, and React in one Webpack bundle and\nreleased [developer239/electron-swords-and-souls-color-bot](https://github.com/developer239/electron-swords-and-souls-color-bot).\n\n## Automation Engine\n\nFast-forward to 2021, and I decided to step outside of my comfort zone and learn C++ and CMake. Some people\nrecommended [C++ Game Engine Programming](https://pikuma.com/courses/cpp-2d-game-engine-development) when I asked how to\nmake a custom game engine. Surprisingly, not only was I able to finish the 30-hour course, but it also gave me\nconfidence to build something on my own.\n\nWhen I was deciding about what to build, I remembered my old automation project and chose to rewrite it in C++.\nOnly the goal was to write general purpose solution and implemented a sort of engine so that I could automate anything\nwriting simple scripts.\n\nYou can find 📚 [TypeScript (Lua) scripts here](https://github.com/developer239/automation-engine-scripts).\n\n[![video preview](./docs/video-preview.png)](https://www.youtube.com/watch?v=u59b_5Q-SEs)\n\n## Technologies\n\n### CMake\n\nI wanted to learn how to use CMake and how to write CMakeLists files. All dependencies are\nincluded as git submodules except for SDL2, OpenCV and ONNXRuntime. Currently only MacOS is supported.\n\n```\n├── cmake (FindPACKAGE)\n│   ├── imgui\n│   ├── lua\n│   ├── onnx\n│   ├── sdl2\n│   └── tesseract\n└── externals\n    ├── googletest\n    ├── imgui\n    ├── ImGuiFileDialog\n    ├── lua\n    ├── onnxruntime\n    └── sol2\n\n```\n\nThe project is also split into `apps` and `packages`. `apps` contains only the automation-engine app at the moment and\nin `packages` there are many different modules that should be somewhat easy to reuse in across applications or even\ncopy paste into different C++ projects.\n\n```\n└── src\n    ├── apps\n    │   └── automation-engine\n    │       ├── assets\n    │       └── src\n    │           ├── components\n    │           ├── events\n    │           ├── layout\n    │           ├── main.cpp\n    │           ├── services\n    │           ├── strategies\n    │           ├── structs\n    │           └── systems\n    └── packages\n        ├── core\n        ├── core-imgui\n        ├── devices\n        ├── ecs\n        ├── events\n        ├── logger\n        ├── utility\n        └── yolo\n```\n\n### SDL2 \u0026 ImGui\n\nGUI is handled by SDL2 and ImGui. SDL2 creates a window and handles input events. ImGui is cloned from docking branch so\nthat it can provide extra window and docking features. Everything is then rendered inside ImGui windows.\n\n![preview-2](docs/preview-gui.png)\n\n### OpenCV\n\nOpenCV is used for computer vision and for simple image processing to detect objects on the screen using colors\nor to make ML object detection or OCR easier. It is also important to note that it is possible to capture selected\nscreen or specific screen region.\n\n![preview-opencv](docs/preview-opencv.gif)\n\n### ONNXRuntime \u0026 YOLOv5\n\nYOLOv5 is used for object detection and instance segmentation. In all examples I am using custom trained models.\n\nYou can find more information in\nseparate [yolo-cmake-opencv-onnx-cpp](https://github.com/developer239/yolo-cmake-opencv-onnx-cpp) repository.\n\n### Tesseract\n\nMostly proof of concept at this point because it is not as performant as I would like it to be. However, it works and it\nis possible to detect text in selected parts of the screen.\n\n![tesseract-example-1](docs/tesseract-example-1.png)\n![tesseract-example-2](docs/tesseract-example-2.png)\n\n### Lua \u0026 Sol\n\nSol is to Lua what TypeScript is to JavaScript. Lua is utilized for writing automation scripts that are loaded by the\nengine, making it easy to automate various tasks without requiring knowledge of how the engine operates or the need to\nwrite code in C++. As a bonus, it's also possible to write scripts in TypeScript and compile them to Lua\nusing [TypeScriptToLua](https://typescripttolua.github.io/).\n\nKeep in mind that this has nothing to do with JavaScript or TypeScript in the traditional sense. It is just a way to\nwrite scripts if you don't want to write C++ or Lua.\n\nIt is also possible to write scripts in C++ if you update\nScriptingSystem and replace `lua[\"main\"][\"onUpdate\"]();` with custom logic.\n\nYou can find type definitions here: [./automation-engine.d.ts](./automation-engine.d.ts) There is no need for `npm`\npackage. Treat it like a header file.\n\n```TypeScript\n// configure flappy bird detector\nconst flappyDetector = Registry.Instance().createEntity()\n\n// give entity unique tag\nflappyDetector.setTag('flappy-detector')\n\n// make editable in GUI\nflappyDetector.addComponentEditable()\n\n// add detection component for morphological operations\nflappyDetector.addComponentDetection()\n// crop selected area on the screen\nflappyDetector.addComponentDetectionCropOperation({\n    position: {x: 150, y: 0},\n    size: {width: 170, height: 724},\n})\n// detect flappy bird colors\nflappyDetector.addComponentDetectionColorsOperation({\n    lowerBound: {r: 0, g: 0, b: 30},\n    upperBound: {r: 255, g: 85, b: 255},\n})\n// close gaps in detected objects\nflappyDetector.addComponentDetectionMorphologyOperation('close', {\n    size: {width: 14, height: 14},\n})\n\n// find contours (only flappy bird should be visible)\nflappyDetector.addComponentDetectContours({\n    id: 'flappy',\n    minArea: {width: 10, height: 10},\n    maxArea: {width: 80, height: 80},\n    bboxColor: {r: 255, g: 0, b: 0},\n    bboxThickness: 3,\n    shouldRenderPreview: true\n})\n\n// ... create pipes detector\n\n// ... implement collision avoidance logic\n\nmain = {\n    onUpdate: () =\u003e {\n        // do something special on every frame\n    },\n    screen: {\n        // region size\n        width: 545,\n        height: 724,\n        // region position\n        x: 0,\n        y: 243,\n    },\n}\n```\n\n## Installation\n\nAs long as `Build \u0026 Test` CI is passing it should be easy to make things work.\n\n- `$ brew install pkg-config`\n- `$ brew install cmake`\n- `$ brew install opencv`\n- `$ brew install sdl2`\n- `$ brew install sdl2_ttf`\n- `$ brew install sdl2_image` (not actually used anywhere right now)\n- `$ brew install sdl2_mixer` (not actually used anywhere right now)\n- `$ brew install onnxruntime`\n\n## Build \u0026 Link\n\nIf your IDE supports CMake, you can use that. Otherwise, you can run the following command:\n\n```\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=$(brew --prefix)/bin/ninja -G Ninja -S . -B build\n$ ninja\n```\n\n## Development\n\n### ECS\n\nThe engine is built on top of custom Entity Component System (inspired\nby [C++ Game Engine Programming](https://pikuma.com/courses/cpp-2d-game-engine-development)) the core app logic is\ndefined in `ECSStrategy.h`.\n\n### ECS Strategy\n\n```c++\nclass ECSStrategy : public Core::IStrategy {\n public:\n  void Init(Core::Window\u0026 window, Core::Renderer\u0026 renderer) override {\n    // initialize fonts and other resources\n\n    //\n    // Initialize systems\n\n    // ...\n\n    //\n    // Initialize windows\n\n    // ...\n\n    //\n    // Subscribe to events\n\n    // ...\n  }\n\n  void HandleEvent(SDL_Event\u0026 event) override {\n      // handle SDL events\n  }\n\n  void OnUpdate(Core::Window\u0026 window, Core::Renderer\u0026 renderer) override {\n    // use systems to update what is necessary\n  }\n\n  void OnRender(Core::Window\u0026 window, Core::Renderer\u0026 renderer) override {\n    // use systems to render what is necessary\n  }\n\n  void OnBeforeRender(Core::Window\u0026 window, Core::Renderer\u0026 renderer) override {\n  }\n\n  void OnAfterRender(Core::Window\u0026 window, Core::Renderer\u0026 renderer) override {\n    // garbage collection for GUI\n  }\n\n  ~ECSStrategy() {\n    // destroy what is necessary\n  }\n};\n\n```\n\n### Registry\n\nSingleton class for system and entity component management.\n\n- `ECS::Registry::Instance().CreateEntity()` - creates a new entity\n- `ECS::Registry::Instance().KillEntity(entity)` - destroys an entity\n- `ECS::Registry::Instance().AddComponent\u003cTComponent\u003e(entity, ...args)` - adds a component to an entity\n- `ECS::Registry::Instance().GetComponent\u003cTComponent\u003e(entity)` - gets a component from an entity\n- `ECS::Registry::Instance().HasComponent\u003cTComponent\u003e(entity)` - checks if an entity has a component\n- `ECS::Registry::Instance().GroupEntity(entity, group)` - assigns an entity to a group\n- `ECS::Registry::Instance().GetEntityGroups(group)` - gets all entities in a group\n- `ECS::Registry::Instance().TagEntity(entity, tag)` - assigns an entity to a tag\n- `ECS::Registry::Instance().GetEntityByTag(tag)` - gets an entity by tag\n\nExample:\n\n\u003cdetails\u003e\n  \u003csummary\u003eC++\u003c/summary\u003e\n\n```c++\n// Create entity\nauto entity = ECS::Registry::Instance().CreateEntity();\n\n// Make entity components editable in GUI for easier debugging\nECS::Registry::Instance().AddComponent\u003cEditableComponent\u003e(entity);\n\n// Show text label on screen\nECS::Registry::Instance().AddComponent\u003cTextLabelComponent\u003e(\n  entity,\n  \"Hello World\",\n  App::Position(50, 50), // has its own position\n  App::Color(0, 255, 255) // has its own color\n);\n\n// Show bounding boxes around the component\nECS::Registry::Instance().AddComponent\u003cBoundingBoxComponent\u003e(\n  entity,\n  App::Position(50, 50),  // has its own position\n  App::Size(100, 100),\n  App::Color(0, 255, 255)\n);\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTypeScript\u003c/summary\u003e\n\n```TypeScript\n// Create entity\nconst entity = Registry.Instance().createEntity()\n\n// Make entity components editable in GUI for easier debugging\nentity.addComponentEditable()\n\n// Show text label on screen\nentity.addComponentTextLabel({\n    text: 'Hello World',\n    position: {x: 50, y: 50},\n    color: {r: 0, g: 255, b: 255},\n})\n\n// Show bounding boxes around the component\nentity.addComponentBoundingBox({\n    position: {x: 50, y: 50},\n    size: {width: 100, height: 100},\n    color: {r: 0, g: 255, b: 255},\n    thickness: 2,\n})\n```\n\n\u003c/details\u003e\n\n![preview-1](docs/preview-ecs-component.png)\n\n### Detection Systems\n\nThere are multiple ways to detect objects on the screen, such as OCR, OpenCV color detection and finding contours, or\nusing YOLOv5 object detection or instance segmentation.\n\nThe way detection works is that users create entities with detection components, and those entities try to find objects\non the screen and create new entities. For example, with the `flappy-color-detector` below, there will always be one\nentity present (the detector entity), and then there will be `n` entities present based on how many objects are\ndetected.\nObject tracking is not yet supported, and detected entities are created and destroyed every frame. ECS uses data\noriented design for fast operations. You can access entities by their group or tag.\n\n#### Flappy Bird Color Detection\n\n\u003cdetails\u003e\n  \u003csummary\u003eC++\u003c/summary\u003e\n\n```c++\n// Create entity\nauto entity = ECS::Registry::Instance().CreateEntity();\n\n// Make entity components editable in GUI for easier debugging\nECS::Registry::Instance().AddComponent\u003cEditableComponent\u003e(entity);\n\n// Add detection component\nECS::Registry::Instance().AddComponent\u003cDetectionComponent\u003e(\n  entity,\n  DetectionComponent{\n    // operations are optional\n    // operations are applied in order\n    .operations =\n      {\n        // crop specific region of the screen\n        std::make_shared\u003cCropOperation\u003e(\n          App::Position(150, 0),\n          App::Size(170, 724)\n        ),\n        // detect colors\n        std::make_shared\u003cDetectColorsOperation\u003e(\n          App::Color(0, 0, 30),\n          App::Color(255, 85, 255)\n        ),\n        // apply other morphological transformations https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html \n        std::make_shared\u003cCloseOperation\u003e(App::Size(14, 14))}\n      }\n);\n\n// Add detect contours component for DetectContours system\nECS::Registry::Instance().AddComponent\u003cDetectContoursComponent\u003e(\n  entity,\n  \"flappy-color-detector\", // unique ID so that we can have multiple detectors\n  App::Size(20, 20), // minimal contour size\n  App::Color(255, 0, 0), // bounding box color\n  true, // show contours preview in the window stream\n  App::Size(80, 80) // maximum contour size\n);\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTypeScript\u003c/summary\u003e\n\n  ```TypeScript\n// Create entity\nconst flappyDetector = Registry.Instance().createEntity()\n\nflappyDetector.setTag('flappy-detector')\n\n// Make entity components editable in GUI for easier debugging\nflappyDetector.addComponentEditable()\n\n// Add detection component\nflappyDetector.addComponentDetection()\n\n// operations are optional\n// operations are applied in order\n\n// crop specific region of the screen\nflappyDetector.addComponentDetectionCropOperation({\n    position: {x: 150, y: 0},\n    size: {width: 170, height: 724},\n})\n// detect colors\nflappyDetector.addComponentDetectionColorsOperation({\n    lowerBound: {r: 0, g: 0, b: 30},\n    upperBound: {r: 255, g: 85, b: 255},\n})\n// apply other morphological transformations https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html\nflappyDetector.addComponentDetectionMorphologyOperation('close', {\n    size: {width: 14, height: 14},\n})\n\n// Add detect contours component for DetectContours system\nflappyDetector.addComponentDetectContours({\n    id: 'flappy',\n    minArea: {width: 20, height: 20},\n    maxArea: {width: 80, height: 80},\n    bboxColor: {r: 255, g: 0, b: 0},\n    shouldRenderPreview: false\n})\n  ```\n\n\u003c/details\u003e\n\n![color-detection-1](docs/color-detection-1.png)\n![color-detection-2](docs/color-detection-2.png)\n\n#### Flappy Bird YOLO Object Detection\n\n\u003cdetails\u003e\n  \u003csummary\u003eC++\u003c/summary\u003e\n\n```c++\n// Create Entity\nauto entity = ECS::Registry::Instance().CreateEntity();\n\n// AddDetection component\nECS::Registry::Instance().AddComponent\u003cDetectionComponent\u003e(entity);\n\n// Configure YOLO detector\nECS::Registry::Instance().AddComponent\u003cDetectObjectsComponent\u003e(\n    entity,\n    \"object-detector\",\n    0.9,\n    0.5,\n    \"absolute-path-to-model\",\n    \"absolute-path-to-class-names-file\"\n);\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n  \u003csummary\u003eTypeScript\u003c/summary\u003e\n\n  ```TypeScript\n// Create Entity\nconst objectDetector = Registry.Instance().createEntity()\n\n// AddDetection component \nobjectDetector.addComponentDetection()\n\n// Configure YOLO detector\nobjectDetector.addComponentDetectObjects({\n    id: 'object-detector',\n    confidenceThreshold: 0.9,\n    nonMaximumSuppressionThreshold: 0.5,\n    pathToModel:\n        'absolute-path-to-model',\n    pathToClasses:\n        'absolute-path-to-class-names-file',\n})\n  ```\n\n\u003c/details\u003e\n\n![object-detection-1](docs/object-detection-1.png)\n\n#### Flappy Bird YOLO Instance Segmentation\n\nIt is also possible to load ONNX models to run instance segmentation this allows us to do precise automation logic or more complex image operations like object inpainting (PoC)\n\n![object-detection-1](docs/inpaint-1.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper239%2Fautomation-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeveloper239%2Fautomation-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeveloper239%2Fautomation-engine/lists"}