Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sanchai20/desktop-automationtool-manual
JeeJee Automation tool user manual
https://github.com/sanchai20/desktop-automationtool-manual
Last synced: about 20 hours ago
JSON representation
JeeJee Automation tool user manual
- Host: GitHub
- URL: https://github.com/sanchai20/desktop-automationtool-manual
- Owner: SanChai20
- Created: 2023-11-04T13:16:58.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-24T10:57:42.000Z (5 months ago)
- Last Synced: 2024-06-24T12:17:23.899Z (5 months ago)
- Size: 2.83 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# desktop-automationtool-manual
GO TO [desktop-automationtool](https://github.com/SanChai20/desktop-automationtool) PAGE
# Jee Jee Automation Tool User Manual
> Jee-Jee can be used to capture any desktop window, and then perform simple operations according to the given instruction queue sequence, which can save labor costs and is suitable for scenes with high repetition and high mechanical nature.
## Ⅰ. Operation Guide
### ①. Installation
1. Double click `JeeJee Installer` to install desktop application. After the installation is completed, start the application by desktop shortcut or `Jee Jee` in the installation directory .
### ②. Window Capture
1. Please **click the green icon within the red box in the screen-shot below**
[](#)
2. Then **move the mouse cursor to the desktop window to be captured** . The window to be captured will be covered with a light green area to remind you which window is selected.
3. After confirmation, **please click the left mouse button** anywhere on the target window to capture. After the capture is successful, the capture icon mentioned in step 1 will be displayed in red.
### ③. Configure instructions
1. Then look at the middle part of the main interface and you will find an empty command.
2. ***Configure the first command***
- Click red box to select matching image which has been captured and saved in advance
[](#)
- Click red box to select mouse click event
[](#)
- Click red box to select and save click location on image
[](#)
- Click red box to give suitable time in seconds
[](#)
3. Click [](#) to add new command and repeat step 2 to configure all rest commands
### ④. Window operation
1. Then click [](#) the button at the bottom of the main interface to start running. You can click the button [](#) to turn on the debugging mode before starting , and the command execution will be visualized on the target window.
### ⑤. Export command file
1. In order to save time for the next time using, we can export the command list after configuring it by clicking [](#). The export file format is `.command` and this file can be imported in other capture windows by clicking [](#)
## Ⅱ. Function introduction
> The following will introduce each editor interface module one by one.
### ①. Main interface partition
[
](#)
- As shown in the figure above, the main interface can be divided into three parts according to functions.
- Global Control Area, GCA (red box above)
> Mainly used for control `Window Captured` and global configuration
- Visualization Area, VA (orange box above)
> The visual ribbon contains `Printing Log`, `Thumbnail of Captured Window`and`Funtionality Button Area`
>
> **Each capture window in this area is independent of each other**
- Command Logic Area, CLA (blue box above)
> Configure `Command List` `Import/Export Command File` `Running Logic`
>
> **Each capture window in this area is independent of each other**
### ②. Detailed explanation of icons
> All icons will be explained below. You can refer here when you encounter them.
| `Icon` | `Explanation` | `Area` |
| :------------------------------------------: | :----------------------------------------------------------- | :----: |
| [](#) | This icon appears to the left of the capture window title and is used to distinguish between different windows | GCA |
| [](#) | This icon appears on the far right of each capture window. Click to close this window. This function is not available only when there is only one window remaining or the current window is running. | GCA |
| [](#) | This icon appears first when opening the drop-down menu on the right side of the GCA and is used to add a new capture window | GCA |
| [](#) | This icon appears in the second position when opening the drop-down menu on the right side of the GCA. It is used to open a pop-up window displaying developer contact information. | GCA |
| [](#) | This icon appears in the third position when opening the drop-down menu on the right side of the GCA. It is used to open the pop-up window for controlling global settings. | GCA |
| [](#) | This icon appears in the pop-up window that appears after clicking [](#). Click to copy the current developer's contact information. | GCA |
| [](#) | This icon appears on the left side of [](#) and is used to clear log information | VA |
| [](#) | This icon indicates that the debugging window has been opened. Click to close the debugging window. | VA |
| [](#) | This icon indicates that the debugging window has been closed. Click to open the debugging window. | VA |
| [](#) | This icon appears in the lower right corner of the VA and is used to capture the window. When it is green, it means it is to be captured, and when it is red, it means it has been captured successfully. | VA |
| [](#) | This icon appears at the end of each command and is used to insert a new command below the current command. | CLA |
| [](#) | This icon appears on the left side of each command. Drag this icon up or down to change the position of the command in the queue. | CLA |
| [](#) | This icon will appear in both the selected image click position and the debugging interface position display, used to visualize the recorded mouse click position. | CLA |
| [](#) | This icon appears on the left side of [](#) and is used to delete the current instruction. | CLA |
| [](#) | This icon appears at the bottom of the CLA. Click to select the instruction list file that needs to be imported. | CLA |
| [](#) | This icon appears on the right side of [](#). Click to export the command list of the current window to the local command file. | CLA |
| [](#) | This icon appears at the bottom of the command logic ribbon. Click to run the command task of the current capture window. | CLA |
| [](#) | When this icon appears, it indicates that it has entered the command running state. Click to stop running the command task of the current capture window. | CLA |
| [](#) | This icon will appear in position of [](#) when running a command to show which command is currently being processed | CLA |
### ③. Global configuration instructions
> **If there is no special need, just keep this global configuration as default**.
- `Priority`
> - Custom
>
> > `Matching Algorithm` `Detecting Algorithm` `kNN Threshold` can be adjusted freely
>
> - Accuracy fisrt
>
> > Priority is given to ensuring accuracy, but performance and speed will be sacrificed. `Matching Algorithm` will be fixed as `BRUTE_COMMON`, `Detecting Algorithm`will be fixed as `SIFT`
>
> - Speed priority
>
> > Prioritizes running efficiency, but will be accompanied by problems such as incorrect matching or lack of matching. `Matching Algorithm` will be fixed as `FLANNBASED_KNN`, `Detecting Algorithm` will be fixed as `SURF`, `kNN Threshold `will be fixed as `0.7`
>
> - False positive rate priority
>
> > Try to keep the number of false matches to a minimum and ensure reasonable operating efficiency. However, there is a possibility that the result cannot be matched. `Matching Algorithm` will be fixed as `BRUTE_KNN`, `Detecting Algorithm` will be fixed as `SIFT`, `kNN Threshold` will be fixed as `0.15`
- `Matching Algorithm`
Only used on feature matching. Template matching has nothing to do with this type, see below `Points to note` for details
> - BRUTE_KNN
>
> - BRUTE_COMMON
>
> > The Brute method will perform violent matching, which is too time-consuming, but has higher accuracy. In scenarios where accuracy is higher than speed, the BRUTE method is preferred.
>
> - FLANNBASED_KNN
>
> - FLANNBASED_COMMON
>
> > The FLANNBASED method has obvious advantages in operating efficiency when the image to be matched is large. If you do not worry about the misjudgment rate of matching, you can use the FLANNBASED method first.
>
> > The KNN method is used to quickly sort out results with higher probability, and is better than the COMMON method under reasonable `kNN Threshold` given circumstances.
- `Detecting Algorithm`
> - SURF
>
> > Similar to the algorithmic `SIFT`, but much faster than SIFT.
>
> - SIFT
>
> > It is recommended to choose this algorithm when there is a brightness difference or size scaling between the image to be matched and the capture window image.
>
> - KAZE
>
> > It is recommended to choose this algorithm when there is a rotation difference or blur difference between the image to be matched and the capture window image and there is no obvious brightness change.
- `[kNN Threshold]`
> When selected `Matching Algorithm` is `BRUTE_KNN ` or `FLANNBASED_KNN`, this item will pop up for adjustment. Please adjust it within the range of 0.01 to 1.00. The smaller the value, the stricter the match, but at the same time, the probability of not matching the result will increase. The larger the value, the looser the match. But at the same time the probability of incorrect matching will increase.
- `Execution Strategy`
> The strategy that should be adopted when the target image is not matched after the instruction is executed.
>
> - stop command execution after detection failure
>
> > Stop execution when the target image is not matched. This is **the default configuration.**
>
> - execute current command in a loop after detection failure
>
> > If the target image is not matched, continue to execute the current instruction. The next instruction will not be executed until a reasonable result is matched.
>
> - execute next command after detection failure
>
> > If the target image is not matched, continue to execute the next instruction.
- `Language`
> Currently only supports English
## Ⅲ. Points to note
> The following points will help you avoid invalid operations as much as possible and improve the efficiency of using this application.
- Jee Jee uses the openCV library for image matching. The current program matching logic is performed `Template matching` first. If it does not pass, it will be performed `Feature matching` automatically. The performance overhead of these two methods is quite different. Try to give priority to template matching, that is, there is no rotation and scaling between the screenshot to be matched and the captured window image. If it cannot be avoided, you can adjust `Advanced Settings` to trade off time consumption and accuracy.
- The exported file will only record the path of the image to be matched. Therefore, when the image under the path is deleted or moved, the import failure will be prompted when the `.command` file is re-opened, and the thumbnail of the image to be matched in the command list will be black. **Please pay attention to this to avoid unnecessary loss**.
- Although there is no limits on the number of windows [](#) in the editor, the current version of the Jee Jee application will **occupy a high amount of CPU resources .** It is best not to exceed the number of **`6`** windows . Within this number, it will be allocated to different threads for parallel calculations. If it is exceeded, the overall operating efficiency will be reduced significantly.
- [](#) can help you find problems when trying to match the command list, and it will visually show whether the match is successful. If the match is not successful, then you may need to:
- Appropriately expand the interception range of the image to be matched (contain as many feature points as possible)
- You can adjust `Matching Algorithm` to `Brute`, `Detecting Algorithm` adjust to `SIFT`, or directly adjust `Priority` to `False positive rate priority`
- If you use `kNN` categories, you can increase the `kNN Threshold`.
## Ⅳ. Optimizations to do
> After testing, currently when Jee Jee is running with 6 capture windows at the same time, `Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz 2.90 GHz`the occupancy rate will be as high as over 90%.
- Move the feature matching process to the GPU
- Check if memory allocation is frequent
- CPU processing work adopts a rotation mechanism
- ~~Ask another developer :sweat_smile:~~
## Ⅴ. Contact the developer
> If you have any comments or suggestions, you can contact the developer by sending an email to [[email protected]](mailto:[email protected])