Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/jacopofar/virtualbox-helper
Python library to interact with a Virtualbox instance using simple computer vision
https://github.com/jacopofar/virtualbox-helper
ui-automation virtualbox
Last synced: 21 days ago
JSON representation
Python library to interact with a Virtualbox instance using simple computer vision
- Host: GitHub
- URL: https://github.com/jacopofar/virtualbox-helper
- Owner: jacopofar
- License: mit
- Created: 2020-10-26T19:06:24.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2020-10-27T09:27:48.000Z (about 4 years ago)
- Last Synced: 2024-11-29T09:49:34.768Z (23 days ago)
- Topics: ui-automation, virtualbox
- Language: Python
- Homepage:
- Size: 157 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Virtualbox helper
This library helps to automate UI tasks on a Virtualbox machine using OpenCV.
It's based on `remotevbox` to interact with the SOAP service and `OpenCV` to
perform region detection.
__Features__:
* can start the `vboxwebsrv` SOAP server programmatically and shut it down
* match a region of the screen with a given fragment
* wait for a fragment and click on its center
* store if required a representation of the area it matched
* accepts images as NumPy ndarray, raw bytes of filesystem paths
## Installation
python3 -m pip install virtualbox_helper
## Example usage
```Python 3
from virtualbox_helper import (
ensure_server_running,
get_machine,
wait_for_fragment,
wait_click_on_fragment,
)
# this starts the server and stops it when the interpreter exits
# or you can use the command vboxwebsrv manually
ensure_server_running()
machine = get_machine('vbox', 'yourpassphrase', 'Debian testing')
machine.launch()
no_match = detect_fragment(screenshot_data, 'some/random_image.png')
assert no_match is None
detection = detect_fragment(screenshot_data, 'valid_element.png')
assert detection is not None
# similarity score
assert detection[0] > 0.9
# coordinates of the bounding box that was detected
assert detection[1] == (19, 59)
assert detection[2] == (172, 114)
# raw bytes or a numpy (X, Y, 3) matrix can be used instead of a file path
button_data = open('button.png', 'rb').read()
target = wait_click_on_fragment(machine, button_data, timeout=60.0)
# target has the same structure of match above
wait_for_fragment(machine, 'some/element/to/wait_for.png')
with open('screenshot.png', 'wb') as f:
screenshot_data = machine.take_screenshot_to_bytes()
f.write(screenshot_data)
# this functionality is part of remotevbox
machine.send_character_string('hello world')
machine.send_key_combination([''])
# this returns immediately, None if not found
# store_match indicates where to store the screenshot with the match
# highlighted as a red rectangle
detection = detect_fragment(
machine.take_screenshot_to_bytes(),
'expected_element.png',
store_match='highlight_match.png'
)
machine.poweroff()
```
## When shoulkd I use this?
Hopefully never! If you have a proper API for what you want to automate,
go with that.
To automate UI operations look into `PyAutoGUI`, and `Selenium` or `Puppeteer`
for the web.
However, if you can't do that, for example if you are writing tests for
TempleOS or something weird, then this may be fine.
## Tests
The tests are based on pytest, can be run using `poetry run test`.
They expect a machine called "Debian testing" running Debian 11.
Should be trivial to adapt to some other image.