{"id":23630692,"url":"https://github.com/marqueewinq/shooter","last_synced_at":"2026-05-10T02:22:41.554Z","repository":{"id":269809287,"uuid":"902262852","full_name":"marqueewinq/shooter","owner":"marqueewinq","description":"Shooter: full-page screenshot tool","archived":false,"fork":false,"pushed_at":"2024-12-26T09:44:15.000Z","size":164,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-18T21:35:35.286Z","etag":null,"topics":["celery","fastapi","python3","selenium","selenium-python","selenium-webdriver"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marqueewinq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-12T08:21:59.000Z","updated_at":"2024-12-26T09:44:18.000Z","dependencies_parsed_at":"2024-12-26T10:37:34.064Z","dependency_job_id":null,"html_url":"https://github.com/marqueewinq/shooter","commit_stats":null,"previous_names":["marqueewinq/shooter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marqueewinq%2Fshooter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marqueewinq%2Fshooter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marqueewinq%2Fshooter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marqueewinq%2Fshooter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marqueewinq","download_url":"https://codeload.github.com/marqueewinq/shooter/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marqueewinq%2Fshooter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259198000,"owners_count":22820153,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["celery","fastapi","python3","selenium","selenium-python","selenium-webdriver"],"created_at":"2024-12-28T02:37:38.531Z","updated_at":"2026-05-10T02:22:36.505Z","avatar_url":"https://github.com/marqueewinq.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Shooter: full-page screenshot tool\n\nMain use case of this applications is:\n\n - given a URL, take a full-page/partial screenshot;\n - detect all key elements on the page and label the resulting image with the found elements;\n - follow scripted user actions along the way.\n\nREST API server runs all screenshot tasks in background.\n\n## Docs\n\nEach use case can be done through REST API or CLI.\n\nREST API documentation is accessible by the `/docs` endpoint.\n\nCLI documentation can be read here: `python -m shooter --help`.\n\n### Architecture\n\nThe User sends HTTP request to API server. API server sends task(-s) to Task Broker. Several Workers are subscribed to\nthe Task Broker and pick up the tasks. Each Worker calls it's own `make_screenshot` function. The User can invoke\n`make_screenshot` function by themselves; the HTTP request schema matches the function signature:\n\n\u003cimg src=\"./docs/diagrams/arch.png\"/\u003e\n\nEach request contains one or more `TakeScreenshotConfig` objects (\"configs\"). Each config specifies one task, which will\nbe picked up by worker, and one invocation of the `make_screenshot` function. The config specifies the target URL and\nseveral other parameters, including full page screenshot option, browser \u0026 device selection, whether to additionally\ncapture HTML elements on the page, waiting options, proxy settings etc.\n\n\u003cimg src=\"./docs/diagrams/request.png\"/\u003e\n\nEach task will be put to the broker's queue separately (so different workers will pick up the tasks independently). At\nthe same time, all those tasks will form a _task group_. The task group is considered complete when all it's member\ntasks are complete; task group is considered failed if at least one task failed (other tasks will still be run and\npartial results will still be available).\n\n\u003cimg src=\"./docs/diagrams/group_task.png\"/\u003e\n\nThe result of each task is two groups of artifacts. Given `--output_path`, the function creates them in the given\nfolder. The actual results are: screenshot image, detected HTML elements (as JSON) and same screenshot\nimage labelled with those elements. The debug artifacts are: initial config (as JSON) and log file (as TXT).\n\nDepending on the given parameters (`--capture_visible_elements` and `--capture_invisible_elements`), elements JSON and\nthe labelled image may be absent.\n\n\u003cimg src=\"./docs/diagrams/output.png\"/\u003e\n\nThe user can queue status of his HTTP request (which corresponds to the status of the task group): number of total\ntasks, proceeded, failed and the overall status of the task group. When the task group is finished, user may request\nthe zip file; each task artifacts will be in the separate folder.\n\nHere's the output zip structure:\n\n```\n\u003ctask group UUID\u003e.zip\n└── \u003cURL\u003e__\u003cbrowser\u003e__\u003cfullpage?\u003e__\u003cconfig hash\u003e\n│   ├── config.json\n│   ├── elements.json\n│   ├── log.txt\n│   ├── screenshot.labelled.png\n│   └── screenshot.png\n└── \u003cURL\u003e__\u003cbrowser\u003e__\u003cfullpage?\u003e__\u003cconfig hash\u003e\n│   ...\n```\n\nThe structure of `elements.json` is the following:\n\n```\nelement = {\n    \"id\": int,                     # Unique identifier\n    \"parent_id\": int | null,       # ID of the parent element, or null if it's root element\n    \"bbox\": [int, int, int, int],  # Bounding box as [x, y, width, height]\n    \"tag_name\": string,            # HTML tag name (e.g., \"div\", \"span\")\n    \"label\": string,               # Element's label or role description\n    \"position\": string,            # Element's position (e.g., \"absolute\", \"relative\")\n    \"is_visible\": boolean,         # Visibility status (true/false)\n    \"css_selector\": string         # Unique CSS selector for the element\n}\n\nelements = [element, ...]  # List of elements\n```\n\n## Usage\n\nUse case: based on the provided config, make a screenshot of the site.\n\nExample (REST API):\n\n```bash\ncurl -s -XPOST \"http://$HOST:$PORT/take_screenshots/\" \\\n     -H \"Content-Type: application/json\" \\\n     -d '{\n           \"sites\": [\n              \"https://www.amazon.ae/\"\n           ],\n           \"default_config\": {\n             \"device\": \"IPHONE_X\",\n             \"browser\": \"chrome\",\n             \"actions\": [{\"kind\": \"click_element\", \"element_query_selector\": \"#nav-hamburger-menu i\"}]\n           }\n         }\n```\n\nExample (CLI):\n\n```bash\npython -m shooter https://www.amazon.ae/ ./output/ \\\n  --device=IPHONE_X \\\n  --browser=chrome \\\n  --actions='[{\"kind\": \"click_element\", \"element_query_selector\": \"#nav-hamburger-menu i\"}]'\n```\n\nIn both of these examples the tool will make a screenshot of Amazon Store page using Chrome browser with IPhone X\nparameters clicking the side menu before the capture.\n\nThe REST API endpoint will return the group task id, which then can be used to query the progress of the task group:\n\n - `/take_screenshots/{group_result_id}` will return the task group progress\n - `/take_screenshots/{group_result_id}/zip` will download the task group results as a single zip with the subfolder for\n   each requested site.\n\n#### Proxy usage examples\n\nIn REST API, `proxy` parameter can be passed in the following ways:\n\n - `None` (default) to not use the proxy\n - one instance of `ProxyConfig` objects\n - list of those objects (or `None`-s).\n\nScript will try to connect with the first proxy on the list; if unsuccessful, it will try other proxies in the given\n order until success or the list is exhausted (this will raise `NoDriverRemainingError`).\n\nProxy config is the dict in the following form:\n\n```json\n{\n  \"host\": \"proxyserver.com\",\n  \"port\": 8989,\n  \"username\": \"proxyuser\",\n  \"password\": \"imfirinmylaser\",\n  \"protocol\": \"https\"\n}\n```\n\n`protocol` is optional (defaults to `https`), all other fields are required.\n\nThis `ProxyConfig` will be transformed to _connection string_ and passed to CLI as such.\n\nIn CLI, `--proxy` parameter can also be passed in these three ways. Instead of `ProxyConfig` object, CLI version uses\n _connection strings_. This is a handy way to pass all proxy parameters in one string.\n\nExample of the connection string of the proxy config above:\n\n```bash\npython -m shooter $URL $OUTPUT_DIR --proxy='https://proxyuser:imfirinmylaser@proxyserver.com:8989/'\n```\n\nThe list can also be passed:\n\n```bash\npython -m shooter $URL $OUTPUT_DIR \\\n  --proxy='[\"https://proxyuser:imfirinmylaser@proxyserver.com:8989/\",\\\n            \"https://proxyuser:imfirinmylaser@alternativeserver.com:8989/\"]'\n```\n\n## Development\n\n### Developer checklist\n\n - set up a virtualenv if needed: `virtualenv env --python=python3.12` and activate it\n - install pre-commit hooks: `pre-commit install`\n   - if needed, install `pre-commit` first: `pip install pre-commit`\n\n### Webdriver versioning\n\nGoogle publishes their newer packages into Ubuntu Apt, while removing the old ones. If you get a build-time error with\n\"Version '125.0.6422.*' for 'google-chrome-stable' was not found\", try the following:\n\n```\n$ apt search google-chrome-stable\ngoogle-chrome-stable/stable 131.0.6778.204-1 amd64 [upgradable from: 125.0.6422.112-1]\n  The web browser from Google\n```\n\nThe newer `131.0.6778.204` version should be inserted into the Dockerfile in \"Install Chrome and ChromeDriver\" section.\n\n\n### Run the service with `docker-compose`\n\nYou can run a service in a docker container: `docker-compose --profile all up --build`\n\nFor concurrent execution, upscale the workers: `docker-compose --profile all up --build --scale worker=4`\n\nMonitoring of the concurrent execution can be accessed via Flower: http://localhost:5555/\n\nDescription of the docker-compose profiles:\n\n - `all` profile runs the whole stack (excluding container with e2e tests) -- this is the go-to option for deployment.\n - `api` profile only deploys the REST API service (no tasks will be executed).\n - `e2e` profile runs the whole stack + the container with e2e tests (in `test.sh`).\n\n#### `docker-compose` configuration\n\nThe results are saved to the directory specified by `OUTPUT_PATH` envvar; mount that to a volume to export.\n\nAsync task broker and result backend can be specified with `CELERY_BROKER_URL` and `CELERY_BACKEND_URL` envvars.\n\nContainer `e2e_tests` runs the `test.sh` script on start-up.\n\n### Run the service locally\n\nAlternatively, you can run the REST API from your machine:\n\n - install requirements: `pip install -e .`\n - run the service: `make run`\n\nIf you don't need the REST API server, you can run the screenshot script directly. Parameters are described here:\n `python -m shooter --help`.\n\n### Testing\n\nTo run the unit tests locally:\n\n - install requirements: `pip install -e .[test]`\n - run pytest: `make unit_test`\n\nUnit test coverage is saved in `./htmlcov` folder (open `./htmlcov/index.html` to view).\n\n`docker-compose --profile e2e up --build --scale worker=4` runs end-to-end tests with 4 workers in parallel. See\n`./test.sh` for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarqueewinq%2Fshooter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarqueewinq%2Fshooter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarqueewinq%2Fshooter/lists"}