{"id":13563293,"url":"https://github.com/MontFerret/worker","last_synced_at":"2025-04-03T19:32:47.107Z","repository":{"id":48404137,"uuid":"262316390","full_name":"MontFerret/worker","owner":"MontFerret","description":"Containerized Ferret worker","archived":false,"fork":false,"pushed_at":"2023-03-29T01:59:10.000Z","size":1757,"stargazers_count":14,"open_issues_count":12,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T18:49:28.110Z","etag":null,"topics":["chrome","crawler","docker","dsl","ferret","go","hacktoberfest","hacktoberfest2020","scraping","scraping-websites","service","worker"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MontFerret.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-08T12:22:00.000Z","updated_at":"2024-08-31T09:23:01.000Z","dependencies_parsed_at":"2024-01-23T21:20:45.773Z","dependency_job_id":"b0c13832-e28e-42d4-8fa1-95094e3e924e","html_url":"https://github.com/MontFerret/worker","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MontFerret%2Fworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MontFerret%2Fworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MontFerret%2Fworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MontFerret%2Fworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MontFerret","download_url":"https://codeload.github.com/MontFerret/worker/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247065470,"owners_count":20877785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome","crawler","docker","dsl","ferret","go","hacktoberfest","hacktoberfest2020","scraping","scraping-websites","service","worker"],"created_at":"2024-08-01T13:01:17.446Z","updated_at":"2025-04-03T19:32:45.983Z","avatar_url":"https://github.com/MontFerret.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Worker\n\n\u003cp align=\"center\"\u003e\n\t\u003ca href=\"https://goreportcard.com/report/github.com/MontFerret/worker\"\u003e\n\t\t\u003cimg alt=\"Go Report Status\" src=\"https://goreportcard.com/badge/github.com/MontFerret/worker\"\u003e\n\t\u003c/a\u003e\n\u003c!-- \t\u003ca href=\"https://codecov.io/gh/MontFerret/worker\"\u003e\n\t\t\u003cimg alt=\"Code coverage\" src=\"https://codecov.io/gh/MontFerret/worker/branch/master/graph/badge.svg\" /\u003e\n\t\u003c/a\u003e --\u003e\n\t\u003ca href=\"https://discord.gg/kzet32U\"\u003e\n\t\t\u003cimg alt=\"Discord Chat\" src=\"https://img.shields.io/discord/501533080880676864.svg\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://github.com/MontFerret/worker/releases\"\u003e\n\t\t\u003cimg alt=\"Lab release\" src=\"https://img.shields.io/github/release/MontFerret/worker.svg\"\u003e\n\t\u003c/a\u003e\n\t\u003ca href=\"https://opensource.org/licenses/Apache-2.0\"\u003e\n\t\t\u003cimg alt=\"Apache-2.0 License\" src=\"http://img.shields.io/badge/license-Apache-brightgreen.svg\"\u003e\n\t\u003c/a\u003e\n\u003c/p\u003e\n\n**Worker** is a simple HTTP server that accepts FQL queries, executes them and returns their results.\nOpenAPI v2 schema can be found [here](https://raw.githubusercontent.com/MontFerret/cli/master/reference/ferret-worker.yaml).\n\n## Quick start\n\nThe Worker is shipped with dedicated Docker image that contains headless Google Chrome, so feel free to run queries using `cdp` driver:\n\nDockerHub\n```sh\ndocker run -d -p 8080:8080 montferret/worker\n```\nGitHub\n```sh\ndocker run -d -p 8080:8080 ghcr.io/montferret/worker\n```\n\nAlternatively, if you want to use your own version of Chrome, you can run the Worker locally.\n\nBy installing the binary:\n\n```shell\ncurl https://raw.githubusercontent.com/MontFerret/worker/master/install.sh | sh\nworker\n```\n\nOr by building locally:\n\n```sh\nmake\n```\n\nAnd then just make a POST request:\n\n![worker](https://raw.githubusercontent.com/MontFerret/worker/master/assets/postman.png)\n\n## System Resource Requirements\n- 2 CPU\n- 2 Gb of RAM\n\n## Usage\n\n### Endpoints\n\n#### POST /\nExecutes a given query. The payload must have the following shape:\n\n```\nQuery {\n    text: String!\n    params: Map\u003cstring, any\u003e\n}\n```\n\n#### GET /info\nReturns a worker information that contains details about Chrome, Ferret and itself. Has the following shape:\n\n```\nInfo {\n    ip: String!\n    version: Version! {\n        worker: String!\n        chrome: ChromeVersion! {\n            browser: String!\n            protocol: String!\n            v8: String!\n            webkit: String!\n        }\n        ferret: String!\n    }\n}\n```\n\n\n#### GET /health\nHealth check endpoint (for Kubernetes, e.g.). Returns empty 200.\n\n### Run commands\n\n```bash\n  -log-level=\"debug\"\n    log level\n  -port=8080\n    port to listen\n  -body-limit=1000\n    maximum size of request body in kb. 0 means no limit.\n  -request-limit=20\n    amount of requests per second for each IP. 0 means no limit.\n  -request-limit-time-window=180\n    amount of seconds for request rate limit time window.\n  -cache-size=100\n    amount of cached queries. 0 means no caching.\n  -chrome-ip=\"127.0.0.1\"\n    Google Chrome remote IP address\n  -chrome-port=9222\n    Google Chrome remote debugging port\n  -no-chrome=false\n    disable Chrome driver\n  -version=false\n    show version\n  -help=false\n    show this list\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMontFerret%2Fworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMontFerret%2Fworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMontFerret%2Fworker/lists"}