{"id":23130003,"url":"https://github.com/tetreum/xupopter_runner","last_synced_at":"2025-08-08T06:17:33.887Z","repository":{"id":146311426,"uuid":"595274210","full_name":"tetreum/xupopter_runner","owner":"tetreum","description":"Executes crawling recipes coming from Xupopter Chrome Extension.","archived":false,"fork":false,"pushed_at":"2024-02-21T14:31:13.000Z","size":113,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T17:46:01.152Z","etag":null,"topics":["crawler","scrapper","scrapping","webscraper"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tetreum.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-30T18:49:58.000Z","updated_at":"2024-02-08T07:33:55.000Z","dependencies_parsed_at":"2024-12-17T10:11:59.248Z","dependency_job_id":"c22bf466-db5d-490b-8ba7-15d56659832a","html_url":"https://github.com/tetreum/xupopter_runner","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tetreum%2Fxupopter_runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tetreum%2Fxupopter_runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tetreum%2Fxupopter_runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tetreum%2Fxupopter_runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tetreum","download_url":"https://codeload.github.com/tetreum/xupopter_runner/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135125,"owners_count":20889420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","scrapper","scrapping","webscraper"],"created_at":"2024-12-17T10:11:47.485Z","updated_at":"2025-04-04T06:41:19.192Z","avatar_url":"https://github.com/tetreum.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Xupopter Runner\nExecutes crawling recipes coming from [Xupopter Chrome Extension](https://github.com/tetreum/xupopter_chrome_extension).\n\n\n## Usage\n\nYou can either use the docker container (recommended as contains both the backend and a runner) or manually run it.\n\n### Docker\n```\nversion: \"3.3\"\nservices:\n  xupopter-runner:\n    image: ghcr.io/tetreum/xupopter_runner:latest\n    container_name: xupopter-runner\n    ports:\n      - 8089:8089\n    environment:\n      - JWT_ACCESS_SECRET=SAME_SECRET_AS_XUPOPTER_CLIENT # Write the same secret that xupopter client .env has\n    volumes:\n      - /path/to/config:/app/config # Make sure your local config directory exists\n      - /where/i/want/to/store/scrapped_data:/app/public # Make sure your local config directory exists\n```\n\nThe runner will be available at `http://localhost:8089`\n\n### Non-docker\n\n`JWT_ACCESS_SECRET=test npm start`\n\nThe runner will be available at `http://localhost:8089`\n\nRequest sample:\n```curl\ncurl --request POST \\\n  --url http://localhost:8089/ \\\n  --header 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJPbmxpbmUgSldUIEJ1aWxkZXIiLCJpYXQiOjE3MDEzNDk3NzUsImV4cCI6MTczMjg4NTc3NSwiYXVkIjoid3d3LmV4YW1wbGUuY29tIiwic3ViIjoianJvY2tldEBleGFtcGxlLmNvbSIsIkdpdmVuTmFtZSI6IkpvaG5ueSIsIlN1cm5hbWUiOiJSb2NrZXQiLCJFbWFpbCI6Impyb2NrZXRAZXhhbXBsZS5jb20iLCJSb2xlIjpbIk1hbmFnZXIiLCJQcm9qZWN0IEFkbWluaXN0cmF0b3IiXX0.Ah4sSyoF1QUD65RyMCRjYKta9dOWdEEyCNvd00CqBzM' \\\n  --header 'Content-Type: application/json'\n  --data '{\n\t    \"id\": \"home-crawler\",\n\t\t\"recipe\": {\n\t\t\t\"id\": \"a57ddd92-f32f-4bab-98d5-747a7193d924\",\n\t\t\t\"name\": \"Local test\",\n\t\t\t\"expected_output\": \"item\",\n\t\t\t\"schema\": 1,\n\t\t\t\"blocks\": [\n\t\t\t\t{\n\t\t\t\t\t\"id\": \"f3b85729-b967-4a3a-8cb4-f5c8d465be34\",\n\t\t\t\t\t\"type\": \"start\",\n\t\t\t\t\t\"details\": {\n\t\t\t\t\t\t\"type\": \"url\",\n\t\t\t\t\t\t\"source\": \"https://localhost:8080/\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"id\": \"40a9215d-7888-4ae1-b192-a2ae9ce21097\",\n\t\t\t\t\t\"type\": \"extract\",\n\t\t\t\t\t\"details\": {\n\t\t\t\t\t\t\"name\": \"title\",\n\t\t\t\t\t\t\"selector\": \"#results-container [class=\\\"item row border-bottom p-2\\\"] h3\",\n\t\t\t\t\t\t\"property\": \"text\"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t]\n\t\t}\n}'\n```\n\nRunner results can be downloaded by visiting the following url: `http://localhost:8089/public/SENT_ID/result.json`\nEx: `http://localhost:8089/public/home-crawler/result.json`\nA debugging log will also be available for each run: `http://localhost:8089/public/home-crawler/info.log`  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftetreum%2Fxupopter_runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftetreum%2Fxupopter_runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftetreum%2Fxupopter_runner/lists"}