{"id":23689133,"url":"https://github.com/skykery/urlfetcher","last_synced_at":"2026-01-12T19:30:18.271Z","repository":{"id":268518619,"uuid":"536720432","full_name":"skykery/urlfetcher","owner":"skykery","description":"Microservice for HTTP requests with Tor proxies that can handle HTML parsing","archived":false,"fork":false,"pushed_at":"2022-10-23T14:22:40.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-12-30T00:34:47.395Z","etag":null,"topics":["lxml","proxies","python","requests","tor"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skykery.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-14T18:56:49.000Z","updated_at":"2023-03-10T20:05:23.000Z","dependencies_parsed_at":"2024-12-17T09:28:05.604Z","dependency_job_id":"7ed4aa2a-e50f-4069-8611-2e33135f3da2","html_url":"https://github.com/skykery/urlfetcher","commit_stats":null,"previous_names":["skykery/urlfetcher"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skykery%2Furlfetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skykery%2Furlfetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skykery%2Furlfetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skykery%2Furlfetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skykery","download_url":"https://codeload.github.com/skykery/urlfetcher/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239748914,"owners_count":19690353,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lxml","proxies","python","requests","tor"],"created_at":"2024-12-30T00:35:41.540Z","updated_at":"2026-01-12T19:30:18.216Z","avatar_url":"https://github.com/skykery.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# URLFetcher\nA microservice that can handle HTTP requests and can serve parsed elements using CSS Selectors and XPath.\nThe microservice is running on FastAPI as web framework, relies on LXML for the HTML parsing part and is successfully using Tor nodes and proxies to hide the initial request.\n\nTo run it locally\n`docker-compose -f stack.yaml up -d`\n\nI published a project [URLWorker](https://urlworker.techwetrust.com/) which is based on this microservice and you can try it yourself, for free of course.\n\n### Supports:\n - Retries\n - Free proxies by default\n - Custom proxy\n - CSS selectors\n - JavaScript rendering\n\nYou can make a free account on [https://urlworker.techwetrust.com/](https://urlworker.techwetrust.com/) or watch my tutorials on how to use it right on the [Examples](https://urlworker.techwetrust.com/examples/) section.\n\nDon't forget to star the repo for further updates.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskykery%2Furlfetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskykery%2Furlfetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskykery%2Furlfetcher/lists"}