{"id":17995143,"url":"https://github.com/p0dalirius/crawlersuseragents","last_synced_at":"2025-09-03T16:33:00.394Z","repository":{"id":41381756,"uuid":"428308048","full_name":"p0dalirius/crawlersuseragents","owner":"p0dalirius","description":"Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.","archived":false,"fork":false,"pushed_at":"2023-10-01T18:33:36.000Z","size":309,"stargazers_count":20,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-18T18:50:12.621Z","etag":null,"topics":["bugbounty","crawler","crawlers","pentest","request","tool","user-agent","web"],"latest_commit_sha":null,"homepage":"https://podalirius.net/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/p0dalirius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"p0dalirius","patreon":"Podalirius"}},"created_at":"2021-11-15T14:59:54.000Z","updated_at":"2024-08-15T01:44:30.000Z","dependencies_parsed_at":"2024-10-29T20:44:29.191Z","dependency_job_id":null,"html_url":"https://github.com/p0dalirius/crawlersuseragents","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Fcrawlersuseragents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Fcrawlersuseragents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Fcrawlersuseragents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/p0dalirius%2Fcrawlersuseragents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/p0dalirius","download_url":"https://codeload.github.com/p0dalirius/crawlersuseragents/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231904486,"owners_count":18443685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bugbounty","crawler","crawlers","pentest","request","tool","user-agent","web"],"created_at":"2024-10-29T20:18:02.583Z","updated_at":"2024-12-30T19:26:55.844Z","avatar_url":"https://github.com/p0dalirius.png","language":"Python","funding_links":["https://github.com/sponsors/p0dalirius","https://patreon.com/Podalirius"],"categories":[],"sub_categories":[],"readme":"![](./.github/banner.png)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"GitHub release (latest by date)\" src=\"https://img.shields.io/github/v/release/p0dalirius/crawlersuseragents\"\u003e\n  \u003ca href=\"https://twitter.com/intent/follow?screen_name=podalirius_\" title=\"Follow\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/podalirius_?label=Podalirius\u0026style=social\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.youtube.com/c/Podalirius_?sub_confirmation=1\" title=\"Subscribe\"\u003e\u003cimg alt=\"YouTube Channel Subscribers\" src=\"https://img.shields.io/youtube/channel/subscribers/UCF_x5O7CSfr82AfNVTKOv_A?style=social\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n\u003c/p\u003e\n\nThis Python script can be used to check if there is any differences in responses of an application when the request comes from a search engine's crawler.\n\n![](./.github/four_results.png)\n\n## Features\n\n - [x] 30 crawler's user agent strings.\n - [x] Multithreading.\n - [x] JSON export with `--json outputfile.json`.\n - [x] Auto-detecting responses that stands out.\n\n## Usage\n\n```\n$ ./crawlersuseragents.py -h\n[~] Access web pages as web crawlers User-Agents, v1.1\n\nusage: crawlersuseragents.py [-h] [-v] [-t THREADS] [-x PROXY] [-k] [-L] [-j JSONFILE] url\n\nThis Python script can be used to check if there is any differences in responses of an application\nwhen the request comes from a search engine's crawler.\n\npositional arguments:\n  url                   e.g. https://example.com:port/path\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -v, --verbose         arg1 help message\n  -t THREADS, --threads THREADS\n                        Number of threads (default: 5)\n  -x PROXY, --proxy PROXY\n                        Specify a proxy to use for requests (e.g., http://localhost:8080)\n  -k, --insecure        Allow insecure server connections when using SSL (default: False)\n  -L, --location        Follow redirects (default: False)\n  -j JSONFILE, --jsonfile JSONFILE\n                        Save results to specified JSON file.\n\n```\n\n## Auto-detecting responses that stands out\n\nResults are sorted by uniqueness of their response's length. This means that the results with unique response length will be on top, and results with response's length occurring multiple times at the bottom: \n\n| Two different result lengths | Four different result lengths  |\n|------------------------------|--------------------------------|\n| ![](./.github/two_results.png) | ![](./.github/four_results.png) |\n\n\n## Contributing\n\nPull requests are welcome. Feel free to open an issue if you want to add other features.\n\n## References\n - https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers\n - https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0dalirius%2Fcrawlersuseragents","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fp0dalirius%2Fcrawlersuseragents","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fp0dalirius%2Fcrawlersuseragents/lists"}