{"id":16610397,"url":"https://github.com/itszeeshan/crawlinit","last_synced_at":"2025-06-13T20:05:07.880Z","repository":{"id":223681118,"uuid":"496623977","full_name":"itszeeshan/crawlinit","owner":"itszeeshan","description":"A web crawler written in python3","archived":false,"fork":false,"pushed_at":"2024-02-21T13:36:03.000Z","size":24,"stargazers_count":4,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T00:28:35.662Z","etag":null,"topics":["appsec","bugbounty","bugbounty-tool","bugbountytips","crawler","crawler-python","enumeration","infosec","python","recon","reconnaissance","scanner","url","web"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itszeeshan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-05-26T13:06:16.000Z","updated_at":"2024-01-29T10:38:29.000Z","dependencies_parsed_at":"2024-02-21T14:54:16.667Z","dependency_job_id":null,"html_url":"https://github.com/itszeeshan/crawlinit","commit_stats":null,"previous_names":["itszeeshan/crawlinit"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/itszeeshan/crawlinit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itszeeshan%2Fcrawlinit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itszeeshan%2Fcrawlinit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itszeeshan%2Fcrawlinit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itszeeshan%2Fcrawlinit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itszeeshan","download_url":"https://codeload.github.com/itszeeshan/crawlinit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itszeeshan%2Fcrawlinit/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259712376,"owners_count":22900035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appsec","bugbounty","bugbounty-tool","bugbountytips","crawler","crawler-python","enumeration","infosec","python","recon","reconnaissance","scanner","url","web"],"created_at":"2024-10-12T01:29:59.983Z","updated_at":"2025-06-13T20:05:07.848Z","avatar_url":"https://github.com/itszeeshan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About Crawl in it\nCrawl_in_it is a python tool designed to discover URLs. It will help penteration tester and bugbounty hunters to save time, so that they don't have to go manually through all the pages. This tool will generate logs which can helps penetration tester and bugbounty hunters retrieve the results of actions they performed in the past and also will allow them to see the status codes and length of the pages, the tool has crawled through.\n\n# Screeshots\n![Crawl In It](https://user-images.githubusercontent.com/35112049/170496013-75fc2a89-0a36-426d-bee8-c6f26e163c51.gif)\n![logs](https://user-images.githubusercontent.com/35112049/170496034-25374041-fe3c-42de-9a51-90e3ee4ad102.png)\n\n# Installation\n\n```\ngit clone https://github.com/itszeeshan/crawlinit.git\n```\n\n# Recommended Python\nCrawl in it currently only supports python3.\n* The recommended version for python3 is 3.10.x\n\n# Installing Dependencies\n\n### Installing on Windows\n```\npython -m pip install -r requirements.txt\n```\n\n### Installing on Linux\n```\nsudo pip install -r requirements.txt\n```\n\n### Installing Modules independently\n```\npip install numpy\npip install argparse\npip install torch\npip install traitlets\npip install bs4\npip install requests\npip install typing_extensions\n```\n# Usage\nShort | Long      | Description\n------|-----------|-------------\n-d    | --domain  | Domain to crawl into\n-o    | --output  | Save results to file\n-h    | --help    | Shows help message and exit\n\n### For Example\n* to generate help message\n```\npython Crawler.py -h\n```\n* Specifying domain for crawling\n```\npython Crawler.py -d hackerone.com\n```\n* Specifying domain for crawling and saving results into a file\n```\npython Crawler.py -d hackerone.com -o hackerone.txt\n```\n\n### The tool will automatically generate logs!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitszeeshan%2Fcrawlinit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitszeeshan%2Fcrawlinit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitszeeshan%2Fcrawlinit/lists"}