{"id":13639423,"url":"https://github.com/hueristiq/xcrawl3r","last_synced_at":"2025-04-06T22:12:05.838Z","repository":{"id":167597038,"uuid":"383683096","full_name":"hueristiq/xcrawl3r","owner":"hueristiq","description":"A command-line interface (CLI) based utility to recursively crawl webpages. It is designed to systematically browse webpages' URLs and follow links to discover linked webpages' URLs.","archived":false,"fork":false,"pushed_at":"2024-11-11T14:43:02.000Z","size":143,"stargazers_count":96,"open_issues_count":5,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T21:09:50.329Z","etag":null,"topics":["bug-bounty","bug-bounty-tools","contentdiscovery","crawler","ethical-hacking","ethical-hacking-tools","go","golang","penetration-testing","penetration-testing-tools","reconnaissance","red-teaming","red-teaming-tools","web-security"],"latest_commit_sha":null,"homepage":"https://github.com/hueristiq/xcrawl3r","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hueristiq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-07T05:26:30.000Z","updated_at":"2025-03-27T04:29:46.000Z","dependencies_parsed_at":"2023-07-14T06:09:00.545Z","dependency_job_id":"d1021de7-2489-4909-aac1-9b386d0fd6ef","html_url":"https://github.com/hueristiq/xcrawl3r","commit_stats":null,"previous_names":["hueristiq/xcrawl3r","signedsecurity/sigrawl3r","hueristiq/hqcrawl3r"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hueristiq%2Fxcrawl3r","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hueristiq%2Fxcrawl3r/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hueristiq%2Fxcrawl3r/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hueristiq%2Fxcrawl3r/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hueristiq","download_url":"https://codeload.github.com/hueristiq/xcrawl3r/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247557770,"owners_count":20958047,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bug-bounty","bug-bounty-tools","contentdiscovery","crawler","ethical-hacking","ethical-hacking-tools","go","golang","penetration-testing","penetration-testing-tools","reconnaissance","red-teaming","red-teaming-tools","web-security"],"created_at":"2024-08-02T01:01:00.478Z","updated_at":"2025-04-06T22:12:05.815Z","avatar_url":"https://github.com/hueristiq.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# X Crawler (`xcrawl3r`)\n\n![made with go](https://img.shields.io/badge/made%20with-Go-1E90FF.svg) [![go report card](https://goreportcard.com/badge/github.com/hueristiq/xcrawl3r)](https://goreportcard.com/report/github.com/hueristiq/xcrawl3r) [![release](https://img.shields.io/github/release/hueristiq/xcrawl3r?style=flat\u0026color=1E90FF)](https://github.com/hueristiq/xcrawl3r/releases) [![open issues](https://img.shields.io/github/issues-raw/hueristiq/xcrawl3r.svg?style=flat\u0026color=1E90FF)](https://github.com/hueristiq/xcrawl3r/issues?q=is:issue+is:open) [![closed issues](https://img.shields.io/github/issues-closed-raw/hueristiq/xcrawl3r.svg?style=flat\u0026color=1E90FF)](https://github.com/hueristiq/xcrawl3r/issues?q=is:issue+is:closed) [![license](https://img.shields.io/badge/license-MIT-gray.svg?color=1E90FF)](https://github.com/hueristiq/xcrawl3r/blob/master/LICENSE) ![maintenance](https://img.shields.io/badge/maintained%3F-yes-1E90FF.svg) [![contribution](https://img.shields.io/badge/contributions-welcome-1E90FF.svg)](https://github.com/hueristiq/xcrawl3r/blob/master/CONTRIBUTING.md)\n\n`xcrawl3r` is a command-line interface (CLI) based utility to recursively crawl webpages. It is designed to systematically browse webpages' URLs and follow links to discover linked webpages' URLs.\n\n## Resources\n\n* [Features](#features)\n* [Installation](#installation)\n\t* [Install release binaries (Without Go Installed)](#install-release-binaries-without-go-installed)\n\t* [Install source (With Go Installed)](#install-source-with-go-installed)\n\t\t* [`go install ...`](#go-install)\n\t\t* [`go build ...` the development Version](#go-build--the-development-version)\n\t* [Install on Docker (With Docker Installed)](#install-on-docker-with-docker-installed)\n* [Usage](#usage)\n* [Contributing](#contributing)\n* [Licensing](#licensing)\n* [Credits](#credits)\n\t* [Contributors](#contributors)\n\t* [Similar Projects](#similar-projects)\n\n## Features\n\n* Recursively crawls webpages for URLs.\n* Parses URLs from files (`.js`, `.json`, `.xml`, `.csv`, `.txt` \u0026 `.map`).\n* Parses URLs from `robots.txt`.\n* Parses URLs from sitemaps.\n* Renders pages (including Single Page Applications such as Angular and React).\n* Cross-Platform (Windows, Linux \u0026 macOS)\n\n## Installation\n\n### Install release binaries (Without Go Installed)\n\nVisit the [releases page](https://github.com/hueristiq/xcrawl3r/releases) and find the appropriate archive for your operating system and architecture. Download the archive from your browser or copy its URL and retrieve it with `wget` or `curl`:\n\n* ...with `wget`:\n\n\t```bash\n\twget https://github.com/hueristiq/xcrawl3r/releases/download/v\u003cversion\u003e/xcrawl3r-\u003cversion\u003e-linux-amd64.tar.gz\n\t```\n\n* ...or, with `curl`:\n\n\t```bash\n\tcurl -OL https://github.com/hueristiq/xcrawl3r/releases/download/v\u003cversion\u003e/xcrawl3r-\u003cversion\u003e-linux-amd64.tar.gz\n\t```\n\n...then, extract the binary:\n\n```bash\ntar xf xcrawl3r-\u003cversion\u003e-linux-amd64.tar.gz\n```\n\n\u003e [!TIP]\n\u003e The above steps, download and extract, can be combined into a single step with this onliner\n\u003e \n\u003e ```bash\n\u003e curl -sL https://github.com/hueristiq/xcrawl3r/releases/download/v\u003cversion\u003e/xcrawl3r-\u003cversion\u003e-linux-amd64.tar.gz | tar -xzv\n\u003e ```\n\n\u003e [!NOTE]\n\u003e On Windows systems, you should be able to double-click the zip archive to extract the `xcrawl3r` executable.\n\n...move the `xcrawl3r` binary to somewhere in your `PATH`. For example, on GNU/Linux and OS X systems:\n\n```bash\nsudo mv xcrawl3r /usr/local/bin/\n```\n\n\u003e [!NOTE]\n\u003e Windows users can follow [How to: Add Tool Locations to the PATH Environment Variable](https://msdn.microsoft.com/en-us/library/office/ee537574(v=office.14).aspx) in order to add `xcrawl3r` to their `PATH`.\n\n### Install source (With Go Installed)\n\nBefore you install from source, you need to make sure that Go is installed on your system. You can install Go by following the official instructions for your operating system. For this, we will assume that Go is already installed.\n\n#### `go install ...`\n\n```bash\ngo install -v github.com/hueristiq/xcrawl3r/cmd/xcrawl3r@latest\n```\n\n#### `go build ...` the development Version\n\n* Clone the repository\n\n\t```bash\n\tgit clone https://github.com/hueristiq/xcrawl3r.git \n\t```\n\n* Build the utility\n\n\t```bash\n\tcd xcrawl3r/cmd/xcrawl3r \u0026\u0026 \\\n\tgo build .\n\t```\n\n* Move the `xcrawl3r` binary to somewhere in your `PATH`. For example, on GNU/Linux and OS X systems:\n\n\t```bash\n\tsudo mv xcrawl3r /usr/local/bin/\n\t```\n\n\tWindows users can follow [How to: Add Tool Locations to the PATH Environment Variable](https://msdn.microsoft.com/en-us/library/office/ee537574(v=office.14).aspx) in order to add `xcrawl3r` to their `PATH`.\n\n\n\u003e [!CAUTION]\n\u003e While the development version is a good way to take a peek at `xcrawl3r`'s latest features before they get released, be aware that it may have bugs. Officially released versions will generally be more stable.\n\n### Install on Docker (With Docker Installed)\n\nTo install `xcrawl3r` on docker:\n\n* Pull the docker image using:\n\n    ```bash\n    docker pull hueristiq/xcrawl3r:latest\n    ```\n\n* Run `xcrawl3r` using the image:\n\n    ```bash\n    docker run --rm hueristiq/xcrawl3r:latest -h\n    ```\n\n## Usage\n\nTo start using `xcrawl3r`, open your terminal and run the following command for a list of options:\n\n```bash\nxcrawl3r -h\n```\n\nHere's what the help message looks like:\n\n```text\n\n                             _ _____\n__  _____ _ __ __ ___      _| |___ / _ __\n\\ \\/ / __| '__/ _` \\ \\ /\\ / / | |_ \\| '__|\n \u003e  \u003c (__| | | (_| |\\ V  V /| |___) | |\n/_/\\_\\___|_|  \\__,_| \\_/\\_/ |_|____/|_|\n                                    v0.2.0\n\nUSAGE:\n  xcrawl3r [OPTIONS]\n\nINPUT:\n -d, --domain string               domain to match URLs\n     --include-subdomains bool     match subdomains' URLs\n -s, --seeds string                seed URLs file (use `-` to get from stdin)\n -u, --url string                  URL to crawl\n\nCONFIGURATION:\n     --depth int                   maximum depth to crawl (default 3)\n                                      TIP: set it to `0` for infinite recursion\n     --headless bool               If true the browser will be displayed while crawling.\n -H, --headers string[]            custom header to include in requests\n                                      e.g. -H 'Referer: http://example.com/'\n                                      TIP: use multiple flag to set multiple headers\n     --proxy string[]              Proxy URL (e.g: http://127.0.0.1:8080)\n                                      TIP: use multiple flag to set multiple proxies\n     --render bool                 utilize a headless chrome instance to render pages\n     --timeout int                 time to wait for request in seconds (default: 10)\n     --user-agent string           User Agent to use (default: xcrawl3r v0.2.0 (https://github.com/hueristiq/xcrawl3r))\n                                      TIP: use `web` for a random web user-agent,\n                                      `mobile` for a random mobile user-agent,\n                                       or you can set your specific user-agent.\n\nRATE LIMIT:\n -c, --concurrency int             number of concurrent fetchers to use (default 10)\n     --delay int                   delay between each request in seconds\n     --max-random-delay int        maximux extra randomized delay added to `--dalay` (default: 1s)\n -p, --parallelism int             number of concurrent URLs to process (default: 10)\n\nOUTPUT:\n     --debug bool                  enable debug mode (default: false)\n -m, --monochrome bool             coloring: no colored output mode\n -o, --output string               output file to write found URLs\n     --silent bool                 display output URLs only\n -v, --verbose bool                display verbose output\n\n```\n\n## Contributing\n\nWe welcome contributions! Feel free to submit [Pull Requests](https://github.com/hueristiq/xcrawl3r/pulls) or report [Issues](https://github.com/hueristiq/xcrawl3r/issues). For more details, check out the [contribution guidelines](https://github.com/hueristiq/xcrawl3r/blob/master/CONTRIBUTING.md).\n\n## Licensing\n\nThis utility is licensed under the [MIT license](https://opensource.org/license/mit). You are free to use, modify, and distribute it, as long as you follow the terms of the license. You can find the full license text in the repository - [Full MIT license text](https://github.com/hueristiq/xcrawl3r/blob/master/LICENSE).\n\n## Credits\n\n### Contributors\n\nA huge thanks to all the contributors who have helped make `xcrawl3r` what it is today!\n\n[![contributors](https://contrib.rocks/image?repo=hueristiq/xcrawl3r\u0026max=500)](https://github.com/hueristiq/xcrawl3r/graphs/contributors)\n\n### Similar Projects\n\nIf you're interested in more utilities like this, check out:\n\n[gospider](https://github.com/jaeles-project/gospider) ◇ [hakrawler](https://github.com/hakluke/hakrawler) ◇ [katana](https://github.com/projectdiscovery/katana) ◇ [urlgrab](https://github.com/IAmStoxe/urlgrab)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhueristiq%2Fxcrawl3r","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhueristiq%2Fxcrawl3r","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhueristiq%2Fxcrawl3r/lists"}