{"id":29953093,"url":"https://github.com/xorbit01/webpalm","last_synced_at":"2025-12-15T14:10:05.575Z","repository":{"id":156537788,"uuid":"631288520","full_name":"XORbit01/webpalm","owner":"XORbit01","description":"🕸️  Crawl in the web network","archived":false,"fork":false,"pushed_at":"2025-03-20T20:28:34.000Z","size":5315,"stargazers_count":371,"open_issues_count":1,"forks_count":39,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-02T22:15:46.467Z","etag":null,"topics":["crawler","crawling","data","data-science","datamining","go","golang","hack","mining","osint","redteam","spider","tool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/XORbit01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-04-22T14:47:32.000Z","updated_at":"2025-07-27T04:26:41.000Z","dependencies_parsed_at":"2024-02-18T22:29:49.984Z","dependency_job_id":"d75a146f-2606-4fa8-9f34-cc501e9c4aae","html_url":"https://github.com/XORbit01/webpalm","commit_stats":null,"previous_names":["malwarize/webpalm","xorbit01/webpalm"],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/XORbit01/webpalm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XORbit01%2Fwebpalm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XORbit01%2Fwebpalm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XORbit01%2Fwebpalm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XORbit01%2Fwebpalm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/XORbit01","download_url":"https://codeload.github.com/XORbit01/webpalm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XORbit01%2Fwebpalm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268469221,"owners_count":24255262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","crawling","data","data-science","datamining","go","golang","hack","mining","osint","redteam","spider","tool"],"created_at":"2025-08-03T14:20:17.879Z","updated_at":"2025-12-15T14:10:05.488Z","avatar_url":"https://github.com/XORbit01.png","language":"Go","readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://user-images.githubusercontent.com/130087473/235356807-32b80288-7808-4f66-a6f2-fcbe7ab34b72.png\" alt=\"WebPalm Banner\" width=\"600\"\u003e\n\n\u003ch2\u003e 🌐 Advanced Web Traversal \u0026 Data Extraction 🚀 \u003c/h2\u003e\n\n[![GitHub release](https://img.shields.io/github/v/release/XORbit01/webpalm?color=blue\u0026label=release)]()\n[![GitHub license](https://img.shields.io/github/license/XORbit01/webpalm?color=green)]()\n[![GitHub issues](https://img.shields.io/github/issues/XORbit01/webpalm?color=red)]()\n[![GitHub stars](https://img.shields.io/github/stars/XORbit01/webpalm?color=yellow)]()\n[![GitHub forks](https://img.shields.io/github/forks/XORbit01/webpalm?color=orange)]()\n[![GitHub watchers](https://img.shields.io/github/watchers/XORbit01/webpalm?color=blue)]()\n\n🔍 **Crawl websites efficiently, extract structured data, and visualize connections.** 🕵️‍♂️\n\n\u003cimg src=\"https://github.com/XORbit01/webpalm/assets/130087473/6c601672-f278-431d-854b-0a9876a2fafd\" alt=\"WebPalm Preview\" width=\"600\"\u003e\n\n\u003c/div\u003e\n\n---\n\n## 🗺️ Table of Contents\n- [`📦 Installation`](#-installation)\n- [`⚡ Features`](#-features)\n- [`🚀 Usage`](#-usage)\n- [`📌 Examples`](#-examples)\n- [`📜 Regex Patterns`](#-regex-patterns)\n- [`🤝 Contributing`](#-contributing)\n\n---\n\n## ⚡ Features  \n\n- 🌳 **Structured Web-Tree Generation**  \n\n- 🕵️ **Regex-Based Data Extraction**  \n\n- ⚡ **High-Speed Multi-threading**  \n\n- 📂 **Multiple Export Formats**  \n\n- 🎨 **Colorized Output \u0026 Robust Error Handling**  \n\n---\n\n## 📦 Installation\n\n### 📥 Download Binary\n\n### 📥 Compile from Source\n```sh\ngit clone https://github.com/XORbit01/webpalm.git\ncd webpalm\ngo build -o webpalm \u0026\u0026 ./webpalm\n```  \n👉 [Download Latest Release](https://github.com/XORbit01/webpalm/releases/latest)\n\n### 📥 Install via Go\n```sh\ngo install github.com/XORbit01/webpalm/v2@latest\n```\n\n---\n\n## 🚀 Usage\n\n```sh\nwebpalm -h\n```\n\n### ⚙️ Common Flags\n```yaml\n🌎 -i, --include     # Include only specific domains (e.g., google.com, facebook.com)\n🔗 -u, --url         # Target website  \n📏 -l, --level       # Depth of traversal  \n❌ -x, --exclude     # Exclude status codes (e.g., 404, 500)  \n💾 -o, --output      # Save results (JSON, XML, TXT)  \n🚀 -w, --worker      # Multi-threading workers  \n🔍 --regexes         # Extract data using regex  \n```\n\n---\n\n## 📌 Examples\n\n### 🌲 Generate a Website Map\n```sh\nwebpalm -u https://example.com -l2\n```\n\n### 💬 Extract Comments from Pages\n```sh\nwebpalm -u https://example.com -l1 --regexes comments=\"\\\u003c\\!--.*?--\u003e\" -o results.json\n```\n\n### 🚀 Crawl with Multi-threading\n```sh\nwebpalm -u https://example.com -l3 -w 50\n```\n\n### 💾 Export Results\n```sh\nwebpalm -u https://example.com -l2 -o output.xml\n```\n\n---\n\n## 📜 Regex Patterns\n\n| 🔍 Purpose   | 📜 Regex Pattern |\n|-----------|--------------|\n| 📧 Emails    | `[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+` |\n| 💬 Comments  | `\\\u003c\\!--.*?--\u003e` |\n| 🔑 Tokens    | `[a-zA-Z0-9]{32}` |\n| 🔐 Passwords | `\\bpassword\\b.{0,10}` |\n\n📌 *Escape special characters if needed.*\n\n---\n\n## 🤝 Contributing\n💡 Pull requests are welcome! Open an issue before major changes.  \n📢 **Discord:** `xorbit.`\n\n---\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorbit01%2Fwebpalm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxorbit01%2Fwebpalm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxorbit01%2Fwebpalm/lists"}