{"id":13845886,"url":"https://github.com/knownsec/LSpider","last_synced_at":"2025-07-12T03:32:47.291Z","repository":{"id":39091784,"uuid":"330892154","full_name":"knownsec/LSpider","owner":"knownsec","description":"LSpider 一个为被动扫描器定制的前端爬虫","archived":false,"fork":false,"pushed_at":"2022-11-10T10:57:23.000Z","size":496,"stargazers_count":347,"open_issues_count":6,"forks_count":51,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-06-19T00:08:38.931Z","etag":null,"topics":["python3","security","spider"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/knownsec.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-01-19T07:00:57.000Z","updated_at":"2025-06-14T07:34:55.000Z","dependencies_parsed_at":"2022-08-03T04:16:19.360Z","dependency_job_id":null,"html_url":"https://github.com/knownsec/LSpider","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/knownsec/LSpider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knownsec%2FLSpider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knownsec%2FLSpider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knownsec%2FLSpider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knownsec%2FLSpider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/knownsec","download_url":"https://codeload.github.com/knownsec/LSpider/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/knownsec%2FLSpider/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264930824,"owners_count":23684931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python3","security","spider"],"created_at":"2024-08-04T17:03:39.542Z","updated_at":"2025-07-12T03:32:46.978Z","avatar_url":"https://github.com/knownsec.png","language":"Python","funding_links":[],"categories":["Python","信息搜集"],"sub_categories":[],"readme":"# LSpider\n\nLSpider - 一个为被动扫描器定制的前端爬虫\n\n# 什么是LSpider?\n\n一款为被动扫描器而生的前端爬虫~\n\n由Chrome Headless、LSpider主控、Mysql数据库、RabbitMQ、被动扫描器5部分组合而成。\n\n(1) 建立在Chrome Headless基础上，将模拟点击和触发事件作为核心原理，通过设置代理将流量导出到被动扫描器。\n\n(2) 通过内置任务+子域名api来进行发散式的爬取，目的经可能的触发对应目标域的流量。\n\n(3) 通过RabbitMQ来进行任务管理，支持大量线程同时任务。\n\n(4) 智能填充表单，提交表单等。\n\n(5) 通过一些方式智能判断登录框，并反馈给使用者，使用者可以通过添加cookie的方式来完成登录。\n\n(6) 定制了相应的Webhook接口，以供Webhook统计发送到微信。\n\n(7) 内置了Hackerone、bugcrowd爬虫，提供账号的情况下可以一键获取某个目标的所有范围。\n\n### 为什么选择LSpider?\n\nLSpider是专门为被动扫描器定制的爬虫，许多功能都是为被动扫描器而服务的。\n\n建立在RabbitMQ的任务管理系统相当稳定，可以长期在无人监管的情况下进行发散式的爬取。\n\n### LSpider的最佳实践是什么？\n\n**服务器1（2c4g以上）**: Nginx + Mysql + Mysql管理界面（phpmyadmin）\n\n将被动扫描器的输出位置设置为web路径下，这样可以通过Web同时管理结果以及任务。\n\nLSpider部署5线程以上，设置代理连接被动扫描器（被动扫描器可以设置专门的漏扫代理）\n\n**服务器2**（非必要，但如果部署在服务器1，那么就需要更好的配置）：RabbitMQ\n\n### 还有什么问题？\n\nLSpider从设计之初是为了配合像xray这种被动扫描器而诞生的，但可惜的是，在工具发展的过程中，深刻认识到爬虫是无法和被动扫描器拆分开来的。\n\n强行将应该在被动扫描器实现的功能在爬虫端实现简直是舍本逐末，所以我们发起了另一个被动扫描器项目，如果有机会，后续还会开源出来给大家。\n\n### 设计思路？\n\n[为被动扫描器量身打造一款爬虫-LSpider](https://lorexxar.cn/2021/01/28/lspider-design/)\n\n# Usage\n\n[安装\u0026使用](./docs/init.md)\n\n你可以通过下面的命令来测试是否安装成功\n\n```\npython3 manage.py SpiderCoreBackendStart --test\n```\n\n通过dockerfile安装（不推荐的安装模式）\n```\ncd ./docker\n\ndocker-compose up -d\n```\n\n[dockerfile 安装\u0026使用](./docker/readme.md)\n\n**使用dockerfile安装，推荐修改其中必要的配置信息以避免安全漏洞诞生。**\n\n**值得注意的是，以下脚本可能会涉及到项目路径影响，使用前请修改相应的配置**\n\n建议配合screen来挂起进程\n\n启动LSpider webhook 与漏洞展示页面（默认端口2062）\n\n```\n./lspider_webhook.sh\n```\n\n启动LSpider\n```\n./lspider_start.sh\n```\n\n完全关闭LSpider\n```\n./lspider_stop.sh\n```\n\n启动被动扫描器\n```\n./xray.sh\n```\n\n# 一些关键的配置\n\n[配置说明](./docs/config.md)\n\n# 如何配置扫描任务 以及 其他的配置相关\n\n其中包含了如何配置扫描任务、鉴权信息、webhook。\n\n值得注意的是，文中提到的Cookie配置，格式为浏览器请求包复制即可。\n\n[如何配置扫描任务 以及 其他的配置相关](./docs/manage.md)\n\n扫描器结果输出到配置文件相同目录（默认为vuls/）,则可以通过web界面访问。\n\n![](./docs/6.png)\n\n# 使用内置的hackerone、bugcrowd爬虫获取目标\n\n使用hackerone爬虫，你需要首先配置好hackerone账号\n```\n python3 .\\manage.py HackeroneSpider {appname}\n```\n![](./docs/4.png)\n\n同理，bugcrowd使用\n```\n python3 .\\manage.py BugcrowdSpider {appname}\n```\n\n![](./docs/5.png)\n\n# Contributors\n\n感谢如下贡献者对本工具发展过程中的贡献：\n\n- [QGW](https://github.com/qboy0000)\n\n# 404StarLink\n![](https://github.com/knownsec/404StarLink-Project/raw/master/logo.png)\n\nLSpider 是 404Team [星链计划](https://github.com/knownsec/404StarLink-Project)中的一环，如果对LSpider有任何疑问又或是想要找小伙伴交流，可以参考星链计划的加群方式。\n\n- [https://github.com/knownsec/404StarLink-Project#community](https://github.com/knownsec/404StarLink-Project#community)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknownsec%2FLSpider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknownsec%2FLSpider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknownsec%2FLSpider/lists"}