Crawler | Ecosyste.ms: Awesome

https://github.com/PuerkitoBio/gocrawl

Polite, slim and concurrent web crawler.

crawler robots-txt

Last synced: 25 Mar 2025

https://github.com/elliotgao2/gain

Web crawling framework based on asyncio.

aiohttp asyncio crawler python spider uvloop

Last synced: 08 Apr 2025

https://github.com/rendora/rendora

dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

angular chrome-devtools chrome-headless crawler dynamic-rendering go golang javascript puppeteer react reactjs seo seo-optimization server-side-rendering spa ssr vue vuejs

Last synced: 27 Sep 2025

https://github.com/zorlan/skycaiji

蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

crawler crawling php spider webcrawler

Last synced: 14 May 2025

https://github.com/blankerl/dxy-covid-19-crawler

2019新型冠状病毒疫情实时爬虫及API | COVID-19/2019-nCoV Realtime Infection Crawler and API

2019-ncov crawler realtime-api

Last synced: 15 May 2025

https://github.com/anouarbensaad/vulnx

vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.

auto-exploiter bot cloudflare-detection cms-detector crawler detects-vulnerabilities dorks exploits hacking information-gathering pentest security-tools shell-injection subdomains-gathering vulnerability vulnerability-assessment vulnerability-detection vulnerability-exploit website-vulnerability-scanner wp-scanner

Last synced: 15 May 2025

https://github.com/xianhu/pspider

简单易用的Python爬虫框架，QQ交流群：597510560

crawler multi-threading multiprocessing proxies python python-spider spider web-crawler web-spider

Last synced: 15 May 2025

https://github.com/xianhu/PSpider

简单易用的Python爬虫框架，QQ交流群：597510560

crawler multi-threading multiprocessing proxies python python-spider spider web-crawler web-spider

Last synced: 25 Mar 2025

https://github.com/hu17889/go_spider

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

crawler go pipeline schedule spider

Last synced: 08 Apr 2026

https://github.com/nekmo/dirhunt

Find web directories without bruteforce

crawler dirscanner pentesting python security security-tools websec without-bruteforce

Last synced: 14 May 2025

https://github.com/CatVodTVOfficial/CatVodTVSpider

catvod crawler maotv player spider tv

Last synced: 15 Apr 2025

https://github.com/Nekmo/dirhunt

Find web directories without bruteforce

crawler dirscanner pentesting python security security-tools websec without-bruteforce

Last synced: 28 Mar 2025

https://github.com/lixi5338619/lxspider

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书、大众点评、推特、脉脉、知乎》

12306 andrioid crawler douban douyin douyinsignature kuaishou meituan pdd signature taobao toutiao twitter wechat weibo weixin xiaohongshu xiecheng youku

Last synced: 15 May 2025

https://github.com/howie6879/ruia

Async Python 3.6+ web scraping micro-framework based on asyncio

aiohttp asyncio asyncio-spider crawler crawling-framework middlewares python python-ruia ruia spider uvloop

Last synced: 14 May 2025

https://github.com/diskoverdata/diskover-community

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

crawler disk-space disk-space-analyzer disk-usage duplicate-files duplicatefilefinder elasticsearch file-indexing file-manager file-tagging filesystem filesystem-analysis filesystem-indexer filesystem-visualization metadata php python storage storage-analytics web-application

Last synced: 18 Feb 2026

https://github.com/jsrei/ast-hook-for-js-re

浏览器内存漫游解决方案（探索中...）

crawler js-reverse

Last synced: 15 May 2025

https://github.com/teamnewpipe/newpipeextractor

NewPipe's core library for extracting data from streaming sites

bandcamp crawler extractor mediaccc newpipe peertube scraper soundcloud youtube

Last synced: 05 Feb 2026

https://github.com/extractus/article-extractor

To extract main article from given URL with Node.js

article article-extractor article-parser crawler extract nodejs readability scraper

Last synced: 27 Apr 2025

https://github.com/yoongikim/autocrawler

Google, Naver multiprocess image web crawler (Selenium)

bigdata chromedriver crawler customizable deep-learning google image-crawler multiprocess python selenium thread

Last synced: 06 Oct 2025

https://github.com/YoongiKim/AutoCrawler

Google, Naver multiprocess image web crawler (Selenium)

bigdata chromedriver crawler customizable deep-learning google image-crawler multiprocess python selenium thread

Last synced: 19 Apr 2025

https://github.com/edoardottt/cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

bugbounty crawler crawling endpoint-discovery endpoints go golang hacktoberfest infosec osint penetration-testing pentesting recon reconnaissance redteam scraper secret-keys secrets-detection security security-tools

Last synced: 14 May 2025

https://github.com/JSREI/ast-hook-for-js-RE

浏览器内存漫游解决方案（探索中...）

crawler js-reverse

Last synced: 11 Jul 2025

https://github.com/u3c3/bt-btt

磁力網站U3C3介紹以及域名更新

adult avmoo bittorrent bt btsow crawler download jav javbus javlibrary magnet magnet-link nyaa porn rarbg spider sukebei tracker u3c3

Last synced: 27 Jan 2026

https://github.com/coder-hxl/x-crawl

Flexible Node.js AI-assisted crawler library

ai ai-crawl chromium crawl crawler fingerprint flexible javascript multifunction nodejs puppeteer spider typescript

Last synced: 14 May 2025

https://github.com/AAndyProgram/SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.

crawler download downloader gay image instagram lgbt manager media onlyfans photo pictures pornhub reddit thisvid twitter video xhamster xvideo youtube

Last synced: 09 Apr 2025

https://github.com/aandyprogram/scrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.

crawler download downloader gay image instagram lgbt manager media onlyfans photo pictures pornhub reddit thisvid twitter video xhamster xvideo youtube

Last synced: 13 Apr 2025

https://github.com/goclone-dev/goclone

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

cloning crawler go golang website-cloner website-scraper

Last synced: 14 May 2025

https://github.com/github/lightcrawler

Crawl a website and run it through Google lighthouse

chrome crawler google-lighthouse

Last synced: 28 Sep 2025

https://github.com/imthaghost/goclone

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

cloning crawler go golang website-cloner website-scraper

Last synced: 24 Mar 2025

https://github.com/archiveteam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

archiving crawl crawler spider warc

Last synced: 15 May 2025

https://github.com/ArchiveTeam/grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

archiving crawl crawler spider warc

Last synced: 07 Apr 2025

https://github.com/TeamNewPipe/NewPipeExtractor

NewPipe's core library for extracting data from streaming sites

bandcamp crawler extractor mediaccc newpipe peertube scraper soundcloud youtube

Last synced: 19 Jul 2025

https://github.com/srx-2000/spider_collection

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos视频爬取，有声书爬取，微博爬虫，安居客信息爬取+数据可视化，哔哩哔哩视频封面提取器，ip代理池封装，知乎百万级用户爬虫+数据分析，github用户爬虫

crawler python spider

Last synced: 08 Apr 2025

https://github.com/leonardocardoso/swiftlinkpreview

It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.

carthage cocoapods crawler flow ios macos preview regular-expressions relevant-texts swift swift-package-manager tvos url watchos website

Last synced: 28 Sep 2025

https://github.com/LeonardoCardoso/SwiftLinkPreview

It makes a preview from an URL, grabbing all the information such as title, relevant texts and images.

carthage cocoapods crawler flow ios macos preview regular-expressions relevant-texts swift swift-package-manager tvos url watchos website

Last synced: 06 Aug 2025

https://github.com/openwpm/openwpm

A web privacy measurement framework

crawler firefox privacy python3

Last synced: 24 Dec 2025

https://github.com/lorey/mlscraper

🤖 Scrape data from HTML websites automatically by just providing examples

crawler crawler-python crawling extraction-engine html machine-learning scraper scraping

Last synced: 15 May 2025

https://github.com/dadoonet/fscrawler

Elasticsearch File System Crawler (FS Crawler)

crawler elasticsearch java tika

Last synced: 17 Mar 2026

https://github.com/openwpm/OpenWPM

A web privacy measurement framework

crawler firefox privacy python3

Last synced: 28 Mar 2025

https://github.com/felipecsl/wombat

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

crawler dsl ruby scraper

Last synced: 14 May 2025

https://github.com/adyzng/jd-autobuy

Python爬虫，京东自动登录，在线抢购商品

crawler jingdong python scraper

Last synced: 11 Mar 2026

https://github.com/Adyzng/jd-autobuy

Python爬虫，京东自动登录，在线抢购商品

crawler jingdong python scraper

Last synced: 15 May 2025

https://github.com/kiddyuchina/beanbun

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

beanbun crawler php spider

Last synced: 15 May 2025

https://github.com/kiddyuchina/Beanbun

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

beanbun crawler php spider

Last synced: 30 Mar 2025

https://github.com/darbra/sperm

浏览过的精彩逆向文章汇总，值得一看

crawl crawler frida spider unidbg

Last synced: 14 May 2025

https://github.com/kkoooqq/fakebrowser

🤖 Fake fingerprints to bypass anti-bot systems. Simulate mouse and keyboard operations to make behavior like a real person.

anti-bot-detection anti-fingerprinting automation bot browser-fingerprint cheat crawler fake headless puppeteer puppeteer-extra puppeteer-extra-plugin scrapy spoof stealth

Last synced: 13 Mar 2025

https://github.com/0x676e67/rnet

An ergonomic Python HTTP Client with TLS fingerprint

akamai akamai-fingerprint crawler fingerprint http http-client http2 https ja3 ja4 python rust scraper tls tls-client tls-fingerprint web-scraping websocket websocket-client

Last synced: 18 Feb 2026

https://github.com/0xinfection/xsrfprobe

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

audit crafted-tokens crawler csrf csrf-attacks csrf-poc csrf-scanner csrf-tokens spider token-generation xsrf

Last synced: 13 May 2025

https://github.com/instapy/instagram-profilecrawl

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

automation crawler information instagram instapy python python-script selenium simple

Last synced: 15 May 2025

https://github.com/InstaPy/instagram-profilecrawl

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

automation crawler information instagram instapy python python-script selenium simple

Last synced: 01 Apr 2025

https://github.com/0xInfection/XSRFProbe

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

audit crafted-tokens crawler csrf csrf-attacks csrf-poc csrf-scanner csrf-tokens spider token-generation xsrf

Last synced: 24 Mar 2025

https://github.com/dwisiswant0/go-dork

The fastest dork scanner written in Go.

bing-dorks bugbounty bugbounty-tool crawler dork-scanner dorking golang google-dorking google-dorks infosec security shodan-dorks vulnerability-scanners

Last synced: 16 May 2025

https://github.com/seveniruby/AppCrawler

基于appium的app自动遍历工具

appium appium-app crawler diff scala xpath

Last synced: 16 Apr 2025

https://github.com/yutto-dev/bilili

:beers: bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

bilibili crawler danmaku download downloader multithread python3 requests spider subtitle video

Last synced: 23 Feb 2025

https://github.com/xisuo67/xhs-spider

小红书数据采集、网站图片、视频资源批量下载工具，颜值超高的数据采集工具（批量下载，视频提取，图片，去水印等）Telegram:https://t.me/+ZtLSwuIKTo44MDY1

crawler csharp downloader wpf wpf-notifyicon wpf-ui

Last synced: 13 Apr 2025

https://github.com/the-robot/sqliv

massive SQL injection vulnerability scanner

crawler multiprocessing reverse-ip-scan scanner scanning sql-injection sqli sqli-vulnerability-scanner

Last synced: 24 Mar 2025

https://github.com/dixudx/tumblr-crawler

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

crawler photos python tumblr videos

Last synced: 15 May 2025

https://github.com/orangecoding/fredy

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

appartment appartments crawler ebay-kleinanzeigen fredy immobilien immobilienscout24 immonet immoscout24 immowelt nodejs puppeteer real-estate real-estate-search self-hosted telegrambots web-scraping wg-gesucht wohnungssuche

Last synced: 11 Jun 2026

https://github.com/chenjiandongx/mzitu

👧 美女写真套图爬虫（二）

crawler meizi

Last synced: 12 Apr 2025

https://github.com/Autumn-27/ScopeSentry

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

bug-bounty bugbounty bugbounty-tool crawler dirscanner infosec osint pagemonitor scanner scope-sentry security security-tools src subdomain-enumeration subdomain-takeovers urlscan vulnerability-scanners

Last synced: 07 Sep 2025

https://github.com/elixir-crawly/crawly

Crawly, a high-level web crawling & scraping framework for Elixir.

crawler crawling elixir erlang extract-data scraper scraping scraping-websites spider

Last synced: 11 Dec 2025

https://github.com/vifreefly/kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites

crawler headless-chrome kimurai scraper scrapy

Last synced: 14 May 2025

https://github.com/pea3nut/pxer

A tool for pixiv.net. 人人可用的P站爬虫

add-on batch crawler pixiv tampermonkey userscript

Last synced: 10 Apr 2025

https://github.com/pea3nut/Pxer

A tool for pixiv.net. 人人可用的P站爬虫

add-on batch crawler pixiv tampermonkey userscript

Last synced: 02 Apr 2025

https://github.com/Zeal-L/BiliBili-Manga-Downloader

一个好用的哔哩哔哩漫画下载器，拥有图形界面，支持关键词搜索漫画和二维码登入，黑科技下载未解锁章节，多线程下载，多种保存格式，本地漫画管理，一键检查更新！

bilibili bilibili-download comic-downloader comics crawler downloader gui manga manga-downloader pyside6 python3

Last synced: 16 Mar 2025

https://github.com/webrecorder/browsertrix-crawler

Run a high-fidelity browser-based web archiving crawler in a single Docker container

crawler crawling wacz warc web-archiving web-crawler webrecorder

Last synced: 10 Feb 2026

https://github.com/apache/stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

apache-storm crawler distributed java stormcrawler web-crawler

Last synced: 13 Feb 2026

https://github.com/codelibs/fess

Fess is very powerful and easily deployable Enterprise Search Server.

crawler elasticsearch enterprise-search full-text-search fulltext-search java lucene search search-engine

Last synced: 12 Nov 2025

https://github.com/fredwu/crawler

A high performance web crawler / scraper in Elixir.

crawler elixir files offline scraper scraper-engine spider

Last synced: 15 May 2025

https://github.com/johanneszab/tumblthree

A Tumblr Blog Backup Application

backup crawler csharp downloader hidden-tumblr internationalization mef mvvm password-protected-tumblr safe-mode-tumblr tumblr tumblr-blog tumblr-likes tumblr-search tumblr-tags windows wpf

Last synced: 30 Sep 2025

https://github.com/johanneszab/TumblThree

A Tumblr Blog Backup Application

backup crawler csharp downloader hidden-tumblr internationalization mef mvvm password-protected-tumblr safe-mode-tumblr tumblr tumblr-blog tumblr-likes tumblr-search tumblr-tags windows wpf

Last synced: 05 May 2025

https://github.com/wycm/zhihu-crawler

zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目

crawler java spider zhihu

Last synced: 03 May 2025

https://github.com/skywalkerdarren/chatweb

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

ai chatgpt crawler docx embedding faiss gpt gpt-35-turbo news-extractor newspaper openai pdf pgvector postgresql vector-database

Last synced: 25 Oct 2025

https://github.com/apache/incubator-stormcrawler

A scalable, mature and versatile web crawler based on Apache Storm

apache-storm crawler distributed java stormcrawler web-crawler

Last synced: 12 Apr 2025

https://github.com/Le0nsec/SecCrawler

一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客，持续更新中。

anquanke bot crawler security seebug xianzhi

Last synced: 04 Apr 2025

https://github.com/hellock/icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

bing-image crawler flickr-api google-images python scrapy spider

Last synced: 14 May 2025

https://github.com/kong36088/BaiduImageSpider

一个超级轻量的百度图片爬虫

baidu crawler python3 spider

Last synced: 25 Mar 2025

https://github.com/SkywalkerDarren/chatWeb

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

ai chatgpt crawler docx embedding faiss gpt gpt-35-turbo news-extractor newspaper openai pdf pgvector postgresql vector-database

Last synced: 30 Mar 2025

https://github.com/jomingyu/google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

crawler google-play hacktoberfest hacktoberfest2023 python scraper

Last synced: 14 May 2025

https://github.com/scrapinghub/scrapyrt

HTTP API for Scrapy spiders

crawler crawling hacktoberfest hacktoberfest2021 python scraper scrapy twisted webcrawler webcrawling

Last synced: 15 May 2025

https://github.com/JoMingyu/google-play-scraper

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

crawler google-play hacktoberfest hacktoberfest2023 python scraper

Last synced: 10 Apr 2025

https://github.com/skrapeit/skrape.it

A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.

crawler dom hacktoberfest html-parser integration-testing jsoup kotlin kotlin-dsl parse scraper skrape system-testing test-automation testing

Last synced: 09 Apr 2025

https://github.com/platonai/PulsarRPA

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ai-agents ai-crawler ai-rpa browser-automation crawler llm scraper

Last synced: 04 Apr 2025

https://github.com/platonai/pulsarRPA

Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

ai-agents ai-crawler ai-rpa ai-scrarper crawler rpa scraper scraping web-crawler web-scraping

Last synced: 01 Apr 2025

https://github.com/soskek/bookcorpus

Crawl BookCorpus

bookcorpus corpus crawler nlp scraper

Last synced: 07 Oct 2025

https://github.com/wujunwei928/parse-video

Golang短视频去水印：抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

crawler douyin kuaishou spider video video-crawler video-spider weibo

Last synced: 28 Jan 2026

https://github.com/postmodern/spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

crawler ruby scraper spider spider-links web web-crawler web-scraper web-scraping web-spider

Last synced: 13 May 2025

https://github.com/DataHenHQ/till

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

crawler man-in-the-middle mitm proxy-server scraper scraping web-scraping