Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
- GitHub: https://github.com/topics/crawler
- Wikipedia: https://en.wikipedia.org/wiki/Web_crawler
- Last updated: 2024-11-14 00:05:40 UTC
- JSON Representation
https://github.com/0xsha/ChainWalker
Rapid Smart Contract Crawler
blockchain crawler dataset evm-bytecode geth security smart-contracts web3
Last synced: 04 Aug 2024
https://github.com/elliotxx/zhihu-crawler-people
A simple distributed crawler for zhihu && data analysis
crawler python python-crawler spider web-crawler web-spider
Last synced: 06 Nov 2024
https://github.com/vormkracht10/laravel-seo-scanner
Scan your Laravel application routes for SEO improvements suggestions.
crawler laravel laravel-framework laravel-seo laravel-seo-scanner scanner seo seo-optimization seo-tools seotools
Last synced: 11 Oct 2024
https://github.com/subins2000/search
An Open Source Search Engine
crawler php search search-engine
Last synced: 13 Nov 2024
https://github.com/Josue87/MetaFinder
Search for documents in a domain through Search Engines (Google, Bing and Baidu). The objective is to extract metadata
Last synced: 04 Aug 2024
https://github.com/cocrawler/cocrawler
CoCrawler is a versatile web crawler built using modern tools and concurrency.
aiohttp aiohttp-client async-python concurrency crawler pluggable-modules python3 screenshot warc
Last synced: 29 Oct 2024
https://github.com/viasite/site-audit-seo
Web service and CLI tool for SEO site audit: crawl site, lighthouse all pages, view public reports in browser. Also output to console, json, csv, xlsx
audit cli crawl-site crawler lighthouse puppeteer scraper seo seo-audit seo-site-audit site-audit xlsx
Last synced: 06 Nov 2024
https://github.com/mehmetozkaya/dotnetcrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping
Last synced: 27 Oct 2024
https://github.com/mehmetozkaya/DotnetCrawler
DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c
crawler crawling csharp ddd-architecture dotnetcore entity-framework-core htmlagilitypack scraping scrapy scrapy-crawler webcrawler webcrawler-htmlagilitypack webcrawling webscraper webscraping
Last synced: 09 Nov 2024
https://github.com/Jiramew/spoon
🥄 A package for building specific Proxy Pool for different Sites.
crawler distributed ip proxies proxy proxy-provider proxypool python redis spider spoon
Last synced: 06 Nov 2024
https://github.com/norconex/crawlers
Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.
collector-fs collector-http crawler crawlers filesystem-crawler flexible java search-engine web-crawler
Last synced: 11 Nov 2024
https://github.com/rebrowser/rebrowser-patches
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
automation bot bot-detection chrome chromedriver cloudflare crawler crawling datadome headless headless-chrome playwright puppeteer puppeteer-extra rebrowser scraping selenium stealth web-scraping webdriver
Last synced: 10 Oct 2024
https://github.com/nfx/slrp
rotating open proxy multiplexer
crawler golang proxy proxy-checker proxy-list proxy-pool proxy-server
Last synced: 13 Nov 2024
https://github.com/amerkurev/scrapper
Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.
crawler crawling docker headless readability scraper web-parsers web-parsing web-scraping
Last synced: 01 Nov 2024
https://github.com/N0taN3rd/Squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
browser-automation chrome chrome-headless crawler crawling headless-chrome high-fidelity-preservation puppeteer webarchives webarchiving
Last synced: 05 Nov 2024
https://github.com/n0tan3rd/squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
browser-automation chrome chrome-headless crawler crawling headless-chrome high-fidelity-preservation puppeteer webarchives webarchiving
Last synced: 27 Oct 2024
https://github.com/guilhermecgs/ir
Projeto de calculo de Imposto de Renda em operacoes na bovespa automaticamente. Tags:canal eletronico do investidor, CEI, selenium, bovespa, IRPF, IR, imposto de renda, finance, yahoo finance, acao, fii, etf, python, crawler, webscraping, calculadora ir
acoes b3 bovespa calculadora-ir canal-eletronico-investidor cei crawler etf fii finance imposto-de-renda irpf webscraping
Last synced: 11 Nov 2024
https://github.com/stulzq/HttpCode.Core
简单、易用、高效 一个有态度的开源.Net Http请求框架!可以用制作爬虫，api请求等等。
crawler httpcode httpmock httprequest net-core net-standard
Last synced: 13 Nov 2024
https://github.com/cytopia/urlbuster
Powerful mutable web directory fuzzer to bruteforce existing and/or hidden files or directories.
brute-force bruteforce bruteforce-attacks crawler cytopia-sec url-bruteforcer
Last synced: 14 Nov 2024
https://github.com/oscar-project/ungoliant
:spider: The pipeline for the OSCAR corpus
common-crawl commoncrawl corpus-linguistics crawler fasttext language-classification nlp oscar
Last synced: 04 Nov 2024
https://github.com/sudheer-ranga/aliexpress-product-scraper
Get Aliexpress product details as a json response including feedbacks, variants, shipping info, description, images, etc.,
aliexpress aliexpress-api aliexpress-crawler aliexpress-product-json aliexpress-product-scraper aliexpress-scraper aliexpress-spider crawler dropship dropshipping hacktoberfest hacktoberfest19 hacktoberfest2019 product-json product-reviews product-scraper scraper spider
Last synced: 06 Nov 2024
https://github.com/beb7/gflare-tk
Open-Source Python Based SEO Web Crawler
crawler python robots-txt scraper seo seo-crawler tkinter
Last synced: 03 Aug 2024
https://github.com/tijme/not-your-average-web-crawler
A web crawler (for bug hunting) that gathers more than you can imagine.
bug-bounty callbacks crawler custom get post python request scanner scraper security spider vulnerability
Last synced: 10 Nov 2024
https://github.com/luohaha/jlitespider
A lite distributed Java spider framework :-)
crawler distributed distributed-systems rabbitmq spider
Last synced: 03 Aug 2024
https://github.com/Liu233w/acm-statistics
An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge
acm-icpc codechef-api codeforces-api crawler csharp docker javascript nodejs spoj-api vue
Last synced: 07 Nov 2024
https://github.com/liu233w/acm-statistics
An online tool (crawler) to analyze users performance in online judges (coding competition websites). Supported OJ: POJ, HDU, HYSBZ, CodeForces, UVA, ICPC Live Archive, FZU, SPOJ, Timus (URAL), LeetCode_CN, CSU, LibreOJ, 洛谷, 牛客OJ, Lutece (UESTC), AtCoder, AIZU, CodeChef, El Judge, BNUOJ, Codewars, UOJ, NBUT, 51Nod, DMOJ, VJudge
acm-icpc codechef-api codeforces-api crawler csharp docker javascript nodejs spoj-api vue
Last synced: 13 Nov 2024
https://github.com/bartdag/pylinkvalidator
pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web site and reports errors (e.g., 500 and 404 errors) encountered.
crawler link-checker networking python
Last synced: 14 Nov 2024
https://github.com/janreges/siteone-crawler
SiteOne Crawler is a website analyzer and exporter you'll ♥ as a Dev/DevOps, QA engineer, website owner or consultant. Works on all popular platforms - Windows, macOS and Linux (x64 and arm64 too).
analyzer crawler crawling performance qa quality-assessment security seo seotools stress-testing swoole testing website
Last synced: 25 Oct 2024
https://github.com/abaykan/CrawlBox
Easy way to brute-force web directory.
admin-finder crawler python web-crawler wordlist
Last synced: 30 Oct 2024
https://github.com/twiny/spidy
Domain names collector - Crawl websites and collect domain names along with their availability status.
backlinks crawler domain expired-domain golang scraper seotools spider
Last synced: 05 Nov 2024
https://github.com/karust/gogetcrawl
Extract web archive data using Wayback Machine and Common Crawl
commoncrawl concurrency crawler golang wayback-machine webarchive
Last synced: 05 Nov 2024
https://github.com/6677-ai/tap4-ai-crawler
The crawler opened source by tap4.ai
aitoolkit aitools crawler crawler-engine crawler-python
Last synced: 09 Nov 2024
https://github.com/moranzcw/Zhihu-Spider
一个获取知乎用户主页信息的多线程Python爬虫程序。
crawler jupyter-notebook matplotlib python requests zhihu-spider
Last synced: 31 Oct 2024
https://github.com/hominee/dyer
Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
crawler rust rust-programming-language spider web-crawler web-framework web-scraping
Last synced: 06 Nov 2024
https://github.com/nuhmanpk/webscrapper
Simple and powerfull all in one Telegram Bot to scrap / crawl webpages using Requests, html5lib and Beautifulsoup
beautifulsoup4 crawler crawler-engine crawler-python hacktoberfest hacktoberfest-accepted hacktoberfest2023 pyrogram pyrogram-bot requests scraper scraping selenium telegram telegram-bot web-scraping webscraping webscrapper webscrapping webscrapping-python
Last synced: 09 Nov 2024
https://github.com/TGiles/auto-lighthouse
A utility package for automating lighthouse reporting
audits auto-lighthouse crawler lighthouse-reports robots simplecrawler
Last synced: 05 Nov 2024
https://github.com/teal33t/poopak
POOPAK - TOR Hidden Service Crawler
crawler dark-web darknet deepweb docker flask hidden-services mongo osint redis tor tor-network
Last synced: 13 Oct 2024
https://github.com/tgiles/auto-lighthouse
A utility package for automating lighthouse reporting
audits auto-lighthouse crawler lighthouse-reports robots simplecrawler
Last synced: 01 Nov 2024
https://github.com/karthikuj/sasori
Sasori is a dynamic web crawler powered by Puppeteer, designed for lightning-fast endpoint discovery.
automation crawler crawling dast dynamic endpoint-discovery infosec puppeteer scraping security
Last synced: 01 Nov 2024
https://github.com/duckduckgo/tracker-radar-collector
🕸 Modular, multithreaded, puppeteer-based crawler
crawler puppeteer tracker-radar
Last synced: 14 Nov 2024
https://github.com/JakePartusch/lumberjack
An automated website accessibility scanner and cli
a11y accessibility axe cli crawler lumberjack
Last synced: 03 Aug 2024
https://github.com/alash3al/scraply
Scraply a simple dom scraper to fetch information from any html based website
crawler crawling dom golang scraper scrapers scraping-websites scrapy server
Last synced: 04 Nov 2024
https://github.com/luckylittle/blinkist-m4a-downloader
Grabs all of the audio files from all of the Blinkist books
audiobooks blinkist books crawler data-archiving data-mining data-processing go golang scraper spider
Last synced: 11 Nov 2024
https://github.com/jakepartusch/lumberjack
An automated website accessibility scanner and cli
a11y accessibility axe cli crawler lumberjack
Last synced: 27 Oct 2024
https://github.com/wx-chevalier/sentinel-crawler
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure :dizzy: 多语言执行器，分布式爬虫
crawler etl koa2 monitor nodejs react wx-code
Last synced: 07 Nov 2024
https://github.com/wxyyxc1992/declarative-crawler
Xenomorph Crawler, a Concise, Declarative and Observable Distributed Crawler(Node / Go / Java / Rust) For Web, RDB, OS, also can act as a Monitor(with Prometheus) or ETL for Infrastructure :dizzy: 多语言执行器，分布式爬虫
crawler etl koa2 monitor nodejs react wx-code
Last synced: 04 Aug 2024
https://github.com/nasa-jpl-memex/memex-explorer
Viewers for statistics and dashboarding of Domain Search Engine data
ache anaconda apache crawler dashboard domain-discovery memex-explorer miniconda nutch tika
Last synced: 07 Aug 2024
https://github.com/greengerong/prerender-java
java framework for prerender
angular1 crawler java prerender prerendered-page seo
Last synced: 11 Oct 2024
https://github.com/mazzzystar/baiducrawler
Sample of using proxies to crawl baidu search results.
Last synced: 11 Nov 2024
https://github.com/SimFin/pdf-crawler
SimFin's open source PDF crawler
crawler crawling geckodriver pdf pdf-crawler puppeteer python selenium-webdriver
Last synced: 06 Nov 2024
https://github.com/ethereum/node-crawler
Attempts to crawl the Ethereum network of valid Ethereum execution nodes and visualizes them in a nice web dashboard.
Last synced: 07 Oct 2024
https://github.com/simfin/pdf-crawler
SimFin's open source PDF crawler
crawler crawling geckodriver pdf pdf-crawler puppeteer python selenium-webdriver
Last synced: 11 Oct 2024
https://github.com/SeaQL/starfish-ql
✴️ An experimental graph database
crates-io crawler database graph hacktoberfest network rust sql visualization
Last synced: 11 Nov 2024
https://github.com/hardikvasa/webb
Python: An all-in-one Web Crawler, Web Parser and Web Scrapping library!
crawl-pages crawler python-library
Last synced: 09 Nov 2024
https://github.com/schollz/linkcrawler
Cross-platform persistent and distributed web crawler :link:
Last synced: 08 Nov 2024
https://github.com/pavlovtech/WebReaper
Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.
crawler datamining parser parsing scraper scraping scraping-api scraping-data scraping-tool scraping-web scraping-websites webcrawler webscraping
Last synced: 06 Nov 2024
https://github.com/zytedata/zyte-smartproxy-headless-proxy
A complimentary proxy to help to use SPM with headless browsers
Last synced: 11 Nov 2024
https://github.com/antoinevastel/bots-zoo
bot crawler crawling playwright puppeteer scraper scraping selenium user-agent useragent
Last synced: 02 Nov 2024
https://github.com/ducdev/aliexscrape
Get Aliexpress product details in JSON
aliexpress aliexpress-api aliexpress-crawler aliexpress-scraper aliexpress-spider crawler dropship dropshipping hacktoberfest hacktoberfest19 hacktoberfest2019 json scraper spider
Last synced: 27 Oct 2024
https://github.com/wuchunfu/ipproxypool
Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra
crawler go ip proxy proxy-server proxypool
Last synced: 30 Oct 2024
https://github.com/patrickschur/pappet
A command-line tool to crawl websites using puppeteer.
cli crawler pdf puppeteer screenshot
Last synced: 06 Nov 2024
https://github.com/kostas-pa/LFITester
LFITester is a Python3 program that automates the detection and exploitation of Local File Inclusion (LFI) vulnerabilities on a server.
bugbounty crawler cybersecurity enumeration exploitation fuzzing hacking lfi lfi-detection lfi-exploitation lfi-vulnerability penetration-testing penetration-testing-tools pentest-tool pentesting python web-hacking webhacking
Last synced: 04 Aug 2024
https://github.com/zhaow-de/rotating-tor-http-proxy
A multi-arch image provides one HTTP proxy endpoint with many concurrent tunnels to the Tor network.
amd64 arm64 armv6 armv7 crawler docker-image dockerhub-image haproxy multi-platform privoxy-tor proxy tor
Last synced: 06 Nov 2024
https://github.com/medcl/gopa-abandoned
GOPA, a spider written in Go.（NOTE: this project moved to https://github.com/infinitbyte/gopa ）
crawler golang lightweight spider
Last synced: 04 Aug 2024
https://github.com/hueristiq/xcrawl3r
A command-line interface (CLI) based utility to recursively crawl webpages. It is designed to systematically browse webpages' URLs and follow links to discover linked webpages' URLs.
bug-bounty bug-bounty-tools contentdiscovery crawler ethical-hacking ethical-hacking-tools go golang penetration-testing penetration-testing-tools reconnaissance red-teaming red-teaming-tools web-security
Last synced: 06 Nov 2024
https://github.com/kurogai/deepweb-scappering
Discover hidden deepweb pages
crawler deepweb hacking hacking-tool internet kali python3 scappering scapre tor tor-network
Last synced: 27 Oct 2024
https://github.com/creekorful/bathyscaphe
Fast, highly configurable, cloud native dark web crawler.
architecture crawler crawling elasticsearch golang hidden-services kibana tor web-crawler
Last synced: 27 Oct 2024
https://github.com/foo-git/rewe-discounts
Grabs current REWE discounts and saves them in a markdown file || Holt sich aktuelle REWE-Angebote und exportiert sie in eine Markdown-Liste
Last synced: 26 Oct 2024
https://github.com/jefferyhus/es6-crawler-detect
:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
bots crawler detection es6-javascript spider
Last synced: 27 Oct 2024
https://github.com/JefferyHus/es6-crawler-detect
:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.
bots crawler detection es6-javascript spider
Last synced: 11 Nov 2024
https://github.com/nietaki/crawlie
A simple Elixir library for writing decently-performing crawlers with minimum effort.
crawler elixir elixir-library genstage
Last synced: 26 Oct 2024
https://github.com/tobecrazy/seleniumdemo
Selenium automation test framework
container crawler docker docker-compose jenkins maven pip python selenium selenium-grid selenium-webdriver snapshot
Last synced: 11 Oct 2024
https://github.com/Randark-JMT/Bilibili_manga_download
带图形界面的哔哩哔哩漫画下载工具
bilibili crawler downloader pyside6 python python3 qt spider
Last synced: 27 Oct 2024
https://github.com/yuanxu-li/html-table-extractor
extract data from html table
beautifulsoup crawler extract-data html html-table scraping table
Last synced: 06 Nov 2024
https://github.com/boris-code/feaplat
爬虫管理系统，支持集群，弹性伸缩。支持运行feapder、scrapy、selenium、playwright等各种框架及脚本
crawler feapder feaplat spider
Last synced: 15 Oct 2024
https://github.com/tensojka/instastories-backup
Backup your friends' Instagram Stories forever and get to keep them even after 24 hours.
backup crawler instagram instagram-stories python python-3-6 python3
Last synced: 12 Oct 2024
https://github.com/ArchiveTeam/wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
archiveteam archiving crawl crawler crawlers crawling downloader ftp lua scraper scraping spider warc webarchiving wget wget-lua zstd
Last synced: 06 Aug 2024
