{"id":5101,"url":"https://github.com/BruceDone/awesome-crawler","name":"awesome-crawler","description":"A collection of awesome web crawler,spider in different languages","projects_count":101,"last_synced_at":"2026-08-05T12:00:26.094Z","repository":{"id":37396635,"uuid":"70459317","full_name":"BruceDone/awesome-crawler","owner":"BruceDone","description":"A collection of awesome web crawler,spider in different languages","archived":false,"fork":false,"pushed_at":"2024-06-16T17:03:36.000Z","size":76,"stargazers_count":7254,"open_issues_count":39,"forks_count":756,"subscribers_count":200,"default_branch":"master","last_synced_at":"2026-07-17T06:04:52.686Z","etag":null,"topics":["awesome","crawler","node-crawler","scraper","spider","web-crawler","web-scraper"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BruceDone.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-10-10T06:42:00.000Z","updated_at":"2026-07-16T19:55:07.000Z","dependencies_parsed_at":"2024-04-13T22:55:24.937Z","dependency_job_id":"a3a55d5d-8105-4424-aaf4-b44ea93f5bb2","html_url":"https://github.com/BruceDone/awesome-crawler","commit_stats":{"total_commits":70,"total_committers":29,"mean_commits":2.413793103448276,"dds":0.5285714285714286,"last_synced_commit":"5b6f40dab0518171d9b3c7adec31a257eeab64eb"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/BruceDone/awesome-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BruceDone%2Fawesome-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BruceDone%2Fawesome-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BruceDone%2Fawesome-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BruceDone%2Fawesome-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BruceDone","download_url":"https://codeload.github.com/BruceDone/awesome-crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BruceDone%2Fawesome-crawler/sbom","scorecard":{"id":24773,"data":{"date":"2025-08-11","repo":{"name":"github.com/BruceDone/awesome-crawler","commit":"832efd3e8f8af1350be3450d9b7636a8523fc4b2"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.1,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":8,"reason":"Found 13/15 approved changesets -- score normalized to 8","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 29 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-14T17:37:45.375Z","repository_id":37396635,"created_at":"2025-08-14T17:37:45.375Z","updated_at":"2025-08-14T17:37:45.375Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36311008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-08-05T02:00:06.619Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"created_at":"2024-01-06T20:25:11.714Z","updated_at":"2026-08-05T12:00:26.095Z","primary_language":null,"list_of_lists":false,"displayable":true,"categories":["Java","C#","Go","JavaScript","Ruby","C","Python","PHP","C++","Erlang","Rust","Scala","Perl","R"],"sub_categories":[],"readme":"# Awesome-crawler ![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)\nA collection of awesome web crawler,spider and resources in different languages.\n\n## Contents\n\n- [Python](#python)\n- [Java](#java)\n- [C#](#c)\n- [JavaScript](#javascript)\n- [PHP](#php)\n- [C++](#c-1)\n- [C](#c-2)\n- [Ruby](#ruby)\n- [Rust](#rust)\n- [R](#r)\n- [Erlang](#erlang)\n- [Perl](#perl)\n- [Go](#go)\n- [Scala](#scala)\n\n## Python \n* [Scrapy](https://github.com/scrapy/scrapy) - A fast high-level screen scraping and web crawling framework.\n    * [django-dynamic-scraper](https://github.com/holgerd77/django-dynamic-scraper) - Creating Scrapy scrapers via the Django admin interface.\n    * [Scrapy-Redis](https://github.com/rolando/scrapy-redis) - Redis-based components for Scrapy.\n    * [scrapy-cluster](https://github.com/istresearch/scrapy-cluster) - Uses Redis and Kafka to create a distributed on demand scraping cluster.\n    * [distribute_crawler](https://github.com/gnemoug/distribute_crawler) - Uses scrapy,redis, mongodb,graphite to create a distributed spider.\n* [pyspider](https://github.com/binux/pyspider) - A powerful spider system.\n* [CoCrawler](https://github.com/cocrawler/cocrawler) - A versatile web crawler built using modern tools and concurrency.\n* [cola](https://github.com/chineking/cola) - A distributed crawling framework.\n* [Demiurge](https://github.com/matiasb/demiurge) - PyQuery-based scraping micro-framework.\n* [Scrapely](https://github.com/scrapy/scrapely) - A pure-python HTML screen-scraping library.\n* [feedparser](http://pythonhosted.org/feedparser/) - Universal feed parser.\n* [you-get](https://github.com/soimort/you-get) -  Dumb downloader that scrapes the web.\n* [MechanicalSoup](https://github.com/hickford/MechanicalSoup) - A Python library for automating interaction with websites.\n* [portia](https://github.com/scrapinghub/portia) - Visual scraping for Scrapy.\n* [crawley](https://github.com/jmg/crawley) - Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.\n* [RoboBrowser](https://github.com/jmcarp/robobrowser) - A simple, Pythonic library for browsing the web without a standalone web browser.\n* [MSpider](https://github.com/manning23/MSpider) - A simple ,easy spider using gevent and js render. \n* [brownant](https://github.com/douban/brownant) - A lightweight web data extracting framework.\n* [PSpider](https://github.com/xianhu/PSpider) - A simple spider frame in Python3.\n* [Gain](https://github.com/gaojiuli/gain) - Web crawling framework based on asyncio for everyone.\n* [sukhoi](https://github.com/iogf/sukhoi) - Minimalist and powerful Web Crawler.\n* [spidy](https://github.com/rivermont/spidy) - The simple, easy to use command line web crawler. \n* [newspaper](https://github.com/codelucas/newspaper) - News, full-text, and article metadata extraction in Python 3\n* [aspider](https://github.com/howie6879/aspider) - An async web scraping micro-framework based on asyncio. \n\n## Java\n* [ACHE Crawler](https://github.com/ViDA-NYU/ache) - An easy to use web crawler for domain-specific search.\n* [Apache Nutch](http://nutch.apache.org/) - Highly extensible, highly scalable web crawler for production environment.\n    * [anthelion](https://github.com/yahoo/anthelion) - A plugin for Apache Nutch to crawl semantic annotations within HTML pages.\n* [Crawler4j](https://github.com/yasserg/crawler4j) - Simple and lightweight web crawler.\n* [JSoup](http://jsoup.org/) - Scrapes, parses, manipulates and cleans HTML.\n* [websphinx](http://www.cs.cmu.edu/~rcm/websphinx/) - Website-Specific Processors for HTML information extraction.\n* [Open Search Server](http://www.opensearchserver.com/) - A full set of search functions. Build your own indexing strategy. Parsers extract full-text data. The crawlers can index everything.\n* [Gecco](https://github.com/xtuhcy/gecco) - A easy to use lightweight web crawler\n* [WebCollector](https://github.com/CrawlScript/WebCollector) - Simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.\n* [Webmagic](https://github.com/code4craft/webmagic) - A scalable crawler framework.\n* [Spiderman](https://git.oschina.net/l-weiwei/spiderman) - A scalable ,extensible, multi-threaded web crawler.\n    * [Spiderman2](http://git.oschina.net/l-weiwei/Spiderman2) - A distributed  web crawler framework,support js render.\n* [Heritrix3](https://github.com/internetarchive/heritrix3) -  Extensible, web-scale, archival-quality web crawler project.\n* [SeimiCrawler](https://github.com/zhegexiaohuozi/SeimiCrawler) - An agile, distributed crawler framework.\n* [StormCrawler](http://github.com/DigitalPebble/storm-crawler/) - An open source collection of resources for building low-latency, scalable web crawlers on Apache Storm\n* [Spark-Crawler](https://github.com/USCDataScience/sparkler) - Evolving Apache Nutch to run on Spark.\n* [webBee](https://github.com/pkwenda/webBee) - A DFS web spider.\n* [spider-flow](https://github.com/ssssssss-team/spider-flow) - A visual spider framework, it's so good that you don't need to write any code to crawl the website.\n* [Norconex Web Crawler](https://github.com/Norconex/collector-http) - Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repository of your choice (e.g. a search engine). Can be used as a stand alone application or be embedded into Java applications.\n\n\n## C# \n* [ccrawler](http://www.findbestopensource.com/product/ccrawler) - Built in C# 3.5 version. it contains a simple extension of web content categorizer, which can separate between the web page depending on their content.\n* [SimpleCrawler](https://github.com/lei-zhu/SimpleCrawler) - Simple spider base on mutithreading, regluar expression.\n* [DotnetSpider](https://github.com/zlzforever/DotnetSpider) - This is a cross platfrom, ligth spider develop by C#.\n* [Abot](https://github.com/sjdirect/abot) - C# web crawler built for speed and flexibility.\n* [Hawk](https://github.com/ferventdesert/Hawk) - Advanced Crawler and ETL tool written in C#/WPF.\n* [SkyScraper](https://github.com/JonCanning/SkyScraper) - An asynchronous web scraper / web crawler using async / await and Reactive Extensions.\n* [Infinity Crawler](https://github.com/TurnerSoftware/InfinityCrawler) - A simple but powerful web crawler library in C#.\n\n## JavaScript\n* [scraperjs](https://github.com/ruipgil/scraperjs) - A complete and versatile web scraper.\n* [scrape-it](https://github.com/IonicaBizau/scrape-it) - A Node.js scraper for humans.\n* [simplecrawler](https://github.com/cgiffard/node-simplecrawler) - Event driven web crawler.\n* [node-crawler](https://github.com/bda-research/node-crawler) - Node-crawler has clean,simple api.\n* [js-crawler](https://github.com/antivanov/js-crawler) - Web crawler for Node.JS, both HTTP and HTTPS are supported.\n* [webster](https://github.com/zhuyingda/webster) - A reliable web crawling framework which can scrape ajax and js rendered content in a web page.\n* [x-ray](https://github.com/lapwinglabs/x-ray) - Web scraper with pagination and crawler support.\n* [node-osmosis](https://github.com/rchipka/node-osmosis) - HTML/XML parser and web scraper for Node.js.\n* [web-scraper-chrome-extension](https://github.com/martinsbalodis/web-scraper-chrome-extension) - Web data extraction tool implemented as chrome extension.\n* [supercrawler](https://github.com/brendonboshell/supercrawler) - Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits. \n* [headless-chrome-crawler](https://github.com/yujiosaka/headless-chrome-crawler) - Headless Chrome crawls with jQuery support\n* [Squidwarc](https://github.com/n0tan3rd/squidwarc) - High fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head\n* [crawlee](https://github.com/apify/crawlee) - A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast. \n\n\n## PHP\n* [Goutte](https://github.com/FriendsOfPHP/Goutte) - A screen scraping and web crawling library for PHP.\n    * [laravel-goutte](https://github.com/dweidner/laravel-goutte) - Laravel 5 Facade for Goutte.\n* [dom-crawler](https://github.com/symfony/dom-crawler) - The DomCrawler component eases DOM navigation for HTML and XML documents.\n* [QueryList](https://github.com/jae-jae/QueryList) - The progressive PHP crawler framework.\n* [pspider](https://github.com/hightman/pspider) - Parallel web crawler written in PHP.\n* [php-spider](https://github.com/mvdbos/php-spider) - A configurable and extensible PHP web spider.\n* [spatie/crawler](https://github.com/spatie/crawler) - An easy to use, powerful crawler implemented in PHP. Can execute Javascript.\n* [crawlzone/crawlzone](https://github.com/crawlzone/crawlzone) - Crawlzone is a fast asynchronous internet crawling framework for PHP.\n* [PHPScraper](https://github.com/spekulatius/PHPScraper) - PHPScraper is a scraper \u0026 crawler built for simplicity.\n\n## C++\n* [open-source-search-engine](https://github.com/gigablast/open-source-search-engine) - A distributed open source search engine and spider/crawler written in C/C++.\n\n## C\n* [httrack](https://github.com/xroche/httrack) - Copy websites to your computer.\n\n## Ruby\n* [Nokogiri](https://github.com/sparklemotion/nokogiri) - A Rubygem providing HTML, XML, SAX, and Reader parsers with XPath and CSS selector support.\n* [upton](https://github.com/propublica/upton) - A batteries-included framework for easy web-scraping. Just add CSS(Or do more).\n* [wombat](https://github.com/felipecsl/wombat) - Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.\n* [RubyRetriever](https://github.com/joenorton/rubyretriever) - RubyRetriever is a Web Crawler, Scraper \u0026 File Harvester.\n* [Spidr](https://github.com/postmodern/spidr) - Spider a site, multiple domains, certain links or infinitely.\n* [Cobweb](https://github.com/stewartmckee/cobweb) - Web crawler with very flexible crawling options, standalone or using sidekiq.\n* [mechanize](https://github.com/sparklemotion/mechanize) - Automated web interaction \u0026 crawling.\n\n## Rust\n* [spider](https://github.com/spider-rs/spider) - The fastest web crawler and indexer.\n* [crawler](https://github.com/a11ywatch/crawler) - A gRPC web indexer turbo charged for performance.\n\n## R\n* [rvest](https://github.com/hadley/rvest) - Simple web scraping for R.\n\n## Erlang \n* [ebot](https://github.com/matteoredaelli/ebot) - A scalable, distribuited and highly configurable web cawler.\n\n## Perl\n* [web-scraper](https://github.com/miyagawa/web-scraper) - Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions.\n\n## Go\n* [pholcus](https://github.com/henrylee2cn/pholcus) -  A distributed, high concurrency and powerful web crawler.\n* [gocrawl](https://github.com/PuerkitoBio/gocrawl) - Polite, slim and concurrent web crawler.\n* [fetchbot](https://github.com/PuerkitoBio/fetchbot) - A simple and flexible web crawler that follows the robots.txt policies and crawl delays.\n* [go_spider](https://github.com/hu17889/go_spider) - An awesome Go concurrent Crawler(spider) framework. \n* [dht](https://github.com/shiyanhui/dht) - BitTorrent DHT Protocol \u0026\u0026 DHT Spider.\n* [ants-go](https://github.com/wcong/ants-go) - A open source, distributed, restful crawler engine in golang.\n* [scrape](https://github.com/yhat/scrape) - A simple, higher level interface for Go web scraping.\n* [creeper](https://github.com/wspl/creeper) - The Next Generation Crawler Framework (Go).\n* [colly](https://github.com/asciimoo/colly) - Fast and Elegant Scraping Framework for Gophers.\n* [ferret](https://github.com/MontFerret/ferret) - Declarative web scraping.\n* [Dataflow kit](https://github.com/slotix/dataflowkit) - Extract structured data from web pages. Web sites scraping.\n* [Hakrawler](https://github.com/hakluke/hakrawler) - Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application\n\n\n## Scala\n* [crawler](https://github.com/bplawler/crawler) - Scala DSL for web crawling.\n* [scrala](https://github.com/gaocegege/scrala) - Scala crawler(spider) framework, inspired by scrapy.\n* [ferrit](https://github.com/reggoodwin/ferrit) - Ferrit is a web crawler service written in Scala using Akka, Spray and Cassandra.\n","projects_url":"https://awesome.ecosyste.ms/api/v1/lists/brucedone%2Fawesome-crawler/projects"}