Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-crawler

A collection of awesome web crawler,spider in different languages
https://github.com/asciimoo/awesome-crawler

Last synced: about 20 hours ago
JSON representation

  • Python

    • Scrapy - A fast high-level screen scraping and web crawling framework.
    • django-dynamic-scraper - Creating Scrapy scrapers via the Django admin interface.
    • scrapy-cluster - Uses Redis and Kafka to create a distributed on demand scraping cluster.
    • distribute_crawler - Uses scrapy,redis, mongodb,graphite to create a distributed spider.
    • pyspider - A powerful spider system.
    • Demiurge - PyQuery-based scraping micro-framework.
    • Scrapely - A pure-python HTML screen-scraping library.
    • Scrapy-Redis - Redis-based components for Scrapy.
    • cola - A distributed crawling framework.

  • Java

    • Spiderman2 - A distributed web crawler framework,support js render.
    • websphinx - Website-Specific Processors for HTML information extraction.

  • C#

    • SimpleCrawler - Simple spider base on mutithreading, regluar expression.
    • ccrawler - Built in C# 3.5 version. it contains a simple extension of web content categorizer, which can saparate between the web page depending on their content.
Programming Languages
Python 8 HTML 1
Categories
Python 9 C# 2 Java 2
Sub Categories
Keywords
python 4 scrapy 3 scraping 3 crawler 3 redis 2 distributed 2 kafka 1 webscraping 1 spider 1 scraper 1 django 1 web-scraping-python 1 web-scraping 1 framework 1 crawling 1