Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Crawler

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

https://github.com/adileo/MicroFrontier

A lightweight crawler frontier implementation in TypeScript using Redis.

crawler frontier microservice redis robots-txt spider

Last synced: 14 Nov 2024

https://github.com/tsoliangwu0130/spotify-news

A Flask application to retrieve the singers' latest news according to your Spotify current playing song.

bootstrap crawler flask oauth2 python3 restful-api spotify-api

Last synced: 11 Nov 2024

https://github.com/shawon922/jobs-crawler

Crawl IT/Telecommunication jobs from bdjobs.com

beautifulsoup4 crawler python3

Last synced: 09 Nov 2024

https://github.com/spekulatius/spatie-crawler-cached-queue-example

Example to demonstrate the usage of cached queues across multiple requests.

crawler crawler-engine laravel php-crawler php-scraper queues spatie-crawler

Last synced: 12 Nov 2024

https://github.com/logocomune/botdetector

BotDetector is a golang library that detects Bot/Spider/Crawler from user agent

botdetector bots crawler go golang golang-library spider user-agent

Last synced: 11 Nov 2024

https://github.com/tosone/githubtraveler

Travel all of the GitHub users, orgs, repos.

crawler github golang

Last synced: 06 Nov 2024

https://github.com/sabinbajracharya/Insta-crawler

Pulls data from instagram and saves it to Firebase for storage and Algolia for search

accounts algolia algolia-search crawler firebase firebase-database instagram instagram-feed instagram-post javascript nodejs public scraper

Last synced: 07 Nov 2024

https://github.com/trungdq88/movie-showtimes

Web Service & Android Application to look up Vietnam movie showtimes

crawler java movie-showtimes theater

Last synced: 31 Oct 2024

https://github.com/eight04/ptt-mail-backup

一個用來抓取 PTT 站內信的 BBS Bot

bbs cli crawler ptt ptt-crawler python python3

Last synced: 28 Oct 2024

https://github.com/igeligel/BackpackLogin

:arrow_forward: A .NET core library to handle the login to Backpack.tf. Backpack.tf is a trading site for Team Fortress 2, Counter-Strike: Global Offensive, and Dota 2. Community item pricing, item trading and stats, and much more.

bot bot-framework crawler csgo csgo-bot steam steam-api steambot steamweb teamfortress2

Last synced: 13 Nov 2024

https://github.com/bbc2/discolinks

Command-line tool which checks a website for broken links.

broken-links crawler html http link-checker link-checkers link-checking validator web

Last synced: 28 Oct 2024

https://github.com/umihico/minigun-requests

Web scraping API to outsource tons of GET & xpath to cloud computing

crawler crawling scraping scraping-api scraping-framework scraping-python web-scraping

Last synced: 15 Nov 2024

https://github.com/fanzeyi/torchic

A generic search engine built using Go & Spring & Redis. Project for Google's CodeU event.

bm25 crawler search-engine

Last synced: 21 Oct 2024

https://github.com/bioinformatist/py3_scripts

Life is short, *****.

blast crawler gtf pacbio scrapy

Last synced: 10 Nov 2024

https://github.com/pawod/gis-berlin-rents

A web crawler for ImmobilienScout24.de, that has been implemented for a small project at the institue of geographic sciences of the Free University of Berlin.

apartment-rents berlin crawler gis immobilienscout24

Last synced: 04 Nov 2024

https://github.com/sebobo/shel.crawler

Neos based crawler for nodes and sites

crawler neos-cms

Last synced: 14 Oct 2024

https://github.com/luyadev/luya-module-crawler

Crawle a Website and provide intelligent search results

crawler hacktoberfest intelligent-search luya search yii2

Last synced: 10 Oct 2024

https://github.com/yerkopalma/bash-crawler

:computer: Get a site links with bash

bash crawler

Last synced: 13 Oct 2024

https://github.com/nakabonne/webcrawlerforserps

Web crawler that scrapes Google search results

cli crawler golang

Last synced: 24 Oct 2024

https://github.com/29dch/word_cloud

python制作词云项目

crawler jieba wordcloud

Last synced: 11 Nov 2024

https://github.com/khaleddallah/LinkedinScraper

Python Scrapy project parse people profiles of Linkedin Search and arrange result content in Excel and Json file

crawler excel json linkedin python scraper scrapy spider

Last synced: 05 Nov 2024

https://github.com/amirhoseinsb/Cloud_Player_V2

You can use the cloudplayer tool to listen to the music of the singer you want without going to a specific website and at a very high speed.

cloud-player crawler crawling music music-player programming python url-player

Last synced: 04 Aug 2024

https://github.com/softmarshmallow/inked-news-crawler

🕷 korean news source crawler (realtime & bulk)

crawler naver-news python3 scrapy

Last synced: 11 Oct 2024

https://github.com/yaroslaff/bulk-http-check

Very fast and simple concurrent HTTP client (3500 HTTP req/s)

bulk check concurrent connections crawler header http https multiple parallel spider status

Last synced: 07 Nov 2024

https://github.com/aurelius84/pycrawler

A flexible spider based on mysql

crawler etl mysql scrapy spider

Last synced: 06 Nov 2024

https://github.com/omilab/internet-archive-link-extractor

Tool for extracting external links of a URL from Internet Archive snapshots

crawler internetarchive

Last synced: 07 Aug 2024

https://github.com/appliedsoul/promise-crawler

Promise support for node-crawler (Web Crawler/Spider for NodeJS + server-side jQuery)

crawler node-crawler nodejs promise-node-crawler spider

Last synced: 08 Nov 2024

https://github.com/fedebotu/neurips2022-openreviewdata

Crawl & Visualize NeurIPS 2022 Data from OpenReview

crawler dataset neurips neurips-2022 openreview peer-review review scraper

Last synced: 06 Nov 2024

https://github.com/drogbadvc/crawlit

This project is a web crawler based on Scrapy, visualization 2D, PageRank

crawler scrapy seo streamlit

Last synced: 08 Nov 2024

https://github.com/duongdev/facebook-group-crawler

Facebook Groups Discussions Crawler

crawler facebook groups puppeteer

Last synced: 12 Nov 2024

https://github.com/thesp0nge/nightcrawler

A python program that crawls a website and tries to stress it, polluting forms with bogus data

crawler offensive-scripts offensive-security stress-test web-crawler web-crawling

Last synced: 12 Oct 2024

https://github.com/windfarer/biu

biubiubiu~~ I'm a tiny web crawler framework

crawler python spider spider-framework web-crawler

Last synced: 28 Oct 2024

https://github.com/bfwg/node-tinycrawler

Tiny web-crawler in a nute shell for Node.js

crawler nodejs redis

Last synced: 11 Oct 2024

https://github.com/the1812/bingwallpapers

A tool for downloading wallpapers from Bing.

crawler csharp wpf

Last synced: 04 Nov 2024

https://github.com/jean-baptiste-camps/iiif-crawler

Interrogate IIIF servers and get images of manuscripts

crawler iiif iiif-image manuscripts

Last synced: 11 Oct 2024

https://github.com/capturr/jsonld-extract

A damn simple tool to extract json-ld metadata from webpage using jquery like api (jQuery, Cheerio, CashDom ...).

cashdom cheerio crawler crawling data extract extractor javascript jquery json jsonld metadata nodejs parser scraper scraping spider typescript

Last synced: 28 Oct 2024

https://github.com/gabfl/sitecrawl

Simple Python module to crawl a website and extract URLs

crawl crawler crawler-python crawling-sites

Last synced: 13 Oct 2024

https://github.com/eric2788/platformscrawler

多平台爬蟲 + 模塊化管理，用於搜集資料並經 redis pubsub 發送

bilibili crawler crawling pubsub redis twitter youtube

Last synced: 11 Oct 2024

https://github.com/dotenorio/freeloader-of-data

A simple crawler or scraper to get open graph and other meta data from any website.

crawler graph hacktoberfest meta-data open-graph scraper

Last synced: 25 Oct 2024

https://github.com/ajcerejeira/base.gov.pt

A crawler that fetches data from base.gov.pt

crawler csv python scrapy

Last synced: 06 Nov 2024

https://github.com/simin75simin/libgencrawl

crawl all books from a library genesis search

crawler free-software libgen python3 scraper

Last synced: 05 Nov 2024

https://github.com/chusiang/crawler-book-info

A crawler for quick parser the book information

book crawler python

Last synced: 07 Nov 2024

https://github.com/tikazyq/github-crawler

Github repositories crawler

crawler scrapy

Last synced: 11 Oct 2024

https://github.com/luizppa/web-crawler

A web crawler that collects and indexes web pages. Made with chilkat and gumbo parser.

chilkat cpp crawler webcrawler

Last synced: 28 Oct 2024

https://github.com/AmirAref/DivarCrawler

an script to crawl divar.ir and extract phone numbers

crawler scraper selenium

Last synced: 05 Aug 2024

https://github.com/AmirAref/Torobot

an inline telegram robot to easy access and search in torob.com products from telegram.

crawler python python-telegram-bot scraper telegtam-bot

Last synced: 05 Aug 2024

https://github.com/bugfishtm/bugfish-image-downloader

💾 Bugfish Image Downloader: Effortless web image downloads, subsite exploration, and HD selection. Windows app, .NET 4.5, no registry usage. Download now!

bugfish bugfish-software bugfishtm crawler downloader downloadmanager downloadtool gplv3 image imagedownloader imagedownloadertool imageprocessing portable-executable portableapps software utilityapp webscraping windows windows-desktop

Last synced: 06 Nov 2024

https://github.com/feedeo/youtube-channel-crawler

YouTube Channel :tv: Crawler

crawler youtube youtube-channel

Last synced: 11 Oct 2024

https://github.com/bernabe9/render-it

Render any JavaScript content to create static sites ready for SEO

crawler javascript prerender prerenderio puppeteer render seo seo-tools server-side-rendering static-site static-site-generator

Last synced: 07 Nov 2024

https://github.com/s045pd/magicworld

环球网-神奇世界看看看爬虫

crawler python3 sanic telepot

Last synced: 07 Nov 2024

https://github.com/dynesshely/everydaynews

A repo fetched most of news and infomation, where stored and organized them.

crawler data fetcher network news

Last synced: 05 Nov 2024

https://github.com/oscarnevarezleal/ecommerce-crawler

Parallel ecommerce crawler using Docker and Puppeter on GCP

crawler gcp nodejs pubnub puppeteer

Last synced: 09 Aug 2024

https://github.com/Antosser/web-crawler

Rust Web Crawler that finds every page, image, and script on a website (and downloads it)

crawler html rust seo web

Last synced: 24 Sep 2024

https://github.com/karambir/ugc-colleges

Python Script to extract college names from UGC, India website.

college crawler extract html-parser python python-script ugc

Last synced: 24 Oct 2024

https://github.com/integralist/go-web-crawler

A web crawler built in the Go programming language

concurrency crawler go golang web-crawler

Last synced: 11 Oct 2024

https://github.com/Hound-fm/podcatcher

Audio media crawler for lbry.

crawler lbry python

Last synced: 03 Aug 2024

https://github.com/baraja-core/webcrawler

Simple crawling websites by following links.

bot crawler crawling-websites fast php robot speed

Last synced: 06 Nov 2024

https://github.com/aprilnea/xjtlu

This is how to get all the network resources of XJTLU.

crawler gateway http-auth python spider web-crawler xjtlu

Last synced: 15 Nov 2024

https://github.com/foolin/scrago

An simpe, fast, extensible crawl page framework for golang

crawler go scrago scrapy

Last synced: 09 Nov 2024

https://github.com/dist1ll/hltv-rust

A client to fetch and parse data from HLTV.org

api crawler hltv parser rust

Last synced: 14 Oct 2024

https://github.com/ernesto-jimenez/crawler

Easily crawl websites in Go.

crawler golang

Last synced: 13 Oct 2024

https://github.com/licoy/java-crawler

通过java使用jsoup爬虫框架爬取数据

crawler java jsoup

Last synced: 19 Oct 2024

https://github.com/vmdang/historycrawler

The OOP project collects historical data in Vietnam and displays

crawler gson java javafx jsoup

Last synced: 11 Oct 2024

https://github.com/floscha/genius-lyrics-crawler

A concurrent crawler to retrieve song lyrics from Genius

celery crawler fluentd genius lyrics mongodb python

Last synced: 09 Nov 2024

https://github.com/roccomuso/is-bing

Verify that a request is from Bing crawlers using Bing's DNS verification steps

bing bot check crawler dns ip js nodejs verify

Last synced: 17 Oct 2024

https://github.com/coghost/iparse

To extract HTML/json content identified by CSS selectors(with bs4) with yaml config support

crawler parser parser-library python xkcd yaml

Last synced: 09 Nov 2024

https://github.com/sayakie/pixiv-crawler

Crawls images from Pixiv 🚀

crawler nodejs pixiv typescript

Last synced: 28 Oct 2024

https://github.com/surelle-ha/dogma

Dogma is a CLI tool that enables interaction with the GitHub API for the purpose of searching .env files with specified keywords. You can configure a GitHub token and use the crawler to search for keys in .env files across public repositories.

cli crawler github nodejs

Last synced: 10 Nov 2024

https://github.com/danielmorell/se_bot_checker

Validate search engine user agents and IP addresses.

crawler googlebot python search-engine spider

Last synced: 15 Oct 2024

https://github.com/spencerlepine/readme-crawler

A Node.js web crawler to download README files and follow contained links. Fetch repositories from a valid GitHub URL

crawler javascript node nodejs readme scraper web-crawler webcrawer

Last synced: 13 Nov 2024

https://github.com/iml1111/toonkor_collector

툰코 만화 수집기

crawler python

Last synced: 21 Oct 2024

https://github.com/vinitkumar/pycrawler

Crawler in Python 3.7, 3.8. 3.9. Pypy3

crawler python python35 python36 utils

Last synced: 28 Oct 2024

https://github.com/leelow/nightmare-screenshot-selector

👻 📷 A Nightmare plugin to easily take screenshots.

crawler headless-browsers javascript js nightmare nightmarejs nodejs plugin webcrawler

Last synced: 15 Nov 2024

https://github.com/pjt3591oo/golang-crawler

golang으로 크롤러 만들기

crawler golang

Last synced: 06 Nov 2024

https://github.com/frectonz/rampilo

A telegram crawler

crawler rust telegram telegram-crawler

Last synced: 14 Nov 2024

https://github.com/kernelerr/pixivsync

Pixiv图片下载及同步工具

crawler pixiv pixiv-crawler python

Last synced: 12 Oct 2024

https://github.com/vinouno/BilibiliDanmuCrawler

一个从 bilibili.com 爬取弹幕并生成词云的 Python 项目

crawler python

Last synced: 27 Oct 2024

https://github.com/juliandavidmr/raptor

Lightweight tool for scanning web sites, works as spider. Once executed, starts scanning pages looking for websites to visit, with automatic indexing.

crawler kotlin mysql spider

Last synced: 09 Nov 2024

https://github.com/stopka/fedicrawl

Collect feeds to follow on Fediverse nodes.

crawler docker fediverse nodejs prisma typescript

Last synced: 05 Nov 2024

https://github.com/leomaurodesenv/smm-course-search

A package to searching courses - Super Mario Maker

bookmark-site crawler javascript json mario-game mario-maker nodejs

Last synced: 02 Nov 2024

https://github.com/alishahbazi81/jobcrawler

Job crawler robot which finds jobs on job board platforms like LinkedIn, Glassdoor, and indeed based on their post time and send them to a telegram channel

asp-net-core crawler jobs jobsearch telegram telegram-bot

Last synced: 11 Nov 2024

https://github.com/liyifeng1994/go-crawler

基于golang的分布式爬虫项目

crawler elastic elasticsearch golang

Last synced: 12 Nov 2024

Crawler Awesome Lists
awesome-crawler 101 awesome-python-primer 68 awesome-digital-preservation 45
Crawler Categories
2.6 机器学习 50 Python 18 Replay tools 18 1.1 语言基础 13 2.4 Web 前端 10 2.1 爬虫基础 9 3\. 数据库 8 Java 7 2.5 数据分析 7 Web archiving 7 4\. 异步IO 6 Other digital objects 6 Standards and specifications 4 Social Networks 4 2.3 Django 框架 4 2.2 Flask 框架 4 1.2 语言进阶 3 C# 3 Related lists 2 Organizations 2