Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
- GitHub: https://github.com/topics/crawler
- Wikipedia: https://en.wikipedia.org/wiki/Web_crawler
- Last updated: 2026-06-22 00:06:47 UTC
- JSON Representation
https://github.com/ging-dev/sitemap-crawler
Collect links through the sitemap.xml or robots.txt
crawler php php8 sitemap sitemap-crawler
Last synced: 10 Jan 2026
https://github.com/destan0098/go-agent
you can use this package to make random user agent
crawler security security-tools user-agent user-agents
Last synced: 20 Sep 2025
https://github.com/airtoxin/stackable-crawler
middleware based lightweight crawler framework
crawler javascript lightweight
Last synced: 13 Apr 2025
https://github.com/agricolamz/2017_andan_course
Course for ANDAN Summer School about strings and texts in R
crawler language-detection r regular-expressions rstats string-distance string-manipulation strings teaching teaching-materials text-analysis tf-idf tidytext
Last synced: 14 Jun 2025
https://github.com/tbarnes94/fortnite-weapons-bot
A bot that returns fortnite weapon statistics based on input from Discord users. Written in TypeScript.
crawler discord discord-bot discord-js typescript2
Last synced: 06 Jan 2026
https://github.com/joeri-abbo/python-credly-scraper
This project is a set of Python scripts designed to crawl and extract data from the Credly platform, focusing on skills, organizations, and badges. The scripts allow users to perform searches using command-line arguments, predefined search terms, or skills listed in a JSON file. The collected data is then saved to JSON files for further analysis an
badges crawler credly data-extraction json organizations python python3 requests-library skills web-crawling
Last synced: 23 Sep 2025
https://github.com/genfuture/cryptocurrency-scraper
Cryptocurrency Data Crawler 🚀 Updates CoinData Every 12 hours. High-performance Node.js crawler that fetches comprehensive data for 1500+ cryptocurrencies from CoinGecko API. Collects market data, and blockchain details with built-in rate limiting and resume capability. Perfect for crypto analysis, research, and building market intelligence tools
binance bitcoin bsc coingecko coingecko-api crawler crypto-bot cryptocurrencies cryptocurrency ethereum scraper
Last synced: 28 Jan 2026
https://github.com/anzo52/jcrawl
Java web crawler
crawler java java-web-crawler web web-crawler
Last synced: 06 Mar 2026
https://github.com/tikazyq/colly-crawlers
Crawlers using Golang-based web crawling framework Colly
Last synced: 15 Jun 2025
https://github.com/litingyes/cobweb
Collect, store and distribute meaningful static data
apis bing-image bing-wallpapers crawler image random-image
Last synced: 31 Jul 2025
https://github.com/ozakboy/taiwan-news-crawlers
.net-based Crawlers for news of Taiwan (.net 台灣新聞爬蟲，數據物件化，方便使用)
crawler data-collection dataset-generation dotnet news taiwan webcrawlers
Last synced: 15 Apr 2025
https://github.com/andreoliwa/scrapy-tegenaria
🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢
crawler flask postgresql python python3 scrapy
Last synced: 13 Apr 2025
https://github.com/eeriemyxi/nosori
Online image viewer for https://coomer.su and https://kemono.su
api coomer crawler docker image javascript kemono server typescript video viewer web
Last synced: 01 Aug 2025
https://github.com/bimmr/site-crawler
Chromium Extension: Crawl a website
chrome-extension crawler downloader sitemap
Last synced: 12 Mar 2026
https://github.com/soakit/book-download
book-download
crawler html2epub nodejs novel-downloader
Last synced: 01 May 2026
https://github.com/telanflow/scrago
A micro crawler framework. achieved by GOLANG.
crawler go micro-framework spider
Last synced: 25 Jun 2025
https://github.com/lucassmacedo/tvtime-scrapy
A simple crawly to get personal watched movies/series/animes from my user profile
Last synced: 10 Oct 2025
https://github.com/norconex/committer-neo4j
Implementation of Norconex Committer for Neo4j.
crawler neo4j neo4j-committer norconex-committer
Last synced: 19 Jan 2026
https://github.com/kluhan/kraken
Kraken is a generic, mid-scale web crawler specifically built to crawl vertical data-sources, like Youtube or the Google Play Store.
celery crawler google-play-store python web-crawling
Last synced: 07 Sep 2025
https://github.com/brunojppb/airport-crawler
Simple and powerful CLI app to get worldwide airport information in JSON format
Last synced: 09 Jun 2026
https://github.com/sachin-kumar-2003/seocrawler
SEO Link Checker | Find Broken Links & Improve SEO I have built an SEO Link Checker that helps businesses, marketers, and site owners scan their websites, detect broken or harmful links, and fix them fast. This improves site health, user experience, and search rankings. Features: -Scan entire website for broken internal and external links
beautifulsoup crawler fastapi reactjs seo seo-optimization
Last synced: 15 Apr 2026
https://github.com/sebi75/lightweight-sitemapper
A lightweight sitemapper written in typescript, built on top of fast-xml-parser and relying on few dependencies
Last synced: 21 Jan 2026
https://github.com/qianbinbin/moebooru-crawler
Retrieve links of images from moebooru-based sites, like yande.re and konachan.com .
Last synced: 22 Oct 2025
https://github.com/gabrielrf/bsbdf
Telegram Public Channel
crawler python telegram telegram-channel telegraph
Last synced: 05 Feb 2026
https://github.com/akagi201/spy
A lightweight distributed web crawler
crawler distributed lightweight nsq
Last synced: 26 Feb 2025
https://github.com/wangshouh/icourse163_script
A python script designed for like and comments to MOOC. 用于中国大学MOOC点赞和评论的Python脚本
crawler icourse163 python requests
Last synced: 28 Mar 2025
https://github.com/rudrakshi99/web_crawler
A Spider🕷 or search engine bot that downloads and indexes content from all over the Internet.
Last synced: 22 Jul 2025
https://github.com/sharmadhiraj/free-json-datasets
Collection of free JSON data that are scraped and parsed from different websites.
collection crawler data data-scraping datasets json sports statistics web-scraping
Last synced: 28 Mar 2025
https://github.com/tsonglew/spidreat
Article Spider with Python & Node.js :beetle:
Last synced: 06 Apr 2025
https://github.com/yjg30737/pyqt-google-image-crawler
Crawling image files from Google search result with Python and icrawler
beautifulsoup4 crawler icrawler image-crawler pyqt pyqt5 pyqt5-desktop-application
Last synced: 14 Dec 2025
https://github.com/darealfreak/figure-tracker
application to keep watch of wished figures on multiple sites and notify you about auctions, sales or sudden price drops
crawler figure-tracker monitoring
Last synced: 30 Mar 2025
https://github.com/saadali1996/goose-rest-api
https://github.com/advancedlogic/GoOse based REST API for article content extraction
Last synced: 09 Mar 2026
https://github.com/dean9703111/shopee_find_mac
用最快的速度找到便宜符合自己要求規格的mac
argparse crawler mac pip python python2 xlsxwriter
Last synced: 14 Apr 2026
https://github.com/zabuzard/songcrawler
Crawles all song files available on 'http://downloads.khinsider.com/'. Creates a list of direct download links for all such songs, intended for use with JDownloader or similar.
command-line-tool crawler download-musics downloadmanager jdownloader multithreading song-files songs web-crawler
Last synced: 09 Jun 2026
https://github.com/exasol/error-code-crawler-maven-plugin
Validator and crawler for exasol-error-codes in Java code
catalog crawler error-handling error-report error-reporting exasol exasol-integration java unification
Last synced: 13 Oct 2025
https://github.com/fabrix-app/spool-scraper
Spool: Webscraper
cheerio crawler fabrix nodejs scraping spools typescript webscraper
Last synced: 09 May 2026
https://github.com/mzazakeith/puppetmaster
Puppeteer & Crawl4AI microservice for web automation, scraping, and AI processing with Bull queues
agent ai automation bull bullmq chrome crawl4ai crawler data data-extraction extraction gemini llm llms openai playwright puppeteer web-automation
Last synced: 13 May 2025
https://github.com/maxbubblegum47/spotydump
Spotify Scraper combined with a Genius Scraper. Scrape artist of a certain period of time/region of the world and dump all their songs!
crawler dump genius lyrics python spotify unimore-informatica
Last synced: 22 Mar 2025
https://github.com/pedrohs1771/hyenzy-x-anime-scraper
A powerful all-in-one media scraper for Anime and Games with 4K Upscale (MPV) and Discord RPC.
anime-scrapper anime4k crawler discord-rpc game-downloader mpv-player playwright python upscale
Last synced: 30 May 2026
https://github.com/merrier/overwatch-spider
:beetle: Overwatch Spider with NodeJS + node-crawler
crawler javascript jquery nodejs overwatch spider
Last synced: 29 Mar 2025
https://github.com/spaceemotion/goodreads-browser
Custom crawler + interface to have better filtering and sorting of the goodreads database 📚🔍
Last synced: 22 Jan 2026
https://github.com/anjackson/scrapy-url-frontier
A Scrapy module for URL Frontier integration
crawler frontier scrapy spider
Last synced: 23 Feb 2025
https://github.com/santhin/real-estate
Real estate crawler with ML on scraped data
crawler jupyter-notebook ml real-estate scrapy
Last synced: 17 Mar 2025
https://github.com/jiannei/github-trending
Github trending crawling based on lumen.
crawler github-trending lumen php
Last synced: 30 Apr 2025
https://github.com/dr4g0nsr/sitemap-crawler
Crawler using sitemap to crawl site/regenerate cache
Last synced: 13 Jan 2026
https://github.com/nzrsky/useragent-generator
High-performance User-Agent generator for Go. Zero-alloc bots, auto-updated browser versions from real usage data.
bot browser crawler go golang http scraping user-agent useragent
Last synced: 14 Apr 2026
https://github.com/btheu/estivate
Mapping from DOM to POJO with CSS Query Syntax and annotations.
annotations crawler csquery css-selector html java jsoup jsoup-operation pojo
Last synced: 12 Jan 2026
https://github.com/supadata-ai/py
Official Python SDK for the Supadata API.
ai api crawler llm markdown scraping sdk transcript web-scraper youtube
Last synced: 22 Mar 2025
https://github.com/epigos/newsbot
A news bot written in Go for Dialogflow and Facebook messenger
autocert chatbot crawler datastore dialogflow facebook-messenger-bot golang letsencrypt newsfeed
Last synced: 22 Mar 2025
https://github.com/omkarcloud/dentalkart-scraper
🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖
beautifulsoup crawler crawling crawling-framework crawling-python dentalkart dentalkart-product-scraper dentalkart-scraper dentalkart-scraping node-crawler scraper scraping scraping-framework scraping-python selenium web-crawler web-crawling web-scraper web-scraping webscraping
Last synced: 07 Sep 2025
https://github.com/fernandod1/yahoo-finance-scraper
This python script scraps "Open" and "Previous Close" values from any company in Yahoo Finance and save them in a local text file.
crawler python python3 scraper scraping scraping-websites scrapper scrapping spider yahoo yahoo-finance yahoo-finance-api
Last synced: 24 Aug 2025
https://github.com/qiubits2007/xml-sitemap
Multi-domain XML sitemap generator with support for robots.txt, meta tags, email logging & search engine pinging
crawler generator gzip multi-domain php8 robots-txt seo seotools sitemap-builder sitemap-generator sitemap-xml
Last synced: 25 Feb 2026
https://github.com/ian-lin8239/javdb_magnet
JavDB 磁力鏈接專用工具 - 自動獲取有碼月榜前30部影片的磁力鏈接，支持智能過濾和多格式導出
cli-tool crawler javdb magnet-link python web-scraping
Last synced: 12 Feb 2026
https://github.com/kokseen1/chii
A minimal marketplace bot maker.
auction automation bidding bot carousell crawler ecommerce marketplace python python-telegram-bot scraper telegram telegram-bot web-scraping yahoo yahoo-auction
Last synced: 20 Aug 2025
https://github.com/pinpox/go-random-downloader
Download Html using "Random Page"
Last synced: 17 Aug 2025
https://github.com/1970mr/link-crawler
Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.
clawler crawler crawler-python link-crawler link-crawler-python link-scraper link-scraper-python links python scraper scraper-python website-crawler website-scraper
Last synced: 06 Feb 2026
https://github.com/viper373/xovideos
一个为用户打造的个性化视频下载工具
boto3 crawler downloader githubactions m3u8 mongodb mp4 pornhub python s3-storage
Last synced: 16 Jun 2025
https://github.com/roccomuso/is-twitter
Verify that a request is from Twitter crawlers using DNS verification steps
bot crawler dns ip js nodejs twitter verification
Last synced: 14 Sep 2025
https://github.com/qin2dim/istockphoto-go
📸 Gracefully download dataset from iStockPhoto.
Last synced: 05 Apr 2025
https://github.com/eklem/browsercrawler
Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.
crawler search-engine website-generation
Last synced: 02 Aug 2025
https://github.com/nemmusu/free-vpn-downloader
This repository contains three Python scripts designed to simplify the process of downloading and configuring free VPN .ovpn files for use with OpenVPN.
automation crawler download downloader free freevpn openvpn ovpn ovpn-files vpn
Last synced: 07 Feb 2026
https://github.com/thecloer/crawler-himym
How I met your mother script PDF generator for learning English
crawler pdf pdf-generation typescript web-scraping webscraping
Last synced: 21 Jun 2025
https://github.com/highbreed/web-crawler
A web crawler script that crawls the target website and lists its links
Last synced: 07 Jun 2026
https://github.com/arif98741/deadlink-checker-python
A Python tool to crawl websites and check for broken/dead links with detailed reporting in both text and PDF formats.
crawler crawling python python3 website-scraper
Last synced: 18 Apr 2026
https://github.com/carloocchiena/python_url_crawler
A script that starting from a webpage, iterate thru all its link, appending them in a list. Sort of proxy to get all pages in a website
beautifulsoup crawler python python3
Last synced: 12 Feb 2026
https://github.com/stevieflyer/quokka
An easy-to-use web crawler framework, supporting parallel crawling without a line of code and headless running.
crawler parallel web-automation
Last synced: 12 Jul 2025
https://github.com/nbdy/prntscrngrb
prnt.sc / lightshot crawler, nudity detection and text extraction to a sqlite database
crawler nudity-detection prntsc text-extraction
Last synced: 04 Oct 2025
https://github.com/skulltech/arachnid
Crawling Instagram for reasons.
crawler instagram instagram-scraper python3 scraper scrapy
Last synced: 13 Jun 2025
https://github.com/becky-dai/flower-knowledge-graph-visualization
A full stack program of knowledge graph visualization 一个关于知识图谱可视化的全栈项目
crawler css django echarts html js knowledge-graph neo4j python
Last synced: 11 Mar 2026
https://github.com/nazanin1369/searchengine
Implementing a search engine using Java, AngularJS and Elastic search
angularjs crawler elasticsearch java search-engine
Last synced: 12 Apr 2026
https://github.com/benderpan/fakeagent.net
Fake Agent for .Net Standard.
agent crawler fake-agent http-headers
Last synced: 12 Apr 2025
https://github.com/lucky845/animetimeline
使用Python脚本爬取动漫信息时间表，并保存为Markdown文件。
Last synced: 09 Jul 2025
https://github.com/raspi/scrapy-kuntavaalit2021-yle
Fetch YLE kuntavaalit 2021 data
crawler mirror python scrapy spider webcrawler
Last synced: 26 Apr 2025
https://github.com/wangyihang/acw-sc-v2-py
Python requests.HTTPAdapter for `acw_sc__v2`
Last synced: 18 Jun 2026
https://github.com/marvnc/pixiv-dump
Pixiv Encyclopedia DB Dumps, updated daily
crawler database dump encyclopedia japanese pixiv pixiv-crawler pixiv-database scraping
Last synced: 12 Jan 2026
https://github.com/tufayellus/linkedin-cv-downloader
A Python based GUI automation software for downloading bulk LinkedIn CV / LinkedIn Resume from a list of profile links
crawler digital-marketing email-marketing email-scraper leads linkedin-bot linkedin-cv linkedin-cv-downloader linkedin-download linkedin-downloader linkedin-resume linkedin-resume-downloader linkedin-scraper scrape-emails scrape-websites scraper scraper-engine
Last synced: 17 Mar 2025
https://github.com/basemax/fakefaces
This repository contains a crawler that downloads thousands of fake human face images from various sources on the internet. Additionally, the repository includes a dataset of thousands of face images of fake humans.
crawler crawler-php crawler-testing crawlers curl dataset datasets face face-fake faces fake-face fake-faces php php-curl
Last synced: 27 Apr 2026
https://github.com/keosariel/ramby
Ramby is a simple way to setup a webscraper
beautifulsoup crawler python3 webscraping
Last synced: 27 Mar 2025
https://github.com/foufou-exe/yspeed
Yspeed is a library that scrapes the Speedtest site
crawler python rich scraper scraping selenium selenium-python speedtest
Last synced: 28 Feb 2026
https://github.com/camara94/crawlers
Web scraping is the process of using bots to extract content and data from a website. Unlike screen scraping, which only copies pixels displayed onscreen, web scraping extracts underlying HTML code and, with it, data stored in a database. The scraper can then replicate entire website content elsewhere
crawler python scraping scrapy spider
Last synced: 09 Apr 2025
