Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
- GitHub: https://github.com/topics/crawler
- Wikipedia: https://en.wikipedia.org/wiki/Web_crawler
- Last updated: 2026-06-22 00:06:47 UTC
- JSON Representation
https://github.com/xiantang/mini_scrapy
模仿scrapy的轻量级爬虫框架
crawler python3 requets scrapy
Last synced: 27 Mar 2025
https://github.com/truethari/fcrawler
Python application that can be used to copy files of a given file type from a folder directory.
copy copy-files crawl crawler crawler-python file files
Last synced: 25 Feb 2025
https://github.com/kahsolt/allchan
An image crawler for xChan(4chan/8ch/...) image board.
4chan 4chan-downloader 8chan crawler image-crawler
Last synced: 12 Jun 2026
https://github.com/yukito0209/is6941-ml-social-media
IS6941 Machine Learning & Social Media Analytics 课程小组项目代码仓库，探索机器学习在社交媒体数据分析中的应用。
bert city-university-of-hong-kong crawler data-collection llama machine-learning python sentiment-analysis social-media
Last synced: 01 Apr 2025
https://github.com/liebki/githubnet
This library allows you to retrieve several things from GitHub, things like trending repositories, profiles of users, the repositories of users and related information.
crawler crawling github github-trending htmlagilitypack microsoft
Last synced: 20 May 2026
https://github.com/thiiagoms/dict-crawler
Simple crawler on UOL dictionary
beautifulsoup4 crawler dic python pythonic
Last synced: 26 May 2026
https://github.com/Juphex/SupremeBot
Demonstrates automated purchasing of the clothing brand "Supreme". This was a fun project and had no further application.
android chrome crawler kivy python3 webscraping windows
Last synced: 10 Mar 2025
https://github.com/nakabonne/netsurfer
netsurfer is a very lightweight scraping framework
Last synced: 01 Apr 2025
https://github.com/spraakbanken/svt-crawler
Programme for crawling SVT's API for news articles and converting the data to XML.
Last synced: 07 Mar 2026
https://github.com/codeforequity-at/botium-crawler
Botium Crawler - Like a Website Crawler, just for Conversation Flows
Last synced: 23 Apr 2025
https://github.com/jemaf/stackoverflow-jobs
A wrapper for crawling data at Stack Overflow Jobs portal
crawler jobs python stack-overflow
Last synced: 14 Jan 2026
https://github.com/empire/go-tse
Go client for http://www.tsetmc.com/
crawler financial-data financial-data-analysis market-data stock stock-prices tehran-stock-exchange tse tsetmc
Last synced: 01 Jun 2026
https://github.com/anyparser/anyparserjs
Anyparser Typescript SDK for RAG/ETL Pipelines - File Content Extraction. Supports extraction from various file formats including PDF, Microsoft Office documents, OCR/Image to Text, Audio to Text, and Website to Text.
anyparser artificial-intelligence cache-augmented-generation crawler etl-pipeline graph-rag knowledgebase langchain microsoft-office microsoft-word ms-office n8n-nodes ocr pdf-extraction rag retrieval-augmented-generation text-extraction web-crawler
Last synced: 17 Feb 2026
https://github.com/vmandic/tris-web-crawler
Tris is a simple NodeJS web crawler tool to help you collect links from visited links of a website's domain.
crawler data-tools nodejs scraping seo-tools web-scraper
Last synced: 20 May 2026
https://github.com/panyanyany/vps_spider
VPS Spider powering https://findallvps.com
Last synced: 28 Feb 2025
https://github.com/zhuruili/spider
一些简单的爬虫代码，会不定时更新，希望能帮到你
crawler drissionpage python requests
Last synced: 18 Mar 2025
https://github.com/jxeng/site-info-crawler
A tool for batch crawling website's title, description, favicon.
Last synced: 30 May 2026
https://github.com/liinen/vocalist-backend
vloom backend implementation in cloud service, with crawling dataset from karaoke website
connection-pool crawler express mysql ncloud-server pagination python3 selenium
Last synced: 13 Apr 2026
https://github.com/first-coding/django-and-web
This is a django and Web front - and back -end separation project.
Last synced: 16 Feb 2026
https://github.com/reisdev/reads
Real Estate Agency Data Scraper
crawler python scraping scrapy selenium-python selenium-webdriver spider
Last synced: 31 Jan 2026
https://github.com/yidas/tw-stock-crawler-php
PHP Crawler for Taiwan Stock Data (台股資料爬蟲)
crawler stock taiwan taiwan-stock-information taiwan-stock-market
Last synced: 25 Mar 2025
https://github.com/travorlzh/temperature-analyzer
Python crawler that helps fetch temperature of Beijing, China
crawler homework python variance
Last synced: 25 Aug 2025
https://github.com/runnin-n-gunnin/geckofxinterceptrequestcaptureresponse
[GeckoFX/Firefox]: Shows how to Intercept request(s), capture response(s), customize GeckoPreferences, handle certificate errors, change useragent++.
browser cefsharp controls crawler crawling firefox gecko geckofx geckofx60 scraping webbrowser windows windowsforms winforms
Last synced: 06 Feb 2026
https://github.com/madis/flatcrawl
Clojure app for crawling apartment information from http://kv.ee
clojure crawler real-estate webapp
Last synced: 05 Jul 2025
https://github.com/zekrotja/r34-crawler
A simple CLI tool to fetch and download images from rule34.xxx
crawler go rest-api rule34 worker-pool xml
Last synced: 06 Mar 2026
https://github.com/jofaval/webscraping
WebScraper providing tools to scrape tons of websites with the same base
crawler e-commerce python scraper webscraper webscraping
Last synced: 06 Oct 2025
https://github.com/congcoi123/crawler-sheis
A small crawler for getting data from the website: https://sheis.vn
crawler webcrawler webcrawling webscraper webscraping
Last synced: 25 Feb 2026
https://github.com/jjlibra/bake-mediacrawler
NanmiCoder‘s self-media data crawling software
Last synced: 06 May 2025
https://github.com/superreal/octopus
Recursive and multi-threaded broken link checker
Last synced: 14 May 2026
https://github.com/nextlevelshit/fick
Fucking Incredible Command line King. Add CLI flavour to any website you like to.
Last synced: 17 Feb 2026
https://github.com/krishpranav/spider
A ruby web spidering tool that can spider a site, multiple domains, certain links or infinitely
crawler ruby spider web-crawler web-scraping
Last synced: 14 Feb 2026
https://github.com/microlinkhq/ua
A simple redis primitives to incr() and top() user agents
crawler redis user-agent user-agent-parser
Last synced: 18 Mar 2026
https://github.com/denrydu/baiduimagecrawler
自己写的两个用来爬取百度图片的脚本，方便CV研究者制作数据集。Two ways to download images from baidu, useful tool for making cv datasets!
Last synced: 04 Nov 2025
https://github.com/imthaghost/gocloneold
Website Cloner - Utilizes powerful go routines to clone websites to your computer within seconds.
Last synced: 05 Apr 2025
https://github.com/galaxiat/galaxiat.serve.seo
Node.JS package to serve React app and prerender path (cron)
crawler cron puppeteer seo seo-optimization ssr
Last synced: 31 Jan 2026
https://github.com/h-alice/piday-crawler
This is a little script to get pi digits and pretty print.
Last synced: 25 Jun 2025
https://github.com/marcus-v-freitas/crawlerbrazilgovdata
Projeto ASP.NET Core .NET 5 para Extração e Parseamento de Dados do governo de São Paulo com integração com Buckets S3, Filas SQS AWS e Persistência realizada via EF Core no Mysql.
api-rest aspnetcore automapper aws crawler csharp efcore government-data htmlagilitypack linux memory-cache mysql net5 onion-architecture parallel-computing parser s3-bucket serilog sqs-queue swagger
Last synced: 17 Jan 2026
https://github.com/sangupta/shopify-burst-crawler
Simple crawler to download meta information for all stock pics from Shopify Burst website
burst crawler java shopify stock-photos
Last synced: 18 Feb 2026
https://github.com/erikmueller/jazmax
Crawl JAZ for different heat pumps depending on flow and return temperatures from the JAZ calculator
crawler data-science efficiency green heatpump jaz
Last synced: 24 Mar 2025
https://github.com/antoinegagne/treewalker
A web crawler in Erlang that respects `robots.txt`.
Last synced: 11 Feb 2026
https://github.com/joelkoen/wls
Easily crawl multiple sitemaps and list URLs
Last synced: 12 Apr 2025
https://github.com/arpan404/spidey
Spidey is a powerful asynchronous web crawler built in Python that can crawl websites and download files with specified extensions. It's designed to be efficient, configurable, and easy to use.
asynchronous crawler dataminer opensource python webscraper
Last synced: 04 Feb 2026
https://github.com/amirhoseinsalimi/boxapi-python
Python client for https://boxapi.ir to crawl and read Instagram data.
crawler instagram instagram-api python python3
Last synced: 26 May 2026
https://github.com/afsh7n/crawly-automation
Crawly Automation is a lightweight, modular, and extensible web crawling framework built on top of Puppeteer. Whether you need to scrape data, automate browser interactions, manage CAPTCHAs, or handle advanced data extraction, Crawly Automation simplifies the process.
automation crawler nodejs puppeteer webscraping
Last synced: 25 Feb 2026
https://github.com/xcrypt0r/watchdog
🐶 Dcinside image crawler that includes NSFW detection (Enhanced version of Hyacinth)
crawler crawling dc dcinside nodejs nsfw parsing tensorflow
Last synced: 18 May 2026
https://github.com/sieep-coding/web-crawler
A simple web crawler implemented in Go.
Last synced: 09 Mar 2026
https://github.com/zanmato/shouting-robin
SEO Crawler focused on E-commerce
crawler developer-tools seo seo-tools
Last synced: 21 Jun 2026
https://github.com/eduardozepeda/go-web-crawler
A concurrent web crawler written in go that looks for exposed .git and .env uris.
crawler environment-variables git go pentesting security-audit
Last synced: 16 Apr 2026
https://github.com/cyberdolfi/serverrawler
ServerRawler is a Minecraft Server Crawler, written in Rust
crawler minecraft ratatui-rs rust seeker servercrawler serverseeker
Last synced: 04 Mar 2026
https://github.com/rebrowser/stubhub-dataset
StubHub secondary ticket market data: event listings with section, row, quantity, delivery type, ticket class, and 500+ venues across US, Canada, and Europe. Updated daily.
concert-tickets crawler data-collection data-science dataset event-tickets live-events open-data resale-tickets scraper secondary-market sports-tickets stubhub tickets web-scraping
Last synced: 03 May 2026
https://github.com/shunk031/lineblogscraper
Scraper for LINE Blog in Scrapy
crawler lineblog scraper scrapy
Last synced: 17 Jun 2026
https://github.com/polakosz/smf-scraper
You know, just for backup :smile: - The only so the best Simple Machines Forum C# scraper on GitHub :cat:
crawler csharp forum machines php scraper simple simplemachines smf
Last synced: 30 Apr 2026
https://github.com/viclafouch/pe-crawler
📌 An automated system that serves data extracted from the Google Help Center
crawler javascript nodejs postgresql sequelize
Last synced: 17 Apr 2026
https://github.com/YGGverse/pulsarss
RSS Aggregator for Gemini Protocol
aggregator cli crawler daemon feed gemini gemini-protocol gemtext parser rss rust
Last synced: 15 Jun 2026
https://github.com/natshah/natshah-crawler
Natshah Crawler works to crawl a selected domain with all it's internal links and internal pages.
crawler database filter natshah-crawler
Last synced: 29 Apr 2026
https://github.com/marabesi/social-crawler
Easy way to find emails from social networks
crawler emails php social-crawler social-network
Last synced: 02 Mar 2026
https://github.com/kapitanluffy/sunny-crawler
That moment when I tried learning things about "Big Data" and "Inverted Indexes"
big-data crawler inverted-index php search
Last synced: 30 Apr 2026
https://github.com/devkoriel/teslalarm-kr
🚀 Teslalarm KR Real-time, AI-powered Tesla news & price alerts tailored for the Korean market. Stay updated on price changes, new model releases, and more – delivered directly to your Telegram. 🔔 Join us and help revolutionize Tesla news in Korea!
Last synced: 04 Apr 2026
https://github.com/mashukui/xhs_pic_tool
用python开发的小红书图片采集软件，支持下载小红书笔记无水印图片、采集笔记数据、评论数据等。小红书爬虫|小红书无水印图片|小红书无水印下载|小红书评论爬虫|小红书采集工具|小红书评论采集|小红书采集软件|小红书爬取数据|xiaohongshu|xhs|XHS
crawler gui gui-application python-spider spider xhs xhs-downloader xhs-spider xiaohongshu xiaohongshu-downloader
Last synced: 04 Apr 2026
https://github.com/restuwahyu13/node-scraper-content
example node scraper all content programming using puppeteer
crawler nodejs puppeter scrapper
Last synced: 14 May 2026
https://github.com/ewertoncodes/mind-crawler
A simple api written in Rails to extract quotations from the Quotes to Scrape site.
Last synced: 14 May 2026
https://github.com/arshamroshannejad/scrapify
Scrapify is a golang library that automates the process of bypassing CAPTCHAs, enabling efficient web scraping and data acquisition.
403-bypass arkose cloudflare crawler golang http-client scraper
Last synced: 18 Apr 2026
https://github.com/mwoss/mors
Application of topic models for information retrieval and search engine optimization.
common-crawl crawler django doc2vec gensim hacktoberfest lda python scrapy search search-engine tfidf
Last synced: 19 Apr 2026
https://github.com/nava45/simplempcrawler
Simple Multiprocessing Crawler in python
crawler multiprocessing python
Last synced: 22 Jun 2026
https://github.com/coverified/spider
A microservice with web-crawler/spider capabilities which only follows and indexes urls of the provided host domain(s)
akka crawler graphql hacktoberfest microservice spider
Last synced: 29 Apr 2026
https://github.com/buaadreamer/buaastar
北航星球网站 北航2021年夏季学期Python英文课大作业
crawler css flask html javascript python
Last synced: 28 Apr 2026
https://github.com/eduardosbcabral/desafio-tecnico-mp
Desafio - Gerador de arquivos em C# utilizando Web Crawler e Buffers para a escrita do arquivo em disco.
Last synced: 08 May 2026
https://github.com/johnvanderton/flysh
HTML web parser powered by jQuery and JSDOM
crawler crawler-engine dom dom-manipulation html javascript javascript-library jquery jsdom parser-library scraper typescript typescript-library web-crawler web-parser
Last synced: 03 Mar 2026
https://github.com/yuminn-k/crawling-tabelog
Crawling store information from tabelog
Last synced: 08 Jun 2026
https://github.com/ph-7/gettermails
GetterMails, Scraper
bot crawler email php python retrieve-web-page scrape scraper scraping scraping-websites scrapper webdriver
Last synced: 20 Apr 2026
https://github.com/maraf/staticsitecrawler
A simple util for crawling links from root URL and saving HTML documents.
Last synced: 21 Apr 2026
https://github.com/manojahi/is-there-any-song-reference-in-article
It will tell if there are any songs references in article from a website.
crawler lyrics-search python webscraping
Last synced: 28 Mar 2026
https://github.com/santhoshse7en/alcoholics-anonymous
Research Project to analyse the knowledge about Alcoholics Anonymous in public
aa-meetings alcoholics alcoholics-anonymous anonymous bs4 crawler data-extraction-and-pre-processing google-search-using-python news-crawler newspaper3k python the-hindu web-scraping without-api
Last synced: 07 May 2026
https://github.com/elky84/lol-crawler
Notification from LOL friend game start & end.
crawler csharp docker dotnet web-crawler
Last synced: 07 May 2026
https://github.com/ysh329/stock-newspaper-crawler
[UNMAINTAINED]Crawl 4 kinds of finance newspaper corpus (from CCSTOCK.CN).
corpus crawled-data crawler database stock-newspaper-crawler
Last synced: 28 Apr 2026
https://github.com/gnujoow/crawl-repo
crawling github's repositories basic info
crawler github github-api python3
Last synced: 03 May 2026
https://github.com/dean9703111/ithelp_total_count
計算 IT邦幫忙文章的瀏覽/Like/留言總數
crawler ithelp total-likes total-responses total-views
Last synced: 07 May 2026
https://github.com/alquipo/dragoongames-api
Dragoon Games Store - API (E-commerce)
crawler e-commerce-project graphql jsdom nodejs postgresql strapi-cms stripe
Last synced: 16 Apr 2026
https://github.com/tctien342/simple-doc-crawler
Craw all sub page from given URL to markdown
Last synced: 03 Mar 2026
https://github.com/sarvarbekup/searchin_v1
Seach system
backend crawler crawling crawling-go frontend go golang nextjs search-engine searching
Last synced: 16 Apr 2026
https://github.com/emarifer/search-engine
A mini Google. Custom web crawler & indexer written in Golang.
crawler dashboard deep-first-search fiber-framework full-text-search golang gorm-orm htmx htmx-go hyperscript indexer inverted-index response-caching search-engine templ worker-pool
Last synced: 16 Apr 2026
https://github.com/poodle64/supacrawl
Zero-infrastructure web scraping for the terminal
cli crawler llm markdown playwright python scraper terminal web-scraping
Last synced: 04 Mar 2026
