Crawler | Ecosyste.ms: Awesome

https://github.com/fernandod1/yahoo-finance-scraper

This python script scraps "Open" and "Previous Close" values from any company in Yahoo Finance and save them in a local text file.

crawler python python3 scraper scraping scraping-websites scrapper scrapping spider yahoo yahoo-finance yahoo-finance-api

Last synced: 24 Aug 2025

https://github.com/tvrcgo/collect

数据采集

crawler scraper

Last synced: 06 Apr 2025

https://github.com/sachin-kumar-2003/seocrawler

SEO Link Checker | Find Broken Links & Improve SEO I have built an SEO Link Checker that helps businesses, marketers, and site owners scan their websites, detect broken or harmful links, and fix them fast. This improves site health, user experience, and search rankings. Features: -Scan entire website for broken internal and external links

beautifulsoup crawler fastapi reactjs seo seo-optimization

Last synced: 15 Apr 2026

https://github.com/galaxiat/galaxiat.serve.seo

Node.JS package to serve React app and prerender path (cron)

crawler cron puppeteer seo seo-optimization ssr

Last synced: 31 Jan 2026

https://github.com/telanflow/scrago

A micro crawler framework. achieved by GOLANG.

crawler go micro-framework spider

Last synced: 25 Jun 2025

https://github.com/soakit/book-download

book-download

crawler html2epub nodejs novel-downloader

Last synced: 01 May 2026

https://github.com/tikazyq/colly-crawlers

Crawlers using Golang-based web crawling framework Colly

crawler

Last synced: 15 Jun 2025

https://github.com/supadata-ai/py

Official Python SDK for the Supadata API.

ai api crawler llm markdown scraping sdk transcript web-scraper youtube

Last synced: 22 Mar 2025

https://github.com/eeriemyxi/nosori

Online image viewer for https://coomer.su and https://kemono.su

api coomer crawler docker image javascript kemono server typescript video viewer web

Last synced: 01 Aug 2025

https://github.com/andreoliwa/scrapy-tegenaria

🕷🕸 Spiders to crawl ads of houses and apartments. 🏠 🏢

crawler flask postgresql python python3 scrapy

Last synced: 13 Apr 2025

https://github.com/mushoffa/scrapy-tokopedia-python

crawler python scraping scrapy spider tokopedia

Last synced: 13 Jul 2025

https://github.com/kokseen1/chii

A minimal marketplace bot maker.

auction automation bidding bot carousell crawler ecommerce marketplace python python-telegram-bot scraper telegram telegram-bot web-scraping yahoo yahoo-auction

Last synced: 20 Aug 2025

https://github.com/litingyes/cobweb

Collect, store and distribute meaningful static data

apis bing-image bing-wallpapers crawler image random-image

Last synced: 31 Jul 2025

https://github.com/akagi201/spy

A lightweight distributed web crawler

crawler distributed lightweight nsq

Last synced: 26 Feb 2025

https://github.com/r3c0ger/liscaps

A LSTM-based intelligent stock crawl, analysis and prediction system.

crawler lstm python pytorch stock streamlit

Last synced: 27 Jan 2026

https://github.com/zabuzard/mplogger

Saves marketprices for items, based on transactions, from the game 'http://www.freewar.de/' in a database by using a bot. Then processes the data and creates corresponding market price articles in 'http://www.fwwiki.de/'.

bot crawler database game mediawiki-api mmorpg mmorpg-freewar php saves-marketprices web-crawler wikipedia-api

Last synced: 03 Feb 2026

https://github.com/liuzl/newsmth

A go crawler for newsmth.net

bigdata crawler newsmth nlp

Last synced: 25 Jun 2026

https://github.com/nbdy/prntscrngrb

prnt.sc / lightshot crawler, nudity detection and text extraction to a sqlite database

crawler nudity-detection prntsc text-extraction

Last synced: 04 Oct 2025

https://github.com/joeri-abbo/python-credly-scraper

This project is a set of Python scripts designed to crawl and extract data from the Credly platform, focusing on skills, organizations, and badges. The scripts allow users to perform searches using command-line arguments, predefined search terms, or skills listed in a JSON file. The collected data is then saved to JSON files for further analysis an

badges crawler credly data-extraction json organizations python python3 requests-library skills web-crawling

Last synced: 23 Sep 2025

https://github.com/jdx/mise-java

A JVM Data Crawler

cli crawler jvm

Last synced: 15 Mar 2026

https://github.com/foufou-exe/yspeed

Yspeed is a library that scrapes the Speedtest site

crawler python rich scraper scraping selenium selenium-python speedtest

Last synced: 28 Feb 2026

https://github.com/madis/flatcrawl

Clojure app for crawling apartment information from http://kv.ee

clojure crawler real-estate webapp

Last synced: 05 Jul 2025

https://github.com/jxeng/site-info-crawler

A tool for batch crawling website's title, description, favicon.

crawler favicon title

Last synced: 30 May 2026

https://github.com/darealfreak/figure-tracker

application to keep watch of wished figures on multiple sites and notify you about auctions, sales or sudden price drops

crawler figure-tracker monitoring

Last synced: 30 Mar 2025

https://github.com/coghost/izen

encapsulation of some useful features

chaos crawler encrypt izen mqtt profig python3 utils

Last synced: 30 Apr 2025

https://github.com/viper373/xovideos

一个为用户打造的个性化视频下载工具

boto3 crawler downloader githubactions m3u8 mongodb mp4 pornhub python s3-storage

Last synced: 16 Jun 2025

https://github.com/epigos/newsbot

A news bot written in Go for Dialogflow and Facebook messenger

autocert chatbot crawler datastore dialogflow facebook-messenger-bot golang letsencrypt newsfeed

Last synced: 22 Mar 2025

https://github.com/qiaocco/crawler

爬虫：百度贴吧、今日头条（阳光宽频网）、笔趣阁

crawler python

Last synced: 27 Mar 2025

https://github.com/santhin/real-estate

Real estate crawler with ML on scraped data

crawler jupyter-notebook ml real-estate scrapy

Last synced: 17 Mar 2025

https://github.com/joelkoen/wls

Easily crawl multiple sitemaps and list URLs

crawler sitemap url

Last synced: 12 Apr 2025

https://github.com/arpan404/spidey

Spidey is a powerful asynchronous web crawler built in Python that can crawl websites and download files with specified extensions. It's designed to be efficient, configurable, and easy to use.

asynchronous crawler dataminer opensource python webscraper

Last synced: 04 Feb 2026

https://github.com/ging-dev/sitemap-crawler

Collect links through the sitemap.xml or robots.txt

crawler php php8 sitemap sitemap-crawler

Last synced: 10 Jan 2026

https://github.com/travorlzh/temperature-analyzer

Python crawler that helps fetch temperature of Beijing, China

crawler homework python variance

Last synced: 25 Aug 2025

https://github.com/omkarcloud/dentalkart-scraper

🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖

beautifulsoup crawler crawling crawling-framework crawling-python dentalkart dentalkart-product-scraper dentalkart-scraper dentalkart-scraping node-crawler scraper scraping scraping-framework scraping-python selenium web-crawler web-crawling web-scraper web-scraping webscraping

Last synced: 07 Sep 2025

https://github.com/zhaoweih/meizitu-crawler

🕷️妹子图爬虫-Scrapy

crawler meizitu python scrapy spider

Last synced: 11 Apr 2025

https://github.com/anyparser/anyparserjs

Anyparser Typescript SDK for RAG/ETL Pipelines - File Content Extraction. Supports extraction from various file formats including PDF, Microsoft Office documents, OCR/Image to Text, Audio to Text, and Website to Text.

anyparser artificial-intelligence cache-augmented-generation crawler etl-pipeline graph-rag knowledgebase langchain microsoft-office microsoft-word ms-office n8n-nodes ocr pdf-extraction rag retrieval-augmented-generation text-extraction web-crawler

Last synced: 17 Feb 2026

https://github.com/pedrohs1771/hyenzy-x-anime-scraper

A powerful all-in-one media scraper for Anime and Games with 4K Upscale (MPV) and Discord RPC.

anime-scrapper anime4k crawler discord-rpc game-downloader mpv-player playwright python upscale

Last synced: 30 May 2026

https://github.com/truethari/fcrawler

Python application that can be used to copy files of a given file type from a folder directory.

copy copy-files crawl crawler crawler-python file files

Last synced: 25 Feb 2025

https://github.com/0fatal/zjxxc-crawl

在浙学爬虫：作业情况和登录

crawler

Last synced: 03 Apr 2025

https://github.com/ductnn/curls

Simple tool crawler URLs form domain

colly crawler domain golang scanning url

Last synced: 04 Apr 2025

https://github.com/arif98741/deadlink-checker-python

A Python tool to crawl websites and check for broken/dead links with detailed reporting in both text and PDF formats.

crawler crawling python python3 website-scraper

Last synced: 18 Apr 2026

https://github.com/roccomuso/is-twitter

Verify that a request is from Twitter crawlers using DNS verification steps

bot crawler dns ip js nodejs twitter verification

Last synced: 14 Sep 2025

https://github.com/liebki/githubnet

This library allows you to retrieve several things from GitHub, things like trending repositories, profiles of users, the repositories of users and related information.

crawler crawling github github-trending htmlagilitypack microsoft

Last synced: 20 May 2026

https://github.com/benderpan/fakeagent.net

Fake Agent for .Net Standard.

agent crawler fake-agent http-headers

Last synced: 12 Apr 2025

https://github.com/laurybueno/crawler-olhovivo

Coletor de dados mapeáveis do transporte público de ônibus em São Paulo

api crawler docker olhovivo python sptrans

Last synced: 27 Apr 2026

https://github.com/codeforequity-at/botium-crawler

Botium Crawler - Like a Website Crawler, just for Conversation Flows

botium chatbots crawler

Last synced: 23 Apr 2025

https://github.com/airtoxin/stackable-crawler

middleware based lightweight crawler framework

crawler javascript lightweight

Last synced: 13 Apr 2025

https://github.com/pinpox/go-random-downloader

Download Html using "Random Page"

crawler golang

Last synced: 17 Aug 2025

https://github.com/btheu/estivate

Mapping from DOM to POJO with CSS Query Syntax and annotations.

annotations crawler csquery css-selector html java jsoup jsoup-operation pojo

Last synced: 12 Jan 2026

https://github.com/lucassmacedo/tvtime-scrapy

A simple crawly to get personal watched movies/series/animes from my user profile

crawler phython scrapy tvtime

Last synced: 10 Oct 2025

https://github.com/wangyihang/acw-sc-v2-py

Python requests.HTTPAdapter for `acw_sc__v2`

acw-sc-v2 crawler waf

Last synced: 18 Jun 2026

https://github.com/pjt3591oo/exchange-crawler

업비트, 코인원 크롤러

crawler data exchange python

Last synced: 27 Oct 2025

https://github.com/ahmed-kamal2004/aurora_searchengine

crawler indexer jar java maven mongodb ranking-algorithm search-engine spring spring-boot spring-data-jpa spring-data-mongodb springboot

Last synced: 12 Apr 2025

https://github.com/norconex/committer-neo4j

Implementation of Norconex Committer for Neo4j.

crawler neo4j neo4j-committer norconex-committer

Last synced: 19 Jan 2026

https://github.com/woojubb/link-collector

웹페이지 주소 및 RSS를 크롤링 해주는 프로그램

crawler crawling rss

Last synced: 28 May 2026

https://github.com/superreal/octopus

Recursive and multi-threaded broken link checker

broken checker crawler links

Last synced: 14 May 2026

https://github.com/tufayellus/linkedin-cv-downloader

A Python based GUI automation software for downloading bulk LinkedIn CV / LinkedIn Resume from a list of profile links

crawler digital-marketing email-marketing email-scraper leads linkedin-bot linkedin-cv linkedin-cv-downloader linkedin-download linkedin-downloader linkedin-resume linkedin-resume-downloader linkedin-scraper scrape-emails scrape-websites scraper scraper-engine

Last synced: 17 Mar 2025

https://github.com/empire/go-tse

Go client for http://www.tsetmc.com/

crawler financial-data financial-data-analysis market-data stock stock-prices tehran-stock-exchange tse tsetmc

Last synced: 01 Jun 2026

https://github.com/sebi75/lightweight-sitemapper

A lightweight sitemapper written in typescript, built on top of fast-xml-parser and relying on few dependencies

crawler node-js sitemap

Last synced: 21 Jan 2026

https://github.com/basemax/fakefaces

This repository contains a crawler that downloads thousands of fake human face images from various sources on the internet. Additionally, the repository includes a dataset of thousands of face images of fake humans.

crawler crawler-php crawler-testing crawlers curl dataset datasets face face-fake faces fake-face fake-faces php php-curl

Last synced: 27 Apr 2026

https://github.com/nakabonne/netsurfer

netsurfer is a very lightweight scraping framework

crawler go library scraping

Last synced: 01 Apr 2025

https://github.com/gabrielrf/bsbdf

Telegram Public Channel

crawler python telegram telegram-channel telegraph

Last synced: 05 Feb 2026

https://github.com/sharmadhiraj/free-json-datasets

Collection of free JSON data that are scraped and parsed from different websites.

collection crawler data data-scraping datasets json sports statistics web-scraping

Last synced: 28 Mar 2025

https://github.com/diogoazevedos/x-ray-build

A helper that build a x-ray based on a schema

crawler schema scraper structure x-ray

Last synced: 24 Feb 2026

https://github.com/tsonglew/spidreat

Article Spider with Python & Node.js :beetle:

crawler

Last synced: 06 Apr 2025

https://github.com/ozakboy/taiwan-news-crawlers

.net-based Crawlers for news of Taiwan (.net 台灣新聞爬蟲，數據物件化，方便使用)

crawler data-collection dataset-generation dotnet news taiwan webcrawlers

Last synced: 15 Apr 2025

https://github.com/spraakbanken/svt-crawler

Programme for crawling SVT's API for news articles and converting the data to XML.

corpus crawler

Last synced: 07 Mar 2026

https://github.com/dr4g0nsr/sitemap-crawler

Crawler using sitemap to crawl site/regenerate cache

crawler php

Last synced: 13 Jan 2026

https://github.com/carloocchiena/python_url_crawler

A script that starting from a webpage, iterate thru all its link, appending them in a list. Sort of proxy to get all pages in a website

beautifulsoup crawler python python3

Last synced: 12 Feb 2026

https://github.com/runnin-n-gunnin/geckofxinterceptrequestcaptureresponse

[GeckoFX/Firefox]: Shows how to Intercept request(s), capture response(s), customize GeckoPreferences, handle certificate errors, change useragent++.

browser cefsharp controls crawler crawling firefox gecko geckofx geckofx60 scraping webbrowser windows windowsforms winforms

Last synced: 06 Feb 2026

https://github.com/thecloer/crawler-himym

How I met your mother script PDF generator for learning English

crawler pdf pdf-generation typescript web-scraping webscraping

Last synced: 21 Jun 2025

https://github.com/genfuture/cryptocurrency-scraper

Cryptocurrency Data Crawler 🚀 Updates CoinData Every 12 hours. High-performance Node.js crawler that fetches comprehensive data for 1500+ cryptocurrencies from CoinGecko API. Collects market data, and blockchain details with built-in rate limiting and resume capability. Perfect for crypto analysis, research, and building market intelligence tools

binance bitcoin bsc coingecko coingecko-api crawler crypto-bot cryptocurrencies cryptocurrency ethereum scraper

Last synced: 28 Jan 2026

https://github.com/saadali1996/goose-rest-api

https://github.com/advancedlogic/GoOse based REST API for article content extraction

crawler golang rest-api

Last synced: 09 Mar 2026

https://github.com/camara94/crawlers

Web scraping is the process of using bots to extract content and data from a website. Unlike screen scraping, which only copies pixels displayed onscreen, web scraping extracts underlying HTML code and, with it, data stored in a database. The scraper can then replicate entire website content elsewhere

crawler python scraping scrapy spider

Last synced: 09 Apr 2025

https://github.com/keosariel/ramby

Ramby is a simple way to setup a webscraper

beautifulsoup crawler python3 webscraping

Last synced: 27 Mar 2025

https://github.com/1970mr/link-crawler

Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.

clawler crawler crawler-python link-crawler link-crawler-python link-scraper link-scraper-python links python scraper scraper-python website-crawler website-scraper

Last synced: 06 Feb 2026

https://github.com/sc0vu/jspachong

Js crawler library.

crawler pachong

Last synced: 06 Feb 2026

https://github.com/erikmueller/jazmax

Crawl JAZ for different heat pumps depending on flow and return temperatures from the JAZ calculator

crawler data-science efficiency green heatpump jaz

Last synced: 24 Mar 2025

https://github.com/lucky845/animetimeline

使用Python脚本爬取动漫信息时间表，并保存为Markdown文件。

anime crawler python-script

Last synced: 09 Jul 2025

https://github.com/afsh7n/crawly-automation

Crawly Automation is a lightweight, modular, and extensible web crawling framework built on top of Puppeteer. Whether you need to scrape data, automate browser interactions, manage CAPTCHAs, or handle advanced data extraction, Crawly Automation simplifies the process.

automation crawler nodejs puppeteer webscraping

Last synced: 25 Feb 2026

https://github.com/exca-dk/node-util

Usefull utils for analyzing p2p crypto networks.

crawler ethereum mev p2p scanner

Last synced: 16 May 2026

https://github.com/zekrotja/r34-crawler

A simple CLI tool to fetch and download images from rule34.xxx

crawler go rest-api rule34 worker-pool xml

Last synced: 06 Mar 2026

https://github.com/reisdev/reads

Real Estate Agency Data Scraper

crawler python scraping scrapy selenium-python selenium-webdriver spider

Last synced: 31 Jan 2026

https://github.com/yggverse/aquatic-crawler

SSD-friendly FS crawler for the Aquatic BitTorrent tracker, based on librqbit API

api aquatic bencode bittorrent btracker crawler daemon info-hash ipv6 librqbit magnet parser resolver rqbit torrent tracker

Last synced: 11 Mar 2026

https://github.com/denrydu/baiduimagecrawler

自己写的两个用来爬取百度图片的脚本，方便CV研究者制作数据集。Two ways to download images from baidu, useful tool for making cv datasets!

baidu crawler dynamic python3

Last synced: 04 Nov 2025

https://github.com/skulltech/arachnid

Crawling Instagram for reasons.

crawler instagram instagram-scraper python3 scraper scrapy

Last synced: 13 Jun 2025

https://github.com/becky-dai/flower-knowledge-graph-visualization

A full stack program of knowledge graph visualization 一个关于知识图谱可视化的全栈项目

crawler css django echarts html js knowledge-graph neo4j python

Last synced: 11 Mar 2026

https://github.com/nemmusu/free-vpn-downloader

This repository contains three Python scripts designed to simplify the process of downloading and configuring free VPN .ovpn files for use with OpenVPN.

automation crawler download downloader free freevpn openvpn ovpn ovpn-files vpn