https://github.com/xscriptor/shinobi

https://github.com/xscriptor/shinobi

Last synced: 9 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/xscriptor/shinobi
• Owner: xscriptor
• Created: 2026-07-03T11:49:51.000Z (14 days ago)
• Default Branch: main
• Last Pushed: 2026-07-03T18:52:03.000Z (14 days ago)
• Last Synced: 2026-07-03T20:27:06.408Z (14 days ago)
• Language: Rust
• Size: 407 KB
• Stars: 0
• Watchers: 0
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Security: SECURITY.md
  • Roadmap: ROADMAP.md

Awesome Lists containing this project

README

          

Shinobi




 

Silent Web Scraper — Anti-Blocking Download System



**Language / Idioma**  

[English](#) | [Español](./docs/esp/README.md)

XWA  submodule focused on silent web scraping with anti-blocking — under active development






Overview ·

Capabilities ·

Anti-Blocking ·

Quick Start ·

Launch Script ·

Docker Compose ·

Standalone ·

Modes ·

Env Variables ·

Docs





Overview


Shinobi is a stealth web scraper with a single web interface. It downloads entire sites — HTML, CSS, JS, images, PDFs — while evading detection through multiple anti-blocking layers.


  

    Interface

    Directory

    Language

    Type

  

  

    Shinobi Web

    / (monorepo root)

    Rust (Axum) + Angular 19

    Web application (standalone or Docker)

  

Capabilities



  


Recursive Crawling — BFS page discovery with configurable depth and max pages

  


JavaScript Rendering — Headless Chromium engine for SPA/React/Vue/Angular sites

  


Asset Download — HTML, CSS, JS, images, PDFs, archives, media, fonts

  


File Type Filtering — Select which file extensions to download

  


Same-Domain Scoping — Stay within target domain or crawl freely

  


Real-Time Progress — SSE live stream with pages scraped, files downloaded, current URL

  


Downloaded File Browser — Browse and open downloaded files from the web UI

  


Two Operation Modes:

    


      


Fast Test — Pure Rust, zero external deps, crawl + download + anti-blocking

      


Deep Research — Python sidecar adds structured data extraction (JSON-LD, microdata, Open Graph, RDFa), NLP analysis (summary, entities, keywords), custom CSS selectors, and enriched email/phone extraction

    

  



Anti-Blocking System



  


User-Agent Rotation — 15 real browser UAs (Chrome, Firefox, Safari, Edge, mobile)

  


Header Randomization — Accept, Accept-Language, Sec-CH-UA, Sec-Fetch-* per request

  


Request Delay + Jitter — Configurable base delay with random jitter

  


Exponential Backoff — Retry with jitter on failures (configurable attempts)

  


Rate Limit Handling — Detects HTTP 429/503, waits, retries with longer backoff

  


Proxy Support — HTTP/HTTPS/SOCKS5 proxy rotation





Quick Start


Launch Script (Recommended)

./shinobi.sh --fast       # Rust backend + frontend (Fast Test mode)

./shinobi.sh --deep        # Rust + Python extractor (Fast Test + Deep Research)

./shinobi.sh --docker      # Everything via docker-compose

./shinobi.sh --help        # Full usage help


Web Version (Docker Compose)

docker compose up -d --build



  
Web UI: http://localhost:8080



  
Extractor API: http://localhost:9090 (Deep Research only)



Standalone (No Docker)

Fast Test mode (Rust only)

cargo run --release


Full stack (Rust + Python extractor)

./shinobi.sh --deep

Web UI at http://localhost:8080. Downloaded files stored in ./downloads/.


Execution Modes


  ModeCommandDescription

  Fast Test./shinobi.sh --fastCrawling + asset download + anti-blocking. No external dependencies.

  Deep Research./shinobi.sh --deepAdds Python sidecar for structured data extraction (JSON-LD, microdata, OG), NLP analysis (summary, entities, keywords), and custom CSS selectors.

  Docker./shinobi.sh --dockerBoth services via docker-compose.

  Python Only./shinobi.sh --python-onlyExtractor standalone for development.

Environment Variables


  VariableDefaultDescription

  PORT8080HTTP listen port (Rust)

  DATA_DIRdownloadsDownloaded files directory

  RUST_LOGshinobi=info,tower_http=infoLogging verbosity

  EXTRACTOR_URLhttp://localhost:9090Python extractor endpoint (Deep Research)



Related Documents


  DocumentDescription

  docs/manual.mdDevelopment and production deployment guide

  docs/ui-architecture.mdFrontend architecture specification

  ROADMAP.mdDevelopment phases and milestones

  docs/project-structure.mdDetailed codebase structure with file-by-file breakdown and API reference





X




  



 & 



  



 &