https://github.com/russellsteadman/netscrape

A Node.js framework for creating good bots
https://github.com/russellsteadman/netscrape

bot crawler crawling exclusion rfc9309 scraper scraping web-scraping

Last synced: 6 days ago
JSON representation

A Node.js framework for creating good bots

• Host: GitHub
• URL: https://github.com/russellsteadman/netscrape
• Owner: russellsteadman
• License: other
• Created: 2022-01-08T19:00:29.000Z (almost 3 years ago)
• Default Branch: main
• Last Pushed: 2023-10-13T02:51:18.000Z (about 1 year ago)
• Last Synced: 2024-03-15T10:22:44.118Z (8 months ago)
• Topics: bot, crawler, crawling, exclusion, rfc9309, scraper, scraping, web-scraping
• Language: TypeScript
• Homepage: https://www.npmjs.com/package/netscrape
• Size: 761 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# NetScrape Mono-Repo

Efficient, simple, and compliant web scraping.

## Packages

- [`netscrape`](https://github.com/russellsteadman/netscrape/tree/main/packages/bot#readme)

  — A simple web scraping client

- [`exclusion`](https://github.com/russellsteadman/netscrape/tree/main/packages/exclusion#readme)

  — A simple and compliant robots.txt parser

## Get Started

Visit the `netscrape` package [on npm](https://www.npmjs.com/packages/netscrape) to get started.

## Licenses

See the individual packages for licenses.