https://github.com/neogeek/raspar

⛏ A simple to use Promise-based web scraper with local caching.
https://github.com/neogeek/raspar

promise web-scraper

Last synced: 19 days ago
JSON representation

⛏ A simple to use Promise-based web scraper with local caching.

• Host: GitHub
• URL: https://github.com/neogeek/raspar
• Owner: neogeek
• License: mit
• Created: 2015-09-03T00:15:24.000Z (about 9 years ago)
• Default Branch: main
• Last Pushed: 2024-06-04T02:53:21.000Z (5 months ago)
• Last Synced: 2024-10-11T10:45:44.738Z (about 1 month ago)
• Topics: promise, web-scraper
• Language: JavaScript
• Homepage: https://doxdox.org/neogeek/raspar
• Size: 336 KB
• Stars: 1
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# raspar

> A simple to use Promise-based web scraper with local caching.

[![Tests](https://github.com/neogeek/raspar/actions/workflows/test.workflow.yml/badge.svg)](https://github.com/neogeek/raspar/actions/workflows/test.workflow.yml)

[![NPM Version](http://img.shields.io/npm/v/raspar.svg?style=flat)](https://www.npmjs.org/package/raspar)

[![Latest Documentation](https://doxdox.org/images/badge-flat.svg)](https://doxdox.org/)

## Usage

```javascript

import fetch from 'raspar';

const contents = await fetch('http://www.google.com/humans.txt');

console.log(contents);

```

### Options

```javascript

import fetch from 'raspar';

const options = {

  cacheDirectory: 'temp/cache/',

  requestOptions: {

    headers: {

      'User-Agent': 'request'

    },

    method: 'POST'

  },

  ttl: 1800

};

const contents = await fetch('http://www.google.com/humans.txt', options);

console.log(contents);

```

| Name           | Description                                                                                                                     | Default Value |

| -------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------- |

| cacheDirectory | Directory to store cache.                                                                                                       | `temp/cache/` |

| requestOptions | Request options object. [Read more github.com/node-fetch/node-fetch](https://github.com/node-fetch/node-fetch/tree/2.x#options) | `{}`          |

| ttl            | TTL (Time to live) in seconds.                                                                                                  | `1800`        |