https://github.com/ganevdev/periodic-crawler

periodic-crawler
https://github.com/ganevdev/periodic-crawler

Last synced: 10 months ago
JSON representation

periodic-crawler

• Host: GitHub
• URL: https://github.com/ganevdev/periodic-crawler
• Owner: ganevdev
• License: mit
• Created: 2019-03-30T06:30:40.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2019-04-08T10:36:40.000Z (almost 7 years ago)
• Last Synced: 2025-04-06T09:24:39.562Z (10 months ago)
• Language: TypeScript
• Homepage: https://www.npmjs.com/package/periodic-crawler
• Size: 31.3 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Periodic Crawler

[![Build Status](https://travis-ci.com/Ganevru/periodic-crawler.svg?branch=master)](https://travis-ci.com/Ganevru/periodic-crawler)

[![npm](https://img.shields.io/npm/v/periodic-crawler.svg?style=flat-square)](http://npm.im/periodic-crawler)

I do NOT recommend using it in real projects.

```

npm i periodic-crawler

```

## Exsampls

```js

const periodicCrawler = require('periodic-crawler');

async () => {

  const res = await periodicCrawler(['www.example.com/'], {

    proxy: {

      useProxy: false

    },

    settings: {

      pauseMin: 1,

      pauseMax: 2

    }

  });

  console.log(res);

};

// in console:

// [

//   {

//     "url": "www.example.com/",

//     "dowDate": "2019-04-01T02:39:55.268Z",

//     "body": "\n\n\n\n\n
\n    
Example Domain
\n    
This domain is established to be used for illustrative examples in documents. You may use this\n    domain in examples without prior coordination or asking for permission.
\n    
More information...
\n
\n\n\n"

//   }

// ]

```


## TODO

- [Puppeteer](https://pptr.dev/) option for scrapping