https://github.com/lsongdev/node-crawler

simple crawler
https://github.com/lsongdev/node-crawler

crawler node-crawler

Last synced: 8 days ago
JSON representation

simple crawler

• Host: GitHub
• URL: https://github.com/lsongdev/node-crawler
• Owner: lsongdev
• License: isc
• Created: 2018-01-27T11:40:27.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2018-01-27T11:41:12.000Z (almost 7 years ago)
• Last Synced: 2024-12-31T10:05:31.516Z (10 days ago)
• Topics: crawler, node-crawler
• Language: JavaScript
• Homepage:
• Size: 3.91 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
## node-crawler

> simple crawler

### Installation

```bash

$ npm install @song940/crawler --save

```

### Example

```js

const Crawler = require('@song940/crawler');

const engine = new Crawler({ 

    url: 'https://news.ycombinator.com'

});

engine.parse = async ($, commit) => {

    $('table.itemlist tr.athing').each((i, row) => {

        const title = $('td.title', row).text();

        commit({ title });

    });

};

engine.on('commit', posts => {

    console.log(posts);

});

for(var i=1;i<100;i++){

    engine.push(`${engine.url}/news?p=${i}`);

}

engine.on('end', () => {

    console.log('all job done');

});

engine.start();

```

### Contributing

- Fork this Repo first

- Clone your Repo

- Install dependencies by `$ npm install`

- Checkout a feature branch

- Feel free to add your features

- Make sure your features are fully tested

- Publish your local branch, Open a pull request

- Enjoy hacking <3

### ISC

This work is licensed under the [ISC license](./LICENSE).

---