https://github.com/darwinz/js-cli-web-crawlers
Some web crawlers run through CLI, developed in JavaScript
https://github.com/darwinz/js-cli-web-crawlers
cli crawlers express javascript nodejs
Last synced: 7 months ago
JSON representation
Some web crawlers run through CLI, developed in JavaScript
- Host: GitHub
- URL: https://github.com/darwinz/js-cli-web-crawlers
- Owner: darwinz
- Created: 2017-12-15T04:58:01.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2023-04-10T20:11:57.000Z (over 2 years ago)
- Last Synced: 2025-03-24T02:52:02.548Z (7 months ago)
- Topics: cli, crawlers, express, javascript, nodejs
- Language: JavaScript
- Size: 391 KB
- Stars: 0
- Watchers: 1
- Forks: 3
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# JS CLI Web Crawlers
### Prerequisites
* [Node.js](https://nodejs.org/en/) 8.5+
* Npm 5.3+
### Setup
After all prerequisites are met, install package dependencies with npm
```bash
$ npm install
```
Then run the config helper to configure the environment (prompts for user configuration options)
```bash
$ npm run config:helper
```
### Running
After all prerequisites are met and setup is complete, crawlers can be run from shell using node, or as a node
web service
##### Example (running from cli)
```bash
$ node crawlers/vivint_solar --run
```
or
```bash
$ npm run crawlers:solar
```
##### Example (running as web service)
```bash
$ npm start
```
```bash
$ curl -XGET http://localhost:3000/crawlers/solar/production
```
### Crawlers
* Vivint Solar - check production output
* Google Analytics - check real time analytics data
### Testing
Integration testing can be run using npm
```bash
$ npm test
```