https://github.com/yoksel/harvester

https://github.com/yoksel/harvester

Last synced: 3 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/yoksel/harvester
• Owner: yoksel
• Created: 2018-09-23T22:01:53.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T16:45:57.000Z (about 2 years ago)
• Last Synced: 2024-04-14T04:05:33.495Z (10 months ago)
• Language: JavaScript
• Size: 1.32 MB
• Stars: 10
• Watchers: 3
• Forks: 2
• Open Issues: 6
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Harvester

[Puppeteer](https://github.com/GoogleChrome/puppeteer)-based tool for collecting different types of data:

* links

* screenshots

* code snippets

Describe tasks in configs and tool will allow you to run it from web interface:



The result will be printed to page:



## Available tasks

### Links

Can be useful for old large sites with vague structure. You can found something unexpected.

Config example: tasks/urls.example.js

### Screenshots

Tool can make screenshots of given pages with given dimensions and device emulation. You can run task twice to compare result with previous.

Config example: tasks/screens.example.js

### Snippets

Useful if you need download all your demos from external service.

Config example: tasks/snippets.example.js

## Usage

1. Clone:

`git clone [email protected]:yoksel/harvester.git --depth 1 && cd harvester`

2. Run `npm i`

3. Rename `credits-example.js` to `credits.js` and fill it with real logins and passwords. It'll allow you to log in and visit a site as a logged in user.

4. Take needed example file in tasks, rename it without `example` (`screens.example.js` -> `screens.js`) and fill it with real data.

5. Run `npm start` and open [localhost:3007](http://localhost:3007/)

You'll see page wich allows you to start and stop tasks, see collected data and to download it in archive.

`credits.js` and task files are in `gitignore` and will not be commited. Don't push your passwords to the public repository.

## Previews

**Collected links**



**Collected links with screenshots**



**Full view of the screenshot**



**Full view of the screenshot with diff**



---

Tool is in development. If you find a bug, [fill an issue](https://github.com/yoksel/harvester/issues/new)