https://github.com/incuca/generic-website-scraper

A generic website scraper made with Strapi, Puppeteer and Cheerio running on a cron with an example set
https://github.com/incuca/generic-website-scraper

Last synced: about 1 year ago
JSON representation

A generic website scraper made with Strapi, Puppeteer and Cheerio running on a cron with an example set

• Host: GitHub
• URL: https://github.com/incuca/generic-website-scraper
• Owner: InCuca
• License: mit
• Created: 2021-01-30T16:26:48.000Z (over 5 years ago)
• Default Branch: main
• Last Pushed: 2021-01-30T16:37:18.000Z (over 5 years ago)
• Last Synced: 2025-03-28T16:13:46.110Z (over 1 year ago)
• Language: JavaScript
• Size: 197 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Scraper

A quick scraper application with a functioning example.

## Start

Install dependencies and run `yarn develop`.

Create your admin credentials on Strapi and create the first Scraper:

```

name: Jamstack.org

enabled: true

frequency: * * * * *

```

The cron tries to run the scraping code every minute, and:

- check if the Scraper is `enabled` before running;

- respects the `frequency` set;

The scraping is done with puppeteer and cherio and is at `/scripts/scrapers`. The content is saved to the **Site generators** model.

## From here...

You can turn it into a generic website scraper or just modify the index.js file to your needs. Possibilities are endless...