https://github.com/ms10596/scrape

https://github.com/ms10596/scrape

Last synced: 11 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/ms10596/scrape
• Owner: ms10596
• Created: 2020-01-16T10:03:25.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2023-02-08T00:44:46.000Z (over 3 years ago)
• Last Synced: 2025-03-04T02:44:19.302Z (over 1 year ago)
• Language: Python
• Size: 121 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 5
• Metadata Files:
  • Readme: readme.md

Awesome Lists containing this project

README

          
# Scrapest

scraping engine that is contenraized and deployed in a kubernetes cluster. It scrapes and sends articles to an elasticsearch instance in the cloud.

## Main Blocks

1. __scraper__: a python program built on the SCRAPY framework. It consists of several spiders that are invoked using a post request.

2. __elasticsearch__: t3.micro instance of elastic search that presents in AWS cloud service.

## Monitored via cloudwatch

![image](images/cloudwatch.png)