Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ms10596/scrape

Last synced: 1 day ago
JSON representation

Host: GitHub
URL: https://github.com/ms10596/scrape
Owner: ms10596
Created: 2020-01-16T10:03:25.000Z (almost 5 years ago)
Default Branch: master
Last Pushed: 2023-02-08T00:44:46.000Z (almost 2 years ago)
Last Synced: 2023-03-04T03:17:09.053Z (almost 2 years ago)
Language: Python
Size: 121 KB
Stars: 1
Watchers: 2
Forks: 0
Open Issues: 5
Metadata Files:
- Readme: readme.md

Awesome Lists containing this project

README

# Scrapest
scraping engine that is contenraized and deployed in a kubernetes cluster. It scrapes and sends articles to an elasticsearch instance in the cloud.

## Main Blocks
1. __scraper__: a python program built on the SCRAPY framework. It consists of several spiders that are invoked using a post request.
2. __elasticsearch__: t3.micro instance of elastic search that presents in AWS cloud service.

## Monitored via cloudwatch
![image](images/cloudwatch.png)