https://github.com/mahmoudimus/genescrape
Using ElasticSearch, Kibana, and Scrapy to build a scraping framework
https://github.com/mahmoudimus/genescrape
Last synced: 2 months ago
JSON representation
Using ElasticSearch, Kibana, and Scrapy to build a scraping framework
- Host: GitHub
- URL: https://github.com/mahmoudimus/genescrape
- Owner: mahmoudimus
- Created: 2020-11-18T17:15:17.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2020-11-18T18:52:14.000Z (over 4 years ago)
- Last Synced: 2025-01-29T15:29:37.188Z (4 months ago)
- Language: Python
- Size: 15.6 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Genescrape
[](https://circleci.com/gh/mahmoudimus/genescrape)
## Getting Started
### Installation
- `brew install docker-credential-gcloud`
- `gcloud components install docker-credential-gcr`
- `ln -s /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin/docker-credential-gcloud /usr/local/bin/`
- `docker-credential-gcr configure-docker`
- `docker-credential-gcloud configure-docker`
### Build
```bash
docker-compose up --build
```
### Running (after build)
```bash
docker-compose up scrapyd
```
```bash
docker-compose up orphanet
```
```bash
docker-compose up kibana
```
Kibana is http://localhost:5601