https://github.com/lucassimao/coinmarketcap-crawler

Crypto currencies market data from http://coinmarketcap.com/ crawled using puppteer + expressjs backend + Elasticsearch + React frontend
https://github.com/lucassimao/coinmarketcap-crawler

elasticsearch express nodejs puppeteer react

Last synced: about 18 hours ago
JSON representation

Crypto currencies market data from http://coinmarketcap.com/ crawled using puppteer + expressjs backend + Elasticsearch + React frontend

• Host: GitHub
• URL: https://github.com/lucassimao/coinmarketcap-crawler
• Owner: lucassimao
• Created: 2021-03-18T05:39:50.000Z (almost 4 years ago)
• Default Branch: master
• Last Pushed: 2021-03-18T14:50:07.000Z (almost 4 years ago)
• Last Synced: 2024-11-23T01:27:32.683Z (2 months ago)
• Topics: elasticsearch, express, nodejs, puppeteer, react
• Language: TypeScript
• Homepage:
• Size: 531 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# How to run

### Overall architecture

We leverage an elastic search instance to store crypto currencies market data from http://coinmarketcap.com/ crawled using a pupetter recurrent job that runs every 5min. 

The react based frontend requests data from an expressjs based backend which by its turn fetches data from elastic search.

[Quick demo](https://drive.google.com/file/d/17Ds9uEjNUTlMUdY74p0SUv2gFOddHJpO/view?usp=sharing)

### using docker-compose (easier way)

at the root of the project:

> docker-compose build

and

> docker-compose up

This will build the crawler, backend, frontend and will set up an elastic search instance 

Then you can point your browser to http://localhost:80

### using npm

First ensure you have a running elastic search instance running on localhost:9200. Then:

Inside the backend folder:

> npm run start:dev

Inside the frontend folder:

> npm start

Inside the crawler folder:

> npm run start:dev

If your local elastic search instance runs on a port other than 9200, edit the .env files inside the `crawler` and `backend` folders

### Tech stack

* Nodejs v15

* Typescript

* React

* pupetter

* elastic search v7.11

* Docker