Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/gforien/flightbeat
Projet INSA 4e année : Heatmap des trajets d'avions les plus demandés et les plus coûteux avec la suite ELK et l'API Skyscanner.
https://github.com/gforien/flightbeat
beats data-vizualisation elasticsearch elk-stack flights-api kibana scrapping
Last synced: about 1 month ago
JSON representation
Projet INSA 4e année : Heatmap des trajets d'avions les plus demandés et les plus coûteux avec la suite ELK et l'API Skyscanner.
- Host: GitHub
- URL: https://github.com/gforien/flightbeat
- Owner: gforien
- License: unlicense
- Created: 2020-10-05T09:08:57.000Z (about 4 years ago)
- Default Branch: master
- Last Pushed: 2021-10-25T09:11:21.000Z (about 3 years ago)
- Last Synced: 2024-11-15T12:35:17.763Z (about 2 months ago)
- Topics: beats, data-vizualisation, elasticsearch, elk-stack, flights-api, kibana, scrapping
- Language: JavaScript
- Homepage:
- Size: 5.8 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# flightbeat :airplane::zap:
#### Beat written in Go to retrieve flight prices from Skyscanner API and insert them into an elasticsearch cluster
## :construction_worker: Build and run the beat
```bash
$ mage build
$ ./flightbeat -e -d "*"
# input : flightbeat.yml (default)
# output: console (default)
```
## :rocket: Deploy
The beat has to always be active on 1 node (out of 16).
A periodic NodeJS job, **leader.js** makes sure that there is always a leader designated to run the beat.
*(and because a NodeJS job doesn't handle hard reboot, a cron job makes sure every hour that **leader.js** is up and running).*
In case of failure of the leader, the nodes perform a leader election to elect a new leader who will run the beat.
## :wrench: Generate your Beat from elastic/beats with mage (oct. 2020)
1. install go
2. `go get github.com/elastic/beats`
3. `go get github.com/magefile/mage`
4. `cd elastic/beats; mage GenerateCustomBeat` (fill everything explicitly, there is no default value)
5. `cd gforien/custombeat`
6. disable W10 App execution alias + copy python.exe to go/bin/python3.exe + disable W10 real-time protection (re-enable after)
7. `make setup; make`
## :books: Sources
This project is sampled from :
- [Beats developer guide](https://www.elastic.co/guide/en/beats/devguide/current/new-beat.html)
- [Go by example](https://gobyexample.com/)
- [HTTP requests in Go](https://medium.com/rungo/making-external-http-requests-in-go-eb4c015f8839)
- [Biggest airports in the world](https://gettocenter.com/airports/top-100-airports-in-world/1000)
- [Flight routes database](https://www.kaggle.com/open-flights/flight-route-database)
##
#### [Gabriel FORIEN](https://github.com/gforien)
