https://github.com/xrmx/primadatapipeline
La mia prima data pipeline talk demo project
https://github.com/xrmx/primadatapipeline
data-engineering go luigi pipeline python
Last synced: 7 months ago
JSON representation
La mia prima data pipeline talk demo project
- Host: GitHub
- URL: https://github.com/xrmx/primadatapipeline
- Owner: xrmx
- License: gpl-3.0
- Created: 2017-04-02T14:34:09.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-04-02T14:34:23.000Z (over 8 years ago)
- Last Synced: 2025-01-24T11:09:38.909Z (9 months ago)
- Topics: data-engineering, go, luigi, pipeline, python
- Language: Python
- Size: 14.6 KB
- Stars: 3
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# La mia prima data pipeline
## setup api
```
cd api
go build
```
## setup pipeline
```
python3 -m venv venv
. ./venv/bin/activate
pip install wheel
cd pipeline
python setup.py install
cd -
```
## setup superset
https://github.com/airbnb/superset/blob/master/docs/installation.rst#making-your-own-build
## quickstart
Start the api on a terminal:
```
./api/api
```
On another one we create the db and execute the pipeline for today:
```
. ./venv/bin/activate
python3 createdb.py
luigi.sh $(date +%Y-%m-%d)
```