https://github.com/chhantyal/luigi-kubernetes

Luigi with Kubernetes demo presented at Europython 2019 in Basel
https://github.com/chhantyal/luigi-kubernetes

Last synced: 2 months ago
JSON representation

Luigi with Kubernetes demo presented at Europython 2019 in Basel

• Host: GitHub
• URL: https://github.com/chhantyal/luigi-kubernetes
• Owner: chhantyal
• License: bsd-3-clause
• Created: 2019-07-10T20:19:54.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-08T05:53:01.000Z (about 2 years ago)
• Last Synced: 2024-10-12T07:25:16.018Z (3 months ago)
• Language: Python
• Homepage: https://www.youtube.com/watch?v=41ubI6a3jzI
• Size: 70.3 KB
• Stars: 27
• Watchers: 3
• Forks: 3
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Luigi examples with Kubernetes

This is Luigi and Kubernetes demo presented at Europython 2019 in Basel.

Luigi pipeline can be easily triggered from Kubernetes cronjob. 

By doing so, one can get all the benefits of container orchestration system like flexible deployment, horizontal scaling and easier maintenance.   

Slides: https://speakerdeck.com/chhantyal/building-data-workflows-with-luigi-and-kubernetes

## Run

Luigi example on local

`pipenv run python -m luigi --module example SalesReport --date=2019-07-11 --local-scheduler`

Luigi example on Kubernetes (minikube)

Minikube:

- Start minikube: `minikube start --memory 4096`

- Open K8S dashboard: `minikube dashboard`

Luigid:

- Deploy luigid: `mkctl apply -f luigid/deployment.yaml && mkctl apply -f luigid/service.yaml`

- Open Luigi UI: `minikube service luigid-service`

Luigi task:

- Deploy as cronjob: `mkctl apply -f cronjob.yaml`

- Run: `kubecron run luigi-example`

## Docker 

Docker images used in above deployments are available in Docker Hub https://hub.docker.com/u/chhantyal