https://github.com/mrintern/etl-using-k8s
A simple example of an ETL job using KinD (kubernetes in docker). Just for fun.
https://github.com/mrintern/etl-using-k8s
Last synced: about 1 year ago
JSON representation
A simple example of an ETL job using KinD (kubernetes in docker). Just for fun.
- Host: GitHub
- URL: https://github.com/mrintern/etl-using-k8s
- Owner: mrintern
- Created: 2023-05-31T13:31:57.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-03T17:29:30.000Z (about 3 years ago)
- Last Synced: 2025-02-10T13:43:51.061Z (over 1 year ago)
- Language: Python
- Size: 23.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Under Construction! You've been warned ;)
# ETL-using-k8s
A simple example of an ETL job using KinD (kubernetes in docker).
Requests data from the weather API and sends to a mysql table.
Eventually, this app will implement airflow and spark to do this in a more scalable way.
# Requirements
- KinD (Kubernetes in Docker)
- Docker
# Demo
1. (if running locally) Setup kind cluster
`kind create cluster --name etl-using-k8s`
2. Run the deployment
`kubectl apply -f manifests/deployment.yml`
### TODO
1. write a mysql pod manifest (https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/)
2. manually interact with mysql pod, create a table
3. update etl-app with mysql connection creds
4. test mysql app again