{"id":18707198,"url":"https://github.com/hellomaxime/data-platform-on-kubernetes","last_synced_at":"2025-11-09T08:30:37.025Z","repository":{"id":223901448,"uuid":"760881299","full_name":"hellomaxime/data-platform-on-kubernetes","owner":"hellomaxime","description":"Open Source Data Platform on Kubernetes","archived":false,"fork":false,"pushed_at":"2024-04-22T16:45:58.000Z","size":157,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-28T07:19:39.167Z","etag":null,"topics":["bigdata","data","data-pipeline","dbt","druid","etl","kubernetes","ml","open-source","platform","spark","superset"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hellomaxime.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-20T20:51:44.000Z","updated_at":"2024-12-23T10:12:50.000Z","dependencies_parsed_at":"2024-03-27T19:28:31.170Z","dependency_job_id":"fa3eb6b0-a799-4f45-b44f-feb81b39cf28","html_url":"https://github.com/hellomaxime/data-platform-on-kubernetes","commit_stats":null,"previous_names":["hellomaxime/data-platform-on-kubernetes"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellomaxime%2Fdata-platform-on-kubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellomaxime%2Fdata-platform-on-kubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellomaxime%2Fdata-platform-on-kubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hellomaxime%2Fdata-platform-on-kubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hellomaxime","download_url":"https://codeload.github.com/hellomaxime/data-platform-on-kubernetes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239571540,"owners_count":19661164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","data","data-pipeline","dbt","druid","etl","kubernetes","ml","open-source","platform","spark","superset"],"created_at":"2024-11-07T12:17:06.137Z","updated_at":"2025-11-09T08:30:36.961Z","avatar_url":"https://github.com/hellomaxime.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data platform on Kubernetes\n\nThis project aims to deploy a complete data platform on kubernetes, many services are available to build end-to-end data engineering projects from ingestion to visualization. \n\n## Prerequisites\n- docker\n- kubernetes (minikube cluster for local development)\n- kubectl\n- helm\n\n## Available services\n\n- __Data ingestion__\n    - Nifi\n- __Data integration__\n    - Airbyte\n- __Message queue__\n    - Kafka\n    - RabbitMQ\n- __Change Data Capture__\n    - Debezium\n- __Database__\n    - Cassandra\n    - Druid\n    - MongoDB\n    - MySQL/Phpmyadmin\n    - PostgreSQL/pgAdmin\n- __Data warehouse__\n    - ClickHouse\n- __Datalake__\n    - MinIO\n- __Data transformation__\n    - dbt\n    - Flink\n    - Spark\n- __Data quality__\n    - Great Expectations\n- __Distributed SQL query engine__\n    - Trino\n- __Visualization__\n    - Metabase\n    - Superset\n- __Machine learning__\n    - Kubeflow\n- __Orchestration__\n    - Airflow\n    - Argo Workflows\n- __Monitoring__\n    - Grafana/Prometheus\n- __Notebook__\n    - JupyterHub\n\n## Data formats\n- Delta Lake\n- Apache Iceberg (soon)\n\n## How to deploy the data platform on kubernetes\n\nBefore deploying in the cluster, choose services you want to start in `.config` file. (y|n)  \n\n__Deploy the data plaftorm__  \n`./start.sh`  \n\nYou may need to wait a few minutes for all services to start, you can check pods status with the following command : `kubectl get all -A`.  \n\n\n__Turn off the data plaftorm__  \n`./stop.sh`  \n\n## Helpful:  \n\n__some services are accessible through an URL__  \nexample : `http://dataplatform.\u003cservice-name\u003e.io/`\n\n__access another service from inside__  \n`\u003cservice-name\u003e.\u003cnamespace\u003e.svc.cluster.local:\u003cservice-port\u003e`\n\n__get helm default values__  \n`helm show values \u003crepo/chart\u003e \u003e values.yaml`  \n\n__config file__  \nset .config file to choose services you want to enable/disable\n\n__minikube ingress addons__  \n`minikube addons enable ingress`\n\n__kubernetes dashboard__  \n`minikube dashboard --url`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellomaxime%2Fdata-platform-on-kubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhellomaxime%2Fdata-platform-on-kubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhellomaxime%2Fdata-platform-on-kubernetes/lists"}