{"id":27206543,"url":"https://github.com/ddeutils/data-orchestra","last_synced_at":"2026-05-05T15:34:57.034Z","repository":{"id":152052375,"uuid":"547981026","full_name":"ddeutils/data-orchestra","owner":"ddeutils","description":"❌ Full-Stack Data Orchestration config by Yaml template with Flask \u0026 HTMX","archived":false,"fork":false,"pushed_at":"2024-05-11T13:43:35.000Z","size":4000,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-09T23:45:41.008Z","etag":null,"topics":["data-orchestration","docker","flask","htmx","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ddeutils.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-08T17:37:46.000Z","updated_at":"2025-04-07T10:44:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"81b042bc-0c6e-4128-a533-de41081ede68","html_url":"https://github.com/ddeutils/data-orchestra","commit_stats":null,"previous_names":["ddeutils/ddeapp-flask","ddeutils/data-orchestra"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fdata-orchestra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fdata-orchestra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fdata-orchestra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ddeutils%2Fdata-orchestra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ddeutils","download_url":"https://codeload.github.com/ddeutils/data-orchestra/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248131465,"owners_count":21052819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-orchestration","docker","flask","htmx","python3"],"created_at":"2025-04-09T23:45:44.761Z","updated_at":"2026-05-05T15:34:51.560Z","avatar_url":"https://github.com/ddeutils.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Framework Application: *Flask*\n\n**Table of Contents**:\n\n* [Overviews](#overviews)\n  * [Getting Started](#getting-started)\n  * [Components](#components)\n* [Build Application](#build-application)\n* [API Document](#api-document)\n* [CI/CD Flow](#application-cicd-flow)\n* [Service Reference](#service-reference)\n\nThis is the Full-Stack **Data Framework Application** that was built by \n[Flask](https://flask.palletsprojects.com/en/2.0.x/)\nand does not use any 3th-party of data orchestration or data framework libs like; \n[Airflow](https://airflow.apache.org/), [Dagster](https://dagster.io/), [Prefect](https://www.prefect.io/), or [Airbyte](https://airbyte.com/).\nThen I was making this framework application is easy to enhance, fix, and deploy\non different environments or systems. It has [fewer dependencies](requirements.txt),\nless complex code, and lightweight of coding which mean you can remove some components\nof this framework before deploy without error or with debuggable error.\n\n\u003e [!NOTE]\n\u003e First objective of this framework application is data pipeline orchestration in\n\u003e a retail platform, but the core engine of this application framework can do more\n\u003e than orchestrate data pipeline by requested or scheduler.\n\n## Overviews\n\nThis application framework able to run on **Docker** container service like **AWS ECS**,\nand it was requested from **AWS Batch** or any Platform Web server via **AWS Load Balancing**.\nAfter that, it will generate and deliver SQL statements to be performed in the\n**AWS RDS**, or **PostgresSQL**, for run a data pipeline.\n\nThe **AWS ECS** service is the host for this framework application orchestration,\nso the overhead of resources for running any transformation in data pipeline is\nusing only the database engine, but the part of DataFrame transformation which\nuse CPU bound this framework application can handle this purpose\n(This application framework already implement this function).\n\n### Getting Started\n\nBefore start this application, the needed environment parameters are,\n\n```yaml\nAPIKEY: For request API to application\nDB_HOST: host of database\nDB_NAME: database name\nDB_USER: database user\nDB_PASS: database password\nDB_PORT: database port\nAI_SCHEMA: AI schema name\nMAIN_SCHEMA: The platform schema name\n```\n\nand the needed [python dependencies](requirements.txt). If you want to install it,\nyou can use:\n\n```shell\n# Optional for vendor function.\npip install --no-cache-dir -r requirements.pre.txt\n\n# Main requirement package of this application.\npip install --no-cache-dir -r requirements.txt\n```\n\nAfter setup environment variables and dependencies, we can run by this command:\n\n```shell\n$ python .\\manage.py run --help\n```\n\nFor the first time, you should create all tables to database:\n\n```shell\n$ python .\\manage.py init\n$ python .\\manage.py run --api --debug\n```\n\n### Components\n\n**This data application framework has 2 functional components**:\n\n- [Web Service Component](#web-service)\n- [RestAPI Service Component](#restapi-service)\n\n#### Web Service\n\nThe Web Application framework serve static templates to client. This session will\nshow UI for control any framework components.\n\n- Pipeline\n- Catalog\n- Table\n- Admin \n\n\u003e [!NOTE]\n\u003e This Web App use **HTMX**, and **Ajax** for sync data from server.\n\n#### RestAPI Service\n    \nThe RestAPI service has 3 subcomponents are *analytic*, *ingestion*, and *framework* component.\nFor the *framework* component is the core of this application framework that have\n3 modules are,\n\n- Setup\n- Run Data\n- Retention\n\n**Setup**: all tables that config in the control pipeline table to database and initialize data if set initial\nkey in catalog file.\n\n**Run Data**: transformation or preparation process.\n- *common mode* ( process with incremental tracking )\n- *rerun mode* ( process without incremental tracking )\n\n**Retention**: data to all tables in database that config retention value more than 0.\nThis module contains backup process, which mean a dumping data from current schema to the backup schema.\n\n## Build Application\n\nThere are 3 ways to build application with input parameters. If both types were \ncreated at the same time, the application inherits from `environment parameter` \nfirst.\n\n- [Built with setting `environment variables` in local](#built-with-setting-environment-variables-in-local)\n- [Built with the `.env` file](#built-with-the-env-file)\n- [Built with Docker Compose](#built-with-docker-compose)\n\n### Built with setting `environment variables` in local\n     \n*Docker image*:\n\n```shell\nsudo docker build -t ${env}-application .\nsudo docker images\n# REPOSITORY          TAG        IMAGE ID       CREATED          ...\n# ${env}-application  latest     b50e8cdd83ed   10 seconds ago   ...\n ```\n\n*Docker container*:\n  \n```shell\n$ mkdir -p log\n$ export env=\"development\"\n\n$ sudo docker run --${env}-application \\\n  -e APIKEY='\u003capi-key-in-env\u003e' \\\n  -e DB_HOST='\u003chost\u003e' \\\n  -e DB_NAME='\u003cdatabase-name\u003e' \\\n  -e DB_USER='\u003cuser\u003e' \\\n  -e DB_PASS='\u003cpassword\u003e' \\\n  -e DB_PORT='5432' \\\n  -e AI_SCHEMA='ai' \\\n  -e MAIN_SCHEMA='public' \\\n  -e APP_ENV=${env} \\\n  --restart=always -d -p 5000:5000 ${env}-application\n\n$ sudo docker ps\n# CONTAINER ID      IMAGE               COMMAND                   CREATED         ...\n# 873eca95a051      ${env}-application  \"python ./manage.py run\"  10 seconds ago  ...\n```\n\n### Built with the `.env` file\n\nset environment variables in *[.env](.env.%7Bdemo%7D)* file.\n\n```yaml\n# Main Configurations\nAPIKEY: \"\u003capi-key-in-env\u003e\"\nDB_HOST: \"\u003chost\u003e\"\nDB_NAME: \"\u003cdatabase-name\u003e\"\nDB_USER: \"\u003cuser\u003e\"\nDB_PASS: \"\u003cpassword\u003e\"\nDB_PORT: \"5432\"\nAI_SCHEMA: \"ai\"\nMAIN_SCHEMA: \"public\"\n\n# Optional for SSH Tunnel to Private Database in Local Machine\nSSH_FLAG: \"True\"\nSSH_HOST: \"\u003chost\u003e\"\nSSH_USER: \"\u003cuser\u003e\"\nSSH_PRIVATE_KEY: \"\u003c`.pem` file in ./conf\u003e\"\nSSH_PORT: \"22\"\n```\n\n*Docker image*:\n\n```shell\n$ export env = \"development\"\n$ sudo docker build -t ${env}-application .\n$ sudo docker images\n# REPOSITORY          TAG        IMAGE ID       CREATED          ...\n# ${env}-application  latest     b50e8cdd83ed   10 seconds ago   ...\n```\n\n*Docker container*:\n\n```shell\n$ mkdir -p log\n$ sudo docker run --name=${env}-application --restart=always \\\n  -v $(pwd)/.env:/app/.env \\\n  --restart=always -d -p 5000:5000 ${env}-application\n$ sudo docker ps\n# CONTAINER ID      IMAGE               COMMAND                   CREATED         ...\n# 873eca95a051      ${env}-application  \"python ./manage.py run\"  10 seconds ago  ...\n```\n\n### Built with Docker Compose\n\nStart Docker Compose: \n\n```shell\ndocker-compose  -f .\\docker-compose.pg.yml up -d --build --force-recreate\n```\n\nClear all Docker Compose component:\n\n```shell\n$ docker-compose -f .\\docker-compose.pg.yml down --rmi all --volumes\nStopping postgres ... done\nRemoving postgres ... done\nRemoving network flask-rds-data-engine_backend-network\nRemoving volume flask-rds-data-engine_pgdata\nRemoving image postgres:latest \n```\n\n## API Document\n\nThe first thing you should do after running this application is to perform an API\nhealth checking with below the curl command,\n\n*Without `APIKEY`*:\n\n```shell\n$ curl --location --request GET 'http://127.0.0.1:5000/api'\n{'message': \"Success: Application was running ...\"}\n```\n\n*Included `APIKEY`*:\n```shell\n$ curl --location --request GET 'http://127.0.0.1:5000/apikey' \\\n  --header 'APIKEY: \u003capi-key-in-env\u003e'\n{\"message\": \"Success: Connect with the apikey, the application was running ...\"}\n```\n\u003e [!NOTE]\n\u003e If the `APIKEY` does not match with the application config, it will respond with\n\u003e `{ \"message\": \"Error: Unauthorised with 'APIKEY'\" }`.\n\nRead more information, [API documents](docs/README.md)\n\n## Application CI/CD Flow\n\nThis application framework use [GoCD](https://www.gocd.org/) for create CI/CD pipeline\nto deploy with difference environments (like dev, sit, or prod)\nwhen the source code in this main repository was merged.\n\nIn the GoCD pipeline, it has 3 steps on the CI/CD pipeline,\n\n1) Build container image\n2) Push the image to GoCD image registry\n3) Deploy the image from registry to ECS\n\n![Application CI/CD Flow](docs/image/application_cicd_flow.png)\n\n## Service Reference\n\n- AWS\n  - ~~AWS EC2~~\n    - virtual machine / virtual server\n    - `python version 3.8`\n  - AWS RDS (PostgreSQL)\n    - database service: `postgrest version 12.6`\n  - AWS ECR\n    - image service for keep application image\n  - AWS ECS\n    - container service for running application\n  - AWS Systems Manager\n    - use Parameter Store for control secret parameters\n    - such as `APIKEY`, `DB_HOST`, ...\n  - ~~AWS Athena (additional)~~\n    - monitor source files that was uploaded by users\n    - config AWS Glue catalog and sync catalog to dataset in AWS Athena\n  - AWS Cloudwatch (additional)\n    - monitor logging of AWS ECS\n  - ~~AWS Lambda (additional)~~\n    - create export task for generate dashboard data for platform\n  - ~~AWS Glue~~\n- GoCD\n  - for create CI/CD pipeline with difference environments\n- GitHub\n  - code repository\n- Confluence\n  - project document for deliver to customer\n- Jira\n  - project timeline planning and task assignment\n- Slack\n  - meeting and communication with team\n  - ~~alert error log from AWS ECS~~\n\n## License\n\nThis project was licensed under the terms of the [MIT license](LICENSE.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddeutils%2Fdata-orchestra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fddeutils%2Fdata-orchestra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fddeutils%2Fdata-orchestra/lists"}