{"id":23238482,"url":"https://github.com/zkan/building-data-pipelines-with-apache-airflow","last_synced_at":"2025-08-19T23:32:56.949Z","repository":{"id":61689426,"uuid":"540262547","full_name":"zkan/building-data-pipelines-with-apache-airflow","owner":"zkan","description":"Building Data Pipelines with Apache Airflow","archived":false,"fork":false,"pushed_at":"2022-09-25T14:54:53.000Z","size":11122,"stargazers_count":2,"open_issues_count":0,"forks_count":11,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-14T13:48:40.140Z","etag":null,"topics":["apache-airflow","data-pipelines","docker"],"latest_commit_sha":null,"homepage":"","language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zkan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-23T03:21:25.000Z","updated_at":"2023-03-16T13:30:43.000Z","dependencies_parsed_at":"2022-10-20T05:00:52.763Z","dependency_job_id":null,"html_url":"https://github.com/zkan/building-data-pipelines-with-apache-airflow","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkan%2Fbuilding-data-pipelines-with-apache-airflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkan%2Fbuilding-data-pipelines-with-apache-airflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkan%2Fbuilding-data-pipelines-with-apache-airflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkan%2Fbuilding-data-pipelines-with-apache-airflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zkan","download_url":"https://codeload.github.com/zkan/building-data-pipelines-with-apache-airflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230374249,"owners_count":18216044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-airflow","data-pipelines","docker"],"created_at":"2024-12-19T04:17:57.701Z","updated_at":"2024-12-19T04:17:58.286Z","avatar_url":"https://github.com/zkan.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Building Data Pipelines with Apache Airflow\n\n## Starting Airflow\n\n```sh\nmkdir -p mnt/dags mnt/logs mnt/plugins\necho -e \"AIRFLOW_UID=$(id -u)\" \u003e .env\ndocker-compose up\n```\n\n## Airflow S3 Connection to MinIO\n\nSince MinIO offers S3 compatible object storage, we can set the connection type to S3. However, we'll need to set an extra option, so that Airflow connects to MinIO instead of S3.\n\n- Connection Name: `minio` or any name you like\n- Connection Type: S3\n- Login: `\u003creplace_here_with_your_minio_access_key\u003e`\n- Password: `\u003creplace_here_with_your_minio_secret_key\u003e`\n- Extra: a JSON object with the following properties:\n  ```json\n  {\n    \"host\": \"http://minio:9000\"\n  }\n  ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzkan%2Fbuilding-data-pipelines-with-apache-airflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzkan%2Fbuilding-data-pipelines-with-apache-airflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzkan%2Fbuilding-data-pipelines-with-apache-airflow/lists"}