{"id":13485721,"url":"https://github.com/dsaidgovsg/airflow-pipeline","last_synced_at":"2025-03-27T19:31:43.762Z","repository":{"id":41871819,"uuid":"69946281","full_name":"dsaidgovsg/airflow-pipeline","owner":"dsaidgovsg","description":"An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR","archived":false,"fork":false,"pushed_at":"2023-11-15T10:00:39.000Z","size":143,"stargazers_count":174,"open_issues_count":3,"forks_count":59,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-03-25T16:02:15.895Z","etag":null,"topics":["airflow","docker","hadoop","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dsaidgovsg.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2016-10-04T08:22:54.000Z","updated_at":"2024-12-06T10:24:31.000Z","dependencies_parsed_at":"2023-11-15T11:24:59.573Z","dependency_job_id":"a8a62f05-4f47-4465-8639-f51b26c6de3c","html_url":"https://github.com/dsaidgovsg/airflow-pipeline","commit_stats":null,"previous_names":["datagovsg/airflow-pipeline"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsaidgovsg%2Fairflow-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsaidgovsg%2Fairflow-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsaidgovsg%2Fairflow-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dsaidgovsg%2Fairflow-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dsaidgovsg","download_url":"https://codeload.github.com/dsaidgovsg/airflow-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245806452,"owners_count":20675298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","docker","hadoop","spark"],"created_at":"2024-07-31T18:00:30.500Z","updated_at":"2025-03-27T19:31:43.420Z","avatar_url":"https://github.com/dsaidgovsg.png","language":"Python","funding_links":[],"categories":["Soluções de deployment do Airflow"],"sub_categories":[],"readme":"# Airflow Pipeline Docker Image Set-up\n\n![CI Status](https://img.shields.io/github/workflow/status/dsaidgovsg/airflow-pipeline/CI/master?label=CI\u0026logo=github\u0026style=for-the-badge)\n\nThis repo is a GitHub Actions build matrix set-up to generate Docker images of\n[Airflow](https://airflow.incubator.apache.org/), and other major applications\nas below:\n\n- Airflow\n- Spark\n- Hadoop integration with Spark\n- Python\n- SQL Alchemy\n\nAdditionally, `poetry` is used to perform all Python related installations at a\npredefined global project directory, so that it is easy to add on new packages\nwithout conflicting dependency package versions, which raw `pip` cannot achieve.\nSee\n\u003chttps://github.com/dsaidgovsg/spark-k8s-addons#how-to-properly-manage-pip-packages\u003e\nfor more information.\n\nFor builds involving Airflow v2 onwards, note that `poetry` is not officially\nsupported as an installation tool, but it is used anyway to make sure dependencies\nare compatible and tested to work across multiple builds with different versions.\n\nSee \u003chttps://github.com/apache/airflow/issues/13149\u003e for a related discussion\nand how to resolve possible conflicts when installing packages on top of this\nbase image.\n\n## Entrypoint\n\nAlso, for convenience, the current version runs both the `webserver` and\n`scheduler` together in the same instance by the default entrypoint, with the\n`webserver` being at the background and `scheduler` at the foreground. All the\nconvenient environment variables only works on the basis that the entrypoint is\nused without any extra command.\n\nIf there is a preference to run the various Airflow CLI services separately,\nyou can simply pass the full command into the Docker command, but it will no\nlonger trigger any of the convenient environment variables / functionalities.\n\nThe above convenience functionalities include:\n\n1. Discovering if database (`sqlite` and `postgres`) is ready\n2. Automatically running `airflow db init` and `airflow db upgrade`\n3. Easy creation of Airflow Web UI admin user by simple env vars.\n\nSee [`entrypoint.sh`](entrypoint.sh) for more details\nand the list of convenient environment variables.\n\nAlso note that the command that will be run will also be run as `airflow`\nuser/group, unless the host overrides the user/group to run the Docker\ncontainer.\n\n## Running locally\n\nYou will need `docker-compose` and `docker` command installed.\n\n### Default Combined Airflow Webserver and Scheduler\n\n```bash\ndocker-compose up --build\n```\n\nNavigate to `http://localhost:8080/`, and log in using the following RBAC\ncredentials to try out the DAGs:\n\n- Username: `admin`\n- Password: `Password123`\n\nNote that the `webserver` logs are suppressed by default.\n\n`CTRL-C` to gracefully terminate the services.\n\n### Separate Airflow Webserver and Scheduler\n\n```bash\ndocker-compose -f docker-compose.split.yml up --build\n```\n\nNavigate to `http://localhost:8080/` to try out the DAGs.\n\nBoth `webserver` and `scheduler` logs are shown separately.\n\n`CTRL-C` to gracefully terminate the services.\n\n## Versioning\n\nStarting from Docker tags that give self-version `v1`, any Docker image usage\nrelated breaking change will generate a new self-version so that this will\nminimize any impact on the user-facing side trying to use the most updated\nimage.\n\nThese are considered breaking changes:\n\n- Change of Linux distro, e.g. Alpine \u003c-\u003e Debian. This will automatically lead\n  to a difference in the package management tool used such as `apk` vs `apt`.\n  Note that however this does not include upgrading of Linux distro that may\n  affect the package management, e.g. `alpine:3.9` vs `alpine:3.10`.\n- Removal of advertized installed CLI tools that is not listed within the\n  Docker tag. E.g. Spark and Hadoop are part of the Docker tag, so they are not\n  part of the advertized CLI tools.\n- Removal of advertized environment variables\n- Change of any environment variable value\n\nIn the case where a CLI tool is known to perform a major version upgrade, this\nset-up will try to also release a new self-version number. But note that this is\nat a best effort scale only because most of the tools are inherited upstream,\nor simply unable / undesirable to specify the version to install.\n\n## Airflow provider packages\n\nAirflow provider packages have been removed from the image from version `v8`\nonwards and users will have to manually install them instead. Note that\nprovider packages follow their own versioning independent of Airflow's.\n\nSee \u003chttps://airflow.apache.org/docs/apache-airflow/2.1.0/backport-providers.html#backport-providers\u003e\nfor more details.\n\n```\n# Airflow V2\npoetry add apache-airflow-provider-apache-spark==1.0.3\n\n# Airflow V1\npoetry add apache-airflow[spark]==1.10.z\n```\n\n## Changelogs\n\nAll self-versioned change logs are listed in [`CHANGELOG.md`](CHANGELOG.md).\n\nThe advertized CLI tools and env vars are also listed in the detailed change\nlogs.\n\n## How to Manually Build Docker Image\n\nExample build command:\n\n```bash\nAIRFLOW_VERSION=2.3\nSPARK_VERSION=3.3.0\nHADOOP_VERSION=3.3.2\nSCALA_VERSION=2.12\nJAVA_VERSION=11\nPYTHON_VERSION=3.9\nSQLALCHEMY_VERSION=1.4\ndocker build -t airflow-pipeline \\\n  --build-arg \"AIRFLOW_VERSION=${AIRFLOW_VERSION}\" \\\n  --build-arg \"SPARK_VERSION=${SPARK_VERSION}\" \\\n  --build-arg \"HADOOP_VERSION=${HADOOP_VERSION}\" \\\n  --build-arg \"SCALA_VERSION=${SCALA_VERSION}\" \\\n  --build-arg \"PYTHON_VERSION=${PYTHON_VERSION}\" \\\n  --build-arg \"JAVA_VERSION=${JAVA_VERSION}\" \\\n  --build-arg \"SQLALCHEMY_VERSION=${SQLALCHEMY_VERSION}\" \\\n  .\n```\n\nYou may refer to the [vars.yml](templates/vars.yml) to have a sensing of all the\npossible build arguments to combine.\n\n## Caveat\n\nBecause this image is based on Spark with Kubernetes compatible image, which\nalways generates Debian based Docker images, the images generated from this\nrepository are likely to stay Debian based as well. But note that there is no\nguarantee that this is always true, but such changes are always marked with\nDocker image release tag.\n\nAlso, currently the default entrypoint without command logic assumes that\na Postgres server will always be used (the default `sqlite` can work as an\nalternative). As such, when using in this mode, an external Postgres server\nhas to be made available for Airflow services to access.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsaidgovsg%2Fairflow-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdsaidgovsg%2Fairflow-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdsaidgovsg%2Fairflow-pipeline/lists"}