{"id":13627415,"url":"https://github.com/ml-tooling/universal-build","last_synced_at":"2025-04-24T16:30:48.622Z","repository":{"id":46060236,"uuid":"305420552","full_name":"ml-tooling/universal-build","owner":"ml-tooling","description":"🔨 Universal build utilities for containerized build pipelines.","archived":false,"fork":false,"pushed_at":"2022-11-09T08:14:25.000Z","size":1145,"stargazers_count":25,"open_issues_count":1,"forks_count":7,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-23T06:06:08.972Z","etag":null,"topics":["act","build","build-automation","build-tool","ci","cicd","containerization","docker","github-actions"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ml-tooling.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null}},"created_at":"2020-10-19T14:57:21.000Z","updated_at":"2025-01-12T14:59:12.000Z","dependencies_parsed_at":"2022-09-18T00:00:41.961Z","dependency_job_id":null,"html_url":"https://github.com/ml-tooling/universal-build","commit_stats":null,"previous_names":[],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-tooling%2Funiversal-build","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-tooling%2Funiversal-build/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-tooling%2Funiversal-build/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-tooling%2Funiversal-build/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ml-tooling","download_url":"https://codeload.github.com/ml-tooling/universal-build/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250663544,"owners_count":21467366,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["act","build","build-automation","build-tool","ci","cicd","containerization","docker","github-actions"],"created_at":"2024-08-01T22:00:33.822Z","updated_at":"2025-04-24T16:30:48.325Z","avatar_url":"https://github.com/ml-tooling.png","language":"Python","readme":"\u003c!-- markdownlint-disable MD033 MD041 --\u003e\n\u003ch1 align=\"center\"\u003e\n    universal-build\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003cstrong\u003eUniversal build utilities for containerized build pipelines.\u003c/strong\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://pypi.org/project/universal-build/\" title=\"PyPi Version\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/universal-build?color=green\u0026style=flat\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/universal-build/\" title=\"Python Version\"\u003e\u003cimg src=\"https://img.shields.io/badge/Python-3.6%2B-blue\u0026style=flat\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/ml-tooling/universal-build/actions?query=workflow%3Abuild-pipeline\" title=\"Build status\"\u003e\u003cimg src=\"https://img.shields.io/github/workflow/status/ml-tooling/universal-build/build-pipeline?style=flat\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/ml-tooling/universal-build/blob/main/LICENSE\" title=\"Project License\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg?style=flat\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://gitter.im/ml-tooling/universal-build/\" title=\"Chat on Gitter\"\u003e\u003cimg src=\"https://badges.gitter.im/ml-tooling/universal-build.svg\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://twitter.com/mltooling\" title=\"ML Tooling on Twitter\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/mltooling.svg?label=follow\u0026style=social\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#getting-started\"\u003eGetting Started\u003c/a\u003e •\n  \u003ca href=\"#features\"\u003eFeatures\u003c/a\u003e •\n  \u003ca href=\"#documentation\"\u003eDocumentation\u003c/a\u003e •\n  \u003ca href=\"#support--feedback\"\u003eSupport\u003c/a\u003e •\n  \u003ca href=\"#contribution\"\u003eContribution\u003c/a\u003e •\n  \u003ca href=\"https://github.com/ml-tooling/universal-build/releases\"\u003eChangelog\u003c/a\u003e •\n  \u003ca href=\"#faq--known-issues\"\u003eFAQ\u003c/a\u003e\n\u003c/p\u003e\n\nUniversal-build is a set of utilities designed to build, test, package, and release software. It enables you to implement your build and release pipeline with Python scripts once and run it either on your local machine, in a containerized environment via [Act](https://github.com/nektos/act), or automated via [Github Actions](https://github.com/features/actions). It supports a monorepo or polyrepo setup and can be used with any programming language or technology. It also provides a full release pipeline for automated releases with changelog generation.\n\n\u003e _**WIP**: This project is still an alpha version and not ready for general usage._\n\n## Highlights\n\n- 🐳\u0026nbsp; Implement once and run locally, containerized, or on Github Actions.\n- 🧰\u0026nbsp; Build utilities for Python, Docker, React \u0026 MkDocs.\n- 🔗\u0026nbsp; Predefined Github Action Workflows for CI \u0026 CD.\n- 🛠\u0026nbsp; Integrated with [devcontainer](https://code.visualstudio.com/docs/remote/containers) for containerized development.\n\n## Getting Started\n\n### Installation\n\n\u003e _Requirements: Python 3.6+._\n\n```bash\npip install universal-build\n```\n\n### Usage\n\nTo make use of universal build for your project, create a build script with the name `build.py` in your project root. The example below is for a single yarn-based webapp component:\n\n```python\nfrom universal_build import build_utils\n\nargs = build_utils.parse_arguments()\n\nversion = args.get(build_utils.FLAG_VERSION)\n\nif args.get(build_utils.FLAG_MAKE):\n    build_utils.log(\"Build the component:\")\n    build_utils.run(\"yarn build\")\n\nif args.get(build_utils.FLAG_CHECK):\n    build_utils.log(\"Run linters and style checks:\")\n    build_utils.run(\"yarn run lint:js\")\n    build_utils.run(\"yarn run lint:css\")\n\nif args.get(build_utils.FLAG_TEST):\n    build_utils.log(\"Test the component:\")\n    build_utils.run(\"yarn test\")\n\nif args.get(build_utils.FLAGE_RELEASE):\n    build_utils.log(\"Release the component:\")\n    # TODO: release the component to npm with version\n\n```\n\nNext, copy the [`build-environment`](https://github.com/ml-tooling/universal-build/blob/main/actions/build-environment) action from the [actions](https://github.com/ml-tooling/universal-build/tree/main/actions) folder into the `.github/actions` folder of your repository. In addition, you need to copy the [build-](https://github.com/ml-tooling/universal-build/blob/main/workflows/build-pipeline.yml) and [release-pipeline](https://github.com/ml-tooling/universal-build/blob/main/workflows/release-pipeline.yml) workflows from the [workflows](https://github.com/ml-tooling/universal-build/tree/main/workflows) folder into the `.github/workflows` folder of your repository as well. Your repository should now contain atleast the following files:\n\n```\nyour-repository\n  - build.py\n  - .github:\n    - actions:\n      - build-enviornment:\n        - Dockerfile\n        - actions.yaml\n    - workflows:\n      - release-pipeline.yml\n      - build-pipeline.yml\n```\n\nOnce you have pushed the `build-environment` action and the [build-](https://github.com/ml-tooling/universal-build/blob/main/workflows/build-pipeline.yml) and [release-pipelines](https://github.com/ml-tooling/universal-build/blob/main/workflows/release-pipeline.yml), please look into the [Automated Build Pipeline](#automated-build-pipeline-ci) and [Automated Release Pipeline](#automated-release-pipeline-cd) sections for information on how to run your build- and release-pipelines.\n\nYou can find a more detailed example project with multiple components in the [examples](https://github.com/ml-tooling/universal-build/tree/main/examples) folder.\n\n---\n\n\u003cbr\u003e\n\n## Support \u0026 Feedback\n\nThis project is maintained by [Benjamin Räthlein](https://twitter.com/raethlein), [Lukas Masuch](https://twitter.com/LukasMasuch), and [Jan Kalkan](https://www.linkedin.com/in/jan-kalkan-b5390284/). Please understand that we won't be able to provide individual support via email. We also believe that help is much more valuable if it's shared publicly so that more people can benefit from it.\n\n| Type                           | Channel                                                                                                                                                                                                                                                                                                                                        |\n| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| 🚨\u0026nbsp; **Bug Reports**       | \u003ca href=\"https://github.com/ml-tooling/universal-build/issues?utf8=%E2%9C%93\u0026q=is%3Aopen+is%3Aissue+label%3Abug+sort%3Areactions-%2B1-desc+\" title=\"Open Bug Report\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/ml-tooling/universal-build/bug.svg?label=bug\"\u003e\u003c/a\u003e                                                                         |\n| 🎁\u0026nbsp; **Feature Requests**  | \u003ca href=\"https://github.com/ml-tooling/universal-build/issues?q=is%3Aopen+is%3Aissue+label%3Afeature+sort%3Areactions-%2B1-desc\" title=\"Open Feature Request\"\u003e\u003cimg src=\"https://img.shields.io/github/issues/ml-tooling/universal-build/feature.svg?label=feature%20request\"\u003e\u003c/a\u003e                                                              |\n| 👩‍💻\u0026nbsp; **Usage Questions**   | \u003ca href=\"https://stackoverflow.com/questions/tagged/ml-tooling\" title=\"Open Question on Stackoverflow\"\u003e\u003cimg src=\"https://img.shields.io/badge/stackoverflow-ml--tooling-orange.svg\"\u003e\u003c/a\u003e \u003ca href=\"https://gitter.im/ml-tooling/universal-build\" title=\"Chat on Gitter\"\u003e\u003cimg src=\"https://badges.gitter.im/ml-tooling/universal-build.svg\"\u003e\u003c/a\u003e |\n| 🗯\u0026nbsp; **General Discussion** | \u003ca href=\"https://gitter.im/ml-tooling/universal-build\" title=\"Chat on Gitter\"\u003e\u003cimg src=\"https://badges.gitter.im/ml-tooling/universal-build.svg\"\u003e\u003c/a\u003e \u003ca href=\"https://twitter.com/mltooling\" title=\"ML Tooling on Twitter\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/mltooling.svg?style=social\"\u003e\u003c/a\u003e                                   |\n| ❓\u0026nbsp; **Other Requests**    | \u003ca href=\"mailto:team@mltooling.org\" title=\"Email ML Tooling Team\"\u003e\u003cimg src=\"https://img.shields.io/badge/email-ML Tooling-green?logo=mail.ru\u0026logoColor=white\"\u003e\u003c/a\u003e                                                                                                                                                                             |\n\n---\n\n\u003cbr\u003e\n\n## Documentation\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#build-script-cli\"\u003eBuild Script CLI\u003c/a\u003e •\n  \u003ca href=\"#default-flags\"\u003eDefault Flags\u003c/a\u003e •\n  \u003ca href=\"#api-reference\"\u003eAPI Reference\u003c/a\u003e •\n  \u003ca href=\"#update-universal-build\"\u003eUpdate Universal Build\u003c/a\u003e\n\u003c/p\u003e\n\n### Build Script CLI\n\nAny build script that utilizes the `build_utils.parse_arguments()` method to parse the CLI arguments can be executed with the following options:\n\n```bash\npython build.py [OPTIONS]\n```\n\n**Options**:\n\n\u003e _These options correspond to the default flags documented in the next section._\n\n- `--make`: Make/compile/package all artifacts.\n- `--test`: Run unit and integration tests.\n- `--check`: Run linting and style checks.\n- `--release`: Release all artifacts (e.g. to registries like DockerHub or NPM).\n- `--run`: Run the component in development mode (e.g. dev server).\n- `--version VERSION`: Version of the build (`MAJOR.MINOR.PATCH-TAG`).\n- `--force`: Ignore all enforcements and warnings.\n- `--skip-path SKIP_PATH`: Skips the build phases for all (sub)paths provided here. This option can be used multiple times.\n- `--test-marker TEST_MARKER`: Provide custom markers for testing. The default marker for slow tests is `slow`. This option can be used multiple times.\n- `-h, --help`: Show the help message and exit.\n\n### Default Flags\n\nAt its core, universal-build will parse all arguments provided to the build script via `build_utils.parse_arguments()` and returns a sanitized and augmented list of arguments. Those arguments are the building blocks for your build script. You can utilize those arguments in whatever way you like. Here is an example on how to use those arguments in a `build.py` script:\n\n```python\nfrom universal_build import build_utils\n\nargs = build_utils.parse_arguments()\n\nversion = args.get(build_utils.FLAG_VERSION)\n\nif args.get(build_utils.FLAG_MAKE):\n  # Run all relevant build commands.\n\nif args.get(build_utils.FLAG_TEST):\n  # Run all relevant commands for testing\n  test_markers = args.get(build_utils.FLAG_TEST_MARKER)\n  if \"slow\" in test_markers:\n    # Run additional slow tests.\n```\n\nThe following list contains all of the default flags currently supported by universal-build:\n\n| Flag               | Type        | Description                                                                                                              |\n| ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------ |\n| `FLAG_MAKE`        | `bool`      | Build/compile/package all artifacts.                                                                                     |\n| `FLAG_CHECK`       | `bool`      | Run linting and style checks.                                                                                            |\n| `FLAG_TEST`        | `bool`      | Run unit and integration tests.                                                                                          |\n| `FLAG_RELEASE`     | `bool`      | Release all artifacts (e.g. to registries like DockerHub or NPM).                                                        |\n| `FLAG_RUN`         | `bool`      | Run the component in development mode (e.g. dev server).                                                                 |\n| `FLAG_FORCE`       | `bool`      | Ignore all enforcements and warnings.                                                                                    |\n| `FLAG_VERSION`     | `str`       | Semantic version for the build. If not provided via CLI arguments, a valid dev version will be automatically calculated. |\n| `FLAG_TEST_MARKER` | `List[str]` | Custom markers for testing. Can be used to skip or execute certain tests.                                                |\n\n### API Reference\n\nIn addition to argument parsing capabilities, universal-build also contains a variety of utility functions to make building complex projects with different technologies easy. You can find all utilities in the Python API documentation [here](https://github.com/ml-tooling/universal-build/tree/main/docs).\n\n### Update Universal Build\n\nTo update the universal-build version of your project, simply look up the most recent version of build-environment on [DockerHub](https://hub.docker.com/repository/docker/mltooling/build-environment) and set this version in the `.github/actions/build-environment/Dockerfile` file of your repository:\n\n```Dockerfile\nFROM mltooling/build-environment:\u003cUPDATED_VERSION\u003e\n```\n\nIn case you also run your build outside of the build-environment (locally), make sure to also upgrade universal-build on your local machine from [PyPi](https://pypi.org/project/universal-build/):\n\n```bash\npip install --upgrade universal-build\n```\n\nFurthermore, you can also check if the [build-](https://github.com/ml-tooling/universal-build/blob/main/workflows/build-pipeline.yml) and [release-pipeline](https://github.com/ml-tooling/universal-build/blob/main/workflows/release-pipeline.yml) workflows have changed. In case of changes, update the workflows in your `.github/workflows` folder of your repository as well.\n\n---\n\n\u003cbr\u003e\n\n## Features\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#support-for-nested-components\"\u003eSupport for Nested Components\u003c/a\u003e •\n  \u003ca href=\"#automated-build-pipeline-ci\"\u003eAutomated Build Pipeline\u003c/a\u003e •\n  \u003ca href=\"#automated-release-pipeline-cd\"\u003eAutomated Release Pipeline\u003c/a\u003e •\n  \u003ca href=\"#containerized-development\"\u003eContainerized Development\u003c/a\u003e •\n  \u003ca href=\"#simplified-versioning\"\u003eSimplified Versioning\u003c/a\u003e •\n  \u003ca href=\"#mkdocs-utilities\"\u003eMkDocs Utilities\u003c/a\u003e •\n  \u003ca href=\"#python-utilities\"\u003ePython Utilities\u003c/a\u003e •\n  \u003ca href=\"#docker-utilities\"\u003eDocker Utilities\u003c/a\u003e •\n  \u003ca href=\"#extensibility\"\u003eExtensibility\u003c/a\u003e\n\u003c/p\u003e\n\n### Automated Build Pipeline (CI)\n\nUniversal-build enables you to run your build pipeline on your local machine, in a containerized environment via [Act](https://github.com/nektos/act), or automated via [Github Actions](https://github.com/features/actions) (= Continuous Integration).\n\n#### Local machine via build script (not recommended):\n\n\u003e _Requirements: [universal-build](#installation) and all the build requirements that your build script is using (e.g. yarn, pipenv, maven, ...) need to be installed on your machine._\n\nExecute the following command in the root folder of any component with a valid `build.py` script:\n\n```bash\npython build.py --make --check --test\n```\n\nExecuting the build-pipeline directly via the build scripts is not recommended.\n\n#### Containerized environment via Act:\n\n\u003e _Requirements: [Docker](https://docs.docker.com/get-docker/) and [Act](https://github.com/nektos/act#installation) are required to be installed on your machine._\n\nExecute this command in the root folder of your repository:\n\n```bash\nact -b -s BUILD_ARGS=\"--check --make --test\" -j build\n```\n\n#### Manually via Github Actions:\n\nIn the Github UI, go to `Actions` -\u003e select `build-pipeline` -\u003e select `Run Workflow` and provide the build arguments, e.g. `--check --make --test`.\n\n#### Automated via Github Actions (CI):\n\nWith the default configuration, the build pipeline will run automatically via Github Actions on any `push` event to your repository. This automation can be referred to as continuous integration. You can also change the events that trigger the build-pipeline by modifying the `on` section in the `.github/workflows/build-pipeline.yml` file. You can find more information about Github Actions events [here](https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows).\n\n### Automated Release Pipeline (CD)\n\nTo release a new version and publish all relevant artifacts to the respective registries (e.g. Docker image to DockerHub) you can either trigger our release pipeline on your local machine, in a containerized environment via [Act](https://github.com/nektos/act), or automated via [Github Actions](https://github.com/features/actions) (= Continuous Delivery).\n\n#### Local machine via build script (not recommended):\n\n\u003e _Requirements: [universal-build](#installation) and all the build requirements that your build script is using (e.g. yarn, pipenv, maven, ...) need to be installed on your machine._\n\nExecute the following command in the root folder of any component with a valid `build.py` script:\n\n```bash\npython build.py --make --check --test --release --version=\"\u003cMAJOR.MINOR.PATCH\u003e\"\n```\n\nExecuting the release step directly via the build scripts is not recommended.\n\n#### Containerized environment via Act:\n\n\u003e _Requirements: [Docker](https://docs.docker.com/get-docker/) and [Act](https://github.com/nektos/act#installation) are required to be installed on your machine._\n\nExecute this command in the root folder of your repository:\n\n```bash\nact -b -s VERSION=\"\u003cMAJOR.MINOR.PATCH\u003e\" -j release\n```\n\nIn case you also want to automatically create a valid Github release, you also need to provide a valid `GITHUB_TOKEN` as a secret (`-s GITHUB_TOKEN=\u003ctoken\u003e`). Please refer to the next section for information on how to finish and publish the release.\n\n#### On Github Actions (CD):\n\n\u003e _Make sure that all required secrets for you release pipeline are configured in your Github repository. More information [here](ttps://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository)._\n\nTo trigger our release pipeline from Github UI, you can either close a milestone that has a valid version name (`vMAJOR.MINOR.PATCH`) or execute the release pipeline manually via the `workflow_dispatch` UI in the Action Tab (`Actions -\u003e release-pipeline -\u003e Run Workflow`). The release pipeline will automatically run the build, check, test, and release steps, and create a pull request for the new version as well as a draft release on Github. This automation can be referred to as continuous delivery.\n\nAfter successful execution of the release pipeline, the following steps are required to finish the release:\n\n1. Merge the release PR into `main`. Preferably via merge commit to keep the version tag in the `main` branch. We suggest to use the following message for the merge commit: `Finalize release for version \u003cVERSION\u003e (#\u003cPR\u003e)`.\n2. Adapt the changelog of the draft release on Github (in the release section). Mention all other changes that are not covered by pull requests.\n3. Publish the release.\n\n#### Resolve an unsuccessful release:\n\nIn case the release pipeline fails at any step, we suggest to fix the problem based on the release pipeline logs and create a new release with an incremented `patch` version. To clean up the unsuccessful release, make sure to delete the following artifacts (if they exist): the release branch, the release PR, the version tag, the draft release, and any release artifact that was already published (e.g. on DockerHub, NPM or PyPi).\n\n### Support for Nested Components\n\n\u003e _You can find the implementation of this multi-nested example in the [examples](https://github.com/ml-tooling/universal-build/tree/main/examples) folder._\n\nUniversal-build has excellent support for repositories that contain multiple nested components (aka Monorepo). The following [`examples`](https://github.com/ml-tooling/universal-build/tree/main/examples) repository has four components: `docs`, `react-webapp`, `docker`, and `python-lib`:\n\n```plain\nexamples:\n  - build.py\n  - docs:\n    - build.py\n  - react-webapp:\n    - build.py\n  - docker:\n    - build.py\n  - python-lib:\n    - build.py\n```\n\nEvery component needs its own `build.py` script in the component root folder that implements all the logic to build, check, test, and release the given component. The `build.py` script in the repo root folder contains the build logic that orchestrates all component builds. Universal-build provides the [`build_utils.build()`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.build_utils.md#function-build) function that allows to call the build script of a sub-component with the parsed arguments (find more info on `build` function in the [API documentation](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.build_utils.md#function-build)).\n\nIn between the build steps, you can execute any required operations, for example, duplicating build artifacts from one component to another. The following example, shows the `build.py` script that would support the `examples` repository structure:\n\n```python\nfrom universal_build import build_utils\n\nargs = build_utils.parse_arguments()\n\nbuild_utils.build(\"react-webapp\", args)\nbuild_utils.build(\"python-lib\", args)\nbuild_utils.duplicate_folder(\"./python-lib/docs/\", \"./docs/docs/api-docs/\")\nbuild_utils.build(\"docker\", args)\nbuild_utils.build(\"docs\", args)\n```\n\nWith this setup, you can execute the build pipeline for the full project or any individual component. In case you only apply changes to a single component, you only need to execute the `build.py` script of the given component. This is a major advantage since it might massively speed up your development time.\n\nTo run the build pipeline on you local machine only for a specific component, navigate to the component and run the `build.py` script in the component root folder (you can find all CLI build arguments [here](#build-script-cli)):\n\n```bash\ncd \"./docs\" \u0026\u0026 python build.py [BUILD_ARGUMENTS]\n```\n\nAlternatively, you can also run the component build containerized via Act:\n\n```bash\nact -b -s BUILD_ARGS=\"[BUILD_ARGUMENTS]\" -s WORKING_DIRECTORY=\"./docs\" -j build\n```\n\nOr directly from the Github UI: `Actions` -\u003e `build-pipeline` -\u003e `Run workflow`. The Github UI will allow you to set the build arguments and working directory.\n\n### Simplified Versioning\n\n\u003e Only [semantic versioning](https://semver.org/) is supported at the moment.\n\nIf you do not provide an explicit version via the build arguments (`--version`), universal-build will automatically detect the latest version via Git tags and pass a dev version to your build scripts. The dev version will have the following format: `\u003cMAJOR\u003e.\u003cMINOR\u003e.\u003cPATCH\u003e-dev.\u003cBRANCH\u003e`. This should be sufficient for the majority of development builds. However, the release step still requires to have a valid semantic version provided via the arguments.\n\n### Python Utilities\n\nThe [`build_python`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_python.md) module of universal-build provides a collection of utilities to simplify the process of building and releasing Python packages. Refer to the [API documentation](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_python.md) for full documentation on all python utilities. An example for a build script for a Python package is shown below:\n\n```python\nfrom universal_build import build_utils\nfrom universal_build.helpers import build_python\n\n# Project specific configuration\nMAIN_PACKAGE = \"template_package\"\n\nargs = build_python.parse_arguments()\n\nversion = args.get(build_utils.FLAG_VERSION)\n\n# Update version in __version__.py\nbuild_python.update_version(os.path.join(HERE, f\"src/{MAIN_PACKAGE}/__version__.py\"), str(version))\n\nif args.get(build_utils.FLAG_MAKE):\n  # Install pipenv dev requirements\n  build_python.install_build_env()\n  # Build distribution via setuptools\n  build_python.build_distribution()\n\nif args.get(build_utils.FLAG_CHECK):\n  build_python.code_checks()\n\nif args.get(build_utils.FLAG_TEST):\n  build_utils.run('pipenv run pytest -m \"not slow\"')\n\n  if \"slow\" in args.get(build_utils.FLAG_TEST_MARKER):\n    build_python.test_with_py_version(python_version=\"3.6.12\")\n\nif args.get(build_utils.FLAG_RELEASE):\n  # Publish distribution on pypi\n  build_python.publish_pypi_distribution(pypi_token=args.get(build_python.FLAG_PYPI_TOKEN),pypi_repository=args.get(build_python.FLAG_PYPI_REPOSITORY))\n```\n\nThe [`build_python.parse_arguments()`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_python.md#function-parse_arguments) argument parser has the following additional flags:\n\n| Flag                   | Type  | Description                               |\n| ---------------------- | ----- | ----------------------------------------- |\n| `FLAG_PYPI_TOKEN`      | `str` | Personal access token for PyPI account.   |\n| `FLAG_PYPI_REPOSITORY` | `str` | PyPI repository for publishing artifacts. |\n\nAnd the following additional CLI options:\n\n- `--pypi-token`: Personal access token for PyPI account.\n- `--pypi-repository`: PyPI repository for publishing artifacts.\n\n### Docker Utilities\n\nThe [`build_docker`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_docker.md) module of universal-build provides a collection of utilities to simplify the process of building and releasing Docker images. Refer to the [API documentation](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_docker.md) for full documentation on all docker utilities. An example for a build script for a Docker image is shown below:\n\n```python\nfrom universal_build import build_utils\nfrom universal_build.helpers import build_docker\n\nIMAGE_NAME = \"build-environment\"\nDOCKER_IMAGE_PREFIX = \"mltooling\"\n\nargs = build_docker.parse_arguments()\n\nversion = args.get(build_utils.FLAG_VERSION)\n\nif args.get(build_utils.FLAG_MAKE):\n  build_docker.build_docker_image(COMPONENT_NAME, version)\n\nif args.get(build_utils.FLAG_CHECK):\n  build_docker.lint_dockerfile()\n\nif args.get(build_utils.FLAG_RELEASE):\n  build_docker.release_docker_image(IMAGE_NAME, version, DOCKER_IMAGE_PREFIX)\n```\n\nThe [`build_docker.parse_arguments()`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_docker.md#function-parse_arguments) argument parser has the following additional flags:\n\n| Flag                       | Type  | Description                                                                                                    |\n| -------------------------- | ----- | -------------------------------------------------------------------------------------------------------------- |\n| `FLAG_DOCKER_IMAGE_PREFIX` | `str` | Docker image prefix. This should be used to define the container registry where the image should be pushed to. |\n\nAnd the following additional CLI options:\n\n- `--docker-image-prefix`: Docker image prefix. This should be used to define the container registry where the image should be pushed to.\n\n### MkDocs Utilities\n\nThe [`build_mkdocs`](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_mkdocs.md) module of universal-build provides a collection of utilities to simplify the process of building and releasing MkDocs documentation. Refer to the [API documentation](https://github.com/ml-tooling/universal-build/blob/main/docs/universal_build.helpers.build_mkdocs.md) for full documentation on all MkDocs utilities. An example for a build script for MkDocs documentation is shown below:\n\n```python\nfrom universal_build import build_utils\nfrom universal_build.helpers import build_mkdocs\n\nargs = build_utils.parse_arguments()\n\nif args.get(build_utils.FLAG_MAKE):\n  # Install pipenv dev requirements\n  build_mkdocs.install_build_env()\n  # Build mkdocs documentation\n  build_mkdocs.build_mkdocs()\n\nif args.get(build_utils.FLAG_CHECK):\n  build_mkdocs.lint_markdown()\n\nif args.get(build_utils.FLAG_RELEASE):\n  # Deploy to Github pages\n  build_mkdocs.deploy_gh_pages()\n```\n\n### Extensibility\n\n#### Extend your build-environment image with additional tools\n\nInstall the tools in the Dockerfile in your `.github/actions/build-environment/Dockerfile` as demonstrated in this example:\n\n```Dockerfile\nFROM mltooling/build-environment:0.6.18\n\n# Install Go Runtime\nRUN apt-get update \\\n    \u0026\u0026 apt-get install -y golang-go\n```\n\n#### Extend the entrypoint of the build-environment\n\nYou can extend or overwrite the default entrypoint with your custom entrypoint script (e.g. `extended-entrypoint.sh`) as shown below:\n\n```Dockerfile\nFROM mltooling/build-environment:0.6.18\n\nCOPY extended-entrypoint.sh /extended-entrypoint.sh\n\nRUN chmod +x /extended-entrypoint.sh\n\nENTRYPOINT [\"/tini\", \"-g\", \"--\", \"/extended-entrypoint.sh\"]\n```\n\nThe following `extended-entrypoint.sh` example demonstrates how to extend and reuse the existing default entrypoint:\n\n```bash\n# Stops script execution if a command has an error\nset -e\n\necho \"Setup Phase\"\n\n# TODO: Do your custom setups here\n\n# Call the default build-environment entrypoint.\n# Disable the immediate script execution stop so that the cleanup phase can run in any case\nset +e\n# Thereby, you can reuse the existing implementation:\n/bin/bash /entrypoint.sh \"$@\"\n# Save the exit code of the previous command\nexit_code=$?\n\necho \"Cleanup Phase\"\n\n# TODO: Do additional cleanup\n\n# Exit the script with the exit code of the actual entrypoint execution\nexit $exit_code\n\n```\n\n#### Support additional build arguments\n\nThe following example demonstrates how you can support custom build arguments (CLI) in your `build.py` script:\n\n```python\nimport argparse\n\nfrom universal_build import build_utils\n\nparser = argparse.ArgumentParser()\nparser.add_argument(\"--deployment-token\", help=\"Token to deploy component.\", default=\"\")\n\nargs = build_utils.parse_arguments(argument_parser=parser)\n\ndeployment_token = args.get(\"deployment_token\")\n```\n\nOnce it is implemented in your build script, you can provide the build argument via the CLI options: `python build.py --deployment-token=my-token`. If your custom argument is a `string` and has a default string value (e.g. `default=\"\"`), you can also provide the build argument via environment variables: `DEPLOYMENT_TOKEN=mytoken python build.py`.\n\nTo use your custom build arguments inside the release pipeline, you need to add the `DEPLOYMENT_TOKEN` as a secret to your Github repository (more info [here](https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository)) and adapt the `.github/workflows/release-pipeline.yml` file by adding the `DEPLOYMENT_TOKEN` as an environment variable (`env`) to the steps that need this build argument, for example:\n\n```yaml\n- name: release-components\n  uses: ./.github/actions/build-environment\n  env:\n    DEPLOYMENT_TOKEN: ${{ secrets.DEPLOYMENT_TOKEN }}\n```\n\n#### Use custom test markers to select tests for execution\n\nYou can provide any number of custom test markers via the `--test-marker` build argument. The following example shows how to react to custom test markers in your build script:\n\n```python\nif args.get(build_utils.FLAG_TEST):\n  # Run your default tests\n  if \"integration\" in args.get(build_utils.FLAG_TEST_MARKER):\n    # Run integration tests\n```\n\n### Containerized Development\n\nThe [build-environment](./build-environment) can also be used for development inside a container. It is fully compatible with the [devcontainer](https://code.visualstudio.com/docs/remote/containers#_create-a-devcontainerjson-file) standard that is used by VS Code and Github Codespaces. The big advantage of using the build-environment for containerized development is that you only have to define your project dependencies in one location, and use this for development, local builds, and automated CI / CD pipelines.\n\nTo use the build-environment for containerized development, just define a `.devcontainer/devcontainer.json` configuration inside your repository and link the `build.dockerfile` to the build-environment action in the `.github/actions/build-environment/Dockerfile` folder. A minimal `devcontainer.json` configuration could look like this:\n\n```\n{\n  \"name\": \"build-environment\",\n  \"build\": {\n    \"dockerfile\": \"../.github/actions/build-environment/Dockerfile\"\n  },\n  \"settings\": {\n    // Set default container specific vs code settings\n    \"terminal.integrated.shell.linux\": \"/bin/bash\"\n  },\n  \"extensions\": [\n    // Add required extensions\n  ]\n}\n```\n\nYou can find a full example [here](https://github.com/ml-tooling/universal-build/blob/main/.devcontainer/devcontainer.json).\n\n---\n\n\u003cbr\u003e\n\n## FAQ \u0026 Known Issues\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003cb\u003eAct: Error response from daemon - volume is in use\u003c/b\u003e (click to expand...)\u003c/summary\u003e\n\nSometimes the act containers are not removed properly and are blocking any subsequent act executions of your workflow. As a workaround, you can just remove all act containers:\n\n```bash\ndocker rm -f $(docker ps -a --filter=\"name=^act-\" -q)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHow to access the host from Docker Containers in GitHub Actions / Act or containers from the host\u003c/b\u003e (click to expand...)\u003c/summary\u003e\n\nIf you want to access the host (in act the pipeline container and on GitHub Actions the Linux VM) from within a container, you can set an environment variable in the workflow file with this step:\n\n```yaml\n- name: set-host-ip\n  run: echo \"::set-env name=_HOST_IP::$(hostname -I | cut -d ' ' -f 1)\"\n  # new syntax which is not yet supported on act:\n  # run: echo \"_HOST_IP=$(hostname -I | cut -d ' ' -f 1)\" \u003e\u003e \"$GITHUB_ENV\"\n```\n\nand then access the environment variable from within a container. This way you can, for example, access other containers with published ports or other host services.\n\nIf you want to access a container directly without going through the host, you can get the IP address for example in the following way:\n\n```bash\ncontainer_id=\u003cCONTAINER-ID-OR-NAME\u003e\ncontainer_ip=$(docker inspect $container_id | jq -r '.[0].NetworkSettings.Networks.bridge.IPAddress')\n```\n\n\u003e Note that the tool `jq` has to be installed. If you run a python script and use the Docker client, the command looks different, of course.\n\nWhen you don't put starting containers into a custom network, the container is now reachable from the host (GitHub Actions \u0026 Act) as well as other containers under this `$container_ip` address. Yet, it is _not_ reachable from your local machine (e.g. your Mac). For that, you have to publish the port and use the `$_HOST_IP` address as explained above. The host port should be assigned randomly so that the setup is as host-independent as possible. To dynamically get the random port you can get it in the following way via bash:\n\n```bash\ncontainer_id=\u003cCONTAINER-ID-OR-NAME\u003e\ncontainer_port=\u003cINNER-CONTAINER-PORT\u003e\ncontainer_host_port=$(docker inspect $container_id | jq -r '.[0].NetworkSettings.Ports[\"'$container_port'/tcp\"][0].HostPort')\n```\n\nIn your code, you should then check whether the `$_HOST_IP` variable is set and if not, use `localhost`. This way, it will work on GitHub Actions, Act, and your local machine. Here is a Python example:\n\n```python\nimport docker\n\nclient = docker.from_env()\ncontainer_name = \"test-container\"\ncontainer_port = 8080\ncontainer = client.containers.run(\n    \"some-image:1.2.3\",\n    name=container_name,\n    ports={f\"{container_port}/tcp\": None},\n    detach=True,\n)\n\ncontainer.reload()\nip_address = os.getenv(\"_HOST_IP\", \"localhost\")\nos.environ[\"CONTAINER_NAME\"] = container_name\nos.environ[\"CONTAINER_IP\"] = ip_address\ncontainer_host_port = container.attrs[\"NetworkSettings\"][\"Ports\"][f\"{container_port}/tcp\"][0][\"HostPort\"]\nos.environ[\"CONTAINER_HOST_PORT\"] = container_host_port\n```\n\n\u003c/details\u003e\n\n---\n\n\u003cbr\u003e\n\n## Contribution\n\n- Pull requests are encouraged and always welcome. Read our [contribution guidelines](https://github.com/ml-tooling/universal-build/tree/main/CONTRIBUTING.md) and check out [help-wanted](https://github.com/ml-tooling/universal-build/issues?utf8=%E2%9C%93\u0026q=is%3Aopen+is%3Aissue+label%3A\"help+wanted\"+sort%3Areactions-%2B1-desc+) issues.\n- Submit Github issues for any [feature request and enhancement](https://github.com/ml-tooling/universal-build/issues/new?assignees=\u0026labels=feature\u0026template=02_feature-request.md\u0026title=), [bugs](https://github.com/ml-tooling/universal-build/issues/new?assignees=\u0026labels=bug\u0026template=01_bug-report.md\u0026title=), or [documentation](https://github.com/ml-tooling/universal-build/issues/new?assignees=\u0026labels=documentation\u0026template=03_documentation.md\u0026title=) problems.\n- By participating in this project, you agree to abide by its [Code of Conduct](https://github.com/ml-tooling/universal-build/blob/main/.github/CODE_OF_CONDUCT.md).\n- The [development section](#development) below contains information on how to build and test the project after you have implemented some changes.\n\n## Development\n\n\u003e _**Requirements**: [Docker](https://docs.docker.com/get-docker/) and [Act](https://github.com/nektos/act#installation) are required to be installed on your machine to execute the build process._\n\nTo simplify the process of building this project from scratch, we provide build-scripts that run all necessary steps (build, check, test, and release) within a containerized environment. To build and test your changes, execute the following command in the project root folder:\n\n```bash\nact -b -j build\n```\n\nRefer to our [contribution guides](https://github.com/ml-tooling/universal-build/blob/main/CONTRIBUTING.md#development-instructions) for more detailed information on our build scripts and development process.\n\n---\n\nLicensed **MIT**. Created and maintained with ❤️\u0026nbsp; by developers from Berlin.\n","funding_links":[],"categories":["Build Tools","Development","github-actions"],"sub_categories":["Updating Best-of Generator"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fml-tooling%2Funiversal-build","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fml-tooling%2Funiversal-build","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fml-tooling%2Funiversal-build/lists"}