{"id":13416443,"url":"https://github.com/avirshup/DockerMake","last_synced_at":"2025-03-14T23:31:51.791Z","repository":{"id":53535286,"uuid":"47433737","full_name":"avirshup/DockerMake","owner":"avirshup","description":"A reproducible Docker image build system for complex software stacks","archived":false,"fork":false,"pushed_at":"2022-10-20T16:59:08.000Z","size":488,"stargazers_count":138,"open_issues_count":10,"forks_count":26,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-03-08T07:36:54.650Z","etag":null,"topics":["dependency-graph","docker","docker-image","dockerfile"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/avirshup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-04T23:21:29.000Z","updated_at":"2025-02-17T19:53:58.000Z","dependencies_parsed_at":"2022-09-05T12:30:40.859Z","dependency_job_id":null,"html_url":"https://github.com/avirshup/DockerMake","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avirshup%2FDockerMake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avirshup%2FDockerMake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avirshup%2FDockerMake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avirshup%2FDockerMake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/avirshup","download_url":"https://codeload.github.com/avirshup/DockerMake/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243663516,"owners_count":20327300,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dependency-graph","docker","docker-image","dockerfile"],"created_at":"2024-07-30T21:00:58.932Z","updated_at":"2025-03-14T23:31:46.781Z","avatar_url":"https://github.com/avirshup.png","language":"Python","readme":"# Docker-make\n[![Codeship Status for avirshup/DockerMake](https://app.codeship.com/projects/d4a701b0-2114-0138-0bfb-1a499be1ccac/status?branch=master)](https://app.codeship.com/projects/382713)\n[ ![PyPI version](https://badge.fury.io/py/DockerMake.svg)](https://badge.fury.io/py/DockerMake)\n\n## Table of Contents\n+ [What is it?](#what-is-it)\n+ [Install it](#install-it)\n+ [Run it](#run-it)\n+ [What you can do with it](#what-you-can-do-with-it)\n    * [Build automation](#build-automation)\n    * [Secrets and squashing](#secrets-and-squashing)\n    * [File handling](#file-handling)\n    * [Cache control](#cache-control)\n+ [How to write DockerMake.yml](#how-to-write-dockermakeyml)\n  - [Defining an image](#defining-an-image)\n  - [Image definition reference](#image-definition-reference)\n  - [Special fields](#special-fields)\n  - [Notes](#notes-on-dockermake.yml)\n+ [Example](#example)\n+ [Command line usage](#command-line-usage)\n\n## What is it?\nA command line tool to build and manage stacks of docker images. You can mix and match different sets of build instructions as a dependency graph to create maintainable and extensible stacks of docker images.\n\n\n## Install it\n\nRequires [Docker](https://www.docker.com/products/docker) and Python 2.7 or 3.5+.\n\n```\npip install DockerMake \n```\n\nThis will install the command line tool, `docker-make`, and its supporting python package, which you can import as `import dockermake`. \n\n\n## Run it\n\nTo build some illustrative examples, try running the example in this repository:\n\n```bash\ngit clone https://github.com/avirshup/DockerMake\ncd DockerMake/example\ndocker-make --list\ndocker-make final\n```\n\n\n## What you can do with it\n\nThe punchline: define small pieces of configuration or functionality, then mix them together into production container images.\n\n#### Build steps\nA `DockerMake.yml` file contains discrete build \"steps\". These steps can depend on each other, forming a dependency graph. DockerMake solves the dependency graph and drives building of each image as necessary. This makes it easy to keep your images up-to-date while still taking advantage of docker's shared fileystem layers and build cache.\n\n#### Build automation\n * **new**: beta support for dockerfile build arguments\n * **new**: specify custom `.dockerignore` files for any given build step\n * Automated registry login and image pushes\n \n#### Secrets and squashing\n - [squash](#squash) arbitrary parts of your build (using `squash: true` in a step definition) without busting the cache\n - Designate [`secret_files`](#secret_files) to erase them from intermediate layers (In the step definition, use `secret_files: ['path1', 'path2', ...]`)\n \n**WARNING:** these features are in alpha - use with extreme caution\n \n#### File handling\n * Create builds that ADD or COPY files from anywhere on your file system\n * Build artifacts in one image, then copy them into smaller images for deployment\n\n#### Cache control\n - Invalidate docker's build cache at a specific step in the build using `--bust-cache [stepname]`\n - **new**: Use specific images to [resolve docker's build cache](https://github.com/moby/moby/issues/26065) (using `--cache-repo [repo]` and/or `--cache-tag [tag]`)\n - Force a clean rebuild without using the cache (using `--no-cache`)\n \n \n## How to write DockerMake.yml\nDockerMake.yml lets you split your image build up into discrete, easy to manage _steps_ that can be mixed together (as a dependency graph) to produce your desired container image.\n\n### Defining an image\n\nThe DockerMake.yml file is a YAML-formatted file that lists build _steps_. To create an extremely basic image, define a step with a base image and a series of dockerfile commands:\n\n```yaml\nFIRST_STEP_NAME:\n  FROM: BASE_IMAGE_NAME\n  build: |\n    RUN [something]\n    ADD [something else]\n    [Dockerfile commands go here]\n```\n\nUse the `requires` field to define additional steps that extend the first one:\n\n```yaml\nFIRST_STEP_NAME:\n   [...]\n\nNEXT_STEP_NAME:\n   requires:\n     - FIRST_STEP_NAME\n   build: |\n     [additional Dockerfile instructions]\n```\n\n### Image definition reference\nImage definitions can include any of the following fields:\n\n* [**`FROM`/`FROM_DOCKERFILE`**](#fromfrom_dockerfile)\n* [**`build`**](#build)\n* [**`requires`**](#requires)\n* [**`build_directory`**](#build_directory)\n* [**`ignore`/`ignorefile`**](#ignoreignorefile)\n* [**`description`**](#description)\n* [**`copy_from`**](#copy_from)\n* [**`squash`**](#squash)\n* [**`secret_files`**](#secret_files)\n\n#### **`FROM`/`FROM_DOCKERFILE`**\nThe docker image to use as a base for this image (and those that require it). This can be either the name of an image (using `FROM`) or the path to a local Dockerfile (using `FROM_DOCKERFILE`).\n\n*Example:*\n```yaml\nbaseimage:\n   FROM: python:3.6-slim\n```\nor\n```yaml\nbaseimage:\n   FROM_DOCKERFILE: ../myproject/Dockerfile\n   ```\n#### **`build`**\nMulti-line string defining dockerfile commands to build this step. Note that these commands CANNOT contain 'FROM'. See also [Notes on multi-line strings](#Notes) below.\n\n*Example:*\n```yaml\nbuild-image:\n   requires:\n     - baseimage\n   build: |\n     RUN apt-get update \\\n      \u0026\u0026 apt-get install -y gcc vi\n     ENV editor=vi\n```\n\n#### **`requires`**\nList of other image definitions to include in this one. `docker-make` will create a new image from a single DockerFile that includes an amalgamation of all image definitions.\n\n*Example:*\n ```yaml\n my-tools:\n   build: |\n     RUN pip install numpy jupyter pandas\n     [...]\n\n data-sci-environment:\n    requires:\n      - baseimage\n      - my-tools\n ```\n\n#### **`build_directory`**\nPath to a directory on your filesystem. This will be used to locate files for `ADD` and `COPY` commands in your dockerfile. See [Notes on relative paths](#Notes) below.\n\n*Example:*\n```yaml\ndata-image:\n    build_directory: ./datafiles\n    build: |\n      COPY data /opt/data\n    [...]\n```\n\n#### **`ignore`/`ignorefile`**\nA custom [.dockerignore](https://docs.docker.com/engine/reference/builder/#dockerignore-file) for this step. This overrides any existing `.dockerignore` file in the build context. Only relevant for `ADD` or `COPY` commands when the `build_directory` is specified. This can either be a multi-line string (using the `ignore` field)  or the path to a file (using the `ignorefile` field).\n\n*Example:*\n```yaml\ndata-image:\n    build_directory: ./datafiles\n    build: |\n      ADD [...]\n    ignore: |\n      *.pyc\n      *~\n      *.tmp\n```\n  \n#### **`description`**\nAn arbitrary comment (ignored by `docker-make`)\n\n#### **`copy_from`**\nUsed to copy files into this image _from other images_ (to copy from your filesystem or a URL, use the standard `ADD` and `COPY` dockerfile commands). This is a mapping of mappings of the form:\n```yaml\n[image-name]:\n   [...]\n   copy_from:\n     [source_image1]:\n        [source path 1]: [dest path 1]\n        [source path 2]: [dest path 2]\n     [source image2]:\n        [...]\n```\n\nNote that, for historical reasons, these copies are performed _after_ any _build_ instructions are executed.\n\n#### **`squash`**\n**NOTE**: this feature requires that your [docker daemon's experimental features be enabled.](https://github.com/docker/docker-ce/blob/master/components/cli/experimental/README.md)\n\nUsed to squash all layers produced in a given step. This can be helfpul both for keeping image sizes low, especially when it's necessary to add a lot of data via the `ADD` or `COPY` dockerfile commands.\n\nNote that setting `squash: True` for a step only squashes the layers generated by that step. All layers in the base image are left intact.\n\nAdditionally, unlike the vanilla `docker build --squash` command, downstream image builds can use the squashed image in their cache, so that squashing doesn't force you to repeatedly re-run the same downstream build steps.\n\n*Example:*\nIn this example, we create a huge file in the image, do something with it, then erase it. \n\n```yaml\ncount-a-big-file:\n    FROM: alpine\n    build: |\n        RUN dd if=/dev/zero of=/root/bigfile count=16384 bs=1024\n        RUN wc /root/bigfile \u003e /root/numbiglines\n        RUN rm /root/bigfile\n```\n\nLet's build it and check the size:\n```bash\n$ docker-make count-a-big-file\n[...]\ndocker-make finished.\nBuilt: \n * count-a-big-file\n$ docker images count-a-big-file\nREPOSITORY         ...   SIZE\ncount-a-big-file   ...   20.9MB\n```\n\nBut, take them same definition and add a `squash: true` to it:\n```yaml\ncount-a-big-file:\n    FROM: alpine\n    squash: true\n    build: |\n        RUN dd if=/dev/zero of=/root/bigfile count=16384 bs=1024\n        RUN wc /root/bigfile \u003e /root/numbiglines\n        RUN rm /root/bigfile\n```\n  \nAnd we find that the deleted file is no longer taking up space:  \n```bash\n$ docker-make count-a-big-file\n[...]\ndocker-make finished.\nBuilt: \n * count-a-big-file\n$ docker images count-a-big-file\nREPOSITORY         ...   SIZE\ncount-a-big-file   ...   4.15MB\n```\n\n\n#### **`secret_files`**\n**Read these caveats first**\n\n - This is an alpha-stage feature. DO NOT rely on it as a security tool. You must carefully verify that your final image, and all its layers AND its history, are free of sensitive information before deploying or publishing them.\n - It relies on [experimental docker daemon features](https://github.com/docker/docker-ce/blob/master/components/cli/experimental/README.md).\n - Although your final image won't contain your secrets, they will be present in intermediate images on your build machine. Your secrets will be exposed to all `docker` users on your build machine.\n - When you define `secret_files` for a step, it only erases files that are added in the `build` definition _for that step_. Files added in other steps will remain exposed in your image's layers. \n\n**Background**\n\nIt's often necessary to perform some form of authentication during a build - for instance, you might need to clone a private git repository or download dependencies from a private server. However, it's quite challenging to do so without leaving your credentials inside a layer of the final docker image or its history.\n\nFiles added or created in a given step can be designated as `secret_files` in DockerMake.yml. These files will be automatically erased at the end of the step, and the step's layers will be squashed to keep the files out of the history.\n \n **Example**\n```yaml\nmy-secret-steps:\n    FROM: python:3.6\n    build: |\n        ADD my-credentials /opt/credentials\n        RUN some-process --credentials /opt/credentials\n    secret_files:\n        - /opt/credentials\n```\n\n### Special fields\n\n#### `_SOURCES_`\n\nYou can include step definitions from other DockerMake.yml files by listing them in the `_SOURCES_`. For example:\n\n```yaml\n_SOURCES_:\n  - ~/mydefinitions/DockerMake.yml\n  - ./other/file.yml\n  [...]\n```\n\nPlease note that relative file paths in each file are always interpreted _relative to the directory containing that file_.\n\n#### `_ALL_`\n\nBy default, running `docker-make --all` will build all well-defined images defined in a file (and any files included via `_SOURCES_`). Images without a `FROM` or `FROM_DOCKERFILE` field in any of their requirements will be ignored.\n\nAlternatively, you can use the `_ALL_` field to designate specific images to build. For example, in the following definition, `docker-make --all` will only build `imgone` and `imgtwo`, not `baseimage`: \n\n```yaml\n_ALL_:\n - imgone\n - imgtwo\n \nbaseimage:\n  FROM: [...]\n  [...]\n \nimgone: [...]\n\nimgtwo: [...]\n```\n\nNote that the `_ALL_` fields from any files included via `_SOURCES_` are ignored.\n\n \n### Notes on DockerMake.yml\n\n**Relative paths**: Several of these fields include paths on your local filesystem. They may be absolute or relative; _relative_ paths are resolved relative to the DockerMake.yml file they appear in. Use of `~` is allowed to denote the home directory.\n\n**Multiline strings**: You'll usually want to express the `build` and `ignore` fields as multiline strings. To do so, use the following [YML \"literal block scalar\" style](http://yaml-multiline.info/), as in all examples above.\n\n```yaml\nfield-name: |\n  [line 1]\n  [line 2]\n  [...]\nnext field: [...]\n```\n\n\n\n## Example\n[(See also this production example)](https://github.com/molecular-toolkit/chemistry-docker-images/tree/master/makefiles)\n\nThis example builds a single docker image called `data_science`. It does this by mixing together three components: `devbase` (the base image), `airline_data` (a big CSV file), and `python_image` (a python installation). `docker-make` will create an image that combines all of these components.\n\nHere's the `DockerMake.yml` file:\n\n```yaml\ndevbase:\n FROM: phusion/baseimage\n build: |\n  RUN apt-get -y update \u0026\u0026 apt-get -y install \n      build-essential \n   \u0026\u0026 mkdir -p /opt\n\nairline_data:\n build_directory: sample_data/airline_data\n build: |\n  ADD AirPassengers.csv /data\n\nplant_data:\n build_directory: sample_data/plant_growth\n build: |\n  ADD Puromycin.csv /data\n\npython_image:\n requires:\n  - devbase\n build: |\n  RUN apt-get install -y python python-pandas\n\ndata_science:\n requires:\n  - python_image\n  - airline_data\n  - plant_data\n\n```\n\nTo build an image called `alice/data_science`, you can run:\n```bash\ndocker-make data_science --repository alice\n```\nwhich will create an image with all the commands in `python_image` and `airline_data`.\n\nThis works by dynamically generating a new Dockerfile every time you ask to build something. However, most of the commands will be cached, especially if you have a large hierarchy of base images. This actually leads to _less_ rebuilding than if you had a series of Dockerfiles linked together with `FROM` commands.\n\nHere's the dependency graph and generated Dockerfiles:\n\n![dependency graph](img/step1.png)\n![dockerfiles](img/step2.png)\n\n\n\n## Command line usage \n```\nusage: docker-make [-h] [-f MAKEFILE] [-a] [-l] [--build-arg BUILD_ARG]\n                   [--requires [REQUIRES [REQUIRES ...]]] [--name NAME] [-p]\n                   [-n] [--dockerfile-dir DOCKERFILE_DIR] [--pull]\n                   [--cache-repo CACHE_REPO] [--cache-tag CACHE_TAG]\n                   [--no-cache] [--bust-cache BUST_CACHE] [--clear-copy-cache]\n                   [--keep-build-tags] [--repository REPOSITORY] [--tag TAG]\n                   [--push-to-registry] [--registry-user REGISTRY_USER]\n                   [--registry-token REGISTRY_TOKEN] [--version] [--help-yaml]\n                   [--debug]\n                   [TARGETS [TARGETS ...]]\n\nNOTE: Docker environmental variables must be set. For a docker-machine, run\n`eval $(docker-machine env [machine-name])`\n\noptional arguments:\n  -h, --help            show this help message and exit\n\nChoosing what to build:\n  TARGETS               Docker images to build as specified in the YAML file\n  -f MAKEFILE, --makefile MAKEFILE\n                        YAML file containing build instructions\n  -a, --all             Print or build all images (or those specified by\n                        _ALL_)\n  -l, --list            List all available targets in the file, then exit.\n  --build-arg BUILD_ARG\n                        Set build-time variables (used the same way as docker\n                        build --build-arg), e.g., `... --build-arg VAR1=val1\n                        --build-arg VAR2=val2`\n  --requires [REQUIRES [REQUIRES ...]]\n                        Build a special image from these requirements.\n                        Requires --name\n  --name NAME           Name for custom docker images (requires --requires)\n\nDockerfiles:\n  -p, --print-dockerfiles, --print_dockerfiles\n                        Print out the generated dockerfiles named\n                        `Dockerfile.[image]`\n  -n, --no_build        Only print Dockerfiles, don't build them. Implies\n                        --print.\n  --dockerfile-dir DOCKERFILE_DIR\n                        Directory to save dockerfiles in (default:\n                        ./docker_makefiles)\n\nImage caching:\n  --pull                Always try to pull updated FROM images\n  --cache-repo CACHE_REPO\n                        Repository to use for cached images. This allows you\n                        to invoke the `docker build --build-from` option for\n                        each image.For instance, running `docker-make foo bar\n                        --cache-repo docker.io/cache` will use\n                        docker.io/cache/foo as a cache for `foo` and\n                        docker.io/cache/bar as a cachefor `bar`.\n  --cache-tag CACHE_TAG\n                        Tag to use for cached images; can be used with the\n                        --cache-repo option (see above).\n  --no-cache            Rebuild every layer\n  --bust-cache BUST_CACHE\n                        Force docker to rebuilt all layers in this image. You\n                        can bust multiple image layers by passing --bust-cache\n                        multiple times.\n  --clear-copy-cache, --clear-cache\n                        Remove docker-make's cache of files for `copy-from`.\n  --keep-build-tags     Don't untag intermediate build containers when build\n                        is complete\n\nRepositories and tags:\n  --repository REPOSITORY, -r REPOSITORY, -u REPOSITORY\n                        Prepend this repository to all built images, e.g.\n                        `docker-make hello-world -u quay.io/elvis` will tag\n                        the image as `quay.io/elvis/hello-world`. You can add\n                        a ':' to the end to image names into tags: `docker-\n                        make -u quay.io/elvis/repo: hello-world` will create\n                        the image in the elvis repository:\n                        quay.io/elvis/repo:hello-world\n  --tag TAG, -t TAG     Tag all built images with this tag. If image names are\n                        ALREADY tags (i.e., your repo name ends in a \":\"),\n                        this will append the tag name with a dash. For\n                        example: `docker-make hello-world -u elvis/repo: -t\n                        1.0` will create the image \"elvis/repo:hello-world-1.0\n  --push-to-registry, -P\n                        Push all built images to the repository specified\n                        (only if image repository contains a URL) -- to push\n                        to dockerhub.com, use index.docker.io as the registry)\n  --registry-user REGISTRY_USER, --user REGISTRY_USER\n                        For pushes: log into the registry using this username\n  --registry-token REGISTRY_TOKEN, --token REGISTRY_TOKEN\n                        Token or password to log into registry (optional; uses\n                        $HOME/.dockercfg or $HOME/.docker/config.json if not\n                        passed)\n\nHelp:\n  --version             Print version and exit.\n  --help-yaml           Print summary of YAML file format and exit.\n  --debug\n```\n\n\nCopyright (c) 2015-2017, Autodesk Inc.\nCopyright (c) 2017-2018, Docker-Make contributors.\nReleased under the Apache 2.0 License.\n","funding_links":[],"categories":["Docker Images","Python"],"sub_categories":["Builder"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favirshup%2FDockerMake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Favirshup%2FDockerMake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favirshup%2FDockerMake/lists"}