{"id":13645237,"url":"https://github.com/gitpod-io/dazzle","last_synced_at":"2025-04-07T18:14:54.436Z","repository":{"id":40514906,"uuid":"211308352","full_name":"gitpod-io/dazzle","owner":"gitpod-io","description":"dazzle is a rather experimental Docker image builder which builds independent layers","archived":false,"fork":false,"pushed_at":"2024-04-19T12:21:41.000Z","size":17070,"stargazers_count":226,"open_issues_count":13,"forks_count":15,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-03-31T15:19:23.726Z","etag":null,"topics":["containers","docker","docker-images","dockerfile","independent-layers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gitpod-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2019-09-27T11:54:17.000Z","updated_at":"2025-03-25T14:38:13.000Z","dependencies_parsed_at":"2023-02-18T19:46:01.012Z","dependency_job_id":"e07e34da-7644-4b66-930f-739fcec09c60","html_url":"https://github.com/gitpod-io/dazzle","commit_stats":null,"previous_names":["32leaves/dazzle","csweichel/dazzle"],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitpod-io%2Fdazzle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitpod-io%2Fdazzle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitpod-io%2Fdazzle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gitpod-io%2Fdazzle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gitpod-io","download_url":"https://codeload.github.com/gitpod-io/dazzle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247704571,"owners_count":20982298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","docker","docker-images","dockerfile","independent-layers"],"created_at":"2024-08-02T01:02:31.874Z","updated_at":"2025-04-07T18:14:54.405Z","avatar_url":"https://github.com/gitpod-io.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003cimg src=\"logo.png\" width=\"100\" style=\"padding: 1em; background-color: white; border-radius: 10px;\"\u003e\n\n[![Setup Automated](https://img.shields.io/badge/setup-automated-blue?logo=gitpod)](https://gitpod.io/#https://github.com/gitpod-io/dazzle)\n[![Go Report Card](https://goreportcard.com/badge/github.com/gitpod-io/dazzle)](https://goreportcard.com/report/github.com/gitpod-io/dazzle)\n[![Stability: Experimental](https://masterminds.github.io/stability/experimental.svg)](https://masterminds.github.io/stability/experimental.html)\n\ndazzle is a rather experimental Docker/OCI image builder. Its goal is to build independent layers where a change to one layer does *not* invalidate the ones sitting \"above\" it.\n\n**Beware** Recently the format for [dazzle builds was changed](https://github.com/gitpod-io/dazzle/commit/ceaa19ef6562e03108c8ea9474d2c627a452a7ca), moving from a single Dockerfile to one per \"chunk\"/layer. It is also about 5x faster, more reliable and less hacky.\n\n## How does it work?\ndazzle has three main capabilities.\n1. _build independent layer chunks_: in a dazzle project there's a `chunks/` folder which contains individual Dockerfiles (e.g. `chunks/something/Dockerfile`). These chunk images are built independently of each other. All of them share the same base image using a special build argument `${base}`. Dazzle can build the base image (built from `base/Dockerfile`), as well as the chunk images. After each chunk image build dazzle will remove the base image layer from that image, leaving just the layers that were produced by the chunk Dockerfile.\n2. _merge layers into one image_: dazzle can merge multiple OCI images/chunks (not just those built using dazzle) by building a new manifest and image config that pulls the layers/DiffIDs from the individual chunks and the base image they were built from.\n3. _run tests against images_: to ensure that an image is capable of what we think it should be - especially after merging - dazzle supports simple tests and assertions that run against Docker images.\n\n## Would I want to use this?\n\nNot ordinarily, no. For example, if you're packing your service/app/software/unicorn you're probably better of with a regular Docker image build and well established means for optimizing that one (think multi-stage builds, proper layer ordering).\n\nIf however you are building images which consist of a lot of independent \"concerns\", i.e. chunks that can be strictly separated, then this might for you.\nFor example, if you're building an image that serves as a collection of tools, the layer hierarchy imposed by regular builds doesn't fit so well.\n\n## Limitations and caveats\n\n- build args are not supported at the moment\n- there are virtually no tests covering this so things might just break\n- consider this alpha-level software\n\n### Requirements\nInstall and run [buildkit](https://github.com/moby/buildkit/releases) - currently 0.10.1 - in the background.\nPull and run a docker registry.\n\nNOTE: if you are running it in Gitpod this is done for you!\n\n```bash\nsudo su -c \"cd /usr; curl -L https://github.com/moby/buildkit/releases/download/v0.10.1/buildkit-v0.10.1.linux-amd64.tar.gz | tar xvz\"\ndocker run -p 5000:5000 --name registry --rm registry:2\n```\n\n## Getting started\n\n```bash\n# start a new project\ndazzle project init\n\n# add our first chunk\ndazzle project init helloworld\necho hello world \u003e chunks/helloworld/hello.txt\necho \"COPY hello.txt /\" \u003e\u003e chunks/helloworld/Dockerfile\n\n# add another chunk, just for fun\ndazzle project init anotherchunk\necho some other chunk \u003e chunks/anotherchunk/message.txt\necho \"COPY message.txt /\" \u003e\u003e chunks/anotherchunk/Dockerfile\n\n# register a combination which takes in all the chunks\ndazzle project add-combination full helloworld anotherchunk\n\n# build the chunks\ndazzle build eu.gcr.io/some-project/dazzle-test\n\n# build all combinations\ndazzle combine eu.gcr.io/some-project/dazzle-test --all\n```\n\n# Usage\n\n## init\n\n```shell\n$ dazzle project init\nStarts a new dazzle project\n\nUsage:\n  dazzle project init [chunk] [flags]\n\nFlags:\n  -h, --help   help for init\n\nGlobal Flags:\n      --addr string      address of buildkitd (default \"unix:///run/buildkit/buildkitd.sock\")\n      --context string   context path (default \"/workspace/workspace-images\")\n  -v, --verbose          enable verbose logging\n```\n\nStarts a new dazzle project. If you don't know where to start, this is the place.\n\n## build\n\n```shell\n$ dazzle build --help\nBuilds a Docker image with independent layers\n\nUsage:\n  dazzle build \u003ctarget-ref\u003e [flags]\n\nFlags:\n      --chunked-without-hash   disable hash qualification for chunked image\n  -h, --help                   help for build\n      --no-cache               disables the buildkit build cache\n      --plain-output           produce plain output\n\nGlobal Flags:\n      --addr string      address of buildkitd (default \"unix:///run/buildkit/buildkitd.sock\")\n      --context string   context path (default \"/workspace/workspace-images\")\n  -v, --verbose          enable verbose logging\n```\n\nDazzle can build regular Docker files much like `docker build` would. `build` will build all images found under `chunks/`.\n\nDazzle cannot reproducibly build layers but can only re-use previously built ones. To ensure reusable layers and maximize Docker cache hits, dazzle itself caches the layers it builds in a Docker registry.\n\n## combine\n\n```shell\n$ dazzle combine --help\nCombines previously built chunks into a single image\n\nUsage:\n  dazzle combine \u003ctarget-ref\u003e [flags]\n\nFlags:\n      --all                  build all combinations\n      --build-ref string     use a different build-ref than the target-ref\n      --chunks string        combine a set of chunks - format is name=chk1,chk2,chkN\n      --combination string   build a specific combination\n  -h, --help                 help for combine\n      --no-test              disables the tests\n\nGlobal Flags:\n      --addr string      address of buildkitd (default \"unix:///run/buildkit/buildkitd.sock\")\n      --context string   context path (default \"/workspace/workspace-images\")\n  -v, --verbose          enable verbose logging\n```\n\nDazzle can combine previously built chunks into a single image. For example `dazzle combine some.registry.com/dazzle --chunks foo=chunk1,chunk2` will combine `base`, `chunk1` and `chunk2` into an image called `some.registry.com/dazzle:foo`.\nOne can pre-register such chunk combinations using `dazzle project add-combination`.\n\nThe `dazzle.yaml` file specifies the list of available combinations. Those combinations can also reference each other:\n\n```yaml\ncombiner:\n  combinations:\n  - name: minimal\n    chunks:\n    - golang\n  - name: some-more\n    ref:\n    - minimal\n    chunks:\n    - node\n```\n\n## Testing Chunks and Combinations\n\nDuring a dazzle build one can test the individual chunks and the combination images.\nDuring the build dazzle will execute the layer tests for each individual layer, as well as the final image.\nThis makes finding and debugging issues created by the layer merge process tractable.\n\nEach chunk gets its own set of tests found under `tests/chunk.yaml`.\n\nFor example:\n\n```YAML\n- desc: \"it should demonstrate tests\"\n  command: [\"echo\", \"hello world\"]\n  assert:\n  - status == 0\n  - stdout.indexOf(\"hello\") != -1\n  - stderr.length == 0\n- desc: \"it should handle exit codes\"\n  command: [\"sh\", \"-c\", \"exit 1\"]\n  assert:\n  - status == 1\n- desc: \"it should have environment variables\"\n  command: [\"sh\", \"-c\", \"echo $MESSAGE\"]\n  env:\n  - MESSAGE=foobar\n  assert:\n  - stdout.trim() == \"foobar\"\n- desc: \"it should have right binary version\"\n  entrypoint: [bash, -i, -c]\n  command: [foo -version]\n  assert:\n  - stderr.indexOf(\"1.8.0_312\") != -1\n\n```\n\nFollowing fields are available in the test spec.\n\n### `assert`\n\nField `assert` is used to add assertions on the test.\nIt accepts an array input.\nAll test assertions are written in [ES5 Javascript](https://github.com/robertkrimen/otto).\nThree variables are available in an assertion:\n\n- `stdout` contains the standard output produced by the command\n- `stderr` contains the standard error output produced by the command\n- `status` contains the exit code of the command/container.\n\nThe assertion itself must evaluate to a boolean value, otherwise the test fails.\n\n### `desc`\n\nField `desc` is used to add description of the test.\nIt accepts a string input.\n\n### `command`\n\nField `command` contains the test command.\nIt accepts an array of string.\n\n### `entrypoint`\n\nField `entrypoint` defines the entrypoint in the image.\nThis is especially handy when the default entrypoint of the image is not a shell.\nIt accepts an array of string.\n\n### `skip`\n\nField `skip` is used to decide if the test should run.\nIt accepts a boolean input.\n\n### `user`\n\nField `user` is used to define the user as whom the tests should run.\nIt accepts a string input.\n\n\n### `env`\n\nField `env` is used to define the user as whom the tests should run.\nIt accepts an array of string.\nEach string is a key value pair separated by `=`.\n\n## Testing approach\n\nWhile the test runner is standalone, the linux+amd64 version is embedded into the dazzle binary using [go.rice](https://github.com/GeertJohan/go.rice) and go generate - see [build.sh](./pkg/test/runner/build.sh).\nTODO: use go:embed?\nNote that if you make changes to code in the test runner you will need to re-embed the runner into the binary in order to use it via dazzle.\n\n```bash\ngo generate ./...\n```\n\nThe test runner binary is extracted and copied to the generated image where it is run using an encoded JSON version of the test specification - see [container.go](pkg/test/buildkit/container.go).\nThe exit code, stdout \u0026 stderr are captured and returned for evaluation against the assertions in the test specification.\n\nWhile of limited practical use, it is *possible* to run the test runner standalone using a base64-encoded JSON blob as a parameter:\n\n```bash\n$ go run pkg/test/runner/main.go eyJEZXNjIjoiaXQgc2hvdWxkIGhhdmUgR28gaW4gdmVyc2lvbiAxLjEzIiwiU2tpcCI6ZmFsc2UsIlVzZXIiOiIiLCJDb21tYW5kIjpbImdvIiwidmVyc2lvbiJdLCJFbnRyeXBvaW50IjpudWxsLCJFbnYiOm51bGwsIkFzc2VydGlvbnMiOlsic3Rkb3V0LmluZGV4T2YoXCJnbzEuMTFcIikgIT0gLTEiXX0=\n{\"Stdout\":\"Z28gdmVyc2lvbiBnbzEuMTYuNCBsaW51eC9hbWQ2NAo=\",\"Stderr\":\"\",\"StatusCode\":0}\n```\n\nThe stdout/err are returned as base64-encoded values.\nThey can be extracted using jq e.g.:\n\n```bash\n$ go run pkg/test/runner/main.go eyJEZXNjIjoiaXQgc2hvdWxkIGhhdmUgR28gaW4gdmVyc2lvbiAxLjEzIiwiU2tpcCI6ZmFsc2UsIlVzZXIiOiIiLCJDb21tYW5kIjpbImdvIiwidmVyc2lvbiJdLCJFbnRyeXBvaW50IjpudWxsLCJFbnYiOm51bGwsIkFzc2VydGlvbnMiOlsic3Rkb3V0LmluZGV4T2YoXCJnbzEuMTFcIikgIT0gLTEiXX0= | jq -r '.Stdout | @base64d'\ngo version go1.16.4 linux/amd64\n```\n\n## Integration tests\n\nThere is an integration test for the build command in pkg/dazzle/build_test.go - TestProjectChunk_test_integration and a shell script to run it.\nThe integration test does an end-to-end check along with editing a test and re-running to ensure only the test image is updated.\n\nIt requires a running Buildkitd instance at unix:///run/buildkit/buildkitd.sock and a docker registry on 127.0.0.1:5000 (i.e. as this workspace is setup on startup).\n\nOverride the env vars BUILDKIT_ADDR and TARGET_REF as required prior to running in a different environment.\n\n```bash\n$ ./integration_tests.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitpod-io%2Fdazzle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgitpod-io%2Fdazzle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitpod-io%2Fdazzle/lists"}