{"id":19070708,"url":"https://github.com/powerdns/pdns-builder","last_synced_at":"2025-04-15T17:46:58.467Z","repository":{"id":24550712,"uuid":"101764200","full_name":"PowerDNS/pdns-builder","owner":"PowerDNS","description":"Infrastructure for creating Dockerfiles for package building","archived":false,"fork":false,"pushed_at":"2024-09-12T07:52:15.000Z","size":112,"stargazers_count":17,"open_issues_count":14,"forks_count":23,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-03-28T23:11:19.193Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PowerDNS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-29T13:25:53.000Z","updated_at":"2024-12-10T03:20:35.000Z","dependencies_parsed_at":"2024-05-31T10:05:27.912Z","dependency_job_id":"1e90d21d-05a7-42b7-a872-0046c9800c0d","html_url":"https://github.com/PowerDNS/pdns-builder","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Fpdns-builder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Fpdns-builder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Fpdns-builder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Fpdns-builder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PowerDNS","download_url":"https://codeload.github.com/PowerDNS/pdns-builder/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249124354,"owners_count":21216658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-11-09T01:20:08.363Z","updated_at":"2025-04-15T17:46:58.449Z","avatar_url":"https://github.com/PowerDNS.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PowerDNS Builder \n\n[![Build Status](https://travis-ci.org/PowerDNS/pdns-builder.svg?branch=master)](https://travis-ci.org/PowerDNS/pdns-builder)\n\nA reusable Docker based distribution package builder.\n\n## Quickstart\n\nTo only build generic source packages:\n\n    ./builder/build.sh sdist\n    \nTo build for CentOS 7:\n\n    ./builder/build.sh centos-7\n\nPackages will end up in `builder/tmp/\u003cversion\u003e/`.\n\nThe build script supports various commandline options. See:\n\n    ./builder/build.sh -h\n\n\n## Build requirements\n\n* Docker \u003e= 17.05\n* git\n* bash\n* tree (optional)\n\n\n## How does this work?\n\nThe build process for distribution packages consists of three steps:\n\n1. Create generic source distributions for all components. This step also\n   performs any more complicated generic steps, like building plain dist assets\n   using webpack.\n\n2. Create rpms or debs from these source packages (and build specs) only, with\n   no other access to the source, in a container with all build dependencies\n   installed.\n\n3. Install the distribution packages and test them in a clean container without\n   the build dependencies.\n\nThe `sdist` target only performs the first step. The install test is skippable\nwith `-s`, but using this option is not recommended.\n\nThe builder expects to be put in `builder/` in the repository to build and to\nfind a `builder-support/` directory next to it with all repository specific\nbuild configurations.\n\nThe implementation uses [Docker 17.05+ multi-stage builds][multistage] to \nimplement these steps. `builder-support/dockerfiles/` is expecte to contain\n[simple templates][templ] for Dockerfiles that will be combined into a \ntemporary Dockerfile for the build.\nThis allows us to split different build steps and substeps into separate\nstages, while only having to trigger a single Docker build. \n\nThe build script does not know or care how the actual build is performed, it just \nexpects the build artifacts to end up in `/sdist` and `/dist` inside \nthe final image after the build.\n\n[multistage]: https://docs.docker.com/engine/userguide/eng-image/multistage-build/\n[templ]: ./templating/templating.sh\n\n\n## Autobuild deployment notes\n\nIf you want to run these builds on a frequent basis, such as in a buildbot that\nautomatically builds on new commits, keep the following in mind:\n\n### Disk usage\n\nEvery build will create a fair amount of new Docker layers that will take up \ndisk space. Make sure you have something like 20 GB or more disk space \navailable for builds, and run the included [docker-cleanup.sh](docker-cleanup.sh)\nonce a day in a cron job to remove containers and images that are no longer\nreferenced by a tag.\n\nDo keep in mind that after a cleanup, a new build will have to start from\nscratch, so you do not want to run it too often. Once a day is probably a\nfair compromise between build time and disk usage, assuming you will not be\nbuilding hundreds of times per day.\n\n### Base image freshness\n\nDocker will never pull newer versions of the base images by itself. After a \nwhile the base images might become outdated. You could add an \n`apt update \u0026\u0026 apt upgrade` or equivalent to the start of each base image, but \nthis will take longer and longer to execute.\n\nInstead, you could `docker pull \u003cimage\u003e` in a cron job every night for every\nbase image used in the builds. It's best to do this before the docker cleanup\nscript, so that it can cleanup all the old layers from the previous image.\n\nScript to find all official images and pull them:\n\n    images=`docker images --format '{{.Repository}}:{{.Tag}}' --filter dangling=false | grep -v '[_/-]'`\n    for image in $images; do docker pull \"$image\"; done\n\nYou probably do not want to add `set -e` here.\n\nNOTE: This will also try to pull any images you tagged locally without any \n`-`, `_` or `/` in the name, and skip any non-official Docker images. Please\nadapt to your use case.\n\n### Concurrent builds\n\nWith the current build script it is unsafe run several builds for the same\ntarget at the same time, because the temporary Dockerfile name and image tag\nwill clash. We considered adding the version number to the tag, but this would\nmake cleanup harder. Maybe we need to add an option to the build script to\noverride the tag, if we need concurrent builds.\n\nConcurrent builds for different targets (like oraclelinux-6.8 and centos-7\nin parallel) should be safe, though.\n\n\n## Implementation details\n\n### Dockerfiles\n\nThe Dockerfile templates are expected in `builder-support/dockerfiles/`.\nNote that these Dockerfiles are repository specific and not included with\nthe builder distribution.\n\nThe files that start with `Dockerfile.target.` are used as build targets.\nFor example, `Dockerfile.target.centos-7` would be used for the `centos-7` target.\n`Dockerfile.target.sdist` is used for the `sdist` target, but also included by\nall the other targets to performs the source builds.\n\nTo allow for reusability of include files, the following stage naming conventions\nshould be observed:\n\n* `sdist` is the final source dist stage that contains all source dists in `/sdist`,\n  which will be copied by the binary package builder.\n* `dist-base` is the stage used as base image for both the package builder and\n  the installation test.\n* `package-builder` is the final binary package build stage that contains binary\n  packages in `/dist`, which will be installed in the installation test.\n\nThe last stage to appear in the Dockerfile will be the resulting image of the\ndocker build. This one must have source dists in `/sdist` and binaries in\n`/dist`, as this is where the build scripts copies the result artifacts from.\nPlease keep in mind that the test stage could be skipped, so these also have to\nexist at the end of the package builder stage.\n\n#### Docker caching\n\nIf editing Dockerfiles, try to maximize the efficiency of docker layer caches.\nFor example:\n\n* Only COPY/ADD files that are really needed at that point in the build process.\n  For example, the installation tests live in a different folder than the \n  build helpers, so that updating installation tests does not invalidate the\n  layers that build the RPMs.\n* Vendor specs should be built before you COPY your source artifacts, so that \n  they are only rebuilt if their spec files change and not every time your code\n  changes.\n* For the same reason, build ARGs should be set as late as possible.\n* If you have a slow build step, like building an Angular project using Webpack,\n  you should consider doing this in a separate stage and only apply any\n  versioning after the actual build, so that the expensive steps can be cached.\n\nIf you have a build step that relies on external, changing state (such as\n`apt-get update`), you may want to avoid caching this step forever. To do so,\nput `ARG BUILDER_CACHE_BUSTER=` before the step, and pass `-b daily` or `-b\nweekly` to build.sh.\n\n#### Templating\n\nTemplating is done using a simple template engine written in bash. \n\nExample text template:\n\n    Lines can start with @INCLUDE, @EVAL or @EXEC for special processing:\n    @INCLUDE foo.txt\n    @EVAL My home dir is $HOME\n    @EXEC uname -a\n    @EXEC [ \"$foo\" = \"bar\" ] \u0026\u0026 include bar.txt\n    @IF [ \"$foo\" = \"bar\" ]\n    This line is only printed if $foo = \"bar\" (cannot be nested)\n    @INCLUDE bar.txt\n    @ENDIF\n    Other lines are printed unchanged.   \n\nThe commands behind `@EXEC` and `@IF` can be any bash commands. `include` is\nan internal bash function used to implement `@INCLUDE`. Note that `@IF`\ncurrently cannot be nested.\n\nThe templating implementation can be found in `templating/templating.sh`.\n\n#### Post Build steps\n\nWhen certain steps or commands are needed after building, add an executable\nfile called `post-build` to `builder-support`. After a build, this file will\nbe run.\n\n\n### Reproducible builds\n\nThe builder has a few features to help with creating reproducible builds.\n\nThe builder sets a `SOURCE_DATE_EPOCH` build argument with the timestamp of the last\ncommit as the value. This is not automatically propagated to the build environment.\nIf you want to use this, add this to your Dockerfile at the place where you want to\nstart using it:\n\n```\nARG SOURCE_DATE_EPOCH\n```\n\nThis will probably be the same place that you inject the `BUILDER_VERSION`.\n\nFor vendor dependency builds, you probably do not want to use it, as it could make their\nartifacts change with every version change. Instead, you may want to set the\n`BUILDER_SOURCE_DATE_FROM_SPEC_MTIME` env var when building RPMs. If this is set, the\nbuild script will use the modification time of the spec file as the `SOURCE_DATE_EPOCH`.\nExample usage:\n\n```\nRUN BUILDER_SOURCE_DATE_FROM_SPEC_MTIME=1 builder/helpers/build-specs.sh builder-support/vendor-specs/*.spec\n```\n\nThe RPM build script always defines the following variables for reproducible RPM builds:\n\n```\n--define \"_buildhost reproducible\"\n--define \"source_date_epoch_from_changelog Y\"\n--define \"clamp_mtime_to_source_date_epoch Y\"\n--define \"use_source_date_epoch_as_buildtime Y\"\n```\n\nThe `source_date_epoch_from_changelog` variable only has effect when no `SOURCE_DATE_EPOCH` is set.\nThese variables are only supported in RHEL 8+ and derived distributions. RHEL 7 does not appear\nto support reproducible RPM builds.\n\nKeep in mind that the builder an only do so much, as any part of your build pipeline\nthat creates non-reproducible artifacts will result in non-reproducible build output.\nFor example, if the base image you use upgrades the compiler, the compiled output\nwill likely change.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpowerdns%2Fpdns-builder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpowerdns%2Fpdns-builder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpowerdns%2Fpdns-builder/lists"}