{"id":36856058,"url":"https://github.com/ppc64le-cloud/docker-ce-build","last_synced_at":"2026-01-20T17:19:58.575Z","repository":{"id":37775166,"uuid":"409520182","full_name":"ppc64le-cloud/docker-ce-build","owner":"ppc64le-cloud","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-16T12:33:36.000Z","size":707,"stargazers_count":0,"open_issues_count":4,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-12T19:50:29.776Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ppc64le-cloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-09-23T09:01:39.000Z","updated_at":"2025-12-16T12:33:41.000Z","dependencies_parsed_at":"2023-02-19T07:01:05.779Z","dependency_job_id":"19191f47-ed15-4495-b6ca-f222a459e3b0","html_url":"https://github.com/ppc64le-cloud/docker-ce-build","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ppc64le-cloud/docker-ce-build","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppc64le-cloud%2Fdocker-ce-build","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppc64le-cloud%2Fdocker-ce-build/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppc64le-cloud%2Fdocker-ce-build/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppc64le-cloud%2Fdocker-ce-build/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ppc64le-cloud","download_url":"https://codeload.github.com/ppc64le-cloud/docker-ce-build/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ppc64le-cloud%2Fdocker-ce-build/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28607626,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-20T16:10:39.856Z","status":"ssl_error","status_checked_at":"2026-01-20T16:10:39.493Z","response_time":117,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-12T14:43:49.638Z","updated_at":"2026-01-20T17:19:58.546Z","avatar_url":"https://github.com/ppc64le-cloud.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scripts for Docker-CE and containerd build and test prow jobs\n\nThis repository contains the scripts called by the Docker-CE and containerd build and test prow jobs in https://github.com/ppc64le-cloud/test-infra repository.\n\nThe goal of these scripts and the two associated prow jobs is to automate the process of building the docker-ce and containerd packages for ppc64le and of testing them. The packages would then be shared with the Docker team and be available on the https://download.docker.com package repositories.\n\nTo build these packages, we use the [docker-ce-packaging](https://github.com/docker/docker-ce-packaging) and the [containerd-packaging](https://github.com/docker/containerd-packaging/) repositories.\n\nThe corresponding prow jobs are:\n1. postsubmit-build-docker\n2. postsubmit-build-test-containerd\n3. postsubmit-test-docker-staging\n4. postsubmit-test-docker-release\n\nWe also run the following 4 jobs to run upstream CI tests against github.com/moby/moby.\n1. periodic-config-docker\n2. periodic-build-dev-image-docker\n3. periodic-unit-test-docker\n4. periodic-integration-test-docker\n\nFor now, this process is semi-automated, since we still need to manually edit the env.list file with the versions and the hash commits.\n1. First prow job : [postsubmit-build-docker.yaml](https://github.com/ppc64le-cloud/test-infra/blob/master/config/jobs/ppc64le-cloud/build-docker/postsubmit-build-docker.yaml#L2:L59)\n\nThis postsubmit prow job is triggered by the editing of the [env.list](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/env/env.list). This file contains the information we need to build the packages : \n- DOCKER_TAG : latest version of docker, that we want to build\n- DOCKER_PACKAGING_HASH : commit associated to the latest version of docker-packaging\n- CONTAINERD_BUILD : if set to 1, it means that a new containerd version has been released that we have not built it yet ; if set to 0, it means that we have already built it in a previous prow job and that we do not need to build it again (we will still verify that no new distribution has been added).\n- CONTAINERD_TAG : latest version of containerd\n- CONTAINERD_PACKAGING_HASH : commit associated to the latest version of containerd\n- RUNC_VERS : runc version used to build the static packages\n- DIND_IMG_STATIC_HASH: The image hash of the Docker-in-Docker image that will be used as the base image to build static binaries. It can be obtained from https://quay.io/repository/powercloud/docker-ce-build\n\nThis prow job builds the dynamic docker packages and then pushes them to our internal COS bucket, before creating the file 'env/date.list' which contains the current date (timestamp). We use the date in the directory where we store the docker packages in the COS bucket, so that we don't confuse the different builds.\n\n1. [Start the docker daemon](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-docker.sh#L20:L22)\n2. [Access to the internal COS Bucket and set up the environmental variables](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-docker.sh#L25:L26)\n3. [Build the dynamic and static docker packages](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-docker.sh#L31:L35)\n4. [Push to the github repository the timestamp content in to the job/postsubmit-build-docker file from the prow-job-tracking branch](https://github.com/ppc64le-cloud/docker-ce-build/blob/prow-job-tracking/job/postsubmit-build-docker)\n\n2. Second prow job : [postsubmit-build-test-containerd.yaml](https://github.com/ppc64le-cloud/test-infra/blob/master/config/jobs/ppc64le-cloud/build-docker/postsubmit-build-docker.yaml#L61:L131)\n\nThis postsubmit prow job is triggered by the editing of the [job/postsubmit-build-docker](https://github.com/ppc64le-cloud/docker-ce-build/blob/prow-job-tracking/job/postsubmit-build-docker), which was edited at the end of the first prow job.\nThis prow job builds the dynamic containerd packages (if CONTAINERD_BUILD is set to 1 in the [env.list](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/env/env.list)), the static packages, and tests all packages.\n\n1. [Start the docker daemon](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-test-containerd.sh#L23)\n2. [Access to the internal COS Bucket and set up the environmental variables](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-test-containerd.sh#L27:L30)\n3. [Get the dockertest and containerd directories if CONTAINERD_BUILD=0 from the COS bucket](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-docker.sh#L35)\n4. [Build the containerd packages](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-test-containerd.sh#L37:L41)\n5. [Test the dynamic and static packages and check if there are any errors](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-test-containerd.sh#L48:L58)\n6. [Push to the COS bucket shared with the Docker team the docker and containerd packages](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-test-containerd.sh#L80:83)\n\n2. Third set of prow jobs for upstream CI: [periodic-ci-docker.yaml](https://github.com/ppc64le-cloud/test-infra/blob/master/config/jobs/ppc64le-cloud/build-docker/periodic-ci-docker.yaml)\n\nThis periodic prow job is triggered once a day. It runs test against the [moby repository](https://github.com/moby/moby/). The 4 jobs in the file trigger the following scripts.\n\n1. [Check the kernel configuration](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-info-docker.sh)\n2. [Build the dev image from the moby repository](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-build-dev-image.sh)\n3. [Run unit tests defined in moby/hack/test/unit](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-unit-test-docker.sh)\n3. [Run integration tests defined in moby/Makefile](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/prow-integration-test-docker.sh)\n\n### The 9 scripts in detail\n- [trigger-prow-job-from-git.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/trigger-prow-job-from-git.sh)\n\nTrigger the execution of the next prow job by pushing a file change on a tracking branch of a github\nrepository. \nThe tracking branch is [prow-job-tracking] https://github.com/ppc64le-cloud/docker-ce-build/tree/prow-job-tracking.\n\n- [dockerctl.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/dockerctl.sh)\n**Usage**: dockerctl [start] | [stop]\n\nStart or Stop the dockerd daemon.\nThis script runs the **dockerd-entrypoint.sh** in the background and then checks if the docker daemon has started and is running. We specify the MTU. See the reason [here](https://sylwit.medium.com/how-we-spent-a-full-day-figuring-out-a-mtu-issue-with-docker-4d81fdfe2caf).\n\n- [get-env.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/get-env.sh)\n\nThis script mounts the internal COS bucket for further uses.\nIt clones the [docker-ce-packaging](https://github.com/docker/docker-ce-packaging) using the hash commit specified in the [env.list](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/env/env.list) and gets the list of distributions in the **env-distrib.list**.\n\n- [get-env-containerd.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/get-env-containerd.sh)\n\nThis script mounts the internal COS bucket, if it has not already been mounted.\nIt gets the dockertest directory from the COS bucket. \nIt also gets the latest containerd directory in the COS bucket, if the latest version has already been built. We get the latest containerd directory for the tests.\n\n- [build-docker.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/build-docker.sh)\n\nThis script builds the version of the dynamic docker packages, which is specified in the [env.list](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/env/env.list).\nWe build in parallel to gain some time. We build 4 distributions at the same time.\nAfter each package successfully built, we push the package to our internal COS bucket, to ensure that we have them stored in case the prow job fails before finishing.\n\n- [build-containerd.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/get-build-containerd.sh)\n\nThis script builds the version of the dynamic docker packages, which is specified in the [env.list](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/env/env.list) and the static packages. As already mentionned, it only builds the containerd packages if CONTAINERD_BUILD is set to 1. \nWe cannot build the packages in parallel, due to a ``git`` command in the Makefile.\nAs for the **build-docker.sh**, the packages are pushed to the internal COS bucket.\n\n- [test.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test.sh)\n\nThis script sets up the tests for both the docker-ce and containerd packages and the static binaries.\nIt takes an optional 'test mode' argument:\n    - local (default): the test is done on the packages that were just built.\n    - staging: the test is done by installing the packages from the docker's staging repo (yum and apt).\n    - release: the test is done by installing the packages from the docker's official repo (yum and apt).\nNote: static packages are only tested for the 'local' mode only at the moment at those packages are not published.\n\nLocal mode:\nIn this script, for each distribution, we build an image, where we install the newly built packages. We then run a docker based on this said image, in which we run **test-launch.sh**.\nWe do this for each distribution, for the docker-ce packages and the static binaries.\nIt generates an **errors.txt** file with a summary of all tests, containing the exit codes of each test.\n\nStaging mode:\nThis script is used to check the packages published by Docker on https://download-stage.docker.com are correct.\nIt uses the same basis as **test.sh** but uses different images (test-repo-DEBS and test-repo-RPMS).\n\nRelease mode:\nThis script is used to check the packages published by Docker on https://download.docker.com are correct.\n\n- [test-launch.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-launch.sh)\n\nThis script is called in the **test.sh**. This runs three tests for every distro we have built, using the [powercloud/dockertest](https://github.ibm.com/powercloud/dockertest). It uses gotestsum to generate xml files.\n- test 1 : TestDistro\n- test 2 : TestDistroInstallPackage\n- test 3 : TestDistroPackageCheck\n\n- [check-tests.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/check-tests.sh)\n\nThis script checks the **errors.txt**, generated by the test.sh, to determine if there are any errors in the tests of the packages.\n\n- [push-COS.sh](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/push-COS.sh)\n\nThis script should push all packages built to the COS bucket shared with Docker. \n### The 7 images in detail\n\n- [dind-docker-build](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/images/docker-in-docker/Dockerfile)\n\nThis Dockerfile is used for getting a docker-in-docker container. It is used for the basis of the prow job, as well as for the container building the packages and the one testing the packages. It also installs s3fs to get directly access to the COS buckets.\n\n- [test-DEBS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-DEBS/Dockerfile) and [test-RPMS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-RPMS/Dockerfile)\n\nThese two Dockerfiles are used for testing the docker-ce and containerd packages. Depending on the distro type (debs or rpms), we use them to build a container to test the packages and run **test-launch.sh**.\n\n- [test-static-DEBS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-static-DEBS/Dockerfile) and [test-static-RPMS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-static-RPMS/Dockerfile)\n\nThese two Dockerfiles are used for testing the static binaries. Like the two aforementioned Dockerfiles : depending on the distro type (debs or rpms), we use them to build a container to test the packages and run **test-launch.sh**.\n\n- [test-repo-DEBS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-repo-DEBS/Dockerfile) and [test-repo-RPMS](https://github.com/ppc64le-cloud/docker-ce-build/blob/main/test-repo-RPMS/Dockerfile)\n\nThese two Dockerfiles are used for testing the packages after Docker has published them on https://download-stage.docker.com or https://download-stage.docker.com. As well as for the previous Dockerfiles, depending on the distro type, we use them to build a container and test the packages with the script **test-launch.sh**.\n\n## How to test the scripts manually in a pod\n### Set up the secrets and the pod\n\nYou need first to set up the secrets docker-token and docker-s3-credentials with kubectl.\n\n```bash\n# docker-token\ndocker login\nkubectl create secret generic docker-token \\\n    --from-file=.dockerconfigjson=$HOME/.docker/config.json \\\n    --type=kubernetes.io/dockerconfigjson\n```\nTemplate of secret for the **secret-s3.yaml** and the **git-credentials** (the latter only for tests), you just need to add the password in base64.\n```yaml\napiVersion: v1\nkind: Secret\nmetadata:\n  name: #add name\ntype: Opaque\ndata:\n  password: #add password in base64\n```\n\n```bash\nkubectl apply -f secret-s3.yaml\n```\nYou also need the **dockerd-entrypoint.sh**, which is the script that starts the docker daemon :\n```\nwget -O /usr/local/bin/dockerd-entrypoint.sh https://raw.githubusercontent.com/docker-library/docker/094faa88f437cafef7aeb0cc36e75b59046cc4b9/20.10/dind/dockerd-entrypoint.sh\nchmod +x /usr/local/bin/dockerd-entrypoint.sh\n```\n**pod.yaml**\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: pod-docker-build\nspec:\n  automountServiceAccountToken: false\n  containers:\n  - name: test\n    command:\n    - /usr/local/bin/dockerd-entrypoint.sh\n    args:\n    - \"--mtu=1440\"\n    image: quay.io/powercloud/docker-ce-build\n    resources:\n      requests:\n        cpu: \"4000m\"\n        memory: \"8Gi\"\n      limits:\n        cpu: \"4000m\"\n        memory: \"8Gi\"\n    terminationMessagePolicy: FallbackToLogsOnError\n    securityContext:\n      privileged: true\n    volumeMounts:\n    - name: docker-graph-storage\n      mountPath: /var/lib/docker\n    env:\n      - name: DOCKER_SECRET_AUTH\n        valueFrom:\n          secretKeyRef:\n            name: docker-token\n            key: .dockerconfigjson\n      - name: S3_SECRET_AUTH\n        valueFrom:\n          secretKeyRef:\n            name: docker-s3-credentials\n            key: password\n  terminationGracePeriodSeconds: 18\n  volumes:\n  - name: docker-graph-storage\n    emptyDir: {}\n```\n```bash\nkubectl apply -f pod.yaml\nkubectl exec -it pod/pod-docker-build -- /bin/bash\n```\n\nExplanations :\n- The MTU needs to be specified, see [Investigate the docker daemon configuration for docker-in-docker](https://github.ibm.com/powercloud/container-dev/issues/1543)\n- The resources : [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)\n- The securityContext must be \"privileged\" and the volume /var/lib/docker must be specified : see [Docker can now run within Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/)\n\n### Run the scripts\n\nRun **prow-build-docker.sh** or **prow-build-test-containerd.sh** except for the line calling **dockerctl.sh**. The **dockerd-entrypoint.sh** script has already been called as entrypoint of the pod, so it should not be called a second time.\n\n## How to test the whole prow job on a cluster\n\nIf the cluster was already created, get only the config file containing the necessary information to connect to the cluster and point the KUBECONFIG variable to the file.\nIf there is no cluster, you can create a ppc64le cluster with kubeadm.\nThe script that can run the prow job on the ppc64le cluster must be used on an x86 machine (no ppc64le support for kind).\n\n### Set up a ppc64le cluster with kubeadm\n\nOn a ppc64le machine :\nSee https://github.com/ppc64le-cloud/test-infra/wiki/Creating-Kubernetes-cluster-with-kubeadm-on-Power\n\n\nOn an x86 machine:\n```bash\nrm -rf $HOME/.kube/config/admin.conf\nnano $HOME/.kube/config/admin.conf\n# Copy the admin.conf from the ppc64le machine\nexport KUBECONFIG=$HOME/.kube/config/admin.conf\n# Check if the cluster is running\nkubectl cluster-info\n```\nOn either of these machines, where the ppc64le cluster is running, configure the secrets (docker-s3-credentials and docker-token if needed).\n\n### Run the prow job on a x86 machine\nOn the x86 machine :\n```bash\n# Set CONFIG_PATH and JOB_CONFIG_PATH with an absolute path\nexport CONFIG_PATH=\"$(pwd)/test-infra/config/prow/config.yaml\"\nexport JOB_CONFIG_PATH=\"$(pwd)/test-infra/config/jobs/periodic/docker-in-docker/periodic-build-docker.yaml\"\n\n./test-infra/hack/test-pj.sh -j ${JOB_NAME}\n# The job name is specified in your yaml.\n```\n#### Things to know when running a prow job against a ppc64le cluster :\n\n- If you don't need it and you are asked about **Volume \"ssh-keys-bot-ssh-secret\"**, answer empty, or you can remove these lines from the [config.yaml](https://github.com/ppc64le-cloud/test-infra/blob/master/config/prow/config.yaml#L77:L78).\n- In the [test-pj.sh](https://github.com/ppc64le-cloud/test-infra/blob/master/hack/test-pj.sh#L16), the **--local** flag specifies the local directory in which the logs will be stored. If you want it to be pushed to a COS bucket or if you want the logs to be displayed in the UI, you need to remove the **--local** flag and the directory specified afterwards.\n- Namespace : test-pods (namespace where to test the prow jobs)\n- The [prow UI](https://prow.ppc64le-cloud.cis.ibm.net/) and the [job-history of my prow job postsubmit-build-docker](https://prow.ppc64le-cloud.cis.ibm.net/job-history/s3/prow-logs/logs/postsubmit-build-docker)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fppc64le-cloud%2Fdocker-ce-build","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fppc64le-cloud%2Fdocker-ce-build","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fppc64le-cloud%2Fdocker-ce-build/lists"}