{"id":17320343,"url":"https://github.com/gdraheim/docker-mirror-packages-repo","last_synced_at":"2025-06-30T23:04:03.100Z","repository":{"id":84104686,"uuid":"146992909","full_name":"gdraheim/docker-mirror-packages-repo","owner":"gdraheim","description":"Allows to simulate upstream centos / ubuntu / opensuse package repository servers by using a docker container which contains a local package mirror","archived":false,"fork":false,"pushed_at":"2025-03-25T12:57:27.000Z","size":1158,"stargazers_count":13,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-21T14:54:15.667Z","etag":null,"topics":["centos","docker","mirroring","mirrors","opensuse","ubuntu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gdraheim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"SUPPORTED.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-09-01T11:30:23.000Z","updated_at":"2025-03-25T12:40:33.000Z","dependencies_parsed_at":"2024-04-08T22:34:08.311Z","dependency_job_id":"b6e5eb65-9555-4f78-82d4-fda9aab4bea8","html_url":"https://github.com/gdraheim/docker-mirror-packages-repo","commit_stats":{"total_commits":460,"total_committers":2,"mean_commits":230.0,"dds":"0.0021739130434782483","last_synced_commit":"ad8f1b5193365ce9ba0b32b5f4ffc8e73009d00f"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/gdraheim/docker-mirror-packages-repo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdraheim%2Fdocker-mirror-packages-repo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdraheim%2Fdocker-mirror-packages-repo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdraheim%2Fdocker-mirror-packages-repo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdraheim%2Fdocker-mirror-packages-repo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gdraheim","download_url":"https://codeload.github.com/gdraheim/docker-mirror-packages-repo/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gdraheim%2Fdocker-mirror-packages-repo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262864118,"owners_count":23376455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["centos","docker","mirroring","mirrors","opensuse","ubuntu"],"created_at":"2024-10-15T13:30:13.641Z","updated_at":"2025-06-30T23:04:03.066Z","avatar_url":"https://github.com/gdraheim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Lint Check](https://github.com/gdraheim/docker-mirror-packages-repo/actions/workflows/lintcheck.yml/badge.svg?event=push)](https://github.com/gdraheim/docker-mirror-packages-repo/actions/workflows/lintcheck.yml)\n[![Type Check](https://github.com/gdraheim/docker-mirror-packages-repo/actions/workflows/typecheck.yml/badge.svg?event=push)](https://github.com/gdraheim/docker-mirror-packages-repo/actions/workflows/typecheck.yml)\n[![PyPI version](https://badge.fury.io/py/docker-mirror-packages-repo.svg)](https://pypi.org/project/docker-mirror-packages-repo/)\n\n# docker mirror packages repo\n\nAllows to simulate upstream centos / ubuntu / opensuse\npackage repository servers by using a docker container\nwhich contains a local package mirror.\n\nThe scripts in this project allow to build a local package\nmirror using rsync. When ready the rpm/deb packages get\ncopied into a docker image which has a small web script\nthat mimics the mirrorlist function of the original\nupstream package repository servers.\n\nThere is a helper script docker_mirror which can start\na mirror docker image as container. The --add-hosts\noption prints the line for \"docker run\" so that calls\nto the original upstream package server get diverted\nto the ip address of the locally running docker container.\n\nEffectively all `\"yum install\"` or `\"zypper install\"` or\n`\"apt-get install\"` commands run locally in the docker \nnetwork. No internet access required. Without changing \nrepo files. Without special proxy host setups. This is\nfaster, allows for reproducible install tests, and it\ncan be used in a disconnected environment.\n\nThe scripting has also been used with docker-compose\nand ansible deployment setups. Over the years a number\nof package repos have been tested - starting with\ndistros from 2016 until today. Even when the upstream\npackage repositories get disabled, for being out of\nsupport, the local docker mirrors continue to work.\n\n----\n\nTested repos are\n\n| docker image                                  | size\n| --------------------------------------------- | ----\n| mirror-packages/epel-repo:9.x.2406\t| 44.9GB\n| mirror-packages/epel-repo:7.x.2012\t| 49GB\n| mirror-packages/debian-repo/contrib:12\t| 132GB\n| mirror-packages/debian-repo/main:12\t| 127GB\n| mirror-packages/debian-repo/contrib:10\t| 104GB\n| mirror-packages/debian-repo/main:10\t| 99.9GB\n| mirror-packages/ubuntu-repo/universe:24.04\t| 134GB\n| mirror-packages/ubuntu-repo/restricted:24.04\t| 30.6GB\n| mirror-packages/ubuntu-repo/main:24.04\t| 27GB\n| mirror-packages/ubuntu-repo/universe:22.04\t| 178GB\n| mirror-packages/ubuntu-repo/restricted:22.04\t| 35GB\n| mirror-packages/ubuntu-repo/main:22.04\t| 29.2GB\n| mirror-packages/ubuntu-repo/universe:20.04\t| 198GB\n| mirror-packages/ubuntu-repo/restricted:20.04\t| 111GB\n| mirror-packages/ubuntu-repo/main:20.04\t| 107GB\n| mirror-packages/ubuntu-repo/restricted:18.04\t| 26GB\n| mirror-packages/ubuntu-repo/main:18.04\t| 25.8GB\n| mirror-packages/ubuntu-repo/universe:16.04\t| 186GB\n| mirror-packages/ubuntu-repo/restricted:16.04\t| 45.4GB\n| mirror-packages/ubuntu-repo/main:16.04\t| 44.4GB\n| mirror-packages/almalinux-repo/plus:9.5\t| 71.3GB\n| mirror-packages/almalinux-repo/main:9.5\t| 35.7GB\n| mirror-packages/almalinux-repo/plus:9.4\t| 46.5GB\n| mirror-packages/almalinux-repo/main:9.4\t| 23.5GB\n| mirror-packages/almalinux-repo/plus:9.4\t| 46.4GB\n| mirror-packages/almalinux-repo/main:9.4\t| 23.4GB\n| mirror-packages/almalinux-repo/plus:9.3\t| 59.6GB\n| mirror-packages/almalinux-repo/main:9.3\t| 26.1GB\n| mirror-packages/almalinux-repo/plus:9.1\t| 23.1GB\n| mirror-packages/almalinux-repo/main:9.1\t| 22.4GB\n| mirror-packages/centos-repo/sclo:7.9.2009\t| 22.4GB\n| mirror-packages/centos-repo7:7.9.2009\t| 22.4GB\n| mirror-packages/centos-repo/main:7.9.2009\t| 15.1GB\n| mirror-packages/opensuse-repo/update:15.6\t| 210GB\n| mirror-packages/opensuse-repo/main:15.6\t| 205GB\n| mirror-packages/opensuse-repo/update:15.5\t| 112GB\n| mirror-packages/opensuse-repo/main:15.5\t| 70.8GB\n| mirror-packages/opensuse-repo/update:15.4\t| 143GB\n| mirror-packages/opensuse-repo/main:15.4\t| 59.8GB\n| mirror-packages/opensuse-repo/update:15.2\t| 118GB\n\nRemember that the internet package repos contain updates for the last\ndistribution release, so they do grow in size over time. As such the\nimage sizes (as shown above) may differ after each sync.\n\n## BACKGROUND\n\nConsider that you have a Dockerfile with\n\n    FROM centos:7.5.1804\n    RUN yum install -y httpd httpd-tools\n\nThis will reach out to the internet downloading 20+ rpm packages.\n\nIt would be better to have a local mirror of the default yum \nrepositories for centos. When using a docker environment you \ncan reassign the systems' package repo URL to a local container \nthat can answer the yum download requests.\n\nThat mode is not only faster but you can also commit the yum repo\nmirror in your binary repository. New updates on the external \ninternet yum repository will not suddenly change rebuild results. \nThis has been extensively used in testing docker-related tools \nwhere the result image was dropped right away after testing it.\n\n## CENTOS\n\nMimics the default URL of http://mirrorlist.centos.org\n\nThe default package repositories in CentOS look like this:\n\n    # /etc/yum.repos.d/CentOS-Base.repo\n    [base]\n    name=CentOS-$releasever - Base\n    mirrorlist=http://mirrorlist.centos.org/?release=$releasever\u0026arch=$basearch\u0026repo=os\n    #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/\n\nSo instead of pointing directly to the storage host \"mirror.centos.org\" \nthere is a reference to a web service on \"mirrorlist.centos.org\" that \nwill return a list of very different storage host options.\n\nThe provided \"centos-repo\" image is able to answer mirrorlist requests\non its port 80/http. It will simply return a single URL pointing back\nto itself. A storage path is attached to serve as the download baseurl.\n\nExample: with --add-host \"mirrorlist.centos.org:172.22.0.2\"\n\n    REQUEST: http://mirrorlist.centos.org/?release=7\u0026repo=os\u0026arch=x86_64\n    REALLLY: http://172.22.0.2/?release=7\u0026repo=os\u0026arch=x86_64\n    Answered Mirrorlist:\n       http://172.22.0.2/7/os/x86_64\n\nThe provided \"centos-repo\" image will furthermore answer download requests\nfor yum packages on 80/http - simply the url path is mapped to a local path\nin the container and a sendfile will return the content. These are mostly \n`*.rpm` packages as well as some package index files.\n\nFor more information check the [centos-repo mirror info](./centos-repo-mirror.info.md)\n\n## ALMALINUX\n\nMimics the default URL of http://mirrors.almalinux.org\n\nThe default package repositories in AlmaLinux look like this:\n\n    # /etc/yum.repos.d/almalinux-baseos.repo\n    [baseos]\n    name=AlmaLinux $releasever - BaseOS\n    mirrorlist=https://mirrors.almalinux.org/mirrorlist/$releasever/baseos\n    # baseurl=https://repo.almalinux.org/almalinux/$releasever/BaseOS/$basearch/os/\n\nThis looks similar to centos but actually it runs on https and the release and\nrepo queries are not added as `?parameters` but as path parts. The new style has \nbeen merged into the centos mirrorlist script.\n\nThe same image does also server package requests on https. Be sure that your\ndockerfiles have a line `echo sslverify=false \u003e\u003e /etc/yum.conf`.\n\nOther than that, the AlmaLinux images are considered the default continuation\nof the CentOS image series. So `centos-docker-mirror.py 8.4` assumes centos\nwhile `centos-docker-mirror.py 8.9` assumes almalinux as the base image.\n\n## EPEL\n\nMimics the default URL of http://mirrors.fedoraproject.org\n\nThis is an extra yum repository that works on top of CentOS or AlmaLinux.\n\nBe sure to add \"--epel\" to the docker_mirror.py call to get it started as\na second container, which will get included into the `--add-hosts` value\nresult.\n\nNote that EPEL does not have a release series. You will always get some\nlatest package from it. Instead the mirror images get an extension number\nwhich reflects the date when docker-mirror-packages-repo was created. The\ndocker_mirror.py script knows the release dates of the CentOS or AlmaLinux\nrelease, so that it can find an epel-repo which is recent enough.\n\n## OPENSUSE\n\nMimics the default URL of http://download.opensuse.org\n\nUnlike for centos there is no mirrorlist involved here. All the default \nzypper targets use that single URL. The package indexes are served at \nthe same path locations.\n\nThe provided \"opensuse-repo\" image will answer download requests \nfor zypper packages on on 80/http - simply the url path is \nmapped to a local path in the container and a sendfile will \nreturn the content. These are mostly `*.rpm` packages as well\nas some package index files.\n\nFor mor information check the [opensuse-repo mirror info](./opensuse-repo-mirror.info.md)\n\n## UBUNTU\n\nMimics the default URL of http://archive.ubuntu.com\n\nMost of the default apt-get targets use that URL. The\nonly exception is http://security.ubuntu.com for getting\nupdate packages.\n\nThe provided \"ubuntu-repo\" image will answer download requests \nfor ubuntu packages on on 80/http - simply the url path \nis mapped to a local path in the container and a sendfile \nwill return the content. These are mostly `*.deb` packages \nas well as some package index files.\n\nFor more information check the [ubuntu-repo mirror info](./ubuntu-repo-mirror.info.md)\n\n## UBUNTU UNIVERSE\n\nUnlike with Redhat- and Suse-style extra repositories, the\nUbuntu \"universe\" packages are hosted on the same server. That\nrequires the \"docker_mirror.py\" script to start just one container\nwhen it sees the \"--universe\" option on the command line. It does\ntry to find a \"/universe\" docker-mirror-packages-repo variant then.\n\nSince the universe mirror is bigger by a good factor, the default\nfor docker_mirror.py is to not try to get a universe repo. Likewise\nthe ubuntu-docker-mirror.py script will not try to sync and include\nthe universe packages unless it sees the \"--universe\" option.\n\nThere was not attempt made to include multiverse package.\n\n# DOCKER MIRROR\n\nThe testing projects get a copy of the `docker_mirror.py`.\nWhen the mirror-packages are ready then the script can find\nthem in the `docker images` list and start them as a container.\nJust say\n\n* `./docker_mirror.py start ubuntu:24.04 --add-hosts`\n\nThe add-hosts says that it should print the line that is\nneeded to be added to some `docker run` for a testing container\nthat should be able to install packages locally. Thease are\nusally the URLs of the offical package repo of the distro \nbut diverted to  the IP address of the locally running repo-container. \nYou can find the repo-container as\n\n* `docker ps -f name=ubuntu-repo-24.04`\n\nSee an example in [zziplib/testbuilds.py](https://github.com/gdraheim/zziplib/blob/master/testbuilds.py)\n\n## docker_mirror.ini\n\nThe `docker_mirror.py` script will load override values in \na config file usually found in `~/.config/docker_mirror.ini`\non newer Linux distros.\n\nIn most cases that is not needed - unless you want to reduce\ndisk pressure (in the docker space) by using the disk/base\napproach where the repo data is mounted into repo container\ninstead of having a copy.\n\nFor details see [DISKBASE.md](DISKBASE.md)\n\n## IMPLEMENTATION\n\nEvery package repo has a different way to store data in its\nfile tree. In general it is not a good idea to just rsync\nall of it. Sure you can but...\n\n...for example Ubuntu uses one host for all distro versions.\nAll `*.deb` packages for all versions are stored in a single\n`$host:/pool/` subdirectory. The deb packages for a single\nversion are listed in a `$host:/dists/$version/Packages.gz` \nfile. The distro version is not given by number (16.04) but \nby code name (xenial).\n\nThe mirror scripts in this project know about that - so the \nrsync will first download the `Packages.gz` files, in order \nto unpack/filter them into a list of `/pool/x/y/*.deb` paths\nin a temporary file. Then `rsync --files-from=pool-files.tmp` \nwill do the rest.\n\nWhen the rsync is complete, only a little web service script\nis added on top of building the image. So when the image is\nstarted as a container, it does accept requests on 80/http\nserving the rsync'ed files back (except epel-repo since the\nusual Fedora URL is shown running on 443/https).\n\nFor tips and tricks please see [PROBLEMS FAQ](./PROBLEMS.md).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgdraheim%2Fdocker-mirror-packages-repo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgdraheim%2Fdocker-mirror-packages-repo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgdraheim%2Fdocker-mirror-packages-repo/lists"}