{"id":26792958,"url":"https://github.com/osbuild/rpmrepo","last_synced_at":"2025-07-25T20:11:15.299Z","repository":{"id":37093955,"uuid":"344071585","full_name":"osbuild/rpmrepo","owner":"osbuild","description":"RPM Repository Snapshot Management","archived":false,"fork":false,"pushed_at":"2025-03-04T13:48:52.000Z","size":218,"stargazers_count":16,"open_issues_count":2,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-29T16:18:11.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/osbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-03T09:29:59.000Z","updated_at":"2025-03-04T13:48:56.000Z","dependencies_parsed_at":"2023-11-09T13:46:33.453Z","dependency_job_id":"0d939b91-fc50-4fe5-8d6b-9a0060d5b322","html_url":"https://github.com/osbuild/rpmrepo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Frpmrepo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Frpmrepo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Frpmrepo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/osbuild%2Frpmrepo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/osbuild","download_url":"https://codeload.github.com/osbuild/rpmrepo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250264909,"owners_count":21402004,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-29T16:18:18.962Z","updated_at":"2025-04-22T15:11:48.714Z","avatar_url":"https://github.com/osbuild.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"RPMrepo Snapshots\n=================\n\nRPM Repository Snapshot Management\n\nThe RPMrepo project creates persistent and immutable snapshots of RPM\nrepositories. It provides tools and infrastructure to create such snapshots, as\nwell as host and serve them.\n\n### Project\n\n * **Website**: \u003chttps://www.osbuild.org\u003e\n * **Bug Tracker**: \u003chttps://github.com/osbuild/rpmrepo/issues\u003e\n\n### Requirements\n\nThe requirements for this project are:\n\n * `python \u003e= 3.8`\n\n### About\n\nRPMrepo is comprised of a set of different utilities and infrastructure. The\nmain goal is to regularly create immutable snapshots of a set of public and\nRedHat-private RPM repositories and provide them for a fixed amount of time\non our own infrastructure.\n\nOur infrastructure is maintained via the OSBuild Terraform configuration. See\nthe\n[image-builder-terraform](https://github.com/osbuild/image-builder-terraform)\nrepository, in particular the\n[rpmrepo.tf](https://github.com/osbuild/image-builder-terraform/blob/main/rpmrepo.tf)\nconfiguration.\n\nFor user documentation on RPMrepo, see:\n\u003chttps://osbuild.org/docs/developer-guide/projects/rpmrepo/\u003e\n\nThe backend implementation of RPMrepo involves the following steps:\n\n  * Target Repository Configuration\n\n    When running snapshot operations, we need to know the list of target RPM\n    repositories to snapshot, what to call the snapshots, and where to store\n    the data. This information is currently stored as JSON files in this\n    repository (see the `./repo/` subdirectory).\n\n    For each target repository, we store a JSON dictionary with the following\n    information:\n\n      * \"base-url\": The RPM repository base-url to create the snapshot of. An\n                    RPM repository base-url requires the root-level metadata\n                    file to be accessible as `repodata/repomd.xml`. See the\n                    DNF / RPM documentation for more information, if desired.\n\n      * \"platform-id\": The DNF Platform ID to use. This allows to group\n                       multiple snapshots together and share the backend\n                       storage. We use this to deduplicate RPMs in our backend.\n                       This ID can be freely chosen, but all snapshots that\n                       share an ID can only be deleted together. We usually\n                       pick the actual DNF Platform ID (see the DNF\n                       `module_platform_id` for details) here, but this is\n                       not required.\n\n      * \"singleton\": We usually create snapshots regularly. In case a target\n                     repository is already immutable by design, this key can\n                     be set to make sure only a single snapshot of this\n                     repository is ever taken. Simply set this key to the\n                     snapshot suffix to use for the singleton snapshot, and\n                     all snapshot operations will use this suffix (and thus\n                     skipping the operation if it already exists).\n\n      * \"snapshot-id\": The name of the snapshot to store it as. Usually this\n                       is the same as the name of this file without extension.\n                       This name can be freely chosen. We usually name\n                       snapshots as:\n\n                       ```\n                       \u003cplatform-id\u003e-\u003carch\u003e-\u003crepo\u003e[-\u003crepo-version\u003e].\n                       ```\n\n                       Note that the actual snapshots will get a suffix like\n                       `-\u003cdate\u003e` appended automatically. This field must not\n                       include this suffix in the snapshot ID.\n\n      * \"storage\": The ID of the storage location to use. We have different\n                   storage locations for different access rights. For now, this\n                   is just a string that specifies the directory in our backend\n                   storage. See the backend information for possible values.\n\n  * Snapshot Creation\n\n    To create snapshots, we use the `reposync` dnf module. See `dnf reposync`\n    for more information. This tool just downloads an entire RPM repository\n    to local storage. We then index this data for our backend storage and\n    upload it.\n\n    The `./src/ctl/` directory implements the command-line control client\n    that we use for this. It is a python module that just wraps `dnf reposync`\n    to download a repository, provides indexing helpers, and then wraps the\n    AWS `boto3` API to upload everything to our storage.\n\n    Note that a single snapshot might store up to 100GiB of data intermittently\n    and can take up to 8h. Therefore, none of the default script execution\n    engines can be used, since they either have limited disk-space or limited\n    execution time.\n\n    We provide a container (see the `osbuild/containers` repository) called\n    `rpmrepo-snapshot` which reads the configuration in `./repo/` and uses the\n    python module in `./src/ctl/` to create a snapshot. The container supports\n    batched execution, thus can be used to create many snapshots in parallel.\n\n  * Storage\n\n    We currently store all snapshots in a dedicated AWS S3 bucket called\n    `rpmrepo-storage`. Since we store a lot of data, we employ a data\n    deduplication strategy. All actual data files are stored with their sha256\n    checksum as name in `data/\u003cstorage\u003e/\u003cplatform-id\u003e/sha256-\u003cchecksum\u003e`. This\n    means matching files will be deduplicated if they are stored in the same\n    storage-directory with the same platform-id.\n\n    Since we dropped file-names and paths, we cannot serve an RPM repository\n    from this checksum-based storage. Therefore, we create shim wrapping layers\n    that refer to this storage. In `data/ref/\u003cplatform-id\u003e/\u003csnapshot-id\u003e/...`\n    we store the entire RPM repository, but with empty files. We then attach\n    AWS S3 metadata to all these empty files and fill in the checksum of their\n    content. This way, all objects underneath the `data/ref/...` directory is\n    empty, and thus free of charge.\n\n    Our frontend thus only needs to redirect requests from `data/ref/` to\n    the correct underlying file, by reading the checksum metadata.\n\n  * Gateway\n\n    The frontend to the RPM repository snapshots is a simple HTTP REST API. It\n    uses AWS API Gateway to create a simple catch-all REST API that forwards\n    all HTTP requests to an AWS Lambda script. This script is sourced from\n    `./src/gateway/` in this repository.\n\n    This scripts provides a multitude of legacy interfaces for all kinds of\n    operations. See its implementation for details. Its main job is to read\n    requests to a snapshot, find the file in `data/ref/...`, read the checksum\n    from the metadata of this empty file, and then return a 301 HTTP redirect\n    to the right file in `data/\u003cstorage\u003e/\u003cplatform-id\u003e/sha256-\u003cchecksum\u003e`. It\n    is then the client's job to follow this redirect and directly download the\n    file.\n\n    Note that we simply redirect clients to the public HTTP interface to AWS\n    S3. The gateway never transmits any data of the repositories. This keeps\n    our charges low and makes sure large files are always directly transferred\n    between AWS S3 and the client.\n\n    Several paths in the `rpmrepo-storage` S3 bucket are publicly accessible.\n    In particular, `data/public/`, `data/ref/`, and `data/thread/`. The\n    `data/rhvpn/` path is *NOT* publicly accessible. Instead, we have an AWS\n    VPC Endpoint that opens up this path to all clients from within the RH\n    VPN. Hence, data stored in this directory is only accessible from within\n    RH.\n    Note that `data/ref/` is public, and as such all snapshots can be listed\n    and enumerated publicly. Only the file content is possibly protected from\n    public access. This is intentional, but can be changed in the future if\n    it poses a problem.\n\n    Apart from redirects, the gateway also provides utility functions to\n    enumerate all snapshots, or redirect to old legacy storage locations of\n    older RPMrepo revisions.\n\n  * Snapshot Routine\n\n    As a single snapshot operation requires a lot of storage and time, we use\n    custom infrastructure to run this. This used to be Beaker, but for better\n    reliability, we now schedule the snapshot jobs on AWS Batch. The\n    previously mentioned `rpmrepo-snapshot` container is scheduled on AWS Batch\n    and then will create the requested snapshots.\n\n    The snapshot routine can be scheduled as a single job, or as an array job.\n    If scheduled as single job, you must specify the name of the target\n    configuration in `./repo/` to run. If scheduled as an array job, you should\n    size the array as big as the number of files in `./repo/` (bigger is fine,\n    those excess jobs will be no-ops; smaller is less fine, as it will miss\n    snapshots). The array jobs will then each pick one file in `./repo/` based\n    on their ARRAY-JOB-ID.\n\n    Furthermore, the snapshot routine requires you to specify the branch and\n    commit of the `rpmrepo` repository to use. You can use `main`+`HEAD`, but\n    this will be subject to concurrent changes in the upstream repository. You\n    are strongly advised to use `main`+`\u003ccommit-sha\u003e` instead.\n\n    Lastly, you can specify the suffix to be used for the snapshots. If you\n    specify `auto`, it will use the current date and time (except for singleton\n    snapshots; see above). You should specify this suffix manually to make\n    sure all snapshots share a suffix. Otherwise, updating users will be a\n    hassle.\n\n    The AWS Batch interface will allow you to track all the snapshot jobs, see\n    which failed, and allow you to reschedule individual jobs, if desired.\n\n  * Updating snapshot configurations\n\n    The `./repo/` directory contains all the configuration files for the\n    snapshots. Each file is a JSON file that specifies the configuration for\n    one snapshot.\n\n    Individual snapshot configurations can be generated using the helper script\n    `./gen-repos.py`. Multiple snapshot configurations can be generated by\n    defining them in `./repo-definitions.yaml` and then running\n    `./gen-all-repos.py`, which internally calls `./gen-repos.py`.\n\n    Updating the snapshot configurations usually consists of deleting unused\n    configurations, and adding new ones. The most convenient way to do this is\n    to update the `./repo-definitions.yaml` file, and then run the\n    `snapshot-configs` Makefile target. This will automatically delete all\n    configurations that are no longer present in the definition file, and\n    generate new ones.\n\n### List Available Snapshots\n\nIf you just need a list of the available snapshots you can query the API like\nthis:\n\n```\ncurl https://rpmrepo.osbuild.org/v2/enumerate | jq .\n```\n\nWhich will return a JSON list of the snapshots names.\n\n### Repository:\n\n - **web**:   \u003chttps://github.com/osbuild/rpmrepo\u003e\n - **https**: `https://github.com/osbuild/rpmrepo.git`\n - **ssh**:   `git@github.com:osbuild/rpmrepo.git`\n\n### License:\n\n - **Apache-2.0**\n - See LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosbuild%2Frpmrepo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosbuild%2Frpmrepo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosbuild%2Frpmrepo/lists"}