{"id":18584674,"url":"https://github.com/fedora-modularity/baseruntime-package-lists","last_synced_at":"2025-08-19T22:08:29.411Z","repository":{"id":70470633,"uuid":"87204384","full_name":"fedora-modularity/baseruntime-package-lists","owner":"fedora-modularity","description":null,"archived":false,"fork":false,"pushed_at":"2018-01-16T21:12:26.000Z","size":46284,"stargazers_count":0,"open_issues_count":11,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-17T16:52:09.023Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fedora-modularity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-04T15:35:37.000Z","updated_at":"2017-06-30T11:15:29.000Z","dependencies_parsed_at":"2023-02-28T20:00:10.151Z","dependency_job_id":null,"html_url":"https://github.com/fedora-modularity/baseruntime-package-lists","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedora-modularity%2Fbaseruntime-package-lists","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedora-modularity%2Fbaseruntime-package-lists/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedora-modularity%2Fbaseruntime-package-lists/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fedora-modularity%2Fbaseruntime-package-lists/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fedora-modularity","download_url":"https://codeload.github.com/fedora-modularity/baseruntime-package-lists/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254474280,"owners_count":22077252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T00:28:25.297Z","updated_at":"2025-05-16T05:33:38.912Z","avatar_url":"https://github.com/fedora-modularity.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Generating Base Runtime Lists\n\nThis repository provides a series of BASH and Python tools for creating\nand updating lists and dist-git hashes for modules in the Fedora Project.\n\n## Setup\nStorage requirements:\n* You will need up to 15GiB of free space for the repository data. Note\n  that this number keeps changing, so add another 5GiB for safety.\n\nThese tools require the following pre-requisites:\n* bash\n* python 2\n* python 3\n* rsync\n* mock\n* rpmdevtools\n* koji\n* depchase (See below)\n* perl\n* perl(autodie), perl(Getopt::Std), perl(IPC::Open3), perl(List::Util),\n  perl(Template), perl(Text::CSV_XS) and perl(Text::Wrap)\n\nTo hack on these tools, you may also need:\n* argbash\n\nThe user that will be downloading the RPM repos to work with must have\nthe `mock` package installed on their system and be a member of the\n'mock' group.  This can be accomplished by:\n```\ndnf install mock\nusermod -a -G mock \u003cyourloginname\u003e\n```\n\nThe user that will be operating on the override repositories must be a\nmember of the `modularity-wg` group in the Fedora Account system, that\nyour SSH keys are properly set up for use with Fedora infrastructure\nand also that you have an identical group present in `/etc/group` such as:\n```\nmodularity-wg:x:189842:\u003cyourloginname\u003e\n```\nThis is so the sync tools can ensure that the files on Fedora People\nare accessible to all members of the modularity-wg team. When you add\nthis, log out of your current session and back in to have the new group\ntake effect.\n\n## Legend/Glossary\nI will use the following terms or variables in this document:\n\n`\u003crelease\u003e`: The path on disk representing a Fedora release or\npre-release.  For example: Fedora 25 would be `25`, but Fedora 26 Beta\nwould be `test/26_beta`.\n\n`\u003carch\u003e`: The processor architecture. Supports\n* `aarch64`\n* `armv7hl`\n* `i686`\n* `ppc64`\n* `ppc64le`\n* `s390x`\n* `x86_64`\n\n(Note: some infrastructure tools use other architecture names for things;\nthese scripts will automatically translate from e.g. `armv7hl` to `armhfp`\nwhere needed, so always use one of these for command-line arguments.)\n\n### Installing depchase\n\n```\n$ git clone https://github.com/fedora-modularity/depchase.git\n$ cd depchase\n$ python3 setup.py install --user\n```\nThis will install the `depchase` command as `~/.local/bin/depchase`\n\nYou may wish to add this to your default PATH variable in `~/.bashrc`\nby doing:\n```\n$ echo \"export PATH=$PATH:$HOME/.local/bin\" \u003e\u003e ~/.bashrc\n```\n\n## Preparing the local repository data\n\nDepchase (a tool we use for processing the data) requires local copies\nof the repository metadata against which to operate. This repository\nprovides some convenient tools for retrieving this data.\n\n### Syncing the existing repositories (the easy way)\n\nInvoke `make -B repo/devel` to download or update your local copy of\nthe development tree repodata.  This operation may take several minutes.\n\n### Syncing the existing repositories (the manual way)\n\nAlternatively you may download the repodata tree manually.\n\nUse the `download_repo.sh` tool located in the root of this repository\nto retrieve the repository metadata and override repositories.\n\nFor example:\n\n```\n./download_repo.sh --release=devel \\\n                   --milestone=Beta \\\n                   --archful-srpm-file=./archful-srpms.txt \\\n                   --arch=aarch64 \\\n                   --arch=armv7hl \\\n                   --arch=i686 \\\n                   --arch=ppc64 \\\n                   --arch=ppc64le \\\n                   --arch=x86_64 \\\n                   --overrides\n```\n\nThis will download the frozen Fedora 26 Beta repository metadata, the\noverrides repository for that milestone and will automatically regenerate\nthose SRPMs that are known to have arch-specific `BuildRequires` and\nmerge them locally to the repo metadata. The packages will be stored in\nthe `repo/test/26_beta/` path relative to the root of this repository.\n\nUse `--release=devel` for a snapshot of rawhide suitable for development\nor `--release=rawhide` for the latest Rawhide compose.  Note Fedora 27\nand newer also support s390x.  Not all `--release` arguments support\nall other options.\n\nThis process may take a long time, as it will transfer approximately\n1GiB of data per architecture to the local system. This tool uses rsync,\nso subsequent calls to it will only download updated information (plus\nregenerating the archful SRPMs).\n\n### Adding content to the override repositories\n\nCopy any binary (or noarch) RPMs for inclusion into\n`repo/\u003crelease\u003e/override/\u003carch\u003e/os`. For source RPMs, copy them to the\n`repo/\u003crelease\u003e/override/\u003carch\u003e/sources` path. Run \n`./repo/generate-repo-data.sh [\u003crelease\u003e]` to automatically update \nthe repo metadata and `./repo/rsync-push-repo.sh [\u003crelease\u003e]` to rsync the\ncontents to the fedorapeople repository.\n\nThere is a helper utility in the root of the git repository called\n`dl_pkgs.sh`.  This utility must be called with one or two arguments.\nFirst, a path to a file containing one package NVR or Koji build-id\nper line. If it is an NVR, the package must be an official build in\nKoji. For a build-id, a scratch-build may work (but is untested). The\nsecond argument should be `\u003crelease\u003e` .  This tool should be used to\nprep the override repository whenever a content change is made to the\nmodule metadata, to keep the lists up-to-date.\n\nOnce completed, run `./repo/generate-repo-data.sh [\u003crelease\u003e]` to \nregenerate all the repository information.  You can then run whatever \ntests you want.  When you are happy with the results, run \n`./repo/rsync-push-repo.sh [\u003crelease\u003e]` to update the fedorapeople repository.\n\n## Dealing with SRPMs with arch-specific BuildRequires\n\nSome packages have differing BuildRequires depending on which platform\nis being built. In these cases, in order to have the list generation\nfind the right values, we need to pull down those SRPMs and regenerate\nthem for every architecture that we process.\n\nThere is a helper utility in the root of the git repository called\n`mock_wrapper.sh` which takes a single argument: a list of NVRs of\nofficial Koji builds that should be regenerated. It will construct a mock\nbuildroot with limited packages (to reduce the risk that the output could\nbe different depending on the packages available in the host system),\ncheck out each of the requested SRPM packages and regenerate them on\neach architecture. It will then drop them into a directory `./output`\n(clobbering any existing content!) relative to the working directory\nthat was executed.\n\nOnce that content is present, you can run `./populate_srpm_repo.sh\n\u003crelease\u003e/override` from the root of the git repository, which will move\nall of the `./output` contents into the `repo/` hierarchy, after which you\ncan follow the \"Pushing to the override repositories\" instructions above.\n\nThis functionality is automatically run whenever the `download_repo.sh`\nscript is invoked with the `--with-archful-srpm-file` argument.\n\n## How to generate package lists (the easy way)\n\nTo generate the relevant package lists for the development snapshot tree,\nsimply invoke `make`.  This will run depchase to resolve runtime and build\ntime dependencies for components listed in the toplevel package lists\nand will also generate complete modulemd files for all supported modules.\n\n## How to generate package lists (the manual way)\n\nEnsure that the contents of `toplevel-binary-packages.txt` and `hints.txt`\nin each of the `data/Fedora/$VERSION[_$MILESTONE]/$MODULE/$ARCH`\ndirectories is correct.  These files must be present for each architecture\nto be processed.\n\n`toplevel-binary-packages.txt` contains the set of package names that\nthe base runtime must contain in order to provide the requisite APIs.\n\n`hints.txt` is a manually-curated set of packages that is used to resolve\nambiguities in dependencies. For example, the `glibc` package `Requires:\nglibc-langpack` and many packages may `Provides: glibc-langpack`,\nso we include `glibc-minimal-langpack` in the hints.txt to resolve\nthis decision.\n\nThen call `generate_module_lists.sh` like so:\n\n```\n./generate_module_lists.sh --version 26 \\\n                           --milestone=Beta \\\n                           --module=base-runtime \\\n                           --arch=aarch64 \\\n                           --arch=armv7hl \\\n                           --arch=i686 \\\n                           --arch=ppc64 \\\n                           --arch=ppc64le \\\n                           --arch=x86_64\n```\n\nThis will produce a set of files in the\n`data/Fedora/$VERSION[_$MILESTONE]/$MODULE/$ARCH` directories named:\n* runtime-binary-packages-full.txt\n* runtime-binary-packages-short.txt\n* runtime-source-packages-full.txt\n* runtime-source-packages-short.txt\n* selfhosting-binary-packages-full.txt\n* selfhosting-binary-packages-short.txt\n* selfhosting-source-packages-full.txt\n* selfhosting-source-packages-short.txt\n\nTo generate modulemd files, use `make_modulemd.pl` and point it to the\ndirectory with package lists and modulemd templates.\n\nTo generate Host \u0026 Platform: `./make_modulemd.pl ./data/Fedora/devel/hp`\n\nTo generate Bootstrap: `./make_modulemd.pl ./data/Fedora/devel/bootstrap`\n\nThis will produce one or more modulemd files in the respective module\ndirectories.  These files should be dist-git-ready, with no further\nmanual editing required.\n\n## Making sure everything is okay\n\nRun `make test` and resolve any issues before pushing the repodata or\nany changes to this repository.\n\nThe simple test suite, under `./tests`, verifies that the data looks\nsane -- such as that there are no empty package lists or that no package\nappears in more than one version.\n\n## And that's it!\n\nCongratulations, you've made it to the end!\n\nA few final words, however.  If you're hacking on Host \u0026 Platform,\nplease, file a pull request for `fedora-modularity/hp` so that your\ncontent change be properly tracked and the toplevel lists \u0026 module data\nfiles regenerated.\n\nHost \u0026 Platform modules dist-git repositories include handy Makefiles\nthat allow for easy modulemd updates.  Just run `make update` after you\npush your changes to the repo and your modulemd files will be synced.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedora-modularity%2Fbaseruntime-package-lists","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffedora-modularity%2Fbaseruntime-package-lists","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffedora-modularity%2Fbaseruntime-package-lists/lists"}