https://github.com/fedora-modularity/baseruntime-package-lists
https://github.com/fedora-modularity/baseruntime-package-lists
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/fedora-modularity/baseruntime-package-lists
- Owner: fedora-modularity
- License: other
- Created: 2017-04-04T15:35:37.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2018-01-16T21:12:26.000Z (over 8 years ago)
- Last Synced: 2025-02-17T16:52:09.023Z (over 1 year ago)
- Language: Shell
- Size: 44.1 MB
- Stars: 0
- Watchers: 2
- Forks: 13
- Open Issues: 11
-
Metadata Files:
- Readme: README.md
- License: COPYING
Awesome Lists containing this project
README
# Generating Base Runtime Lists
This repository provides a series of BASH and Python tools for creating
and updating lists and dist-git hashes for modules in the Fedora Project.
## Setup
Storage requirements:
* You will need up to 15GiB of free space for the repository data. Note
that this number keeps changing, so add another 5GiB for safety.
These tools require the following pre-requisites:
* bash
* python 2
* python 3
* rsync
* mock
* rpmdevtools
* koji
* depchase (See below)
* perl
* perl(autodie), perl(Getopt::Std), perl(IPC::Open3), perl(List::Util),
perl(Template), perl(Text::CSV_XS) and perl(Text::Wrap)
To hack on these tools, you may also need:
* argbash
The user that will be downloading the RPM repos to work with must have
the `mock` package installed on their system and be a member of the
'mock' group. This can be accomplished by:
```
dnf install mock
usermod -a -G mock
```
The user that will be operating on the override repositories must be a
member of the `modularity-wg` group in the Fedora Account system, that
your SSH keys are properly set up for use with Fedora infrastructure
and also that you have an identical group present in `/etc/group` such as:
```
modularity-wg:x:189842:
```
This is so the sync tools can ensure that the files on Fedora People
are accessible to all members of the modularity-wg team. When you add
this, log out of your current session and back in to have the new group
take effect.
## Legend/Glossary
I will use the following terms or variables in this document:
``: The path on disk representing a Fedora release or
pre-release. For example: Fedora 25 would be `25`, but Fedora 26 Beta
would be `test/26_beta`.
``: The processor architecture. Supports
* `aarch64`
* `armv7hl`
* `i686`
* `ppc64`
* `ppc64le`
* `s390x`
* `x86_64`
(Note: some infrastructure tools use other architecture names for things;
these scripts will automatically translate from e.g. `armv7hl` to `armhfp`
where needed, so always use one of these for command-line arguments.)
### Installing depchase
```
$ git clone https://github.com/fedora-modularity/depchase.git
$ cd depchase
$ python3 setup.py install --user
```
This will install the `depchase` command as `~/.local/bin/depchase`
You may wish to add this to your default PATH variable in `~/.bashrc`
by doing:
```
$ echo "export PATH=$PATH:$HOME/.local/bin" >> ~/.bashrc
```
## Preparing the local repository data
Depchase (a tool we use for processing the data) requires local copies
of the repository metadata against which to operate. This repository
provides some convenient tools for retrieving this data.
### Syncing the existing repositories (the easy way)
Invoke `make -B repo/devel` to download or update your local copy of
the development tree repodata. This operation may take several minutes.
### Syncing the existing repositories (the manual way)
Alternatively you may download the repodata tree manually.
Use the `download_repo.sh` tool located in the root of this repository
to retrieve the repository metadata and override repositories.
For example:
```
./download_repo.sh --release=devel \
--milestone=Beta \
--archful-srpm-file=./archful-srpms.txt \
--arch=aarch64 \
--arch=armv7hl \
--arch=i686 \
--arch=ppc64 \
--arch=ppc64le \
--arch=x86_64 \
--overrides
```
This will download the frozen Fedora 26 Beta repository metadata, the
overrides repository for that milestone and will automatically regenerate
those SRPMs that are known to have arch-specific `BuildRequires` and
merge them locally to the repo metadata. The packages will be stored in
the `repo/test/26_beta/` path relative to the root of this repository.
Use `--release=devel` for a snapshot of rawhide suitable for development
or `--release=rawhide` for the latest Rawhide compose. Note Fedora 27
and newer also support s390x. Not all `--release` arguments support
all other options.
This process may take a long time, as it will transfer approximately
1GiB of data per architecture to the local system. This tool uses rsync,
so subsequent calls to it will only download updated information (plus
regenerating the archful SRPMs).
### Adding content to the override repositories
Copy any binary (or noarch) RPMs for inclusion into
`repo//override//os`. For source RPMs, copy them to the
`repo//override//sources` path. Run
`./repo/generate-repo-data.sh []` to automatically update
the repo metadata and `./repo/rsync-push-repo.sh []` to rsync the
contents to the fedorapeople repository.
There is a helper utility in the root of the git repository called
`dl_pkgs.sh`. This utility must be called with one or two arguments.
First, a path to a file containing one package NVR or Koji build-id
per line. If it is an NVR, the package must be an official build in
Koji. For a build-id, a scratch-build may work (but is untested). The
second argument should be `` . This tool should be used to
prep the override repository whenever a content change is made to the
module metadata, to keep the lists up-to-date.
Once completed, run `./repo/generate-repo-data.sh []` to
regenerate all the repository information. You can then run whatever
tests you want. When you are happy with the results, run
`./repo/rsync-push-repo.sh []` to update the fedorapeople repository.
## Dealing with SRPMs with arch-specific BuildRequires
Some packages have differing BuildRequires depending on which platform
is being built. In these cases, in order to have the list generation
find the right values, we need to pull down those SRPMs and regenerate
them for every architecture that we process.
There is a helper utility in the root of the git repository called
`mock_wrapper.sh` which takes a single argument: a list of NVRs of
official Koji builds that should be regenerated. It will construct a mock
buildroot with limited packages (to reduce the risk that the output could
be different depending on the packages available in the host system),
check out each of the requested SRPM packages and regenerate them on
each architecture. It will then drop them into a directory `./output`
(clobbering any existing content!) relative to the working directory
that was executed.
Once that content is present, you can run `./populate_srpm_repo.sh
/override` from the root of the git repository, which will move
all of the `./output` contents into the `repo/` hierarchy, after which you
can follow the "Pushing to the override repositories" instructions above.
This functionality is automatically run whenever the `download_repo.sh`
script is invoked with the `--with-archful-srpm-file` argument.
## How to generate package lists (the easy way)
To generate the relevant package lists for the development snapshot tree,
simply invoke `make`. This will run depchase to resolve runtime and build
time dependencies for components listed in the toplevel package lists
and will also generate complete modulemd files for all supported modules.
## How to generate package lists (the manual way)
Ensure that the contents of `toplevel-binary-packages.txt` and `hints.txt`
in each of the `data/Fedora/$VERSION[_$MILESTONE]/$MODULE/$ARCH`
directories is correct. These files must be present for each architecture
to be processed.
`toplevel-binary-packages.txt` contains the set of package names that
the base runtime must contain in order to provide the requisite APIs.
`hints.txt` is a manually-curated set of packages that is used to resolve
ambiguities in dependencies. For example, the `glibc` package `Requires:
glibc-langpack` and many packages may `Provides: glibc-langpack`,
so we include `glibc-minimal-langpack` in the hints.txt to resolve
this decision.
Then call `generate_module_lists.sh` like so:
```
./generate_module_lists.sh --version 26 \
--milestone=Beta \
--module=base-runtime \
--arch=aarch64 \
--arch=armv7hl \
--arch=i686 \
--arch=ppc64 \
--arch=ppc64le \
--arch=x86_64
```
This will produce a set of files in the
`data/Fedora/$VERSION[_$MILESTONE]/$MODULE/$ARCH` directories named:
* runtime-binary-packages-full.txt
* runtime-binary-packages-short.txt
* runtime-source-packages-full.txt
* runtime-source-packages-short.txt
* selfhosting-binary-packages-full.txt
* selfhosting-binary-packages-short.txt
* selfhosting-source-packages-full.txt
* selfhosting-source-packages-short.txt
To generate modulemd files, use `make_modulemd.pl` and point it to the
directory with package lists and modulemd templates.
To generate Host & Platform: `./make_modulemd.pl ./data/Fedora/devel/hp`
To generate Bootstrap: `./make_modulemd.pl ./data/Fedora/devel/bootstrap`
This will produce one or more modulemd files in the respective module
directories. These files should be dist-git-ready, with no further
manual editing required.
## Making sure everything is okay
Run `make test` and resolve any issues before pushing the repodata or
any changes to this repository.
The simple test suite, under `./tests`, verifies that the data looks
sane -- such as that there are no empty package lists or that no package
appears in more than one version.
## And that's it!
Congratulations, you've made it to the end!
A few final words, however. If you're hacking on Host & Platform,
please, file a pull request for `fedora-modularity/hp` so that your
content change be properly tracked and the toplevel lists & module data
files regenerated.
Host & Platform modules dist-git repositories include handy Makefiles
that allow for easy modulemd updates. Just run `make update` after you
push your changes to the repo and your modulemd files will be synced.