Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

https://github.com/anntzer/pypi2pkgbuild

A PyPI to PKGBUILD converter.
https://github.com/anntzer/pypi2pkgbuild

arch-packages python-packages

Last synced: 7 days ago
JSON representation

A PyPI to PKGBUILD converter.

Lists

README 

        
PyPI2PKGBUILD

=============



**NOTE**: This package is currently not under active development.



-----



|PyPI|



.. |PyPI|

   image:: https://img.shields.io/pypi/v/pypi2pkgbuild.svg

   :target: https://pypi.python.org/pypi/pypi2pkgbuild



Convert PyPI packages to Arch Linux packages, inspired from pip2arch_.



Handles packages of all sizes, from the simplest (pure Python, no dependencies)

to the most complex (C-level dependencies, external C libraries, etc., e.g.

most of the scientific Python stack, or GUI toolkits such as PyGObject and

wxPython) [#]_.



.. [#] ... with a bit of help.



.. contents:: :local:



Dependencies and installation

-----------------------------



``pypi2pkgbuild.py`` depends on the Arch Linux packages namcap_, pkgfile_, and

python_ [#]_.



.. _namcap: https://wiki.archlinux.org/index.php/Namcap

.. _pkgfile: https://wiki.archlinux.org/index.php/Pkgfile

.. _python: https://wiki.archlinux.org/index.php/Python



.. [#] Officially, only the latest releases packaged by Arch Linux are

   supported. In practice, the hard requirements that I am aware of are

   pacman≥5.1 (which changed the behavior of ``makepkg --printsrcinfo``) and a

   recent enough Python so that ``python -mvenv`` creates a virtual environment

   with pip≥10 (which changed the default format of ``pip --list``) and

   setuptools.



The script can be installed with ``pip install [--user] .``, or can also be run

directly.



One can even run ``pypi2pkgbuild.py`` on itself to create a proper Arch package

(``pypi2pkgbuild.py git+https://github.com/anntzer/pypi2pkgbuild``).



A minimal test suite (checking that ``pypi2pkgbuild.py`` can indeed package

itself) can by run with unittest (or pytest).



Usage

-----



``pypi2pkgbuild.py PYPINAME`` creates a PKGBUILD for the latest version of the

given PyPI package and the current version of the Python interpreter (Python 3

only).  Prereleases are considered if the ``--pre`` flag is passed.  Because

PyPI's dependency information is somewhat unreliable, it installs the package

in a venv to figure out the dependencies.  Note that thanks to ``pip``'s wheel

cache, the build is later reused; i.e. the procedure entails very little extra

work.



A ``-git`` package can be built with ``pypi2pkgbuild.py git+https://...``.



The package is then built and verified with ``namcap``.



The goal is to make this tool as automated as possible: if all the information

to build a package is (reasonably) accessible, this tool should be able to

build it.



In order to provide additional information to ``makepkg``, edit

``PKGBUILD_EXTRAS`` (which can also be done with the ``--pkgbuild-extras``

flag).  This file is sourced at the *end* of ``PKGBUILD``.  For ease of

patching, the ``build``, ``package``, and, where applicable, ``pkgver``

functions are defined by forwarding to ``_build``, ``_package``, and

``_pkgver``.  A ``_check`` function is also available, but not used (due to the

lack of standard testing CLI).  Some useful examples of ``PKGBUILD_EXTRAS`` are

listed in the ``pkgbuild-extras`` directory.



Usage notes

```````````



- It is suggested to create an alias with standard options set, e.g.



.. code-block:: sh



   alias pypi2pkgbuild.py='PKGEXT=.pkg.tar pypi2pkgbuild.py -g cython -b /tmp/pypi2pkgbuild/ -f'



- By default, the ``pkgrel`` of (standard) packages is set to ``00``.  This

  allows automatic upgrading into official packages (and AUR ones, if an AUR

  helper is used) whenever the repositories are updated.  Additionally, the use

  of ``00`` rather than ``0`` serves as a (weak) marker that the package was

  automatically generated by this tool.  In order to prevent such an upgrade,

  one can use the ``--pkgrel`` flag to set ``pkgrel`` to, e.g., ``99``.



- If one wishes to completely bypass AUR Python packages while maintaining the

  use of an AUR helper for non-Python packages, one can define a shell function

  that excludes ``pypi2pkgbuild.py``-generated packages that do not appear in

  the official repositories, e.g., for ``pacaur``:



  .. code-block:: sh



     pacaur() {

        if [[ "$1" = "-Syu" ]]; then

           # Update, in case some packages moved in or out of the official repos.

           sudo pacman -Sy

           # Upgrade everything except python packages with pkgver=00 or 99.

           PKGEXT=.pkg.tar command pacaur -Su --ignore \

                 "$(pacman -Qm | grep '^python-.*-\(00\|99\)$' | cut -d' ' -f1 | paste -sd,)"

        else

           command pacaur "$@"

        fi

     }



  This function will not bypass Python packages explicitly installed from the

  AUR, as the user may have done so to bypass some incorrect packaging by

  ``pypi2pkgbuild.py``.  It is recommended to use the ``-i`` flag to calls

  to ``pypi2pkgbuild.py`` (e.g. in an alias) to exclude packages that are

  mishandled by ``pypi2pkgbuild.py`` (see `mispackaged packages`_).  The ``-i``

  flag can be passed multiple times; passing an empty argument to it will clear

  the ignore list defined so far.



  .. _mispackaged packages: TODO.rst#mispackaged-packages



- In order to package a locally available git repository, use



  .. code-block:: sh



     $ pypi2pkgbuild.py git+file://$absolute_path_to_repo  # (e.g. file:///home/...)



  In order to package a locally available sdist or wheel, use



  .. code-block:: sh



     $ pypi2pkgbuild.py file://$absolute_path_to_file  # (e.g. file:///home/...)



  Note that in both cases *absolute* paths are necessary.



  Building packages from local repos or wheels needs to be done in topological

  order of the dependencies (so that ``pypi2pkgbuild.py`` can find that

  the dependencies are actually present), or by passing the ``-d`` flag

  ("do not build dependencies"); if it is used, the Arch package may

  not use the correct dependency names (if they are not of the form

  ``python-pep503-normalized-name``).



- By default, ``pypi2pkgbuild.py`` ignores ``pip`` config files such as

  ``~/.config/pip/pip.conf``.  An explicitly set ``PIP_CONFIG_FILE`` will be

  respected, but may cause ``pypi2pkgbuild.py`` to fail as some ``pip`` calls

  will be unexpectedly modified.



  Likewise, user-site packages are ignored unless ``PYTHONNOUSERSITE`` is

  explicitly set to an empty value.



Build-time dependencies of packages

-----------------------------------



``pypi2pkgbuild.py`` attempts to guess whether ``Cython`` and ``SWIG`` are

build-time dependencies by checking for the presence of ``.pyx`` and ``.i``

files, respectively.  If this is not desired, set the ``--guess-makedepends``

option accordingly.



``pypi2pkgbuild.py`` guesses whether ``numpy`` is a build-time dependency by

attempting a build without ``numpy``, then, in case of failure, a build with

``numpy``.



Additional Python build-time dependencies (i.e., ``setup_requires``) can be

specified (as PyPI names) using the ``--setup-requires`` flag, or just as

normal entries using ``--pkgbuild-extras`` (they will be installed into the

build virtualenv).



Additional non-Python build-time dependencies can be set as ``makedepends``

using ``--pkgbuild-extras``; they will be installed *before*

``pypi2pkgbuild.py`` attempts to build a wheel for the package.



Vendored packages

-----------------



Some Arch packages (e.g. ``ipython``) include a number of smaller PyPI

packages.



Because it is not possible to assign a meaningful version automatically,

``pypi2pkgbuild.py`` instead creates an independent Arch package for each of

the PyPI packages (with two dashes in the name, to prevent name conflicts) and

a master package that depends on all of them.  The ``pkgrel`` of the master

package is set to ``$official_pkgrel.99``, so that the package appears more

recent than the current official version but older than any future official

version.  All these packages ``conflict`` with all versions of the official

package (except the newly created package), so upgrading should work fine when

the official package is actually updated.



However, dependencies are still expressed using the master package (to avoid

breakage on upgrade into an official package), so internal dependencies will

appear be circular.



All the packages are placed in a subfolder named ``meta:$pkgname``, so one can

easily install everything by ``cd``'ing there and running



.. code-block:: sh



   $ sudo pacman -U --asdeps **/*.xz

   $ sudo pacman -D --asexplicit $pkgname/$pkgname.tar.xz



Handling Python upgrades

------------------------



When the Python minor version (``x`` in ``3.x``) is upgraded, it is necessary

to regenerate all self-built packages.  This can be done e.g. with



.. code-block:: sh



   $ pypi2pkgbuild.py $(

        ls /usr/lib/python3.$oldver/site-packages |

        grep -Po '.*(?=-.*.dist-info)'

     )



Comparison with other tools

---------------------------



Other similar tools include pip2arch_, pip2pkgbuild_, and fpm_.  To the best

of my knowledge, the features below are unique to PyPI2PKGBUILD; please let me

know if this is incorrect.



- Supports wheels (the default is to prefer ``any``-platform wheels, then

  ``sdist``\s, then ``manylinux1`` wheels, but this can be changed using

  ``--pkgtypes``).

- Resolves Python dependencies via installation in a temporary virtualenv, and

  also creates PKGBUILDs for those that are not available as official packages.

- Resolves binary dependencies via ``namcap`` and adds them to the ``depends``

  array if they are installed (thus, it is suggested to first install

  them as ``--asdeps`` and then let the generated PKGBUILD pick them up as

  dependencies).  Note that some packages are distributed with a copy of the

  required libraries; in this case, ``pypi2pkgbuild.py``’s behavior will depend

  on whether the package defaults to using the system-wide library or its own

  copy.

- Automatically tries to fetch a missing license file from Github, if

  applicable.

- Automatically builds the package (with options given in ``--makepkg=...``)

  and run ``namcap``.

- Automatically builds all outdated dependencies via ``-u``.



.. _pip2arch: https://github.com/bluepeppers/pip2arch

.. _pip2pkgbuild: https://github.com/wenLiangcan/pip2pkgbuild

.. _fpm: https://github.com/jordansissel/fpm