Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/conda-tools/conda-build-all

Last synced: 4 months ago
JSON representation

Host: GitHub
URL: https://github.com/conda-tools/conda-build-all
Owner: conda-tools
License: bsd-3-clause
Created: 2015-11-24T16:03:59.000Z (about 9 years ago)
Default Branch: master
Last Pushed: 2018-05-23T13:26:53.000Z (almost 7 years ago)
Last Synced: 2024-05-21T16:16:36.913Z (9 months ago)
Language: Python
Size: 165 KB
Stars: 30
Watchers: 11
Forks: 24
Open Issues: 10
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

conda build-all
===============

``conda build-all`` is a conda subcommand which allows multiple distributions to be built (and uploaded) in a single command.
It makes use of the underlying machinery developed for ``conda build``, but has a number of advantages:

* Automatically computes a build matrix for a single package, and builds all possible combinations.
* Can be given a directory of recipes, each of which will be identified and built (with each having their own build matrix).
* Will resolve the build order based on the dependency graph from both build and run time dependencies.
* Ability to re-build everything, or only build distributions that don't already exist in a conda channel and/or folder.
* Since the build matrix is computed, ``conda build-all`` avoids the need for special environment variables which control the build.
* Provides a Python API for building programmatically.

Installation
============

The easiest way of installing ``conda-build-all`` is with conda, and the ``conda-forge`` channel:

```
conda install conda-build-all --channel conda-forge
```

Building from source is trivial with the pre-requisite dependencies (see ``requirements.txt``).

Usage
======

```
usage: conda-build-all [-h] [--version]
[--inspect-channels [INSPECT_CHANNELS [INSPECT_CHANNELS ...]]]
[--inspect-directories [INSPECT_DIRECTORIES [INSPECT_DIRECTORIES ...]]]
[--no-inspect-conda-bld-directory]
[--artefact-directory ARTEFACT_DIRECTORY]
[--upload-channels [UPLOAD_CHANNELS [UPLOAD_CHANNELS ...]]]
[--matrix-conditions [MATRIX_CONDITIONS [MATRIX_CONDITIONS ...]]]
[--matrix-max-n-major-versions MATRIX_MAX_N_MAJOR_VERSIONS]
[--matrix-max-n-minor-versions MATRIX_MAX_N_MINOR_VERSIONS]
recipes

Build many conda distributions.

positional arguments:
recipes The folder containing conda recipes to build.

optional arguments:
-h, --help show this help message and exit
--version Show conda-build-all's version, and exit.
--inspect-channels [INSPECT_CHANNELS [INSPECT_CHANNELS ...]]
Skip a build if the equivalent disribution is already
available in the specified channel.
--inspect-directories [INSPECT_DIRECTORIES [INSPECT_DIRECTORIES ...]]
Skip a build if the equivalent disribution is already
available in the specified directory.
--no-inspect-conda-bld-directory
Do not add the conda-build directory to the inspection
list.
--artefact-directory ARTEFACT_DIRECTORY
A directory for any newly built distributions to be
placed.
--upload-channels [UPLOAD_CHANNELS [UPLOAD_CHANNELS ...]]
The channel(s) to upload built distributions to
(requires BINSTAR_TOKEN envioronment variable).
--matrix-conditions [MATRIX_CONDITIONS [MATRIX_CONDITIONS ...]]
Extra conditions for computing the build matrix (e.g.
'python 2.7.*'). When set, the defaults for matrix-
max-n-major-versions and matrix-max-n-minor-versions
are set to 0 (i.e. no limit on the max n versions).
--matrix-max-n-major-versions MATRIX_MAX_N_MAJOR_VERSIONS
When computing the build matrix, limit to the latest n
major versions (0 makes this unlimited). For example,
if Python 1, 2 and Python 3 are resolved by the recipe
and associated matrix conditions, only the latest N
major version will be used for the build matrix.
(default: 2 if no matrix conditions)
--matrix-max-n-minor-versions MATRIX_MAX_N_MINOR_VERSIONS
When computing the build matrix, limit to the latest n
minor versions (0 makes this unlimited). Note that
this does not limit the number of major versions (see
also matrix-max-n-major-version). For example, if
Python 2 and Python 3 are resolved by the recipe and
associated matrix conditions, a total of Nx2 builds
will be identified. (default: 2 if no matrix
conditions)
```

Example
=======

Supposing we have two moderately complex conda recipes in a directory:

```
$ mkdir -p my_recipes/recipe_a my_recipes/recipe_b
$ cat < my_recipes/recipe_a/meta.yaml
package:
name: recipe_a
version: 2.4

requirements:
build:
- python
run:
- python

EOF

$ cat < my_recipes/recipe_b/meta.yaml
package:
name: recipe_b
version: 3.2

requirements:
build:
- recipe_a
- numpy x.x
run:
- recipe_a
- python
- numpy x.x

EOF
```

If we wish to build the lot, we can simply run:

```
$ conda-build-all my_recipes

conda-build-all my_recipes
Fetching package metadata: ........
Resolving distributions from 2 recipes...
Computed that there are 11 distributions from the 2 recipes:
Resolved dependencies, will be built in the following order:
recipe_a-2.4-py26_0 (will be built: True)
recipe_a-2.4-py27_0 (will be built: True)
recipe_a-2.4-py34_0 (will be built: True)
recipe_a-2.4-py35_0 (will be built: True)
recipe_b-3.2-np19py26_0 (will be built: True)
recipe_b-3.2-np110py27_0 (will be built: True)
recipe_b-3.2-np19py27_0 (will be built: True)
recipe_b-3.2-np110py34_0 (will be built: True)
recipe_b-3.2-np19py34_0 (will be built: True)
recipe_b-3.2-np110py35_0 (will be built: True)
recipe_b-3.2-np19py35_0 (will be built: True)

BUILD START: recipe_a-2.4-py26_0
...

```

As you can see, these two unassuming recipes will result in more than 2 builds.
In this case, ``recipe_a`` has been identified to be built against the top two minor versions of the top two major versions of Python - that is, py26, py27, py34, py35 (at the time of writing).
Next, ``recipe_b`` has been identified to be built against the top two minor versions of the top two major versions of Python *and* numpy.
If all built distributions of python and numpy were available, there would be ``4 x 2`` permutations (4 being the number of Python versions available, and 2 being the number of numpy versions, assuming there exists only 1 major version of numpy, otherwise this would double to 4).

We've seen that we can build a *lot* of distributions for our simple recipes. We can tighten the build matrix somewhat by adding or own conditions:

```
$ conda-build-all my_recipes --matrix-condition "python 3.5.*" "numpy >=1.8"
Fetching package metadata: ........
Resolving distributions from 2 recipes...
Computed that there are 3 distributions from the 2 recipes:
Resolved dependencies, will be built in the following order:
recipe_a-2.4-py35_0 (will be built: True)
recipe_b-3.2-np110py35_0 (will be built: True)
recipe_b-3.2-np19py35_0 (will be built: True)
...
```

Here we've used the language provided to us by conda to limit the build matrix to a smaller number of combinations. Alternatively we could use the max ``N`` major and minor arguments to limit the scope:

```
$ conda-build-all my_recipes --matrix-max-n-minor-versions=1 --matrix-max-n-major-versions=1
Fetching package metadata: ........
Resolving distributions from 2 recipes...
Computed that there are 2 distributions from the 2 recipes:
Resolved dependencies, will be built in the following order:
recipe_a-2.4-py35_0 (will be built: True)
recipe_b-3.2-np110py35_0 (will be built: True)
...

```