Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/silvanmelchior/cme_parser

A tiny parser for more flexible conda environment files
https://github.com/silvanmelchior/cme_parser

cme-parser conda conda-environment data-science meta-environment parser python

Last synced: 23 days ago
JSON representation

A tiny parser for more flexible conda environment files

Awesome Lists containing this project

README 

        
# Conda Meta Environment Parser



This tiny parser takes as input a meta-environment file, transforms it into an environment file and invokes conda on it.

The input, usually called **environment.yml.meta**, is a standard conda environment file which can be enriched with conditions for certain lines/blocks, e.g. to only include them on specific platforms or if certain flags are given.



## Example



The following meta-file



```

name: demo_env

dependencies:

  - numpy=1.16.4

  - tensorflow=1.14.0

  - tensorflow-gpu=1.14.0 [use_gpu]

  - winreg-helpers [platform == win32]

```



can be used to set up a conda environment with



`python create_env.py` (A)



or



`python create_env.py --flag use_gpu` (B).



In (A), the conda package *tensorflow-gpu* is not installed. In both cases, *winreg-helpers* is only installed on windows platforms.



## Quickstart



To use CME Parser in your own project, create an **environment.yml.meta** file in your project folder, e.g. like this:



```

name: demo_env2

channels:

  - conda-forge

  - defaults

dependencies:

  - numpy=1.16.4

  - matplotlib=3.1.0 [allow_plot and plot_tool == matplotlib]

  - ggplot=0.11.5    [allow_plot and plot_tool == ggplot]

  - pip=19.1.1

  - some-linux-package=1.2.3 [platform startswith linux]

  - pip:

    - selenium=3.141.0

```



If **create_env.py** gets called without an `--input` argument, it will try to find **environment.yml.meta** either in the same directory or its parent.

Thus, either directly add **create_env.py** to your project folder next to the meta-env file or add this repo as git submodule:



```

$ cd your_project_folder

$ git rm environment.yml  # if exists

$ echo "environment.yml" >> .gitignore

$ git add environment.yml.meta

$ git submodule add https://github.com/silvanmelchior/cme_parser

$ git commit -m "cme parser added"

$ git push

```



If you now execute (in the case of using a submodule)



`python cme_parser/create_env.py --flag allow_plot --variable plot_tool matplotlib`



in your project folder, CME Parser will find the meta-env file and create **environment.yml**.

In general, if not specified differently with a `--output` argument, the parser will write the output into the same location as the input, removing *.meta* from the filename.

If *.meta* is not present, it will append *_out* to the filename instead.

In all cases, the parser will in the end invoke conda to create the environment specified by the output file. 



## Requirements



CME Parser is written in pure Python and does only need standard library packages.

It can be executed e.g. in the conda default environment (called *base*) and was tested with Python 2.7 and 3.6 / 3.7.



## Syntax



If a line contains `[condition]`, the parser only outputs it if the condition is met (removing the condition itself).

If a line contains `[[condition` and a later line `]]`, the parser outputs everything in between only if the condition is met (removing the two lines marking the block borders).

Nested blocks are allowed.



A condition can either be an arbitrary name, which is then interpreted as a flag, or a triplet of an arbitrary name, some operator and a string.

In the latter, the name is interpreted as variable and the operator can be one of `==`, `!=`, `startswith`, `endswith`, `contains`.

The string does not need any special terminators (e.g. no `""`).



Conditions can be combined by adding `and` and `or` in between them.

Furthermore, `not` can be added in front of a condition. The precedence order is `or` (highest), `and`, `not` (lowest), so e.g. `a and not b` is evaluated as `a and (not b)`.

Brackets to enforce another evaluation order are currently not supported.



Flags are evaluated as *false* per default, except given to the parser with `-f flag_name`.

Variables have to be given to the parser with `-v var_name var_value`, otherwise an error is risen.

An exception are variables of the form  `var_name%value`, which are interpreted as variable `var_name` with default value `value`.

Furthermore, the special variable `platform` is reserved and populated with the result of `sys.platform`.



## Command Line Options



| Short  | Long | Description |

| --- | --- | --- |

| -h  | --help | show usage help message |

| -i  | --input | specify meta environment input file |

| -o  | --output | specify environment output file |

| -f  | --flag | set custom flag for parsing |

| -v  | --variable | define custom variable for parsing |

| -q  | --quiet | quietly overwrite output if already exists |

| -p  | --parse-only | do not invoke conda afterwards |

| -c  | --no-comment | do not add auto-generated comment to output |