Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/pilosus/piny

YAML config loader with environment variables interpolation for Python and command line
https://github.com/pilosus/piny

configuration docker-compose environment-variables kubernetes kubernetes-secrets python yaml

Last synced: 8 days ago
JSON representation

YAML config loader with environment variables interpolation for Python and command line

Awesome Lists containing this project

README 

        
Piny

====



|Logo|



|PyPI| |Coverage| |Downloads| |License|



**Piny** is YAML config loader with environment variables interpolation for Python.



Keep your app's configuration in a YAML file.

Mark up sensitive data in the config as *environment variables*.

Set environment variables on application deployment.

Now let the *piny* load your config and substitute environment variables

in it with their values.



Piny is developed with Docker and Kubernetes in mind,

though it's not limited to any deployment system.



Rationale

---------



Piny combines *readability and versioning* you get when using config files,

and *security* that environment variables provide. Read more about this approach

in the `blog post`_.



Help

----



See `documentation`_ for more details.



Installation

------------



Just run::



  pip install -U piny



Usage

-----



Set your environment variables, add them to your YAML configuration file:



.. code-block:: yaml



    db:

      login: user

      password: ${DB_PASSWORD}

    mail:

      login: user

      password: ${MAIL_PASSWORD:-my_default_password}

    sentry:

      dsn: ${VAR_NOT_SET}



Then load your config:



.. code-block:: python



    from piny import YamlLoader



    config = YamlLoader(path="config.yaml").load()

    print(config)

    # {'db': {'login': 'user', 'password': 'my_db_password'},

    # 'mail': {'login': 'user', 'password': 'my_default_password'},

    # 'sentry': {'dsn': None}}



You may want to discourage Bash-style envs with defaults in your configs.

In such case, use a ``StrictMatcher``:



.. code-block:: python



    from piny import YamlLoader, StrictMatcher



    config = YamlLoader(path="config.yaml", matcher=StrictMatcher).load()



Both strict and default matchers produce ``None`` value if environment variable

matched is not set in the system (and no default syntax used in the case of

default matcher).



Piny also comes with *command line utility* that works both with files and standard

input and output:



.. code-block:: bash



  $ export PASSWORD=mySecretPassword

  $ echo "db: \${PASSWORD}" | piny

  db: mySecretPassword



Validation

----------



Piny supports *optional* data validation using third-party libraries:

`Marshmallow`_, `Pydantic`_, `Trafaret`_.



.. code-block:: python



  import marshmallow as ma

  from piny import MarshmallowValidator, StrictMatcher, YamlLoader



  class DBSchema(ma.Schema):

      login = ma.fields.String(required=True)

      password = ma.fields.String()



  class ConfigSchema(ma.Schema):

      db = ma.fields.Nested(DBSchema)



  config = YamlLoader(

      path="database.yaml",

      matcher=StrictMatcher,

      validator=MarshmallowValidator,

      schema=ConfigSchema,

  ).load(many=False)



Exceptions

----------



``LoadingError`` is thrown when something goes wrong with reading or parsing YAML-file.

``ValidationError`` is a wrapper for exceptions raised by the libraries for optional data validation.

Original exception can be accessed by ``origin`` attribute. It comes in handy when you need more than

just an original exception message (e.g. a dictionary of validation errors).



Both exceptions inherit from the ``ConfigError``.



Best practices

--------------



- Maintain a healthy security/convenience balance for your config



- Mark up entity as an environment variable in your YAML if and only if

  it really is a *secret* (login/passwords, private API keys, crypto keys,

  certificates, or maybe DB hostname too? You decide)



- When loading config file, validate your data.

  Piny supports a few popular data validation tools.



- Store your config files in the version control system along with your app’s code.



- Environment variables are set by whoever is responsible for the deployment.

  Modern orchestration systems like `Kubernetes`_ make it easier to keep envs secure

  (see `Kubernetes Secrets`_).



Fun facts

---------



*Piny* is a recursive acronym for *Piny Is Not YAML*.

Not only it's a library name, but also a name for YAML marked up

with environment variables.



Changelog

---------



See `CHANGELOG.rst`_.



Contributing

------------



See `CONTRIBUTING.rst`_.



.. |PyPI| image:: https://img.shields.io/pypi/v/piny

   :alt: PyPI

   :target: https://pypi.org/project/piny/

.. |Coverage| image:: https://img.shields.io/codecov/c/github/pilosus/piny.svg

   :alt: Codecov

   :target: https://codecov.io/gh/pilosus/piny

.. |License| image:: https://img.shields.io/github/license/pilosus/piny.svg

   :alt: MIT License

   :target: https://github.com/pilosus/piny/blob/master/LICENSE

.. |Logo| image:: https://piny.readthedocs.io/en/latest/_static/piny_logo_noborder.png

   :alt: Piny logo

   :target: https://pypi.org/project/piny/

.. |Downloads| image:: https://img.shields.io/pypi/dm/piny

   :alt: PyPI - Downloads

   :target: https://pypistats.org/packages/piny



.. _blog post: https://blog.pilosus.org/blog/application-configs-files-or-environment-variables-actually-both

.. _future releases: https://github.com/pilosus/piny/issues/2

.. _Kubernetes: https://kubernetes.io/

.. _Kubernetes Secrets: https://kubernetes.io/docs/concepts/configuration/secret/

.. _Pydantic: https://pydantic-docs.helpmanual.io/

.. _Marshmallow: https://marshmallow.readthedocs.io/

.. _Trafaret: https://trafaret.readthedocs.io/

.. _tests: https://github.com/pilosus/piny/tree/master/tests

.. _source code: https://github.com/pilosus/piny/tree/master/piny

.. _coming soon: https://github.com/pilosus/piny/issues/12

.. _CONTRIBUTING.rst: https://github.com/pilosus/piny/tree/master/CONTRIBUTING.rst

.. _CHANGELOG.rst: https://github.com/pilosus/piny/tree/master/CHANGELOG.rst

.. _documentation: https://piny.readthedocs.io/