Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/fictive-kin/dynaconf-aws-loader

A Dynaconf loader for AWS Systems Manager Parameter Store that can query slash-delimited hierarchical configuration data.
https://github.com/fictive-kin/dynaconf-aws-loader

Last synced: 12 days ago
JSON representation

A Dynaconf loader for AWS Systems Manager Parameter Store that can query slash-delimited hierarchical configuration data.

Awesome Lists containing this project

README 

        
Dynaconf AWS Systems Manager Parameter Store Loader

====================================================



When configured, this loader will permit Dynaconf to query `AWS Systems Manager Parameter Store `_ for slash-delimited hierarchical configuration data.



Loader Configuration

--------------------



An example:



.. code-block:: python



    from dynaconf import Dynaconf



    settings = Dynaconf(

        environments=True,

        settings_file="settings.toml",

        LOADERS_FOR_DYNACONF=[

            "dynaconf_aws_loader.loader",

            "dynaconf.loaders.env_loader"

        ],

    )



Note that for the basic functioning of this loader, the `environments `_ option for ``Dynaconf`` must be set, and an environment must be used.



Configuration Variables

-----------------------



Both of the following configuration values should be set in the *environment* to avoid a chicken/egg scenario for initializing this custom loader:



- ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF``: Required.

  The ``project`` prefix in the parameter store path. For example, if the parameter hierarchy looks something like ``/baldur/development/database_uri``, then in this case ``SSM_PARAMETER_PROJECT_PREFIX_FOR_DYNACONF=baldur``.



- ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF``: Optional.

  This provides an additional level of grouping once the project and environment have been determined. For example, if the parameter hierarchy looks something like ``/baldur/pr-123/development/database_uri``, then ``SSM_PARAMETER_NAMESPACE_FOR_DYNACONF=pr-123``.



.. note::

   If a namespace is utilized, be aware that namespaced settings will be *merged* with non-namespaced settings. This merge is a naive one, where namespaced settings will completely overwrite non-namespaced settings with the same key.



The following optional variables should be set in your ``settings.toml`` (or equivalent format), if desired:



- ``SSM_ENDPOINT_URL_FOR_DYNACONF``: If your AWS SSM uses a different endpoint than the AWS default. This can be useful for local development when you are running something like `LocalStack `_.

- ``SSM_SESSION_FOR_DYNACONF``: If you require custom `boto3.session.Session `_ arguments, you can specify then as a dictionary here. Note that this will override the default ``boto3`` credential configuration.

- ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF``: Boolean, defaults to ``True``. If you want the SSM loader to load keys under the ``default`` environment name. The key name itself can be set via the Dynaconf setting of ``DEFAULT_ENV_FOR_DYNACONF`` if you want it to be something other than ``default``.



Parameter Store Details

~~~~~~~~~~~~~~~~~~~~~~~



The structure that this loader expects from the path-based organization in SSM is:



.. code-block::



    ///



An optional ``namespace`` can be specified as a sub-project grouping for parameters:



.. code-block::



    ////



Note that if you choose to use a ``namespace`` identifier, it must not conflict with existing ``environment`` identifiers.



If ``SSM_LOAD_DEFAULT_ENV_FOR_DYNACONF`` is set to ``True`` (which is the default value), the loader will add whatever the value of ``DEFAULT_ENV_FOR_DYNACONF`` as an ``environment`` key to load from SSM. The typical use case here is to have a default value for all environments that can be overriden on a per-environment basis as necessary.



Security Considerations

~~~~~~~~~~~~~~~~~~~~~~~



For this loader to function correctly and securely, the use of IAM policies to restrict which parameters can be read/mutated is highly encouraged.



Policies can be enacted on a glob-path basis, which will ensure that the only parameters that can be fetched/hydrated into the local object instance are the ones that the current environment is permitted to load.



The following policy for a fictional account allows a user to call the ``DescribeParameters`` and ``GetParameters`` API operations for parameters that begin with the path ``/testapp/production``:



.. code-block:: json



    {

        "Version": "2012-10-17",

        "Statement": [

            {

                "Effect": "Allow",

                "Action": [

                    "ssm:DescribeParameters"

                ],

                "Resource": "*"

            },

            {

                "Effect": "Allow",

                "Action": [

                    "ssm:GetParameters"

                ],

                "Resource": "arn:aws:ssm:us-east-1:000000000000:parameter/testapp/production*"

            }

        ]

    }



.. warning::



    If a user has access to a path, then the user can access all levels of that path. For example, if a user has permission to access path ``/testapp``, then the user can also access ``testapp/production``. Even if a user has explicitly been denied access in IAM for parameter ``/testapp/production``, they can still call the ``GetParametersByPath`` API operation recursively for ``/testapp`` and view ``/testapp/production``.



Setting Parameters via Boto3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Parameters may be set via the AWS Web Console UI, or one of their many client libraries. The `boto3 `_ library is perhaps the most well-known, and the process is relatively straightforward:



.. code-block:: python



    import boto3

    ssm_client = boto3.client("ssm")



    ssm_client.put_parameter(

        Name="/testapp/development/database/host",

        Value="localhost",

        Type="String",

    )



    ssm_client.put_parameter(

        Name="/testapp/production/database/password",

        Value="sekrit",

        Type="SecureString",

    )



    ssm_client.put_parameter(

        Name="/testapp/production/database/host",

        Value="db.example.com",

        Type="String",

    )



    ssm_client.put_parameter(

        Name="/testapp/production/admin_email",

        Value="[email protected]",

        Type="String",

    )



This creates a parameter hierarchy with the following structure:



.. code-block:: json



    {

        "testapp": {

            "development": {"database": {"host": "localhost"}},

            "production": {

                "database": {"host": "db.example.com", "password": "sekrit"},

                "admin_email": "[email protected]",

            },

        },

    }



Parameter Name Limitations

--------------------------



AWS SSM has the following key (and thus path) limitations:



- Parameter names are case sensitive

- A parameter name must be unique within an Amazon Web Services Region

- A parameter name can't be prefixed with "aws" or "ssm" (case-insensitive)

- Parameter names can include only the following symbols and letters: a-zA-Z0-9\_.-

- The slash character ( ``/`` ) is used to delineate hierarchies in parameter names

- A parameter name can't include spaces

- Parameter hierarchies are limited to a maximum depth of fifteen levels



Testing

~~~~~~~



0. Have Docker installed and running

1. Clone this repository

2. Ensure you have `poetry` available on your system

3. `poetry run pytest`



The test suite will spin up an ephemeral Docker container; it may take a few seconds for it to load. The relevant test fixtures will handle setting parameters and their values in the Localstack SSM service.



TODO

~~~~



- [ ] CI configuration for matrix-based python/dynaconf version testing

- [ ] Handle `Parameter Store references to AWS Secrets Manager `_

- [ ] Make ``tests/docker-compose.yml`` more configurable, e.g. ports, in case a different Localstack container is already running for the user