https://github.com/makukha/pydantic-file-secrets
Use file secrets in nested pydantic-settings models instead of built-in SecretsSettingsSource
https://github.com/makukha/pydantic-file-secrets
config configuration docker docker-secret file-secrets pydantic pydantic-settings pydantic-v2 python python3 secrets settings validation
Last synced: 3 months ago
JSON representation
Use file secrets in nested pydantic-settings models instead of built-in SecretsSettingsSource
- Host: GitHub
- URL: https://github.com/makukha/pydantic-file-secrets
- Owner: makukha
- License: mit
- Created: 2024-08-08T13:00:52.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-03-07T22:15:35.000Z (4 months ago)
- Last Synced: 2025-03-23T18:54:19.507Z (4 months ago)
- Topics: config, configuration, docker, docker-secret, file-secrets, pydantic, pydantic-settings, pydantic-v2, python, python3, secrets, settings, validation
- Language: Python
- Homepage:
- Size: 179 KB
- Stars: 8
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: .github/CONTRIBUTING.md
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Security: .github/SECURITY.md
Awesome Lists containing this project
README
# pydantic-file-secrets 🔑
[](https://pydantic.dev)
> Use file secrets in nested pydantic-settings models instead of built-in SecretsSettingsSource
[](https://github.com/makukha/pydantic-file-secrets/blob/main/LICENSE)
[](https://pypi.org/project/pydantic-file-secrets)
[](https://pypi.org/project/pydantic-file-secrets)
[](https://github.com/makukha/pydantic-file-secrets)
[](https://github.com/makukha/pydantic-file-secrets)
[](https://github.com/makukha/multipython)
[](https://github.com/makukha/docsub)
[](http://mypy.readthedocs.io)
[](https://github.com/astral-sh/ruff)
[](https://github.com/astral-sh/ruff)
[](https://www.bestpractices.dev/projects/10108)
[](https://pypistats.org/packages/pydantic-file-secrets)
This project is inspired by discussions in Pydantic Settings repository and proposes solution to [#30](https://github.com/pydantic/pydantic-settings/issues/30) and [#154](https://github.com/pydantic/pydantic-settings/issues/154).
# Features
* Plain or nested directory layout: `secrets/dir__key` or `secrets/dir/key`
* Respects `env_prefix`, `env_nested_delimiter` and other [config options](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options)
* Implements config options `secrets_prefix`, `secrets_nested_delimiter` [and more](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#configuration-options) to configure secrets and env vars independently
* Drop-in replacement of standard `SecretsSettingsSource`
* Pure Python thin wrapper over standard `EnvSettingsSource`
* No third party dependencies except `pydantic-settings`
* Fully typed
* 100% test coverage
# Installation
```shell
$ pip install pydantic-file-secrets
```
# Motivation
Nested Pydantic config can contain nested models with secret entries, as well as secrets in top level config. In dockerized environment, these entries may be read from file system, e.g. `/run/secrets` when using Docker Secrets:
```python
from pydantic import BaseModel, Secret
from pydantic_settings import BaseSettings, SettingsConfigDict
class DbSettings(BaseModel):
user: str
passwd: Secret[str] # secret in nested model
class Settings(BaseSettings):
app_key: Secret[str] # secret in root model
db: DbSettings
model_config = SettingsConfigDict(
secrets_dir='/run/secrets',
)
```
# Usage
## Plain secrets directory layout
```text
📂 secrets
├── 📄 app_key
└── 📄 db__passwd
```
```python
from pydantic import BaseModel, SecretStr
from pydantic_file_secrets import FileSecretsSettingsSource, SettingsConfigDict
from pydantic_settings import BaseSettings
from pydantic_settings.sources import PydanticBaseSettingsSource
class DbSettings(BaseModel):
passwd: SecretStr
class Settings(BaseSettings):
app_key: SecretStr
db: DbSettings
model_config = SettingsConfigDict(
secrets_dir='secrets',
secrets_nested_delimiter='__',
)
@classmethod
def settings_customise_sources(
cls,
settings_cls: type[BaseSettings],
init_settings: PydanticBaseSettingsSource,
env_settings: PydanticBaseSettingsSource,
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]:
return (
init_settings,
env_settings,
dotenv_settings,
FileSecretsSettingsSource(file_secret_settings),
)
```
```pycon
>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}
```
## Nested secrets directory layout
Config option `secrets_nested_delimiter` overrides `env_nested_delimiter` for files. In particular, this allows to use nested directory layout along with environmemt variables for other non-secret settings:
```text
📂 secrets
├── 📄 app_key
└── 📂 db
└── 📄 passwd
```
```python
model_config = SettingsConfigDict(
secrets_dir='secrets',
secrets_nested_subdir=True,
)
```
```pycon
>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}
```
## Multiple `secrets_dir`
```text
📂 secrets
├── 📂 layer1
│ └── 📄 app_key
└── 📂 layer2
└── 📄 db__passwd
```
```python
model_config = SettingsConfigDict(
secrets_dir=['secrets/layer1', 'secrets/layer2'],
secrets_nested_delimiter='__',
)
```
```pycon
>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}
```
## Experimental syntactic sugar 🧪
> [!CAUTION]
> This syntax may change at any time. Pin current `pydantic-file-secrets` version if decided to use it.
Few important things to note:
- `@with_builtin_sources` decorator enables `NamedTuple` argument `src: BuiltinSources` encapsulating default builtins settings sources
- `BaseSource` alias is shorter than `PydanticBaseSettingsSource` and is easier to use in type hints
- `settings_cls` was removed from `settings_customise_sources` signature: `cls` seems to be sufficient
```python
from pydantic import BaseModel, SecretStr
from pydantic_file_secrets import (
BaseSource,
BuiltinSources,
FileSecretsSettingsSource,
SettingsConfigDict,
with_builtin_sources,
)
from pydantic_settings import BaseSettings
class DbSettings(BaseModel):
passwd: SecretStr
class Settings(BaseSettings):
app_key: SecretStr
db: DbSettings
model_config = SettingsConfigDict(
secrets_dir='secrets',
secrets_nested_delimiter='__',
)
@classmethod
@with_builtin_sources
def settings_customise_sources(cls, src: BuiltinSources) -> tuple[BaseSource, ...]:
return (
src.init_settings,
src.env_settings,
src.dotenv_settings,
FileSecretsSettingsSource(src.file_secret_settings),
)
```
```pycon
>>> Settings().model_dump()
{'app_key': SecretStr('**********'), 'db': {'passwd': SecretStr('**********')}}
```
# Configuration options
## secrets_dir
Path to secrets directory. Same as `SecretsSettingsSource.secrets_dir` if `str` or `Path`. If `list`, the last match wins. If `secrets_dir` is passed in both source constructor and model config, values are not merged (constructor takes priority).
## secrets_dir_missing
If `secrets_dir` does not exist, original `SecretsSettingsSource` issues a warning. However, this may be undesirable, for example if we don't mount Docker Secrets in e.g. dev environment. Now you have a choice:
* `'ok'` — do nothing if `secrets_dir` does not exist
* `'warn'` (default) — print warning, same as `SecretsSettingsSource`
* `'error'` — raise `SettingsError`
If multiple `secrets_dir` passed, the same `secrets_dir_missing` action applies to each of them.
## secrets_dir_max_size
Limit the size of `secrets_dir` for security reasons, defaults to `SECRETS_DIR_MAX_SIZE` equal to 16 MiB.
`FileSecretsSettingsSource` is a thin wrapper around [`EnvSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.EnvSettingsSource), which loads all potential secrets on initialization. This could lead to `MemoryError` if we mount a large file under `secrets_dir`.
If multiple `secrets_dir` passed, the limit applies to each directory independently.
## secrets_case_sensitive
Same as [`case_sensitive`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#case-sensitivity), but works for secrets only. If not specified, defaults to `case_sensitive`.
## secrets_nested_delimiter
Same as [`env_nested_delimiter`](https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values), but works for secrets only. If not specified, defaults to `env_nested_delimiter`. This option is used to implement [nested secrets directory layout](https://github.com/makukha/pydantic-file-secrets?tab=readme-ov-file#nested-secrets-directory-layout) and allows to do even nastier things like `/run/secrets/model/delim/nested1/delim/nested2`.
## secrets_nested_subdir
Boolean flag to turn on nested secrets directory mode, `False` by default. If `True`, sets `secrets_nested_delimiter` to [`os.sep`](https://docs.python.org/3/library/os.html#os.sep). Raises `SettingsError` if `secrets_nested_delimiter` is already specified.
## secrets_prefix
Secret path prefix, similar to `env_prefix`, but works for secrets only. Defaults to `env_prefix` if not specified. Works in both plain and nested directory modes, like `'/run/secrets/prefix_model__nested'` and `'/run/secrets/prefix_model/nested'`.
## Not supported config options
Some config options that are declared in [`SecretsSettingsSource`](https://docs.pydantic.dev/latest/api/pydantic_settings/#pydantic_settings.SecretsSettingsSource) interface are actually [not working](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_original_source.py) and are not supported in `FileSecretsSettingsSource`:
* `env_ignore_empty`
* `env_parse_none_str`
* `env_parse_enums`
However, we [make sure](https://github.com/makukha/pydantic-file-secrets/blob/main/tests/test_ignored_options.py) that the behaviour of `FileSecretsSettingsSource` matches `SecretsSettingsSource` to provide a drop-in replacement, although it is somewhat wierd (e.g. `env_parse_enums` is always `True`).
# Testing
100% test coverage is provided for latest Python and pydantic-settings version. Tests are run for all minor pydantic-settings v2 versions and all minor Python 3 versions supported by them:
* `pyXY` — Python 3.{8,9,10,11,12,13}
* `psXY` — pydantic-settings v2.{2,3,4,5,6,7,8}
| | ps28 | ps27 | ps26 | ps25 | ps24 | ps23 | ps22 |
|-------|--------|--------|--------|--------|--------|--------|--------|
| py313 | ✳️ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
| py312 | ✅ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
| py311 | ✅ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
| py310 | ✅ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
| py39 | ✅ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
| py38 | ✅ | ✅ | ✅ | ✅ | ☑️ | ☑️ | ☑️ |
- ✳️ pytest and mypy passing, coverage report generated
- ✅ pytest and mypy passing
- ☑️ pytest passing, mypy not attempted
- ❌ tests failing or not attempted
# History
* September 2024 — [Multiple secrets_dir](https://github.com/pydantic/pydantic-settings/pull/372) feature was merged to [pydantic-settings v2.5.0](https://github.com/pydantic/pydantic-settings/releases/tag/v2.5.0)
# Contributing
Pull requests, feature requests, and bug reports are welcome!
* [Contribution guidelines](https://github.com/makukha/pydantic-file-secrets/blob/main/.github/CONTRIBUTING.md)
# Authors
* Michael Makukha
# See also
* [Documentation](https://github.com/makukha/pydantic-file-secrets#readme)
* [Issues](https://github.com/makukha/pydantic-file-secrets/issues)
* [Changelog](https://github.com/makukha/pydantic-file-secrets/blob/main/CHANGELOG.md)
* [Security Policy](https://github.com/makukha/pydantic-file-secrets/blob/main/.github/SECURITY.md)
* [Contribution Guidelines](https://github.com/makukha/pydantic-file-secrets/blob/main/.github/CONTRIBUTING.md)
* [Code of Conduct](https://github.com/makukha/pydantic-file-secrets/blob/main/.github/CODE_OF_CONDUCT.md)
* [License](https://github.com/makukha/pydantic-file-secrets/blob/main/LICENSE)