{"id":23311918,"url":"https://github.com/attumm/maat","last_synced_at":"2025-04-09T13:07:42.160Z","repository":{"id":44688149,"uuid":"142082348","full_name":"Attumm/Maat","owner":"Attumm","description":"Validation and transformation library powered by recursive descent validation algorithm. Made to be extended for any kind of project.","archived":false,"fork":false,"pushed_at":"2025-02-15T10:30:55.000Z","size":2148,"stargazers_count":30,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-09T13:07:32.374Z","etag":null,"topics":["dictionary","fast","json-schema","nested-structures","parser","security","serialization","types","validation","validator"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Attumm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-24T00:08:59.000Z","updated_at":"2025-03-09T12:25:38.000Z","dependencies_parsed_at":"2024-11-17T17:35:18.186Z","dependency_job_id":"90c0f550-9fa4-4fbb-bec8-ba55bd140648","html_url":"https://github.com/Attumm/Maat","commit_stats":{"total_commits":96,"total_committers":7,"mean_commits":"13.714285714285714","dds":"0.26041666666666663","last_synced_commit":"350db63c10b12516a8fc121ab763ff949cb759c4"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Attumm%2FMaat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Attumm%2FMaat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Attumm%2FMaat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Attumm%2FMaat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Attumm","download_url":"https://codeload.github.com/Attumm/Maat/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045232,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dictionary","fast","json-schema","nested-structures","parser","security","serialization","types","validation","validator"],"created_at":"2024-12-20T14:17:16.778Z","updated_at":"2025-04-09T13:07:42.135Z","avatar_url":"https://github.com/Attumm.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Maat\n[![PyPI](https://img.shields.io/pypi/v/Maat.svg)](https://pypi.org/project/Maat/)\n[![CI](https://github.com/Attumm/Maat/actions/workflows/ci.yml/badge.svg)](https://github.com/Attumm/Maat/actions/workflows/ci.yml)\n[![Downloads](https://static.pepy.tech/badge/maat/month)](https://pepy.tech/project/maat)\n[![codecov](https://codecov.io/gh/Attumm/Maat/graph/badge.svg?token=CORUIP41EU)](https://codecov.io/gh/Attumm/Maat)\n\n\n\n\nMaat is an extensible transformation and validation library for Python, designed with simplicity and readability in mind, capable of handling corner cases, nested data, and encrypted data with remarkable speed. The project is named after the ancient Egyptian god [Maat](https://en.wikipedia.org/wiki/Maat), who used a magical scale to weigh the heart as described in the Book of the Dead.\n\nSimilar to Maat's scale, this library offers dictionary-to-dictionary validation and transformation, as well as serialization by leveraging the validation schema. A new dictionary is created from two dictionaries, with each value in the dictionary to be validated passed through their respective validation and transformation functions. Maat accomplishes this without relying on external dependencies.\n\nMaat key features:\n\n* Simplicity and Intuitiveness: Maat's validation schema is straightforward and easy to understand, promoting maintainability and ease of use.\n* Encryption and Decryption: Maat supports validation on encrypted data, helping to maintain data privacy and adhere to data protection regulations.\n* Deep Nesting: The library proficiently manages nested data structures, ensuring efficient validation for complex data requirements.\n* Zero Dependencies: Maat relies on no external packages for its core functionality. The packages listed in test-packages.txt are only used for testing purposes.\n* Performance: Pydantic's benchmarks indicate Maat as the top-performing library among popular validation libraries.\n\nMaat's combination of capabilities, including encryption and decryption, nesting, and exceptional performance, makes it a fitting choice for projects requiring a versatile data validation and transformation solution.\n\nThe simplicity and readability of Maat's validation schema contribute to the stability of its implementations in projects. New developers can quickly understand and work with the validation schema, making it an ideal choice for projects that value maintainability and ease of collaboration.\n\n\n```python\ninput_dic = {\n    \"number\": 123,\n    \"street\": \"John Doe Street\",\n}\n\nvalidation = {\n    \"number\": {\"type\": \"int\", \"min_amount\": 1},\n    \"street\": {\"type\": \"str\", \"min_length\": 5, \"max_length\": 99},\n}\n```\n\n## Examples\nMaat is designed with intuitiveness and simplicity in mind, making it accessible for developers who are familiar with dictionaries. The structure of Maat's validation schema corresponds closely to the data being validated, contributing to the ease of understanding and reducing the learning curve. This example demonstrates the intuitive nature of Maat's validation schema:\n\nThis example validates that the name in the input dictionary is of type str and is either \"John Doe\" or \"Jane Doe\". Maat throws an Invalid exception when validation fails due to its fail-fast policy.\n\n```python\n    \u003e\u003e\u003e from maat import validate\n    \u003e\u003e\u003e user = {\"name\": \"John Doe\"}\n    \u003e\u003e\u003e user_validation = {\"name\": {\"type\": \"str\", \"choices\": [\"John Doe\", \"Jane Doe\"]}}\n    \u003e\u003e\u003e validate(user, user_validation)\n    {\"name\": \"John Doe\"}\n    \n    \u003e\u003e\u003e validate({\"name\": \"peter pan\"}, user_validation)\n    Traceback (most recent call last):\n    maat.validation.Invalid: key: \"name\" contains invalid item \"peter pan\": not in valid choices [\"John Doe\", \"Jane Doe\"]\n    \n    \u003e\u003e\u003e validate({\"name\": 42}, user_validation)\n    Traceback (most recent call last)\n    maat.validation.Invalid: key: \"name\" contains invalid item \"42\" with type \"int\": not of type string\n    \n    \u003e\u003e\u003e  validate({\"user\": \"John Doe\"}, user_validation)\n    Traceback (most recent call last)\n    maat.validation.Invalid: invalid keys: user :expected keys: name\n    \n    \u003e\u003e\u003e validate({\"name\": \"Jane Doe\"}, user_validation)\n    {\"name\": \"Jane Doe\"}\n\n    \u003e\u003e\u003e import maat\n    \u003e\u003e\u003e @maat.protected(user_validation)\n        def create_user(name):\n            return f\"created user {name}\"\n\n    \u003e\u003e\u003e create_user(\"peter pan\")\n    Traceback (most recent call last):\n    maat.maat.Invalid: key: \"name\" contains invalid item \"peter pan\": not in valid choices [\"John Doe\", \"Jane Doe\"]\n\n    \u003e\u003e\u003e create_user(\"John Doe\")\n    \"created user John Doe\"\n```\n\n## Starting Point Example\n\n```python\nvalidation = {\n    \"int   \": {\"type\": \"int\", \"cast\": True, \"min_amount\": 1, \"max_amount\": 150},\n    \"float \": {\"type\": \"float\", \"cast\": True, \"min_amount\": 1, \"max_amount\": 150},\n    \"list  \": {\"type\": \"list\", \"min_amount\": 1, \"max_amount\": 5, \"skip_failed\": True},\n    \"dict  \": {\"type\": \"dict\", \"min_amount\": 1, \"max_amount\": 2, \"key_regex\": r\"(\\w+)\"},\n    \"string\": {\"type\": \"str\", \"cast\": True, \"min_length\": 1,\n        \"max_length\": 12, \"regex\": r\"(\\w+ )(\\w+)\", \"choices\": [\"John Doe\", \"Jane Doe\"]\n    }\n}\n```\n\n#### Field Control\nEach field could be nullable, optional, default; they can be added to any field.\nFor lists it's possible to skip failed items with skip_failed.\n```python\n\u003e\u003e\u003e input_dic = {\"str   \": None}\n\u003e\u003e\u003e validation = {\n\t\"int   \": {\"type\": \"int\", \"min_amount\": 1, \"default\": 42},\n\t\"float \": {\"type\": \"float\", \"optional\": True},\n\t\"str   \": {\"type\": \"str\", \"nullable\": True},\n}\n\u003e\u003e\u003e validate(input_dic, validation)\n{\n    \"int   \": 42,\n    \"str   \": None\n}\n```\n#### Nesting\nNesting data structures and fields are treated as first-class citizens, allowing for seamless validation and ensuring code correctness at any level. Maat efficiently handles nesting, with minimal performance impact, and supports a vast number of nesting levels. The limit is set by Python's recursion depth, which defaults to 1k. To increase the maximum nesting depth, you can adjust Python's recursion limit via sys.setrecursionlimit().\nBelow is an example showcasing nesting. For example with very deep nesting [here](tests/test_corner_case.py)\n```python\n\u003e\u003e\u003e input_dic = {\n    \"foo\": {\n        \"foo_bar\": \"John Doe Street\",\n        \"foo_baz\": 123,\n    }\n}\n\u003e\u003e\u003e validation = {\n    \"foo\": {\n        \"type\": \"dict\",\n        \"key_regex\": r\"\\w+\",\n        \"nested\": {\n            \"foo_bar\": {\"type\": \"str\", \"min_length\": 5, \"max_length\": 99},\n            \"foo_baz\": {\"type\": \"int\", \"min_amount\": 1},\n        }\n    }\n}\n```\n\n#### Nesting of Dicts\n```python\n\u003e\u003e\u003e input = {\n    'foobar': [\n        {'name': 'John Doe', 'points': 22},\n        {'name': 'Jane Doe', 'points': 23},\n        {'name': 'willo wanka', 'points': 42},\n    ]\n}\n\u003e\u003e\u003e validation = {\n    'foobar': {\n        'type': 'list_dicts',\n        'nested': {\n            'name': {'type': 'str'},\n            'points': {'type': 'int'},\n        }\n    }\n}\n```\n\n\n### Extending Maat with custom validation\nMaat's flexibility facilitates the creation of custom validation functions tailored to specific needs. The library can be extended with new data types and validation rules according to the project requirements. In the following example, a custom validation function is implemented and integrated for datetime strings in Maat.\nAdditionally, creating specific types for business logic, such as \"valid_address,\" is also possible. For a relevant example, refer to [here](tests/tests.py#L714).\n```python\n\u003e\u003e\u003e from maat import types\n\n\n\u003e\u003e\u003e def datetime_parse(val, key, formats=\"%Y-%m-%dT%H:%M:%S.%f\", *args, **kwargs):\n    \"\"\"Parse datetime string 'val' in ISO format and return a datetime object, raise Invalid exception on error.\"\"\"\n    try:\n        return datetime.strptime(val, formats)\n    except Exception as e:\n        raise Invalid(f'key: \"{key}\" contains invalid item')\n\n\u003e\u003e\u003e types['custom_datetime'] = datetime_parse\n\n\u003e\u003e\u003e input = {\n    \"created\": \"2022-01-28T15:01:46.0000\",\n}\n\n\n\u003e\u003e\u003e validation = {\n    \"created\": {\n        \"type\": \"custom_datetime\",\n    }   \n}\n\n\u003e\u003e\u003e validate(input, validation)\n{'created': datetime.datetime(2022, 1, 28, 15, 1, 46)}\n\n```\n\n\n### Secure Validation with Encryption and Decryption\n\nMaat enables secure validation on encrypted data, ensuring both data privacy and flexibility in the validation process. This powerful feature allows you to maintain encrypted data throughout the validation process by decrypting it only for validation purposes and re-encrypting it afterward.\n##### Integration and Usage\n\nThe code snippet below illustrates the integration of encryption and decryption functions into Maat, using custom functions for encoding and decoding. These custom functions utilize AES encryption. For more information, refer to here.\nFollowing the integration, a series of tests demonstrate the usage of Maat to validate and transform incoming data, as well as securely validate pre-encrypted data.\n\n\n```python\ndef encode(msg_text):\n    \"\"\"\n    Encrypts a message using AES ECB mode with right-justified padding and encodes the result in base64.\n\n    Args:\n        msg_text (str): The plaintext message to be encrypted.\n\n    Returns:\n        str: The base64-encoded encrypted message.\n    \"\"\"\n    secret_key = os.environ.get('secret', 'veryMuchSecret').encode('utf-8')\n    msg_text_bytes = msg_text.rjust(32).encode('utf-8')\n    cipher = AES.new(secret_key, AES.MODE_ECB)\n    encrypted_bytes = cipher.encrypt(msg_text_bytes)\n\n    return base64.b64encode(encrypted_bytes).decode('utf-8')\n\n\ndef decode(encoded):\n    \"\"\"\n    Decrypts a base64-encoded message encrypted using AES ECB mode and removes leading spaces.\n\n    Args:\n        encoded (str): The base64-encoded encrypted message.\n\n    Returns:\n        str: The decrypted message with leading spaces removed.\n    \"\"\"\n    secret_key = os.environ.get('secret', 'veryMuchSecret').encode('utf-8')\n\n    cipher = AES.new(secret_key, AES.MODE_ECB)\n    encrypted_bytes = base64.b64decode(encoded)\n    decrypted_bytes = cipher.decrypt(encrypted_bytes)\n    decrypted_text = decrypted_bytes.decode('utf-8')\n\n    return decrypted_text.lstrip()\n\n\nimport maat\n\nos.environ['secret'] = 'super super secret key'.rjust(32)\nmaat.registered_transformation['encode'] = encode\nmaat.registered_transformation['decode'] = decode\n\n# Example encode \ntest_input = {\n    'name': 'John Doe',\n    'address': 'John Doe Street',\n}\ncounter_dict = {\n    'name': {\n        'type': 'str', 'regex': 'John Doe', 'transform': 'encode'\n    },\n    'address': {\n        'type': 'str', 'regex': 'John Doe Street', 'transform': 'encode'\n    },\n}\n\nvalidated_items = maat.scale(test_input, counter_dict)\n\n\n# Example validation on encrypted data\ntest_input = {\n    'name': 'LcyInWDZsUv22ocRHM3+yg7QO9ArjlhP2R9v5CSZIRc=',\n    'address': 'LcyInWDZsUv22ocRHM3+yryn2OYg2jesvpgClxA/sdQ=',\n}\n\ncounter_dict = {\n    'name': {\n        'type': 'str', 'regex': 'John Doe',\n        'pre_transform': 'decode', 'transform': 'encode'\n    },\n    'address': {\n        'type': 'str', 'regex': 'John Doe Street',\n        'pre_transform': 'decode', 'transform': 'encode'\n    },\n}\nvalidated_items = maat.scale(test_input, counter_dict)\n```\n\n\n#### Benefits\n\nThis approach enables validation of encrypted data while keeping it encrypted, or performing validation and serialization on encrypted data, thereby ensuring data confidentiality and adherence to data protection regulations.\n\n\n## Installation\n```sh\npip install maat\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details\n\n## Note\nThis project is being used in production by multiple companies.\n\n\n## Benchmark\n\nBenchmark open-sourced from [Pydantic](https://github.com/pydantic/pydantic/tree/7f90b2f342ba338957bb8dfff3ead760edcdd9bf/benchmarks)\n\nPackage | Version | Relative Performance | Mean validation time\n--- | --- | --- | ---\nmaat | `3.0.4` |  | 15.8μs\nattrs + cattrs | `21.2.0` | 2.4x slower | 37.6μs\npydantic | `1.8.2` | 2.5x slower | 39.7μs\nvoluptuous | `0.12.1` | 6.2x slower | 98.6μs\nmarshmallow | `3.13.0` | 7.2x slower | 114.1μs\ntrafaret | `2.1.0` | 7.5x slower | 118.5μs\nschematics | `2.1.1` | 26.6x slower | 420.9μs\ndjango-rest-framework | `3.12.4` | 30.4x slower | 482.2μs\ncerberus | `1.3.4` | 55.6x slower | 880.2μs\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattumm%2Fmaat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fattumm%2Fmaat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fattumm%2Fmaat/lists"}