{"id":17654459,"url":"https://github.com/ramiawar/pydastic","last_synced_at":"2025-04-12T23:07:49.299Z","repository":{"id":37480537,"uuid":"479842588","full_name":"RamiAwar/pydastic","owner":"RamiAwar","description":"An Elasticsearch Python ORM based on Pydantic.","archived":false,"fork":false,"pushed_at":"2023-04-02T21:06:03.000Z","size":539,"stargazers_count":125,"open_issues_count":55,"forks_count":6,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-12T23:06:53.273Z","etag":null,"topics":["elasticsearch","orm","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RamiAwar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-09T21:01:08.000Z","updated_at":"2025-01-19T16:53:47.000Z","dependencies_parsed_at":"2024-10-31T05:01:12.752Z","dependency_job_id":"a4b637b7-b66c-4d1a-9a53-bbd71570a1cd","html_url":"https://github.com/RamiAwar/pydastic","commit_stats":{"total_commits":60,"total_committers":3,"mean_commits":20.0,"dds":"0.44999999999999996","last_synced_commit":"87c6ac3e7c1c8c54fa2cbee88b61cc8a9df033ad"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamiAwar%2Fpydastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamiAwar%2Fpydastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamiAwar%2Fpydastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RamiAwar%2Fpydastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RamiAwar","download_url":"https://codeload.github.com/RamiAwar/pydastic/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248643004,"owners_count":21138355,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","orm","python"],"created_at":"2024-10-23T12:14:13.384Z","updated_at":"2025-04-12T23:07:49.264Z","avatar_url":"https://github.com/RamiAwar.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://github.com/RamiAwar/pydastic/raw/main/assets/images/pydastic.png\" width=\"200\" height=\"200\" /\u003e\n\u003ch1\u003ePydastic\u003c/h1\u003e\n\n\u003ca href=\"https://pypi.org/project/pydastic\" target=\"_blank\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/pydastic?color=%2334D058\u0026label=pypi%20package\" alt=\"Package version\"\u003e\n\u003c/a\u003e\n\n[![build](https://github.com/RamiAwar/pydastic/actions/workflows/build.yml/badge.svg)](https://github.com/RamiAwar/pydastic/actions/workflows/build.yml)\n[![Python Version](https://img.shields.io/pypi/pyversions/pydastic.svg)](https://pypi.org/project/pydastic/)\n[![Dependencies Status](https://img.shields.io/badge/dependencies-up%20to%20date-brightgreen.svg)](https://github.com/ramiawar/pydastic/pulls?utf8=%E2%9C%93\u0026q=is%3Apr%20author%3Aapp%2Fdependabot)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n\n[![Security: bandit](https://img.shields.io/badge/security-bandit-green.svg)](https://github.com/PyCQA/bandit)\n[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)](https://github.com/ramiawar/pydastic/blob/master/.pre-commit-config.yaml)\n[![Semantic Versions](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--versions-e10079.svg)](https://github.com/ramiawar/pydastic/releases)\n[![License](https://img.shields.io/github/license/ramiawar/pydastic)](https://github.com/ramiawar/pydastic/blob/master/LICENSE)\n![Coverage Report](https://github.com/RamiAwar/pydastic/raw/main/assets/images/coverage.svg)\n\nPydastic is an elasticsearch python ORM based on Pydantic.\n\n\u003c/div\u003e\n\n## 💾 Installation\n\nPip:\n```bash\npip install pydastic\n```\n\nPoetry:\n```bash\npoetry add pydastic\n```\n\n\n## 🚀 Core Features\n- Simple CRUD operations supported\n- Sessions for simplifying bulk operations (a la SQLAlchemy)\n- Dynamic index support when committing operations\n\n## In Development\n- Search API (for now, exposed client can be used)\n\n\n## 📋 Usage\n\n### Defining Models\n```python\nfrom pydastic import ESModel\n\nclass User(ESModel):\n    name: str\n    phone: Optional[str]\n    last_login: datetime = Field(default_factory=datetime.now)\n\n    class Meta:\n        index = \"user\"\n```\n\n### Establishing Connection\nAn elasticsearch connection can be setup by using the `connect` function. This function adopts the same signature as the `elasticsearch.Elasticsearch` client and supports editor autocomplete.\nMake sure to call this only once. No protection is put in place against multiple calls, might affect performance negatively.\n\n```python\nfrom pydastic import connect\n\nconnect(hosts=\"localhost:9200\")\n```\n\n### CRUD: Create, Update\n```python\n# Create and save doc\nuser = User(name=\"John\", age=20)\nuser.save(wait_for=True)  # wait_for explained below\n\nassert user.id != None\n\n# Update doc\nuser.name = \"Sam\"\nuser.save(wait_for=True)\n```\n\n### CRUD: Read Document\n```python\ngot = User.get(id=user.id)\nassert got == user\n```\n\n### CRUD: Delete\n```python\nuser = User(name=\"Marie\")\nuser.save(wait_for=True)\n\nuser.delete(wait_for=True)\n```\n\n### Sessions\nSessions are inspired by [SQL Alchemy](https://docs.sqlalchemy.org/en/14/orm/tutorial.html)'s sessions, and are used for simplifying bulk operations using the Elasticsearch client. From what I've seen, the ES client makes it pretty hard to use the bulk API, so they created bulk helpers (which in turn have incomplete/wrong docs).\n\n\n```python\njohn = User(name=\"John\")\nsarah = User(name=\"Sarah\")\n\nwith Session() as session:\n    session.save(john)\n    session.save(sarah)\n    session.commit()\n```\n\nWith an ORM, bulk operations can be exposed neatly through a simple API. Pydastic also offers more informative errors on issues encountered during bulk operations. This is possible by suppressing the built-in elastic client errors and extracting more verbose ones instead.\n\nExample error:\n\n```json\npydastic.error.BulkError: [\n    {\n        \"update\": {\n            \"_index\": \"user\",\n            \"_type\": \"_doc\",\n            \"_id\": \"test\",\n            \"status\": 404,\n            \"error\": {\n                \"type\": \"document_missing_exception\",\n                \"reason\": \"[_doc][test]: document missing\",\n                \"index_uuid\": \"cKD0254aQRWF-E2TMxHa4Q\",\n                \"shard\": \"0\",\n                \"index\": \"user\"\n            }\n        }\n    },\n    {\n        \"update\": {\n            \"_index\": \"user\",\n            \"_type\": \"_doc\",\n            \"_id\": \"test2\",\n            \"status\": 404,\n            \"error\": {\n                \"type\": \"document_missing_exception\",\n                \"reason\": \"[_doc][test2]: document missing\",\n                \"index_uuid\": \"cKD0254aQRWF-E2TMxHa4Q\",\n                \"shard\": \"0\",\n                \"index\": \"user\"\n            }\n        }\n    }\n]\n```\n\nThe sessions API will also be available through a context manager before the v1.0 release.\n\n\n### Dynamic Index Support\nPydastic also supports dynamic index specification. The model Metaclass index definition is still mandatory, but if an index is specified when performing operations, that will be used instead.\nThe model Metaclass index is technically a fallback, although most users will probably be using a single index per model. For some users, multiple indices per model are needed (for example in B2B businesses, one user index per client/company is needed to keep the data separated between clients).\n\n```python\nuser = User(name=\"Marie\")\nuser.save(index=\"my-user\", wait_for=True)\n\nuser.delete(index=\"my-user\", wait_for=True)\n```\n\n### Search API\nStill haven't got an idea on how to wrap the underlying API productively. Unless I create a DSL from scratch or use elasticsearch-dsl (which I don't like due to lacking documentation), I can't really provide any value on top of the client's built-in search API. Give this a minute of thought and shoot me your suggestions if you come up with anything!\n\n\n### Notes on testing\nWhen writing tests with Pydastic (even applies when writing tests with the elasticsearch client), remember to use the `wait_for=True` argument when executing operations. If this is not used, then the test will continue executing even if Elasticsearch hasn't propagated the change to all nodes, giving you weird results.\n\nFor example if you save a document, then try getting it directly after, you'll get a document not found error. This is solved by using the wait_for argument in Pydastic (equivalent to `refresh=\"wait_for\"` in Elasticsearch)\n\nHere is [a reference](https://elasticsearch-py.readthedocs.io/en/v8.2.0/api.html#elasticsearch.Elasticsearch.index) to where this argument is listed in the docs. \n\nIt's also supported in the bulk helpers even though its not mentioned in their docs, but you wouldn't figure that out unless you dug into their source and traced back several function calls where `*args` `**kwargs` are just being forwarded across calls.. :)\n\n## Support Elasticsearch Versions\n\nPart of the build flow is running the tests using elasticsearch 7.12.0 DB as well as a 7.12.0 elasticsearch-python client. Another part is using 8.1.2 as well (DB as well as client, as part of a build matrix). This ensures support for multiple versions.\n\n## 📈 Releases\n\nNone yet.\n\nYou can see the list of available releases on the [GitHub Releases](https://github.com/ramiawar/pydastic/releases) page.\n\nWe follow [Semantic Versions](https://semver.org/) specification.\n\nWe use [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.\n\n\n## 🛡 License\n\n[![License](https://img.shields.io/github/license/ramiawar/pydastic)](https://github.com/ramiawar/pydastic/blob/master/LICENSE)\n\nThis project is licensed under the terms of the `MIT` license. See [LICENSE](https://github.com/ramiawar/pydastic/blob/master/LICENSE) for more details.\n\n## 📃 Citation\n\n```bibtex\n@misc{pydastic,\n  author = {Rami Awar},\n  title = {Pydastic is an elasticsearch python ORM based on Pydantic.},\n  year = {2022},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/ramiawar/pydastic}}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Framiawar%2Fpydastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Framiawar%2Fpydastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Framiawar%2Fpydastic/lists"}