{"id":17703765,"url":"https://github.com/proafxin/xin","last_synced_at":"2025-09-10T23:37:17.900Z","repository":{"id":259114625,"uuid":"864335045","full_name":"proafxin/xin","owner":"proafxin","description":"A pydantic powered universal ORM wrapper for databases.","archived":false,"fork":false,"pushed_at":"2025-04-21T18:11:58.000Z","size":308,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"develop","last_synced_at":"2025-04-21T19:26:25.439Z","etag":null,"topics":["aiomysql","asyncio","asyncpg","mongodb","motor","nosql","orm","pandas","polars","pre-commit","pydantic","python","sqlalchemy","sqlserver","uv"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/proafxin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-28T00:53:29.000Z","updated_at":"2025-04-11T07:27:49.000Z","dependencies_parsed_at":"2024-12-23T21:24:22.005Z","dependency_job_id":"ff4d80a6-0135-47de-8fe3-051b3a16f274","html_url":"https://github.com/proafxin/xin","commit_stats":{"total_commits":44,"total_committers":2,"mean_commits":22.0,"dds":"0.022727272727272707","last_synced_commit":"e84d724b9c152eb141caa2e6885a35d7561bb8fb"},"previous_names":["proafxin/xin"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proafxin%2Fxin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proafxin%2Fxin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proafxin%2Fxin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/proafxin%2Fxin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/proafxin","download_url":"https://codeload.github.com/proafxin/xin/tar.gz/refs/heads/develop","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251294383,"owners_count":21566265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aiomysql","asyncio","asyncpg","mongodb","motor","nosql","orm","pandas","polars","pre-commit","pydantic","python","sqlalchemy","sqlserver","uv"],"created_at":"2024-10-24T21:05:46.831Z","updated_at":"2025-04-28T10:28:30.909Z","avatar_url":"https://github.com/proafxin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Univorm\n\nA pydantic powered universal ORM wrapper for databases.\n\nThis is a python package I created to reuse some functionalities I have had to implement in multiple jobs. For some reason, there aren't any ORM wrappers we can just plug and play. This should help in that area to some extent. I am trying to make it as generalized as possible but data storage services that require paid access may never be part of this package.\n\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/proafxin/xin/develop.svg)](https://results.pre-commit.ci/latest/github/proafxin/xin/develop)\n[![Build, Test and Publish](https://github.com/proafxin/xin/actions/workflows/cicd.yaml/badge.svg?branch=develop)](https://github.com/proafxin/xin/actions/workflows/cicd.yaml)\n[![codecov](https://codecov.io/gh/proafxin/xin/graph/badge.svg?token=p2cOg8tQMb)](https://codecov.io/gh/proafxin/xin)\n[![Documentation Status](https://readthedocs.org/projects/xin/badge/?version=latest)](https://xin.readthedocs.io/en/latest/?badge=latest)\n\n## Usage\n\n### Serialize a dataframe (Pandas/Polars)\n\n```python\nnow = datetime.now()\ndata1 = {\n    \"n\": \"xin\",\n    \"id\": 200,\n    \"f\": [\"a\", \"b\", \"c\"],\n    \"c\": now,\n    \"b\": 20.0,\n    \"d\": {\"a\": 1},\n    \"e\": [[1]],\n}\ndata2 = {\n    \"n\": \"xin\",\n    \"id\": 200,\n    \"f\": [\"d\", \"e\", \"f\"],\n    \"c\": now,\n    \"b\": None,\n    \"d\": {\"a\": 1},\n    \"e\": [[2]],\n}\n\npydantic_objects = await serialize_table(table_name=\"some_table\", data=pd.DataFrame(data=[data1, data2]))\n```\n\n### Deserialize a Pydantic Model\n\n```python\nclass DummyModel(BaseModel):\n    x: int\n    y: float\n\nmodels = [DummyModel(x=1, y=2.0), DummyModel(x=10, y=-1.9)]\ndf = await deserialize_pydantic_objects(models=models)\n```\n\n### Flatten NoSQL data to SQL (Pandas/Polars)\n\n```python\ndata = [\n    {\n        \"id\": 1,\n        \"name\": \"Cole Volk\",\n        \"fitness\": {\"height\": 130, \"weight\": 60},\n    },\n    {\"name\": \"Mark Reg\", \"fitness\": {\"height\": 130, \"weight\": 60}},\n    {\n        \"id\": 2,\n        \"name\": \"Faye Raker\",\n        \"fitness\": {\"height\": 130, \"weight\": 60},\n    },\n]\n\ndf_pandas = pd.DataFrame(data=data)\ndf_polars = pl.DataFrame(data=data)\nflat_df = await flatten(data=df_pandas, depth=0)\nflat_df = await flatten(data=df_polars, depth=0)\nflat_df = await flatten(data=df_polars, depth=1)\n```\n\n### Connect to a mongodb client\n\n```python\ndef mongo_client() -\u003e Generator[MongoClient, None, None]:\n    client = nosql_client(\n        user=os.environ[\"MONGO_USER\"],\n        password=os.environ[\"MONGO_PASSWORD\"],\n        host=os.environ[\"MONGO_HOST\"],\n        dialect=NoSQLDatabaseDialect.MONGODB,\n    )\n    yield client\n    client.close()\n```\n\n### Run query on Mongo\n\n```python\ndocuments = [{\"name\": \"test1\"}, {\"name\": \"test2\"}]\nobject_ids = insert_into_collection(\n    documents=documents, client=mongo_client, dbname=\"test\", collection_name=\"test\"\n)\n\ndf = find_in_collection(\n    query={}, client=mongo_client, dbname=\"test\", collection_name=\"test\"\n)\n```\n\n### Query a MySQL engine\n\nCreate the engine:\n\n```python\nasync def mysql_engine() -\u003e AsyncGenerator[AsyncEngine, None]:\n    engine = await async_sql_engine(\n        user=os.environ[\"MYSQL_USER\"],\n        password=os.environ[\"MYSQL_PASSWORD\"],\n        port=int(os.environ[\"MYSQL_PORT\"]),\n        dialect=SQLDatabaseDialect.MYSQL,\n        host=\"localhost\",\n        dbname=os.environ[\"MYSQL_DBNAME\"],\n    )\n\n    yield engine\n    await engine.dispose()\n```\n\nMake query:\n\n```python\nquery = \"SHOW DATABASES\"\ndf = await async_query_with_result(query=query, engine=mysql_engine)\n```\n\n### Query a PostgreSQL engine\n\nCreate the engine:\n\n```python\nasync def postgres_engine() -\u003e AsyncGenerator[AsyncEngine, None]:\n    engine = await async_sql_engine(\n        user=os.environ[\"POSTGRESQL_USER\"],\n        password=os.environ[\"POSTGRESQL_PASSWORD\"],\n        port=int(os.environ[\"POSTGRESQL_PORT\"]),\n        dialect=SQLDatabaseDialect.POSTGRESQL,\n        host=\"localhost\",\n        dbname=\"postgres\",\n    )\n\n    yield engine\n    await engine.dispose()\n```\n\nMake query:\n\n```python\nquery = \"SELECT * FROM pg_database\"\ndf = await async_query_with_result(query=query, engine=postgres_engine)\n```\n\n### Create a SQL Server Engine\n\nCreate the engine:\n\n```python\nasync def sqlserver_engine() -\u003e AsyncGenerator[Engine, None]:\n    engine = sync_sql_engine(\n        user=os.environ[\"SQLSERVER_USER\"],\n        password=os.environ[\"SQLSERVER_PASSWORD\"],\n        port=int(os.environ[\"SQLSERVER_PORT\"]),\n        dialect=SQLDatabaseDialect.SQLSERVER,\n        host=\"localhost\",\n        dbname=\"master\",\n    )\n\n    yield engine\n    engine.dispose()\n```\n\nMake query:\n\n```python\nquery = \"SELECT * FROM master.sys.databases\"\ndf = sync_query_with_result(query=query, engine=sqlserver_engine)\n```\n\nThe primary backend for parsing dataframes is [polars](https://pola.rs/) due to it's superior [performance](https://pola.rs/_astro/perf-illustration.jHjw6PiD_165TDG.svg). `univorm` supports pandas dataframes as well, however, they are internally converted to polars dataframes first to not compromise performance.\n\nThe backend  for interacting with SQL databases is [sqlalchemy](https://www.sqlalchemy.org/) because it supports async features and is the de-facto standard for communicating with SQL databases.\n\n## Databases Supported\n\n* MySQL\n* PostgreSQL\n* SQL Server\n* Mongodb\n\n## Async Drivers Supported\n\n`univorm` is async first. It means that if an async driver is available for a database dialect, it will leverage the async driver for better performance when  applicable. SQL Server driver PyMSSQL does not have an async variation yet.\n\n* [PyMongo](https://pymongo.readthedocs.io/en/stable/index.html) for Mongodb. Currently async support is in beta but since PyMongo is natively supporting async features, it's safer to use it rather than a third party package like [Motor](https://motor.readthedocs.io/en/stable/index.html).\n* [Asyncpg](https://magicstack.github.io/asyncpg/current/) for PostgreSQL.\n* [AioMySQL](https://aiomysql.readthedocs.io/en/stable/) for MySQL.\n\n## Plan for Future Database Support\n\n* [Couchbase Capella](https://www.couchbase.com/products/capella/)\n* [Scylladb](https://www.scylladb.com/)\n* [Apache Cassandra](https://cassandra.apache.org/_/index.html)\n\n## Test Locally\n\nHave `docker compose`, [tox](https://tox.wiki/en/4.25.0/) and [uv](https://docs.astral.sh/uv/getting-started/installation/) installed. Then run `docker compose up -d`. Create the environment\n\n```bash\nuv venv\nuv sync --dev --extra formatting --extra docs\nuv lock\n```\n\nThen run `tox -p`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproafxin%2Fxin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fproafxin%2Fxin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fproafxin%2Fxin/lists"}