{"id":26448494,"url":"https://github.com/amsdal/amsdal-glue","last_synced_at":"2026-01-16T07:45:40.962Z","repository":{"id":246524779,"uuid":"819407531","full_name":"amsdal/amsdal-glue","owner":"amsdal","description":"AMSDAL Glue is a flexible, lightweight, powerful and easy to use opensource ETL interface, built on top of the Python programming language.","archived":false,"fork":false,"pushed_at":"2025-03-13T14:28:15.000Z","size":7374,"stargazers_count":4,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-13T14:37:31.465Z","etag":null,"topics":["data-abstraction","database-schema","django","dynamodb","etl","mongodb","open-source","orm","orm-framework","postgres","postgresql","python","sqlalchemy","sqlite"],"latest_commit_sha":null,"homepage":"https://docs.amsdal.com/amsdal-glue/background/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amsdal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-24T12:49:17.000Z","updated_at":"2025-03-13T14:26:58.000Z","dependencies_parsed_at":"2024-08-05T11:26:53.581Z","dependency_job_id":"b2f2ceb6-a399-488f-b0f6-85663e55cfd0","html_url":"https://github.com/amsdal/amsdal-glue","commit_stats":null,"previous_names":["amsdal/amsdal-glue"],"tags_count":66,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsdal%2Famsdal-glue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsdal%2Famsdal-glue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsdal%2Famsdal-glue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amsdal%2Famsdal-glue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amsdal","download_url":"https://codeload.github.com/amsdal/amsdal-glue/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244240930,"owners_count":20421553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-abstraction","database-schema","django","dynamodb","etl","mongodb","open-source","orm","orm-framework","postgres","postgresql","python","sqlalchemy","sqlite"],"created_at":"2025-03-18T14:34:42.061Z","updated_at":"2025-10-17T12:55:48.169Z","avatar_url":"https://github.com/amsdal.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AMSDAL Glue project\n\nAMSDAL Glue is a flexible, lightweight, yet powerful, Python ETL interface designed to separate your analytics, application, or ORM from its data, allowing you to operate on multiple databases or other data stores simultaneously through a common interface, using SQL, Python objects, or custom parser, query and command planners.\n\n[![PyPI - Version](https://img.shields.io/pypi/v/amsdal-glue.svg)](https://pypi.org/project/amsdal-glue)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/amsdal-glue.svg)](https://pypi.org/project/amsdal-glue)\n\n-----\n\n## Table of Contents\n\n- [Introduction](#introduction)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Performance](#performance)\n- [Roadmap](#roadmap)\n- [Contributing](docs/contributing.md)\n- [License](#license)\n\n## Introduction\n\nAMSDAL Glue is a simple yet critical data technology that can be utilized in almost all data applications.\n\nIn it's most basic sense, AMSDAL Glue provides users a low-level abstraction for their data commands and queries, without limiting the feature-set and funcionality of the underlying database itself. \n\nThis unified, flexible, all-important ETL technology serves as a powerful tool to simplify the development and maintenance process of database-related tasks in Python applications, reducing cost and increasing scalability.\n\nThe project aims to solve several challenges:\n\n1. **Data Interactions**: It provides a unified way to interact with all data (SQL, NoSQL, unstructured, integrations, etc) abstracting the\n   specifics of each database type and allowing developers to focus on their application logic.\n\n2. **Simultaneous Connections**: It enables applications to connect with or integrate to multiple data sources simultaneously, critical in all but the most basic applications, allowing your analytics or application to be agnostic to where it's data is physically located (now or in the future).\n\n3. **Performance**: At it's core [amsdal-glue-core](libs/core/) simply provides you a common interface to interact with data, allowing you to delegate as much of the workload to the underlying database technology as possible. \n\nIn essence, by detaching your application from its data, AMSDAL Glue simplifies the process of writing and maintaining database-related code with a minimal sacrifice of flexibility or perfmance. \n\nFor more information, please refer to the [Overview](docs/overview.md).\n\n## Installation\n\nThe AMSDAL Glue project consists of three main packages: `amsdal-glue-core`, `amsdal-glue-connections`,\nand `amsdal-glue`. You can install each package separately or install the `amsdal-glue` package, which includes all the\ndependencies.\n\nYou can install the AMSDAL Glue project using `pip`:\n\n```bash\npip install amsdal-glue\n```\n\nThis command automatically installs `amsdal-glue-core` and `amsdal-glue-connections` packages as dependencies.\nNote, the `amsdal-glue-connections` will be installed without extra dependencies.\nIf you want to use `PostgresConnection` in order to connect to postgres database, you need explicitly install \nthe `postgres` or `postgres-binary` extra dependency:\n\n```bash\n# It will install the `psycopg` package, usually it will build from source, that is slower than the `postgres-binary` \n# package, but it is recommended for production use.\npip install amsdal-glue[postgres]\n\n# Or you can install the `psycopg-binary` dependency, it is faster than the `psycopg` package installation.\npip install amsdal-glue[postgres-binary]\n```\n\n## Usage\n\nHere is a simple example of how to use the `amsdal-glue` package to connect to a SQLite database and execute a query:\n\n```python\nimport amsdal_glue\nfrom amsdal_glue import interfaces\nfrom amsdal_glue import (\n    init_default_containers,\n    QueryStatement,\n    FieldReference,\n    Field,\n    OrderByQuery,\n    OrderDirection,\n    SchemaReference,\n    Version,\n    DataQueryOperation,\n)\n\n\ndef main() -\u003e None:\n    init_default_containers()\n\n    # Register a connection to a SQLite database\n    connection_mng = amsdal_glue.Container.managers.get(interfaces.ConnectionManager)\n    connection_mng.register_connection_pool(\n        amsdal_glue.DefaultConnectionPool(\n            amsdal_glue.SqliteConnection,\n            db_path='customers.sqlite',\n            check_same_thread=False,  # The default parallel executor works on top of threads\n        ),\n    )\n\n    # Build a query\n    query = QueryStatement(\n        only=[\n            FieldReference(field=Field(name='id'), table_name='c'),\n            FieldReference(field=Field(name='first_name'), table_name='c'),\n        ],\n        table=SchemaReference(name='customers', alias='c', version=Version.LATEST),\n        order_by=[\n            OrderByQuery(\n                field=FieldReference(field=Field(name='id'), table_name='c'),\n                direction=OrderDirection.ASC,\n            ),\n        ],\n    )\n\n    # Execute the query\n    service = amsdal_glue.Container.services.get(interfaces.DataQueryService)\n    data_result = service.execute(\n        query_op=DataQueryOperation(\n            query=query,\n        ),\n    )\n\n    assert data_result.success is True\n    assert [item.data for item in data_result.data] == [\n        {'id': 1, 'first_name': 'John'},\n        {'id': 2, 'first_name': 'Robert'},\n        {'id': 3, 'first_name': 'David'},\n        {'id': 4, 'first_name': 'John'},\n        {'id': 5, 'first_name': 'Betty'},\n    ]\n```\n\nThis example demonstrates how to connect to a SQLite database, build a query, and execute it using the AMSDAL Glue.\nFor more examples and detailed documentation, please see the [Examples](docs/examples.md) and [Templates](templates/) sections.\n\n## Performance\n\nAMSDAL Glue is designed to be highly performant. At it's core [amsdal-glue-core](libs/core/) is simply an interface, allowing you to delegate as much of the workload to the underlying database technology as possible. \n\nIn practice, AMSDAL Glue comes at relatively little cost as seen in our benchmarking [results](https://amsdal.github.io/amsdal-glue/connections_benchmark/index.html)\n\nYou can find more benchmarking results [here](docs/benchmarking.md) \n\n## Roadmap\n\nThe AMSDAL Glue project is under active development, and we have plans to add more features and improvements in the\nfuture. Some of the planned features include:\n\n- Support for data transformation/hooks (coming soon)\n- Support for more database types:\n    - ✅SQLite\n    - ✅PostgreSQL\n    - ⬜MySQL\n    - ⬜MongoDB\n    - ⬜Iceberg support\n- ✅ SQL to AMSDAL Glue query \u0026 command translator to have an ability to integrate with any existing SQL ORM\n  library ([sql-parser-usage](templates/sql-parser-usage/README.md))\n- Add support for more complex queries and commands\n\n## License\n\nThis project is licensed under the Apache License, Version 2.0 - see the [LICENSE](LICENSE.txt) file for details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famsdal%2Famsdal-glue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famsdal%2Famsdal-glue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famsdal%2Famsdal-glue/lists"}