{"id":28455704,"url":"https://github.com/questdb/questdb-connect","last_synced_at":"2025-07-06T05:33:04.264Z","repository":{"id":153735283,"uuid":"629882508","full_name":"questdb/questdb-connect","owner":"questdb","description":"SQLAchemy and Apache Superset extensions for QuestDB","archived":false,"fork":false,"pushed_at":"2025-05-12T14:17:37.000Z","size":480,"stargazers_count":16,"open_issues_count":6,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-27T02:38:37.071Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/questdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-04-19T08:08:13.000Z","updated_at":"2025-06-09T06:13:31.000Z","dependencies_parsed_at":"2023-09-03T03:04:16.253Z","dependency_job_id":"fa9952fb-3f30-42d6-b2a0-8f6336e870b6","html_url":"https://github.com/questdb/questdb-connect","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/questdb/questdb-connect","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/questdb%2Fquestdb-connect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/questdb%2Fquestdb-connect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/questdb%2Fquestdb-connect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/questdb%2Fquestdb-connect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/questdb","download_url":"https://codeload.github.com/questdb/questdb-connect/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/questdb%2Fquestdb-connect/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263853402,"owners_count":23520137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-06T22:10:26.494Z","updated_at":"2025-07-06T05:33:04.222Z","avatar_url":"https://github.com/questdb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://questdb.io/docs/\" target=\"blank\"\u003e\n    \u003cimg alt=\"QuestDB Logo\" src=\"https://questdb.io/img/questdb-logo-themed.svg\" width=\"305px\"/\u003e\n\u003c/a\u003e\n\u003cp\u003e\u003c/p\u003e\n\u003ca href=\"https://slack.questdb.io\"\u003e\n    \u003cimg src=\"https://slack.questdb.io/badge.svg\" alt=\"QuestDB community Slack channel\"/\u003e\n\u003c/a\u003e\n\n## QuestDB Connect\n\nThis repository contains the official implementation of QuestDB's dialect for [SQLAlchemy](https://www.sqlalchemy.org/),\nas well as an engine specification for [Apache Superset](https://github.com/apache/superset/), using\n[psycopg2](https://www.psycopg.org/) for database connectivity.\n\nThe Python module is available here:\n\n\u003ca href=\"https://pypi.org/project/questdb-connect/\"\u003e\n    \u003cimg src=\"https://pypi.org/static/images/logo-small.2a411bc6.svg\" alt=\"PyPi\"/\u003e\n    https://pypi.org/project/questdb-connect/\n\u003c/a\u003e\n\u003cp\u003e\u003c/p\u003e\n\n_Psycopg2_ is a widely used and trusted Python module for connecting to, and working with, QuestDB and other\nPostgreSQL databases.\n\n_SQLAlchemy_ is a SQL toolkit and ORM library for Python. It provides a high-level API for communicating with \nrelational databases, including schema creation and modification. The ORM layer abstracts away the complexities \nof the database, allowing developers to work with Python objects instead of raw SQL statements.\n\n_Apache Superset_ is an open-source business intelligence web application that enables users to visualize and \nexplore data through customizable dashboards and reports. It provides a rich set of data visualizations, including \ncharts, tables, and maps.\n\n## Requirements\n\n* **Python from 3.9 to 3.11** (superset itself use version _3.9.x_)\n* **Psycopg2** `('psycopg2-binary~=2.9.6')`\n* **SQLAlchemy** `('SQLAlchemy\u003e=1.4')`\n\nYou need to install these packages because questdb-connect depends on them. Note that `questdb-connect` v1.1\nis compatible with both `SQLAlchemy` v1.4 and v2.0 while `questdb-connect` v1.0 is compatible with `SQLAlchemy` v1.4 only.\n\n## Versions 0.0.X\n\nThese are versions released for testing purposes.\n\n## Installation\n\nYou can install this package using pip:\n\n```shell\npip install questdb-connect\n```\n\n## SQLAlchemy Sample Usage\n\nUse the QuestDB dialect by specifying it in your SQLAlchemy connection string:\n\n```shell\nquestdb://admin:quest@localhost:8812/main\nquestdb://admin:quest@host.docker.internal:8812/main\n```\n\nFrom that point on use standard SQLAlchemy. Example with raw SQL API:\n```python\nimport datetime\nimport time\nimport uuid\nfrom sqlalchemy import create_engine, text\n\ndef main():\n    engine = create_engine('questdb://admin:quest@localhost:8812/main')\n\n    with engine.begin() as connection:\n        # Create the table\n        connection.execute(text(\"\"\"\n            CREATE TABLE IF NOT EXISTS signal (\n                source SYMBOL,\n                value DOUBLE,\n                ts TIMESTAMP,\n                uuid UUID\n            ) TIMESTAMP(ts) PARTITION BY HOUR WAL;\n        \"\"\"))\n\n        # Insert 2 rows\n        connection.execute(text(\"\"\"\n            INSERT INTO signal (source, value, ts, uuid) VALUES\n            (:source1, :value1, :ts1, :uuid1),\n            (:source2, :value2, :ts2, :uuid2)\n        \"\"\"), {\n            'source1': 'coconut', 'value1': 16.88993244, 'ts1': datetime.datetime.utcnow(), 'uuid1': uuid.uuid4(),\n            'source2': 'banana', 'value2': 3.14159265, 'ts2': datetime.datetime.utcnow(), 'uuid2': uuid.uuid4()\n        })\n\n    # WAL is applied asynchronously, so we need to wait for it to be applied before querying\n    time.sleep(1)\n\n    # Start a new transaction\n    with engine.begin() as connection:\n        # Query the table for rows where value \u003e 10\n        result = connection.execute(\n            text(\"SELECT source, value, ts, uuid FROM signal WHERE value \u003e :value\"),\n            {'value': 10}\n        )\n        for row in result:\n            print(row.source, row.value, row.ts, row.uuid)\n\n\nif __name__ == '__main__':\n    main()\n```\n\nAlternatively, you can use the ORM API:\n```python\nimport datetime\nimport uuid\nimport time\nfrom questdb_connect import Symbol, PartitionBy, UUID, Double, Timestamp, QDBTableEngine\nfrom sqlalchemy import Column, MetaData, create_engine, text\nfrom sqlalchemy.orm import declarative_base, sessionmaker\n\nBase = declarative_base(metadata=MetaData())\n\n\nclass Signal(Base):\n    # Stored in a QuestDB table 'signal'. The tables has WAL enabled, is partitioned by hour, designated timestamp is 'ts'\n    __tablename__ = 'signal'\n    __table_args__ = (QDBTableEngine(None, 'ts', PartitionBy.HOUR, is_wal=True),)\n    source = Column(Symbol)\n    value = Column(Double)\n    ts = Column(Timestamp)\n    uuid = Column(UUID, primary_key=True)\n\n    def __repr__(self):\n        return f\"Signal(source={self.source}, value={self.value}, ts={self.ts}, uuid={self.uuid})\"\n\n\ndef main():\n    engine = create_engine('questdb://admin:quest@localhost:8812/main')\n\n    # Create the table\n    Base.metadata.create_all(engine)\n    Session = sessionmaker(bind=engine)\n    session = Session()\n\n    # Insert 2 rows\n    session.add(Signal(\n        source='coconut',\n        value=16.88993244,\n        ts=datetime.datetime.utcnow(),\n        uuid=uuid.uuid4()\n    ))\n\n    session.add(Signal(\n        source='banana',\n        value=3.14159265,\n        ts=datetime.datetime.utcnow(),\n        uuid=uuid.uuid4()\n    ))\n    session.commit()\n\n    # WAL is applied asynchronously, so we need to wait for it to be applied before querying\n    time.sleep(1)\n\n    # Query the table for rows where value \u003e 10\n    signals = session.query(Signal).filter(Signal.value \u003e 10).all()\n    for signal in signals:\n        print(signal.source, signal.value, signal.ts, signal.uuid)\n\n\nif __name__ == '__main__':\n    main()\n```\nORM (Object-Relational Mapping) API is not recommended for QuestDB due to its fundamental\ndesign differences from traditional transactional databases. While ORMs excel at managing\nrelationships and complex object mappings in systems like PostgreSQL or MySQL, QuestDB is\nspecifically optimized for time-series data operations and high-performance ingestion. It\nintentionally omits certain SQL features that ORMs typically rely on, such as generated\ncolumns, foreign keys, and complex joins, in favor of time-series-specific optimizations.\n\nFor optimal performance and to fully leverage QuestDB's capabilities, we strongly recommend\nusing the raw SQL API, which allows direct interaction with QuestDB's time-series-focused\nquery engine and provides better control over time-based operations.\n\n## Primary Key Considerations\n\nQuestDB differs from traditional relational databases in its handling of data uniqueness. While most databases enforce\nprimary keys to guarantee unique record identification, QuestDB operates differently due to its time-series optimized\narchitecture.\n\nWhen using SQLAlchemy with QuestDB:\n- You can define primary keys in your SQLAlchemy models, but QuestDB won't enforce uniqueness for individual columns\n- Duplicate rows with identical primary key values can exist in the database\n- Data integrity must be managed at the application level\n- QuestDB support [deduplication](https://questdb.io/docs/concept/deduplication/) during ingestion to avoid data duplication, this can be enabled in the table creation\n\n### Recommended Approaches\n\n1. **Composite Keys + QuestDB Deduplication**\n\nComposite keys can be used to define uniqueness based on multiple columns. This approach:\n- Can combine timestamp with any number of additional columns\n- Works with QuestDB's deduplication capabilities\n- Useful for scenarios where uniqueness is defined by multiple attributes\n- Common combinations might include:\n    * timestamp + device_id + metric_type\n    * timestamp + location + sensor_id\n    * timestamp + instrument_id + exchange + side\n\nDeduplication is often enabled in QuestDB regardless of the primary key definition since\nit's required to avoid data duplication during ingestion. \n\nExample:\n```python\nfrom questdb_connect import QDBTableEngine, PartitionBy, Double, Timestamp, Symbol\nclass Measurement(Base):\n    __tablename__ = 'signal'\n    __table_args__ = (QDBTableEngine(None, 'timestamp', PartitionBy.HOUR, is_wal=True),)\n    timestamp = Column(Timestamp, primary_key=True)\n    sensor_id = Column(Symbol, primary_key=True)\n    location = Column(Symbol, primary_key=True)\n    value = Column(Double)\n```\n\n\nChoose your approach based on your data model and whether you need to leverage QuestDB's deduplication capabilities.\n\n2. **UUID-based Identification**\n\nUUIDs are ideal for QuestDB applications because they:\n- Are globally unique across distributed systems\n- Can be generated client-side without database coordination\n- Work well with high-throughput data ingestion\n\nExample:\n```python\nfrom questdb_connect import Symbol, PartitionBy, UUID, Double, Timestamp, QDBTableEngine\nclass Signal(Base):\n    __tablename__ = 'signal'\n    __table_args__ = (QDBTableEngine(None, 'ts', PartitionBy.HOUR, is_wal=True),)\n    source = Column(Symbol)\n    value = Column(Double)\n    ts = Column(Timestamp)\n    uuid = Column(UUID, primary_key=True)\n    # other columns...\n```\n\n## Superset Installation\nThis repository also contains an engine specification for Apache Superset, which allows you to connect\nto QuestDB from within the Superset interface.\n\n\u003cimg alt=\"Apache Superset\" src=\"https://raw.githubusercontent.com/questdb/questdb-connect/refs/heads/main/docs/superset.png\"/\u003e\n\n\nFollow the official [QuestDB Superset guide](https://questdb.io/docs/third-party-tools/superset/) available on the\nQuestDB website to install and configure the QuestDB engine in Superset.\n\n## Contributing\n\nThis package is open-source, contributions are welcome. If you find a bug or would like to request a feature,\nplease open an issue on the GitHub repository. Have a look at the instructions for [developers](DEVELOPERS.md)\nif you would like to push a PR.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquestdb%2Fquestdb-connect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquestdb%2Fquestdb-connect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquestdb%2Fquestdb-connect/lists"}