{"id":20061020,"url":"https://github.com/bryanyang0528/ksql-python","last_synced_at":"2025-04-12T16:36:20.304Z","repository":{"id":40063958,"uuid":"113642564","full_name":"bryanyang0528/ksql-python","owner":"bryanyang0528","description":"A python wrapper for the KSQL REST API.","archived":false,"fork":false,"pushed_at":"2023-06-10T09:41:01.000Z","size":278,"stargazers_count":159,"open_issues_count":23,"forks_count":65,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-03T16:13:53.477Z","etag":null,"topics":["api","kafka","ksql","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bryanyang0528.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-09T04:47:49.000Z","updated_at":"2025-02-08T02:46:31.000Z","dependencies_parsed_at":"2024-06-18T20:12:55.478Z","dependency_job_id":"7097ffa3-e905-496f-9d1e-50351f958106","html_url":"https://github.com/bryanyang0528/ksql-python","commit_stats":{"total_commits":216,"total_committers":24,"mean_commits":9.0,"dds":"0.39814814814814814","last_synced_commit":"6161b481b9c872a0693f30cede5b46ae23fd2336"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyang0528%2Fksql-python","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyang0528%2Fksql-python/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyang0528%2Fksql-python/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bryanyang0528%2Fksql-python/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bryanyang0528","download_url":"https://codeload.github.com/bryanyang0528/ksql-python/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248597322,"owners_count":21130856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","kafka","ksql","python"],"created_at":"2024-11-13T13:18:01.734Z","updated_at":"2025-04-12T16:36:20.284Z","avatar_url":"https://github.com/bryanyang0528.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"ksql-python\n===========\n\nA python wrapper for the KSQL REST API. Easily interact with the KSQL REST API using this library.\n\nSupported KSQLDB version: 0.10.1+\nSupported Python version: 3.5+\n\n.. image:: https://travis-ci.org/bryanyang0528/ksql-python.svg?branch=master\n  :target: https://travis-ci.org/bryanyang0528/ksql-python\n\n.. image:: https://codecov.io/gh/bryanyang0528/ksql-python/branch/master/graph/badge.svg\n  :target: https://codecov.io/gh/bryanyang0528/ksql-python\n\n.. image:: https://pepy.tech/badge/ksql\n  :target: https://pepy.tech/project/ksql\n\n.. image:: https://pepy.tech/badge/ksql/month\n  :target: https://pepy.tech/project/ksql/month\n  \n.. image:: https://img.shields.io/badge/license-MIT-yellow.svg\n  :target: https://github.com/bryanyang0528/ksql-python/blob/master/LICENSE  \n  \nInstallation\n------------\n\n.. code:: bash\n\n    pip install ksql\n\nOr\n\n.. code:: bash\n\n    git clone https://github.com/bryanyang0528/ksql-python\n    cd ksql-python\n    python setup.py install\n\nGetting Started\n---------------\n\nSetup for KSQL\n~~~~~~~~~~~~~~~\n\nThis is the GITHUB page of KSQL. https://github.com/confluentinc/ksql\n\nIf you have installed open source Confluent CLI (e.g. by installing Confluent Open Source or Enterprise Platform), you can start KSQL and its dependencies with one single command:\n\n.. code:: bash\n\n    confluent start ksql-server\n\nSetup for ksql-python API\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-  Setup for the KSQL API:\n\n.. code:: python\n\n    from ksql import KSQLAPI\n    client = KSQLAPI('http://ksql-server:8088')\n\n- Setup for KSQl API with logging enabled:\n\n.. code:: python\n\n    import logging\n    from ksql import KSQLAPI\n    logging.basicConfig(level=logging.DEBUG)\n    client = KSQLAPI('http://ksql-server:8088')\n\n- Setup for KSQL API with Basic Authentication\n\n.. code:: python\n\n    from ksql import KSQLAPI\n    client = KSQLAPI('http://ksql-server:8088', api_key=\"your_key\", secret=\"your_secret\")\n\nOptions\n~~~~~~~\n\n+---------------+-----------+------------+--------------------------------------------------------------+\n| Option        | Type      | Required   | Description                                                  |\n+===============+===========+============+==============================================================+\n| ``url``       | string    | yes        | Your ksql-server url. Example: ``http://ksql-server:8080``   |\n+---------------+-----------+------------+--------------------------------------------------------------+\n| ``timeout``   | integer   | no         | Timout for Requests. Default: ``5``                          |\n+---------------+-----------+------------+--------------------------------------------------------------+\n| ``api_key``   | string    | no         | API Key to use on the requests                               |\n+---------------+-----------+------------+--------------------------------------------------------------+\n| ``secret``    | string    | no         | Secret to use on the requests                                |\n+---------------+-----------+------------+--------------------------------------------------------------+\n\nMain Methods\n~~~~~~~~~~~~\n\nksql\n^^^^\n\nThis method can be used for some KSQL features which are not supported via other specific methods like ``query``, ``create_stream`` or ``create_stream_as``.\nThe following example shows how to execute the ``show tables`` statement:\n\n.. code:: python\n\n    client.ksql('show tables')\n\n-  Example Response ``[{'tables': {'statementText': 'show tables;', 'tables': []}}]``\n\nquery\n^^^^^\n\nIt will execute sql query and keep listening streaming data.\n\n.. code:: python\n\n    client.query('select * from table1')\n\nThis command returns a generator. It can be printed e.g. by reading its values via `next(query)` or a for loop. Here is a complete example:\n\n.. code:: python\n    \n  from ksql import KSQLAPI\n  client = KSQLAPI('http://localhost:8088')\n  query = client.query('select * from table1')\n  for item in query: print(item)\n\n-  Example Response\n\n   ::\n\n       {\"row\":{\"columns\":[1512787743388,\"key1\",1,2,3]},\"errorMessage\":null}\n       {\"row\":{\"columns\":[1512787753200,\"key1\",1,2,3]},\"errorMessage\":null}\n       {\"row\":{\"columns\":[1512787753488,\"key1\",1,2,3]},\"errorMessage\":null}\n       {\"row\":{\"columns\":[1512787753888,\"key1\",1,2,3]},\"errorMessage\":null}\n\nQuery with HTTP/2\n^^^^^^^^^^^^^^^^^\nExecute queries with the new ``/query-stream`` endpoint. Documented `here \u003chttps://docs.ksqldb.io/en/latest/developer-guide/ksqldb-rest-api/streaming-endpoint/#executing-pull-or-push-queries\u003e`_\n\nTo execute a sql query use the same syntax as the regular query, with the additional ``use_http2=True`` parameter.\n\n.. code:: python\n\n    client.query('select * from table1', use_http2=True)\n\nA generator is returned with the following example response\n\n   ::\n\n       {\"queryId\":\"44d8413c-0018-423d-b58f-3f2064b9a312\",\"columnNames\":[\"ORDER_ID\",\"TOTAL_AMOUNT\",\"CUSTOMER_NAME\"],\"columnTypes\":[\"INTEGER\",\"DOUBLE\",\"STRING\"]}\n       [3,43.0,\"Palo Alto\"]\n       [3,43.0,\"Palo Alto\"]\n       [3,43.0,\"Palo Alto\"]\n\nTo terminate the query above use the ``close_query`` call.\nProvide the ``queryId`` returned from the ``query`` call.\n\n.. code:: python\n\n    client.close_query(\"44d8413c-0018-423d-b58f-3f2064b9a312\")\n\nInsert rows into a Stream with HTTP/2\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nUses the new ``/inserts-stream`` endpoint. See `documentation \u003chttps://docs.ksqldb.io/en/0.10.0-ksqldb/developer-guide/ksqldb-rest-api/streaming-endpoint/#inserting-rows-into-an-existing-stream\u003e`_\n\n.. code:: python\n\n    rows = [\n            {\"ORDER_ID\": 1, \"TOTAL_AMOUNT\": 23.5, \"CUSTOMER_NAME\": \"abc\"},\n            {\"ORDER_ID\": 2, \"TOTAL_AMOUNT\": 3.7, \"CUSTOMER_NAME\": \"xyz\"}\n        ]\n\n    results = self.api_client.inserts_stream(\"my_stream_name\", rows)\n\nAn array of object will be returned on success, with the status of each row inserted.\n\n\nSimplified API\n~~~~~~~~~~~~~~\n\ncreate_stream/ create_table\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\n.. code:: python\n\n    client.create_stream(table_name=table_name,\n                         columns_type=columns_type,\n                         topic=topic,\n                         value_format=value_format)\n\nOptions\n^^^^^^^\n\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| Option          | Type      | Required | Description                                                  |\n+=================+===========+==========+==============================================================+\n| ``table_name``  | string    | yes      | name of stream/table                                         |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| ``columns_type``| list      | yes      | ex:``['viewtime bigint','userid varchar','pageid varchar']`` |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| ``topic``       | string    | yes      | Kafka topic                                                  |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| ``value_format``| string    | no       | ``JSON`` (Default) or ``DELIMITED`` or ``AVRO``              |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| ``key``         | string    | for Table| Key (used for JOINs)                                         |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n\n\n-  Responses\n\n:If create table/stream succeed:\n  return True\n\n:If failed:\n  raise a CreateError(respose_from_ksql_server)\n\ncreate_stream_as\n^^^^^^^^^^^^^^^^\n\na simplified api for creating stream as select\n\n.. code:: python\n\n    client.create_stream_as(table_name=table_name,\n                            select_columns=select_columns,\n                            src_table=src_table,\n                            kafka_topic=kafka_topic,\n                            value_format=value_format,\n                            conditions=conditions,\n                            partition_by=partition_by,\n                            **kwargs)\n\n\n.. code:: sql\n\n  CREATE STREAM \u003ctable_name\u003e\n  [WITH ( kafka_topic=\u003ckafka_topic\u003e, value_format=\u003cvalue_format\u003e, property_name=expression ... )]\n  AS SELECT  \u003cselect_columns\u003e\n  FROM \u003csrc_table\u003e\n  [WHERE \u003cconditions\u003e]\n  PARTITION BY \u003cpartition_by\u003e];\n\nOptions\n^^^^^^^\n\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| Option            | Type      | Required | Description                                                  |\n+===================+===========+==========+==============================================================+\n| ``table_name``    | string    | yes      | name of stream/table                                         |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``select_columns``| list      | yes      | you can select ``[*]`` or ``['columnA', 'columnB']``         |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``src_table``     | string    | yes      | name of source table                                         |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``kafka_topic``   | string    | no       | The name of the Kafka topic of this new stream(table).       |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``value_format``  | string    | no       | ``DELIMITED``, ``JSON``(Default) or ``AVRO``                 |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``conditions``    | string    | no       | The conditions in the where clause.                          |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``partition_by``  | string    | no       | Data will be distributed across partitions by this column.   |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n| ``kwargs``        | pair      | no       | please provide ``key=value`` pairs. Please see more options. |\n+-------------------+-----------+----------+--------------------------------------------------------------+\n\nKSQL JOINs\n~~~~~~~~~~~~~~\n\nKSQL JOINs between Streams and Tables are not supported yet via explicit methods, but you can use the ``ksql`` method for this like the following:\n\n.. code:: python\n\n    client.ksql(\"CREATE STREAM join_per_user WITH (VALUE_FORMAT='AVRO', KAFKA_TOPIC='join_per_user') AS SELECT Time, Amount FROM source c INNER JOIN users u on c.user = u.userid WHERE u.USERID = 1\")\n\nFileUpload\n~~~~~~~~~~~~~~\n\nupload\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\nRun commands from a .ksql file. Can only support ksql commands and not streaming queries.\n\n.. code:: python\n\n     from ksql.upload import FileUpload\n     pointer = FileUpload('http://ksql-server:8080')\n     pointer.upload('rules.ksql')\n\n\nOptions\n^^^^^^^\n\n+-----------------+-----------+----------+--------------------------------------------------------------+\n| Option          | Type      | Required | Description                                                  |\n+=================+===========+==========+==============================================================+\n| ``ksqlfile``    | string    | yes      | name of file containing the rules                            |\n+-----------------+-----------+----------+--------------------------------------------------------------+\n\n\n-  Responses\n\n:If ksql-commands succesfully executed:\n  return (List of server response for all commands)\n\n:If failed:\n  raise the appropriate error\n\nMore Options\n^^^^^^^^^^^^\n\nThere are more properties (partitions, replicas, etc...) in the official document.\n\n`KSQL Syntax Reference \u003chttps://github.com/confluentinc/ksql/blob/0.1.x/docs/syntax-reference.md#syntax-reference\u003e`_\n\n-  Responses\n\n:If create table/stream succeed:\n  return True\n\n:If failed:\n  raise a CreatError(respose_from_ksql_server)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanyang0528%2Fksql-python","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbryanyang0528%2Fksql-python","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbryanyang0528%2Fksql-python/lists"}