{"id":51108179,"url":"https://github.com/confluentinc/flink-table-api-python-examples","last_synced_at":"2026-06-24T15:31:09.702Z","repository":{"id":257029317,"uuid":"849441507","full_name":"confluentinc/flink-table-api-python-examples","owner":"confluentinc","description":"Python Examples for running Apache Flink® Table API on Confluent Cloud","archived":false,"fork":false,"pushed_at":"2026-02-19T23:30:35.000Z","size":76,"stargazers_count":17,"open_issues_count":3,"forks_count":5,"subscribers_count":98,"default_branch":"master","last_synced_at":"2026-02-20T03:27:04.588Z","etag":null,"topics":["apache-flink","confluent","confluent-cloud","flink-sql","stream-processing","table-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/confluentinc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-08-29T15:46:55.000Z","updated_at":"2025-11-06T02:33:52.000Z","dependencies_parsed_at":"2025-01-15T11:56:20.880Z","dependency_job_id":"55643b55-1edc-4e1b-9d32-55cbfb95131a","html_url":"https://github.com/confluentinc/flink-table-api-python-examples","commit_stats":null,"previous_names":["confluentinc/flink-table-api-python-examples"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/confluentinc/flink-table-api-python-examples","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fflink-table-api-python-examples","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fflink-table-api-python-examples/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fflink-table-api-python-examples/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fflink-table-api-python-examples/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/confluentinc","download_url":"https://codeload.github.com/confluentinc/flink-table-api-python-examples/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fflink-table-api-python-examples/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34739426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-24T02:00:07.484Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-flink","confluent","confluent-cloud","flink-sql","stream-processing","table-api"],"created_at":"2026-06-24T15:31:09.634Z","updated_at":"2026-06-24T15:31:09.693Z","avatar_url":"https://github.com/confluentinc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Flink® Table API on Confluent Cloud - Examples\n\nThis repository contains examples for running Apache Flink's Table API on Confluent Cloud.\n\n## Introduction to Table API for Python\n\nThe [Table API](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table_api_tutorial/) enables a programmatic\nway of developing, testing, and submitting Flink pipelines for processing data streams.\nStreams can be finite or infinite, with insert-only or changelog data. The latter allows for dealing with *Change Data\nCapture* (CDC) events.\n\nWithin the API, you conceptually work with tables that change over time - inspired by relational databases. Write\na *Table Program* as a declarative and structured graph of data transformations. Table API is inspired by SQL and complements\nit with additional tools for juggling real-time data. You can mix and match Flink SQL with Table API at any time as they\ngo hand in hand.\n\n## Table API on Confluent Cloud\n\nTable API on Confluent Cloud is a client-side library that delegates Flink API calls to Confluent’s public\nREST API. It submits [Statements](https://docs.confluent.io/cloud/current/api.html#tag/Statements-(sqlv1)) and retrieves\n[StatementResults](https://docs.confluent.io/cloud/current/api.html#tag/Statement-Results-(sqlv1)).\n\nTable programs are implemented against [Flink's open source Table API for Python](https://github.com/apache/flink/tree/master/flink-python/pyflink/table).\nThe provided Confluent pip packages repackage Flink's Python API and bundle the Confluent-specific components for powering the `TableEnvironment` without the need\nfor a local Flink cluster. While using those packages, Flink internal components such as\n`CatalogStore`, `Catalog`, `Planner`, `Executor`, and configuration are managed by the plugin and fully integrate with\nConfluent Cloud. Including access to Apache Kafka®, Schema Registry, and Flink Compute Pools.\n\nNote: The Table API plugin is in Open Preview stage. Take a look at the [Known Limitation](#known-limitations) section below.\n\n### Motivating Example\n\nThe following code shows how a Table API program is structured. Subsequent sections will go into more details how you\ncan use the examples of this repository to play around with Flink on Confluent Cloud.\n\n```python\nfrom pyflink.table.confluent import ConfluentSettings, ConfluentTools\nfrom pyflink.table import TableEnvironment, Row\nfrom pyflink.table.expressions import col, row\n\ndef run():\n    # Setup connection properties to Confluent Cloud\n    settings = ConfluentSettings.from_global_variables()\n    env = TableEnvironment.create(settings)\n\n  # Run your first Flink statement in Table API\n    env.from_elements([row(\"Hello world!\")]).execute().print()\n\n    # Or use SQL\n    env.sql_query(\"SELECT 'Hello world!'\").execute().print()\n\n    # Structure your code with Table objects - the main ingredient of Table API.\n    table = env.from_path(\"examples.marketplace.clicks\") \\\n        .filter(col(\"user_agent\").like(\"Mozilla%\")) \\\n        .select(col(\"click_id\"), col(\"user_id\"))\n\n    table.print_schema()\n    print(table.explain())\n\n    # Use the provided tools to test on a subset of the streaming data\n    expected = ConfluentTools.collect_materialized_limit(table, 50)\n    actual = [Row(42, 500)]\n    if expected != actual:\n        print(\"Results don't match!\")\n\nif __name__ == \"__main__\":\n    run()\n```\n\n## Getting Started\n\n### Prerequisites\n\n1. Sign up for Confluent Cloud at [https://confluent.cloud](https://confluent.cloud/signup)\n2. [Create a compute pool](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/create-compute-pool.html#create-a-compute-pool-in-ccloud-console)\n   in the web UI of Confluent's Cloud Console\n3. [Generate an API Key](https://docs.confluent.io/cloud/current/flink/operate-and-deploy/generate-api-key-for-flink.html#generate-an-api-key)\n   for the region where you created your compute pool\n4. Optional: [Create a Kafka cluster](https://docs.confluent.io/cloud/current/clusters/create-cluster.html#manage-ak-clusters-on-ccloud)\n   if you want to run examples that store data in Kafka\n5. Have the correct environment variables set as per [the documentation](https://docs.confluent.io/cloud/current/flink/reference/table-api.html#environment-variables)\n6. We recommend using a tool like [uv](https://docs.astral.sh/uv/) to manage your Python versions and environments and Python 3.9-3.11 are the only versions currently supported.\n\n### Run Examples\n\nAll example files are located in `examples`. Each file contains a `run()`\nfunction that can be executed directly or in `__main__`. Each has multiple table programs that will be executed individually. Every example program covers a different topic to learn\nmore about how Table API can be used. It is recommended to go through the examples in the defined order as they partially\nbuild on top of each other.\n\nClone this repository to your local computer, or download it as a ZIP file and extract it.\n```bash\ngit clone https://github.com/confluentinc/flink-table-api-python-examples.git\n```\n\nChange the current directory.\n```bash\ncd flink-table-api-python-examples\n```\n\nWe recommend using [uv](https://docs.astral.sh/uv/) to run the scripts, will automatically create a virtualenv with the required dependencies.\n\n**Note**: Flink's Python API communicates with a Java process under the hood. Make sure you also have at least Java 11\ninstalled. Check that your `JAVA_HOME` environment variable is correctly set. Only checking `java -version` might not\nbe enough.\n\n```\necho $JAVA_HOME\n```\n\nIf required install openjdk and export the JAVA_HOME\n```bash\nbrew install openjdk \u0026\u0026 export JAVA_HOME=$(/usr/libexec/java_home) \u0026\u0026 echo $JAVA_HOME\n```\n\nRun an example script. No worries the program is read-only so it won't affect your existing\nKafka clusters. All results will be printed to the console.\n```bash\nuv run examples/example_00_hello_world\n```\n\nAn output similar to the following means that you are able to run the examples:\n```text\nio.confluent.flink.plugin.ConfluentFlinkException: Parameter 'client.organization-id' not found.\n```\nConfiguration will be covered in the next section.\n\n### Configure the settings parameters in the `ConfluentSettings` class.\n\nThe Table API plugin needs a set of configuration options for establishing a connection to Confluent Cloud. These can be set as a properties file, passed in via the command line as arguments, defined in the code or via the environment variables. This example uses the environment variables. For more details, please see the [documentation](https://docs.confluent.io/cloud/current/flink/reference/table-api.html#confluentsettings-class).\n\nAll required information can be found in the web UI of Confluent's Cloud Console:\n- `client.organization-id|ORG_ID` from [**Menu** → **Settings** → **Organizations**](https://confluent.cloud/settings/organizations)\n- `client.environment-id|ENV_ID` from [**Menu** → **Environments**](https://confluent.cloud/environments)\n- `client.cloud|CLOUD_PROVIDER`, `client.region|CLOUD_REGION`, `client.compute-pool-id|COMPUTE_POOL_ID` from [**Menu** → **Environments**](https://confluent.cloud/environments) → **your environment** → **Flink** → **your compute pool**\n- `client.flink-api-key|FLINK_API_KEY`, `client.flink-api-secret|FLINK_API_SECRET` from [**Menu** → **Settings** → **API keys**](https://confluent.cloud/settings/api-keys)\n\nExport the environment variables as shown below:\n\n```bash\nexport CLOUD_PROVIDER=\"\u003cmy_cloud\u003e\"\nexport CLOUD_REGION=\"\u003cmy_region\u003e\"\nexport FLINK_API_KEY=\"\u003cmy_key\u003e\"\nexport FLINK_API_SECRET=\"\u003cmy_secret\u003e\"\nexport ORG_ID=\"\u003cmy_organization\u003e\"\nexport ENV_ID=\"\u003cmy_environment\u003e\"\nexport COMPUTE_POOL_ID=\"\u003cmy_compute_pool\u003e\"\n```\n\nExamples should be runnable after setting all configuration options correctly.\n\n### Table API Playground using Python Interactive Shell\n\nFor convenience, the repository also contains an init script for playing around with\nTable API in an interactive manner.\n\n1. Create a virtualenv with `uv sync` and activate it with `source .venv/bin/activate`.\n\n2. Run `python -i start_pyshell.py` to start an interactive repl to explore Table API.\n\n3. The `TableEnvironment` is pre-initialized from environment variables and available under `env`.\n\n4. Run your first \"Hello world!\" using `env.execute_sql(\"SELECT 'Hello world!'\").print()`\n\n## Configuration\n\nThe Table API plugin needs a set of configuration options for establishing a connection to Confluent Cloud.\n\nThe `ConfluentSettings` class is a utility for providing configuration options from various sources.\n\nFor production, external input, code, and environment variables can be combined.\n\nPrecedence order (highest to lowest):\n1. Properties File\n2. Code\n3. Environment Variables\n\nA multi-layered configuration can look like:\n```python\nfrom pyflink.table.confluent import ConfluentSettings\nfrom pyflink.table import TableEnvironment\n\ndef run():\n  # Properties file might set cloud, region, org, env, and compute pool.\n  # Environment variables might pass key and secret.\n\n  # Code sets the session name and SQL-specific options.\n  settings = ConfluentSettings.new_builder_from_file(...) \\\n    .set_context_name(\"MyTableProgram\") \\\n    .set_option(\"sql.local-time-zone\", \"UTC\") \\\n    .build()\n\n  env = TableEnvironment.create(settings)\n```\n\n### Via Properties File\n\nStore options (or some options) in a `cloud.properties` file:\n\n```properties\n# Cloud region\nclient.cloud=aws\nclient.region=us-east-1\n\n# Access \u0026 compute resources\nclient.flink-api-key=key\nclient.flink-api-secret=secret\nclient.organization-id=b0b21724-4586-4a07-b787-d0bb5aacbf87\nclient.environment-id=env-z3y2x1\nclient.compute-pool-id=lfcp-8m03rm\n```\n\nReference the `cloud.properties` file:\n```python\nfrom pyflink.table.confluent import ConfluentSettings\n\n# Arbitrary file location in file system\nsettings = ConfluentSettings.from_file(\"/path/to/cloud.properties\")\n```\n\nA path to a properties file can also be specified by setting the environment variable `FLINK_PROPERTIES`.\n\n### Via Code\n\nPass all options (or some options) in code:\n\n```python\nfrom pyflink.table.confluent import ConfluentSettings\n\nsettings = ConfluentSettings.new_builder() \\\n  .set_cloud(\"aws\") \\\n  .set_region(\"us-east-1\") \\\n  .set_flink_api_key(\"key\") \\\n  .set_flink_api_secret(\"secret\") \\\n  .set_organization_id(\"b0b21724-4586-4a07-b787-d0bb5aacbf87\") \\\n  .set_environment_id(\"env-z3y2x1\") \\\n  .set_compute_pool_id(\"lfcp-8m03rm\") \\\n  .build()\n```\n\n### Via Environment Variables\n\nPass all options (or some options) as variables:\n\n```bash\nexport CLOUD_PROVIDER=\"aws\"\nexport CLOUD_REGION=\"us-east-1\"\nexport FLINK_API_KEY=\"key\"\nexport FLINK_API_SECRET=\"secret\"\nexport ORG_ID=\"b0b21724-4586-4a07-b787-d0bb5aacbf87\"\nexport ENV_ID=\"env-z3y2x1\"\nexport COMPUTE_POOL_ID=\"lfcp-8m03rm\"\n\npoetry run example\n```\n\nIn code call:\n```python\nfrom pyflink.table.confluent import ConfluentSettings\n\nsettings = ConfluentSettings.from_global_variables()\n```\n\nA path to a properties file can also be specified by setting the environment variable `FLINK_PROPERTIES`.\n\n### Configuration Options\n\nThe following configuration needs to be provided:\n\n| Property key              | Environment variable | Required | Comment                                                                      |\n|---------------------------|----------------------|----------|------------------------------------------------------------------------------|\n| `client.cloud`            | `CLOUD_PROVIDER`     | Y        | Confluent identifier for a cloud provider. For example: `aws`                |\n| `client.region`           | `CLOUD_REGION`       | Y        | Confluent identifier for a cloud provider's region. For example: `us-east-1` |\n| `client.flink-api-key`    | `FLINK_API_KEY`      | Y        | API key for Flink access.                                                    |\n| `client.flink-api-secret` | `FLINK_API_SECRET`   | Y        | API secret for Flink access.                                                 |\n| `client.organization-id`  | `ORG_ID`             | Y        | ID of the organization. For example: `b0b21724-4586-4a07-b787-d0bb5aacbf87`  |\n| `client.environment-id`   | `ENV_ID`             | Y        | ID of the environment. For example: `env-z3y2x1`                             |\n| `client.compute-pool-id`  | `COMPUTE_POOL_ID`    | Y        | ID of the compute pool. For example: `lfcp-8m03rm`                           |\n\nAdditional configuration:\n\n| Property key               | Environment variable | Required | Comment                                                                                                  |\n|----------------------------|----------------------|----------|----------------------------------------------------------------------------------------------------------|\n| `client.endpoint-template` | `ENDPOINT_TEMPLATE`  | N        | A template for the endpoint URL. For example: `https://flinkpls-dom123.{region}.{cloud}.confluent.cloud` |\n| `client.principal-id`      | `PRINCIPAL_ID`       | N        | Principal that runs submitted statements. For example: `sa-23kgz4` (for a service account)               |\n| `client.context`           |                      | N        | A name for this Table API session. For example: `my_table_program`                                       |\n| `client.statement-name`    |                      | N        | Unique name for statement submission. By default, generated using a UUID.                                |\n| `client.rest-endpoint`     | `REST_ENDPOINT`      | N        | URL to the REST endpoint. For example: `proxyto.confluent.cloud`                                         |\n| `client.catalog-cache`     |                      | N        | Expiration time for catalog objects. For example: '5 min'. '1 min' by default. '0' disables the caching. |\n\n### Endpoint Configuration\n\nThe Confluent Flink plugin provides options to configure endpoints for connecting to Confluent Cloud services. **The template-based approach is the recommended method.**\n\n### `client.endpoint-template`\n\nThis option provides a template for constructing the Flink statement API endpoint URL.\n\n- **Default**: `https://flink.{region}.{cloud}.confluent.cloud`\n- **Example**: `https://flinkpls-dom123.{region}.{cloud}.confluent.cloud`\n- **Usage**: The template supports placeholders `{region}` and `{cloud}` that are replaced with the configured region and cloud provider values.\n- **Environment Variable**: `ENDPOINT_TEMPLATE`\n\n### `client.rest-endpoint` (Discouraged)\n\nThis option specifies the base domain for REST API calls to Confluent Cloud. While still supported, using the template-based configuration above is preferred.\n\n- **Default**: No default value\n- **Example**: `proxy.confluent.cloud`\n- **Usage**: When specified, the plugin constructs the full Flink statement API endpoint URL as `https://flink.{region}.{cloud}.{rest-endpoint}` where `{region}` and `{cloud}` are replaced with the configured region and cloud provider values.\n- **Important**: `client.endpoint-template` and `client.rest-endpoint` are mutually exclusive. If both are set, an exception is thrown.\n- **Environment Variable**: `REST_ENDPOINT`\n\n### Relationship and Default Behavior\n\n1. **Mutual Exclusivity**:\n    - `client.endpoint-template` and `client.rest-endpoint` cannot be set simultaneously\n\n2. **Default Behavior**:\n    - If neither `client.rest-endpoint` nor `client.endpoint-template` is configured, the default template `https://flink.{region}.{cloud}.confluent.cloud` is used for statement API\n    - If endpoint templates are used, each endpoint is constructed independently with the provided templates\n\n### Example\n\nHere's a simple example showing how to configure an endpoint:\n\n```python\n# cloud.properties:\n# client.region=us-east-1\n# client.cloud=aws\n# client.endpoint-template=https://flinkpls-dom123.{region}.{cloud}.confluent.cloud\n\n# Resolved endpoints:\n# - Statement API: https://flinkpls-dom123.us-east-1.aws.confluent.cloud\nsettings = ConfluentSettings.from_file(\"/cloud.properties\")\n```\n\n## Documentation for Confluent Utilities\n\n### Confluent Tools\n\nThe `ConfluentTools` class adds additional methods that can be useful when developing and testing Table API programs.\n\n#### `ConfluentTools.collect_changelog` / `ConfluentTools.print_changelog`\n\nExecutes the given table transformations on Confluent Cloud and returns the results locally\nas a list of changelog rows. Or prints to the console in a table style.\n\nThis method performs `table.execute().collect()` under the hood and consumes a fixed\namount of rows from the returned iterator.\n\nNote: The method can work on both finite and infinite input tables. If the pipeline is\npotentially unbounded, it will stop fetching after the desired amount of rows has been\nreached.\n\nExamples:\n```python\nfrom pyflink.table.confluent import ConfluentSettings, ConfluentTools\nfrom pyflink.table import TableEnvironment\n\nsettings = ConfluentSettings.from_global_variables()\nenv = TableEnvironment.create(settings)\n# On Table object\ntable = env.from_path(\"examples.marketplace.customers\")\nrows = ConfluentTools.collect_changelog_limit(table, 100)\nConfluentTools.print_changelog_limit(table, 100)\n\n# On TableResult object\ntableResult = env.execute_sql(\"SELECT * FROM examples.marketplace.customers\")\nrows = ConfluentTools.collect_changelog_limit(tableResult, 100)\nConfluentTools.print_changelog_limit(tableResult, 100)\n```\n\nShortcuts:\n```python\n# For finite (i.e. bounded) tables\nConfluentTools.collect_changelog(table)\nConfluentTools.print_changelog(table)\n```\n\n#### `ConfluentTools.collect_materialized` / `ConfluentTools.print_materialized`\n\nExecutes the given table transformations on Confluent Cloud and returns the results locally\nas a materialized changelog. In other words: changes are applied to an in-memory table and\nreturned as a list of insert-only rows. Or printed to the console in a table style.\n\nThis method performs `table.execute().collect()` under the hood and consumes a fixed\namount of rows from the returned iterator.\n\nNote: The method can work on both finite and infinite input tables. If the pipeline is\npotentially unbounded, it will stop fetching after the desired amount of rows has been\nreached.\n\n```python\nfrom pyflink.table.confluent import ConfluentSettings, ConfluentTools\nfrom pyflink.table import TableEnvironment\n\nsettings = ConfluentSettings.from_global_variables()\nenv = TableEnvironment.create(settings)\n# On Table object\ntable = env.from_path(\"examples.marketplace.customers\")\nrows = ConfluentTools.collect_materialized_limit(table, 100)\nConfluentTools.print_materialized_limit(table, 100)\n\n# On TableResult object\ntableResult = env.execute_sql(\"SELECT * FROM examples.marketplace.customers\")\nrows = ConfluentTools.collect_materialized_limit(tableResult, 100)\nConfluentTools.print_materialized_limit(tableResult, 100)\n```\n\nShortcuts:\n```python\n# For finite (i.e. bounded) tables\nConfluentTools.collect_materialized(table)\nConfluentTools.print_materialized(table)\n```\n\n### `ConfluentTools.get_statement_name` / `ConfluentTools.stop_statement`\n\nAdditional lifecycle methods are available to control statements on Confluent Cloud after they have\nbeen submitted.\n\n```python\n# On TableResult object\ntable_result = env.execute_sql(\"SELECT * FROM examples.marketplace.customers\")\nstatement_name = ConfluentTools.get_statement_name(table_result)\nConfluentTools.stop_statement(table_result)\n\n# Based on statement name\nConfluentTools.stop_statement_by_name(env, \"table-api-2024-03-21-150457-36e0dbb2e366-sql\")\n```\n\n### Confluent Table Descriptor\n\nA table descriptor for creating tables located in Confluent Cloud programmatically.\n\nCompared to the regular Flink one, this class adds support for Confluent's system columns\nand convenience methods for working with Confluent tables.\n\n`for_managed` corresponds to `TableDescriptor.for_conector(\"confluent\")`.\n\n```python\nfrom pyflink.table.confluent import ConfluentTableDescriptor\nfrom pyflink.table import Schema, DataTypes\nfrom pyflink.table.expressions import col, lit\n\ndescriptor = ConfluentTableDescriptor.for_managed() \\\n  .schema(\n    Schema.new_builder()\n      .column(\"i\", DataTypes.INT())\n      .column(\"s\", DataTypes.INT())\n      .watermark(\"$rowtime\", col(\"$rowtime\").minus(lit(5).seconds)) # Access $rowtime system column\n      .build()) \\\n  .build()\n\nenv.createTable(\"t1\", descriptor)\n```\n\n## Known Limitations\n\nThe Table API plugin is in Open Preview stage.\n\n### Unsupported by Table API Plugin\n\nThe following features are currently not supported:\n\n- Temporary catalog objects (including tables, views, functions)\n- Custom modules\n- Custom catalogs\n- User-defined functions (including system functions)\n- Anonymous, inline objects (including functions, data types)\n- CompiledPlan features are not supported\n- Batch mode\n- Restrictions coming from Confluent Cloud\n    - custom connectors/formats, including: \n      - from_elements with Python objects\n      - converting to/from_pandas\n    - processing time operations\n    - structured data types\n    - many configuration options\n    - limited SQL syntax\n    - batch execution mode\n\n### Issues in Open Source Flink\n\n- Both catalog/database must be set or identifiers must be fully qualified. A mixture of setting a current catalog and\n  using two-part identifiers can lead to errors.\n- String concatenation with `.plus` leads to errors. Use `Expressions.concat`.\n- Selecting `.rowtime` in windows leads to errors.\n- Using `.limit()` can lead to errors.\n- Python API is not fully on par with the Java API. The API lacks support for: TablePipeline, ResolvedSchema\n\n### Supported API\n\nThe following API methods are considered stable and ready to be used:\n\n```text\n// TableEnvironment\nTableEnvironment.create_statement_st()\nTableEnvironment.create_table(String, TableDescriptor)\nTableEnvironment.execute_sql(String)\nTableEnvironment.explain_sql(String)\nTableEnvironment.from_path(String)\nTableEnvironment.get_config()\nTableEnvironment.get_current_catalog()\nTableEnvironment.get_current_database()\nTableEnvironment.list_catalogs()\nTableEnvironment.list_databases()\nTableEnvironment.list_functions()\nTableEnvironment.list_tables()\nTableEnvironment.list_views()\nTableEnvironment.sql_query(String)\nTableEnvironment.use_catalog(String)\nTableEnvironment.use_database(String)\n\n// from_elements works partially, it should be safe to use it in combination with\n// pyflink.table.expression, passing Python objects is not supported\nTableEnvironment.from_elements(...)\n\n// Table: SQL equivalents\nTable.select(...)\nTable.alias(...)\nTable.filter(...)\nTable.where(...)\nTable.group_by(...)\nTable.distinct()\nTable.join(...)\nTable.left_outer_join(...)\nTable.right_outer_join(...)\nTable.full_outer_join(...)\nTable.minus(...)\nTable.minus_all(...)\nTable.union(...)\nTable.union_all(...)\nTable.intersect(...)\nTable.intersect_all(...)\nTable.order_by(...)\nTable.offset(...)\nTable.fetch(...)\nTable.limit(...)\nTable.window(...)\n\n// Table: API extensions\nTable.print_schema()\nTable.add_columns(...)\nTable.add_or_replace_columns(...)\nTable.rename_columns(...)\nTable.drop_columns(...)\nTable.explain()\nTable.execute()\nTable.execute_insert(...)\n\n// StatementSet\nStatementSet.execute()\nStatementSet.add_insert(...)\nStatementSet.add_insert_sql(...)\n\n// TableResult\nTableResult.get_job_client().cancel()\nTableResult.wait(...)\nTableResult.collect()\nTableResult.print()\n\n// TableConfig\nTableConfig.set(...)\n\n// Expressions\nExpressions.* (except for call())\n\n// Others\nTableDescriptor.*\nFormatDescriptor.*\nTumble.*\nSlide.*\nSession.*\nOver.*\n```\n\nConfluent adds the following classes for more convenience:\n```text\nConfluentSettings.*\nConfluentTools.*\nConfluentTableDescriptor.*\n```\n\n## Support\n\nTable API goes hand in hand with Flink SQL on Confluent Cloud.\nFor feature requests or support tickets, use one of the [established channels](https://docs.confluent.io/cloud/current/flink/get-help.html).\n\n### Frequent Issues\n\n#### 1. `py4j.protocol.Py4JError: ConfluentSettings does not exist in the JVM`\n\nThis indicates that the Python API was unable to find a working Java runtime for starting a JVM process.\n\nThe plugin requires at least Java 11. Check that your `JAVA_HOME` environment variable is correctly set:\n```\necho \"$JAVA_HOME\"\n```\n\nIt should look similar to:\n```java\n/Users/Bob/.jenv/versions/11.0\n```\n\nNote: Only checking `java -version` might not be enough. It might be that it shows a correct Java version, but `JAVA_HOME`\nstill points to an invalid version. Consider using [jenv](https://github.com/jenv/jenv).\n\n#### 2. `io.confluent.flink.plugin.ConfluentFlinkException: Parameter 'client.organization-id' not found.`\n\nThis indicates that something is wrong with your configuration. Make sure to fill out the`./config/cloud.properties` file\nwith the required connection information to Confluent Cloud, or set all properties via environment variables as described\nabove.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconfluentinc%2Fflink-table-api-python-examples","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconfluentinc%2Fflink-table-api-python-examples","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconfluentinc%2Fflink-table-api-python-examples/lists"}