{"id":13569844,"url":"https://github.com/getindata/flink-http-connector","last_synced_at":"2026-01-12T08:31:20.738Z","repository":{"id":37908132,"uuid":"432838863","full_name":"getindata/flink-http-connector","owner":"getindata","description":"Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs. ","archived":false,"fork":false,"pushed_at":"2025-12-19T11:55:56.000Z","size":761,"stargazers_count":200,"open_issues_count":32,"forks_count":56,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-12-21T00:12:33.992Z","etag":null,"topics":["data-streaming","flink","flink-sql","flink-stream-processing","java"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getindata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-11-28T22:21:02.000Z","updated_at":"2025-12-19T11:16:54.000Z","dependencies_parsed_at":"2023-11-20T19:45:48.760Z","dependency_job_id":"883eda26-4485-45c2-9699-e713392cbb8d","html_url":"https://github.com/getindata/flink-http-connector","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"purl":"pkg:github/getindata/flink-http-connector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-http-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-http-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-http-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-http-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getindata","download_url":"https://codeload.github.com/getindata/flink-http-connector/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-http-connector/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28337596,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T06:09:07.588Z","status":"ssl_error","status_checked_at":"2026-01-12T06:05:18.301Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-streaming","flink","flink-sql","flink-stream-processing","java"],"created_at":"2024-08-01T14:00:44.899Z","updated_at":"2026-01-12T08:31:20.732Z","avatar_url":"https://github.com/getindata.png","language":"Java","funding_links":[],"categories":["Java","大数据"],"sub_categories":[],"readme":"# flink-http-connector\n\n[![Maven Central](https://img.shields.io/maven-central/v/com.getindata/flink-http-connector)](https://mvnrepository.com/artifact/com.getindata/flink-http-connector)\n[![javadoc](https://javadoc.io/badge2/com.getindata/flink-http-connector/javadoc.svg)](https://javadoc.io/doc/com.getindata/flink-http-connector) \n\nThe HTTP TableLookup connector that allows for pulling data from external system via HTTP GET method and HTTP Sink that allows for sending data to external system via HTTP requests.\n\n**Note**: The `main` branch may be in an *unstable or even broken state* during development.\nPlease use [releases](https://github.com/getindata/flink-http-connector/releases) instead of the `main` branch in order to get a stable set of binaries.\n\nThe goal for HTTP TableLookup connector was to use it in Flink SQL statement as a standard table that can be later joined with other stream using pure SQL Flink.\n\n`HttpSink` supports both Streaming API (when using [HttpSink](src/main/java/com/getindata/connectors/http/internal/sink/HttpSink.java) built using [HttpSinkBuilder](src/main/java/com/getindata/connectors/http/internal/sink/HttpSinkBuilder.java)) and the Table API (using connector created in [HttpDynamicTableSinkFactory](src/main/java/com/getindata/connectors/http/internal/table/HttpDynamicTableSinkFactory.java)).\n\n## Updating the connector\nIn case of updating http-connector please see [Breaking changes](#breaking-changes) section.\n\n## Prerequisites\n* Java 11\n* Maven 3\n* Flink 1.18+. Recommended Flink 1.20.* \n\n\n\n## Runtime dependencies\nThis connector has few Flink's runtime dependencies, that are expected to be provided.\n* `org.apache.flink.flink-java`\n* `org.apache.flink.flink-clients`\n* `org.apache.flink.flink-connector-base`\n\n## Installation\n\nIn order to use the `flink-http-connector` the following dependencies are required for both projects using a build automation tool (such as Maven or SBT) and SQL Client with SQL JAR bundles. For build automation tool reference, look into Maven Central: [https://mvnrepository.com/artifact/com.getindata/flink-http-connector](https://mvnrepository.com/artifact/com.getindata/flink-http-connector).\n\n## Documentation\n\nYou can read the official JavaDoc documentation of the latest release at [https://javadoc.io/doc/com.getindata/flink-http-connector](https://javadoc.io/doc/com.getindata/flink-http-connector).\n\n## Usage\n\n### HTTP TableLookup Source\nFlink SQL table definition:\n\nEnrichment Lookup Table\n```roomsql\nCREATE TABLE Customers (\n id STRING,\n id2 STRING,\n msg STRING,\n uuid STRING,\n details ROW\u003c\n isActive BOOLEAN,\n nestedDetails ROW\u003c\n balance STRING\n \u003e\n \u003e\n) WITH (\n'connector' = 'rest-lookup',\n'format' = 'json',\n'url' = 'http://localhost:8080/client',\n'asyncPolling' = 'true'\n)\n```\n\nData Source Table\n```roomsql\nCREATE TABLE Orders (\n id STRING,\n id2 STRING,\n proc_time AS PROCTIME()\n) WITH (\n'connector' = 'datagen',\n'rows-per-second' = '1',\n'fields.id.kind' = 'sequence',\n'fields.id.start' = '1',\n'fields.id.end' = '120',\n'fields.id2.kind' = 'sequence',\n'fields.id2.start' = '2',\n'fields.id2.end' = '120'\n);\n```\n\nUsing _Customers_ table in Flink SQL Lookup Join with _Orders_ table:\n\n```roomsql\nSELECT o.id, o.id2, c.msg, c.uuid, c.isActive, c.balance FROM Orders AS o \nJOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c ON o.id = c.id AND o.id2 = c.id2\n```\n\nThe columns and their values used for JOIN `ON` condition will be used as HTTP GET parameters where the column name will be used as a request parameter name. \n\nFor Example: \n``\nhttp://localhost:8080/client/service?id=1\u0026uuid=2\n``\n\nOr for REST POST method they will be converted to Json and used as request body. In this case, json request body will look like this:\n```json\n{\n \"id\": \"1\",\n \"uuid\": \"2\"\n}\n```\n\n#### Http headers\nIt is possible to set HTTP headers that will be added to HTTP request send by lookup source connector.\nHeaders are defined via property key `gid.connector.http.source.lookup.header.HEADER_NAME = header value` for example:\n`gid.connector.http.source.lookup.header.X-Content-Type-Options = nosniff`.\n\nHeaders can be set using http lookup source table DDL. In example below, HTTP request done for `http-lookup` table will contain three headers:\n- `Origin`\n- `X-Content-Type-Options`\n- `Content-Type`\n\n```roomsql\nCREATE TABLE http-lookup (\n id bigint,\n some_field string\n) WITH (\n 'connector' = 'rest-lookup',\n 'format' = 'json',\n 'url' = 'http://localhost:8080/client',\n 'asyncPolling' = 'true',\n 'gid.connector.http.source.lookup.header.Origin' = '*',\n 'gid.connector.http.source.lookup.header.X-Content-Type-Options' = 'nosniff',\n 'gid.connector.http.source.lookup.header.Content-Type' = 'application/json'\n)\n```\n\n#### Custom REST query\nHttp Lookup Source builds queries out of `JOIN` clauses. One can customize how those queries are built by implementing\n[LookupQueryCreator](src/main/java/com/getindata/connectors/http/LookupQueryCreator.java) and\n[LookupQueryCreatorFactory](src/main/java/com/getindata/connectors/http/LookupQueryCreatorFactory.java) interfaces.\nCustom implementations of `LookupQueryCreatorFactory` can be registered along other factories in\n`resources/META-INF.services/org.apache.flink.table.factories.Factory` file and then referenced by their identifiers in\nthe Http Lookup Source DDL property field `gid.connector.http.source.lookup.query-creator`.\n\nA default implementation that builds an \"ordinary\" GET query, i.e. adds `?joinColumn1=value1\u0026joinColumn2=value2\u0026...`\nto the URI of the endpoint,\n\nFor body based queries such as POST/PUT requests, the \n([GenericGetQueryCreator](src/main/java/com/getindata/connectors/http/internal/table/lookup/querycreators/GenericGetQueryCreator.java))\nis provided as a default query creator. This implementation uses Flink's [json-format](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/) to convert RowData object into Json String.\n\nThe `GenericGetQueryCreator` allows for using custom formats that will perform serialization to Json. Thanks to this, users can create their own logic for converting RowData to Json Strings suitable for their HTTP endpoints and use this logic as custom format\nwith HTTP Lookup connector and SQL queries.\nTo create a custom format user has to implement Flink's `SerializationSchema` and `SerializationFormatFactory` interfaces and register custom format factory along other factories in\n`resources/META-INF.services/org.apache.flink.table.factories.Factory` file. This is common Flink mechanism for providing custom implementations for various factories.\n\nThe most flexible query creator is the [GenericJsonAndUrlQueryCreator](src/main/java/com/getindata/connectors/http/internal/table/lookup/querycreators/GenericJsonAndUrlQueryCreator.java)\nwhich allows column content to be mapped as URL, path, body and query parameter request values; it supports\nPOST, PUT and GET operations. This query creator allows you to issue json requests without needing to code\nyour own custom http connector. The mappings from columns to the json request are supplied in the query creator configuration\nparameters `gid.connector.http.request.query-param-fields`, `gid.connector.http.request.body-fields` and `gid.connector.http.request.url-map`. \n\n### Format considerations\n\n#### For http requests\nIn order to use custom format, user has to specify option `'lookup-request.format' = 'customFormatName'`, where `customFormatName` is the identifier of custom format factory.\n\nAdditionally, it is possible to pass custom query format options from table's DDL.\nThis can be done by using option like so: `'lookup-request.format.customFormatName.customFormatProperty' = 'propertyValue'`, for example\n`'lookup-request.format.customFormatName.fail-on-missing-field' = 'true'`.\n\nIt is important that `customFormatName` part match `SerializationFormatFactory` identifier used for custom format implementation.\nIn this case, the `fail-on-missing-field` will be passed to `SerializationFormatFactory::createEncodingFormat(\nDynamicTableFactory.Context context, ReadableConfig formatOptions)` method in `ReadableConfig` object.\n\nWith default configuration, Flink-Json format is used for `GenericGetQueryCreator`, all options defined in [json-format](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/)\ncan be passed through table DDL. For example `'lookup-request.format.json.fail-on-missing-field' = 'true'`. In this case, format identifier is `json`.\n\n#### For http responses\nSpecify your format options at the top level. For example:\n```roomsql\n 'format' = 'json',\n 'json.ignore-parse-errors' = 'true',\n```\n\n\n#### Timeouts\nLookup Source is guarded by two timeout timers. First one is specified by Flink's AsyncIO operator that executes `AsyncTableFunction`.\nThe default value of this timer is set to 3 minutes and can be changed via `table.exec.async-lookup.timeout` [option](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-async-lookup-timeout).\n\nThe second one is set per individual HTTP requests by HTTP client. Its default value is set currently to 30 seconds and can be changed via `gid.connector.http.source.lookup.request.timeout` option. \n\nFlink's current implementation of `AsyncTableFunction` does not allow specifying custom logic for handling Flink AsyncIO timeouts as it is for Java API.\nBecause of that, if AsyncIO timer passes, Flink will throw TimeoutException which will cause job restart.\n\n#### Available Metadata (Lookup source)\n\nThe metadata column `http-status-code`, if specified in the table definition, will get the HTTP status code.\nThe metadata column `http-headers-map `, if specified in the table definition, will get a map of the HTTP headers.\n\nHTTP requests can fail either immediately or after temporary error retries. The usual behaviour after such failures is to end the job. If you would like to continue \nprocessing after these failures then specify `gid.connector.http.source.lookup.continue-on-error` as true. THe lookup join will complete without content in the expected enrichment columns from the http call,\nthis means that these columns will be null for nullable columns and hold a default value for the type for non-nullable columns.\n\nWhen using `gid.connector.http.source.lookup.continue-on-error` as true, consider adding extra metadata columns that will surface information about failures into your stream.\n\nNote that if metadata columns are specified and the status code is ignored, then a row containing metadata columns will be produced. If\nthe status code is ignored and there are no metadata columns defined, then no row will be emitted; this ensures that the expected \ninner join behaviour still occurs.\n\nMetadata columns can be specified and hold http information. They are optional read-only columns that must be declared VIRTUAL to exclude them during an INSERT INTO operation.\n\n| Key | Data Type | Description |\n|-----------------------|----------------------------------|----------------------------------------|\n| error-string | STRING NULL | A string associated with the error |\n| http-status-code | INT NULL | The HTTP status code |\n| http-headers-map | MAP \u003cSTRING, ARRAY\u003cSTRING\u003e\u003e NULL | The headers returned with the response |\n| http-completion-state | STRING NULL | The completion state of the http call. |\n\n##### http-completion-state possible values\n\n| Value | Description |\n|:-------------------------------|-------------------------------------|\n| SUCCESS | Success |\n| HTTP_ERROR_STATUS | HTTP error status code |\n| EXCEPTION | An Exception occurred |\n| UNABLE_TO_DESERIALIZE_RESPONSE | Unable to deserialize HTTP response |\n| IGNORE_STATUS_CODE | Status code is ignored |\n\nIf the `error-string` metadata column is defined on the table and the call succeeds then it will have a null value.\nWhen the HTTP response cannot be deserialized, then the `http-completion-state` will be `UNABLE_TO_DESERIALIZE_RESPONSE`\nand the `error-string` will be the response body.\nWhen the HTTP status code is in the `gid.connector.http.source.lookup.ignored-response-codes`, then the `http-completion-state` will\nbe `IGNORE_STATUS_CODE` and no data is returned; any metadata columns contain information about the API call that\noccurred.\n\nNote that `UNABLE_TO_DESERIALIZE_RESPONSE` and `IGNORE_STATUS_CODE` are new enum values added\nsince [0.22.0], please ensure you amend your applications and SQL appropriately. \n\nWhen a http lookup call fails and populates the metadata columns with the error information, the expected enrichment columns from the http call\nare not populated, this means that they will be null for nullable columns and hold a default value for the type for non-nullable columns. \n\nIf you are using the Table API `TableResult` and have an `await` with a timeout, this Timeout exception will cause the job to terminate,\neven if there are metadata columns defined.\n\n#### Retries and handling errors (Lookup source)\nLookup source handles auto-retries for two scenarios:\n1. IOException occurs (e.g. temporary network outage)\n2. The response contains a HTTP error code that indicates a retriable error. These codes are defined in the table configuration (see `gid.connector.http.source.lookup.retry-codes`).\nRetries are executed silently, without restarting the job.\n\nNotice that HTTP codes are categorized into into 3 groups:\n- successful responses - response is returned immediately for further processing\n- temporary errors - request will be retried up to the retry limit\n- error responses - unexpected responses are not retried. Any HTTP error code which is not configured as successful or temporary error is treated as an unretriable error.\n\nFor temporary errors that have reached max retries attempts (per request) and error responses, the operation will\nsucceed if `gid.connector.http.source.lookup.continue-on-error` is true, otherwise the job will fail.\n\n##### Retry strategy\nUser can choose retry strategy type for source table:\n- fixed-delay - http request will be re-sent after specified delay.\n- exponential-delay - request will be re-sent with exponential backoff strategy, limited by `lookup.max-retries` attempts. The delay for each retry is calculated as the previous attempt's delay multiplied by the backoff multiplier (parameter `gid.connector.http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier`) up to `gid.connector.http.source.lookup.retry-strategy.exponential-delay.max-backoff`. The initial delay value is defined in the table configuration as `gid.connector.http.source.lookup.retry-strategy.exponential-delay.initial-backoff`.\n\n\n#### Lookup multiple results\n\nTypically, join can return zero, one or more results. What is more, there are lots of possible REST API designs and\npagination methods. Currently, the connector supports only two simple approaches (`gid.connector.http.source.lookup.result-type`):\n\n- `single-value` - REST API returns single object.\n- `array` - REST API returns array of objects. Pagination is not supported yet.\n\nPlease be informed that the mechanism will be enhanced in the future. See [HTTP-118](https://github.com/getindata/flink-http-connector/issues/118).\n\n### HTTP Sink\nThe following example shows the minimum Table API example to create a [HttpDynamicSink](src/main/java/com/getindata/connectors/http/internal/table/HttpDynamicSink.java) that writes JSON values to an HTTP endpoint using POST method, assuming Flink has JAR of [JSON serializer](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/formats/json/) installed:\n\n```roomsql\nCREATE TABLE http (\n id bigint,\n some_field string\n) WITH (\n 'connector' = 'http-sink',\n 'url' = 'http://example.com/myendpoint',\n 'format' = 'json'\n)\n```\n\nThen use `INSERT` SQL statement to send data to your HTTP endpoint:\n\n```roomsql\nINSERT INTO http VALUES (1, 'Ninette'), (2, 'Hedy')\n```\n\nDue to the fact that `HttpSink` sends bytes inside HTTP request's body, one can easily swap `'format' = 'json'` for some other [format](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/formats/overview/). \n\nOther examples of usage of the Table API can be found in [some tests](src/test/java/com/getindata/connectors/http/table/HttpDynamicSinkInsertTest.java).\n\n### Request submission\nStarting from version 0.10 HTTP Sink by default submits events in batch. Before version 0.10 the default and only submission type was `single`.\nThis is a breaking compatibility change.\n\nThe submission mode can be changed using `gid.connector.http.sink.writer.request.mode` property using `single` or `batch` as property value.\n\n#### Batch submission mode\nIn batch mode, a number of events (processed elements) will be batched and submitted in one HTTP request.\nIn this mode, HTTP PUT/POST request's body contains a Json array, where every element of this array represents\nindividual event.\n\nAn example of Http Sink batch request body containing data for three events:\n```json\n[\n {\n \"id\": 1,\n \"first_name\": \"Ninette\",\n \"last_name\": \"Clee\",\n \"gender\": \"Female\",\n \"stock\": \"CDZI\",\n \"currency\": \"RUB\",\n \"tx_date\": \"2021-08-24 15:22:59\"\n },\n {\n \"id\": 2,\n \"first_name\": \"Rob\",\n \"last_name\": \"Zombie\",\n \"gender\": \"Male\",\n \"stock\": \"DGICA\",\n \"currency\": \"GBP\",\n \"tx_date\": \"2021-10-25 20:53:54\"\n },\n {\n \"id\": 3,\n \"first_name\": \"Adam\",\n \"last_name\": \"Jones\",\n \"gender\": \"Male\",\n \"stock\": \"DGICA\",\n \"currency\": \"PLN\",\n \"tx_date\": \"2021-10-26 20:53:54\"\n }\n]\n```\n\nBy default, batch size is set to 500 which is the same as Http Sink's `maxBatchSize` property and has value of 500. \nThe `maxBatchSize' property sets maximal number of events that will by buffered by Flink runtime before passing it to Http Sink for processing.\n\nIn order to change submission batch size use `gid.connector.http.sink.request.batch.size` property. For example:\n\nStreaming API:\n```java\nHttpSink.\u003cString\u003ebuilder()\n .setEndpointUrl(\"http://example.com/myendpoint\")\n .setElementConverter(\n (s, _context) -\u003e new HttpSinkRequestEntry(\"POST\", s.getBytes(StandardCharsets.UTF_8)))\n .setProperty(\"gid.connector.http.sink.request.batch.size\", \"50\")\n .build();\n```\nSQL:\n```roomsql\nCREATE TABLE http (\n id bigint,\n some_field string\n) WITH (\n 'connector' = 'http-sink',\n 'url' = 'http://example.com/myendpoint',\n 'format' = 'json',\n 'gid.connector.http.sink.request.batch.size' = '50'\n)\n```\n\n#### Single submission mode\nIn this mode every processed event is submitted as individual HTTP POST/PUT request. \n\nStreaming API:\n```java\nHttpSink.\u003cString\u003ebuilder()\n .setEndpointUrl(\"http://example.com/myendpoint\")\n .setElementConverter(\n (s, _context) -\u003e new HttpSinkRequestEntry(\"POST\", s.getBytes(StandardCharsets.UTF_8)))\n .setProperty(\"gid.connector.http.sink.writer.request.mode\", \"single\")\n .build();\n```\nSQL:\n```roomsql\nCREATE TABLE http (\n id bigint,\n some_field string\n) WITH (\n 'connector' = 'http-sink',\n 'url' = 'http://example.com/myendpoint',\n 'format' = 'json',\n 'gid.connector.http.sink.writer.request.mode' = 'single'\n)\n```\n\n#### Http headers\nIt is possible to set HTTP headers that will be added to HTTP request send by sink connector.\nHeaders are defined via property key `gid.connector.http.sink.header.HEADER_NAME = header value` for example:\n`gid.connector.http.sink.header.X-Content-Type-Options = nosniff`.\nProperties can be set via Sink builder or Property object:\n```java\nHttpSink.\u003cString\u003ebuilder()\n .setEndpointUrl(\"http://example.com/myendpoint\")\n .setElementConverter(\n (s, _context) -\u003e new HttpSinkRequestEntry(\"POST\", s.getBytes(StandardCharsets.UTF_8)))\n .setProperty(\"gid.connector.http.sink.header.X-Content-Type-Options\", \"nosniff\")\n .build();\n```\nor\n\n```java\nProperties properties = Properties();\nproperties.setProperty(\"gid.connector.http.sink.header.X-Content-Type-Options\", \"nosniff\");\n\nHttpSink.\u003cString\u003ebuilder()\n .setEndpointUrl(\"http://example.com/myendpoint\")\n .setElementConverter(\n (s, _context) -\u003e new HttpSinkRequestEntry(\"POST\", s.getBytes(StandardCharsets.UTF_8)))\n .setProperties(properties)\n .build();\n```\n\nIn Table/SQL API, headers can be set using http sink table DDL. In example below, HTTP request done for `http` table will contain three headers:\n- `Origin`\n- `X-Content-Type-Options`\n- `Content-Type`\n\n```roomsql\nCREATE TABLE http (\n id bigint,\n some_field string\n) WITH (\n 'connector' = 'http-sink',\n 'url' = 'http://example.com/myendpoint',\n 'format' = 'json',\n 'gid.connector.http.sink.header.Origin' = '*',\n 'gid.connector.http.sink.header.X-Content-Type-Options' = 'nosniff',\n 'gid.connector.http.sink.header.Content-Type' = 'application/json'\n)\n```\n\nNote that when using OIDC, it adds an `Authentication` header with the bearer token; this will override \nan existing `Authorization` header specified in configuration.\n\n#### Custom request/response callback\n\n- Http Sink processes responses that it gets from the HTTP endpoint along their respective requests. One can customize the\nbehaviour of the additional stage of processing done by Table API Sink by implementing\n[HttpPostRequestCallback](src/main/java/com/getindata/connectors/http/HttpPostRequestCallback.java) and\n[HttpPostRequestCallbackFactory](src/main/java/com/getindata/connectors/http/HttpPostRequestCallbackFactory.java)\ninterfaces. Custom implementations of `HttpPostRequestCallbackFactory\u003cHttpRequest\u003e` can be registered along other factories in\n`resources/META-INF/services/org.apache.flink.table.factories.Factory` file and then referenced by their identifiers in\nthe HttpSink DDL property field `gid.connector.http.sink.request-callback`.\n\n For example, one can create a class `CustomHttpSinkPostRequestCallbackFactory` with a unique identifier, say `rest-sink-logger`,\nthat implements interface `HttpPostRequestCallbackFactory\u003cHttpRequest\u003e` to create a new instance of a custom callback\n`CustomHttpSinkPostRequestCallback`. This factory can be registered along other factories by appending the fully-qualified name \nof class `CustomHttpSinkPostRequestCallbackFactory` in `resources/META-INF/services/org.apache.flink.table.factories.Factory` file \nand then reference identifier `rest-sink-logger` in the HttpSink DDL property field `gid.connector.http.sink.request-callback`.\n\nA default implementation that logs those pairs as *INFO* level logs using Slf4j ([Slf4jHttpPostRequestCallback](src/main/java/com/getindata/connectors/http/internal/table/sink/Slf4jHttpPostRequestCallback.java)) is provided.\nIf you would like to log more http content (that maybe contain sensitive information), then you can provide a customized version\nof this callback; for inspiration on how to customize in this way, look back in the git history of this file.\n\n- Http Lookup Source processes responses that it gets from the HTTP endpoint along their respective requests. One can customize the\nbehaviour of the additional stage of processing done by Table Function API by implementing\n[HttpPostRequestCallback](src/main/java/com/getindata/connectors/http/HttpPostRequestCallback.java) and\n[HttpPostRequestCallbackFactory](src/main/java/com/getindata/connectors/http/HttpPostRequestCallbackFactory.java)\ninterfaces. \n \n For example, one can create a class `CustomHttpLookupPostRequestCallbackFactory` with a unique identifier, say `rest-lookup-logger`,\nthat implements interface `HttpPostRequestCallbackFactory\u003cHttpLookupSourceRequestEntry\u003e` to create a new instance of a custom callback\n`CustomHttpLookupPostRequestCallback`. This factory can be registered along other factories by appending the fully-qualified name\nof class `CustomHttpLookupPostRequestCallbackFactory` in `resources/META-INF/services/org.apache.flink.table.factories.Factory` file \nand then reference identifier `rest-lookup-logger` in the HTTP lookup DDL property field `gid.connector.http.source.lookup.request-callback`.\n\n A default implementation that logs those pairs as *INFO* level logs using Slf4j\n([Slf4JHttpLookupPostRequestCallback](src/main/java/com/getindata/connectors/http/internal/table/lookup/Slf4JHttpLookupPostRequestCallback.java))\nis provided.\n\n\n## HTTP status code handler\n### Sink table\nYou can configure a list of HTTP status codes that should be treated as errors for HTTP sink table.\nBy default all 400 and 500 response codes will be interpreted as error code.\n\nThis behavior can be changed by using below properties in table definition (DDL) or passing it via `setProperty' method from Sink's builder. The property name are:\n- `gid.connector.http.sink.error.code` used to defined HTTP status code value that should be treated as error for example 404.\nMany status codes can be defined in one value, where each code should be separated with comma, for example:\n`401, 402, 403`. User can use this property also to define a type code mask. In that case, all codes from given HTTP response type will be treated as errors.\nAn example of such a mask would be `3XX, 4XX, 5XX`. In this case, all 300s, 400s and 500s status codes will be treated as errors.\n- `gid.connector.http.sink.error.code.exclude` used to exclude a HTTP code from error list.\n Many status codes can be defined in one value, where each code should be separated with comma, for example:\n `401, 402, 403`. In this example, codes 401, 402 and 403 would not be interpreted as error codes.\n\n### Source table\nThe source table categorizes HTTP responses into three groups based on status codes:\n- Retry codes (`gid.connector.http.source.lookup.retry-codes`):\nResponses in this group indicate a temporary issue (it can be e.g., HTTP 503 Service Unavailable). When such a response is received, the request should be retried.\n- Success codes (`gid.connector.http.source.lookup.success-codes`):\nThese are expected responses that should be processed by table function.\n- Ignored responses (`gid.connector.http.source.lookup.ignored-response-codes`):\nSuccessful response, but its content will be ignored. For example, an HTTP 404 Not Found response is valid and indicates that the requested item does not exist, so its content can be ignored.\n- Error codes: \nAny response code that is not classified as a retry or success code falls into this category. Receiving such a response will result in a job failure.\n\n\nAbove parameters support whitelisting and blacklisting. A sample configuration may look like this:\n`2XX,404,!203` - meaning all codes from group 2XX (200-299), with 404 and without 203 ('!' character). Group blacklisting e.g. !2XX is not supported.\n\nThe same format is used in parameter `gid.connector.http.source.lookup.retry-codes`.\n\nExample with explanation:\n```roomsql\nCREATE TABLE [...]\nWITH (\n [...],\n 'gid.connector.http.source.lookup.success-codes' = '2XX',\n 'gid.connector.http.source.lookup.retry-codes' = '5XX,!501,!505,!506',\n 'gid.connector.http.source.lookup.ignored-response-codes' = '404'\n)\n```\nAll 200s codes and 404 are considered as successful (`success-codes`, `ignored-response-codes`). These responses won't cause retry or job failure. 404 response is listed in `ignored-response-codes` parameter, what means content body will be ignored. Http with 404 code will produce just empty record.\nWhen server returns response with 500s code except 501, 505 and 506 then connector will re-send request based on configuration in `gid.connector.http.source.lookup.retry-strategy` parameters. By default it's fixed-delay with 1 second delay, up to 3 times per request (parameter `lookup.max-retries`). After exceeding max-retries limit the job will fail.\nA response with any other code than specified in params `success-codes` and `retry-codes` e.g. 400, 505, 301 will cause job failure.\n\n\n```roomsql\nCREATE TABLE [...]\nWITH (\n [...],\n 'gid.connector.http.source.lookup.success-codes' = '2XX',\n 'gid.connector.http.source.lookup.retry-codes' = '',\n 'gid.connector.http.source.lookup.ignored-response-codes' = '1XX,3XX,4XX,5XX'\n)\n```\nIn this configuration, all HTTP responses are considered successful because the sets `success-codes` and `ignored-response-codes` together cover all possible status codes. As a result, no retries will be triggered based on HTTP response codes. However, only responses with status code 200 will be parsed and processed by the Flink operator. Responses with status codes in the 1xx, 3xx, 4xx, and 5xx ranges are classified under `ignored-response-codes`.\nNote that retries remain enabled and will still occur on IOException.\nTo disable retries, set `'lookup.max-retries' = '0'`.\n\n\n\n## TLS (more secure replacement for SSL) and mTLS support\n\nBoth Http Sink and Lookup Source connectors support HTTPS communication using TLS 1.2 and mTLS.\nTo enable Https communication simply use `https` protocol in endpoint's URL.\n\nTo specify certificate(s) to be used by the server, use `gid.connector.http.security.cert.server` connector property;\nthe value is a comma separated list of paths to certificate(s), for example you can use your organization's CA\nRoot certificate, or a self-signed certificate.\n\nNote that if there are no security properties for a `https` url then, the JVMs default certificates are\nused - allowing use of globally recognized CAs without the need for configuration.\n\nYou can also configure the connector to use mTLS. For this simply use `gid.connector.http.security.cert.client`\nand `gid.connector.http.security.key.client` connector properties to specify paths to the certificate and\nprivate key. The key MUST be in `PCKS8` format. Both PEM and DER keys are\nallowed.\n\nAll properties can be set via Sink's builder `.setProperty(...)` method or through Sink and Source table DDL.\n\nFor non production environments it is sometimes necessary to use Https connection and accept all certificates.\nIn this special case, you can configure connector to trust all certificates without adding them to keystore.\nTo enable this option use `gid.connector.http.security.cert.server.allowSelfSigned` property setting its value to `true`.\n\n## Basic Authentication\nThe connector supports Basic Authentication using a HTTP `Authorization` header.\nThe header value can be set via properties, similarly as for other headers. The connector converts the passed value to Base64 and uses it for the request.\nIf the used value starts with the prefix `Basic`, or `gid.connector.http.source.lookup.use-raw-authorization-header`\nis set to `'true'`, it will be used as header value as is, without any extra modification.\n\n## OIDC Bearer Authentication\nThe connector supports Bearer Authentication using a HTTP `Authorization` header. The [OAuth 2.0 rcf](https://datatracker.ietf.org/doc/html/rfc6749) mentions [Obtaining Authorization](https://datatracker.ietf.org/doc/html/rfc6749#section-4)\nand an authorization grant. OIDC makes use of this [authorisation grant](https://datatracker.ietf.org/doc/html/rfc6749#section-1.3) in a [Token Request](https://openid.net/specs/openid-connect-core-1_0.html#TokenRequest) by including a [OAuth grant type](https://oauth.net/2/grant-types/) and associated properties, the response is the [token response](https://openid.net/specs/openid-connect-core-1_0.html#TokenResponse). \n\nIf you want to use this authorization then you should supply the `Token Request` body in `application/x-www-form-urlencoded` encoding\nin configuration property `gid.connector.http.security.oidc.token.request`. See [grant extension](https://datatracker.ietf.org/doc/html/rfc6749#section-4.5) for\nan example of a customised grant type token request. The supplied `token request` will be issued to the\n[token end point](https://datatracker.ietf.org/doc/html/rfc6749#section-3.2), whose url should be supplied in configuration property \n`gid.connector.http.security.oidc.token.endpoint.url`. The returned `access token` is then cached and used for subsequent requests; if the token has expired then\n a new one is requested. There is a property `gid.connector.http.security.oidc.token.expiry.reduction`, that defaults to 1 second; new tokens will\nbe requested if the current time is later than the cached token expiry time minus `gid.connector.http.security.oidc.token.expiry.reduction`.\n\n## Logging the http content\nDebug level logging has been added for class `com.getindata.connectors.http.internal.HttpLogger`. To enable this, alter the log4j properties.\nThis logging puts out log entries for the HTTP requests and responses. This can be useful for diagnostics to confirm that HTTP requests have been issued and what\nthat HTTP responses or an exception has occurred (for example connection Refused).\n\nLogging HTTP may not be appropriate for production systems; where sensitive information is not allowed into the logs. But in development environments it is useful\nto be able to see HTTP content. Sensitive information can occur in the headers for example authentication tokens and passwords. Also the HTTP request and response bodies\ncould sensitive. The default minimal logging should be used in production. For development, you can specify config option `gid.connector.http.logging.level`.\nThis dictates the amount of content that debug logging will show around HTTP calls; the valid values are:\n\n| log level | Request method | URI | HTTP Body | Response status code | Headers |\n|-------------|----------------|-----|-----------|----------------------|---------|\n| MIN | Y | Y | N | Y | N |\n| REQRESPONSE | Y | Y | Y | Y | N |\n| MAX | Y | Y | Y | Y | Y |\n \nNotes:\n- you can customize what is traced for lookups using the `gid.connector.http.source.lookup.request-callback`. \n- where there is an N in the table the output is obfuscated.\n\n### Restrictions at this time\n* No authentication is applied to the token request. \n* The processing does not use the refresh token if it present. \n\n## Table API Connector Options\n### HTTP TableLookup Source\n\n| Option | Required | Description/Value |\n|--------------------------------------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| connector | required | The Value should be set to _rest-lookup_ |\n| format | required | Flink's format name that should be used to decode REST response, Use `json` for a typical REST endpoint. |\n| url | required | The base URL that should be use for GET requests. For example _http://localhost:8080/client_ |\n| asyncPolling | optional | true/false - determines whether Async Polling should be used. Mechanism is based on Flink's Async I/O. |\n| gid.connector.http.logging.level | optional | Logging levels for HTTP content. Valid values are `MIN` (the default), `REQRESPONSE` and `MAX`. |\n| lookup-method | optional | GET/POST/PUT (and any other) - determines what REST method should be used for lookup REST query. If not specified, `GET` method will be used. |\n| lookup.cache | optional | Enum possible values: `NONE`, `PARTIAL`. The cache strategy for the lookup table. Currently supports `NONE` (no caching) and `PARTIAL` (caching entries on lookup operation in external API). |\n| lookup.partial-cache.max-rows | optional | The max number of rows of lookup cache, over this value, the oldest rows will be expired. `lookup.cache` must be set to `PARTIAL` to use this option. See the following \u003ca href=\"#lookup-cache\"\u003eLookup Cache\u003c/a\u003e section for more details. |\n| lookup.partial-cache.expire-after-write | optional | The max time to live for each rows in lookup cache after writing into the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following \u003ca href=\"#lookup-cache\"\u003eLookup Cache\u003c/a\u003e section for more details. |\n| lookup.partial-cache.expire-after-access | optional | The max time to live for each rows in lookup cache after accessing the entry in the cache. Specify as a [Duration](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/config/#duration). `lookup.cache` must be set to `PARTIAL` to use this option. See the following \u003ca href=\"#lookup-cache\"\u003eLookup Cache\u003c/a\u003e section for more details. |\n| lookup.partial-cache.cache-missing-key | optional | This is a boolean that defaults to true. Whether to store an empty value into the cache if the lookup key doesn't match any rows in the table. `lookup.cache` must be set to `PARTIAL` to use this option. See the following \u003ca href=\"#lookup-cache\"\u003eLookup Cache\u003c/a\u003e section for more details. |\n| lookup.max-retries | optional | The max retry times if the lookup failed; default is 3. See the following \u003ca href=\"#lookup-cache\"\u003eLookup Cache\u003c/a\u003e section for more detail. Set value 0 to disable retries. |\n| gid.connector.http.lookup.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Source, separated with comma. |\n| gid.connector.http.lookup.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.lookup.error.code` list, separated with comma. |\n| gid.connector.http.security.cert.server | optional | Comma separated paths to trusted HTTP server certificates that should be added to the connectors trust store. |\n| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |\n| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |\n| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |\n| gid.connector.http.security.oidc.token.request | optional | OIDC `Token Request` body in `application/x-www-form-urlencoded` encoding |\n| gid.connector.http.security.oidc.token.endpoint.url | optional | OIDC `Token Endpoint` url, to which the token request will be issued |\n| gid.connector.http.security.oidc.token.expiry.reduction | optional | OIDC tokens will be requested if the current time is later than the cached token expiry time minus this value. |\n| gid.connector.http.source.lookup.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |\n| gid.connector.http.source.lookup.http-version | optional | Version of HTTP to use for lookup http requests. The valid values are HTTP_1_1 and HTTP_2, which specify HTTP 1.1 or 2 respectively. This option may be required as HTTP_1_1, if the endpoint is HTTP 1.1, because some http 1.1 endpoints reject HTTP Version 2 calls, with 'Invalid HTTP request received' and 'HTTP/2 upgrade not supported'. |\n| gid.connector.http.source.lookup.request.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 8 threads will be used. |\n| gid.connector.http.source.lookup.response.thread-pool.size | optional | Sets the size of pool thread for HTTP lookup response processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 4 threads will be used. |\n| gid.connector.http.source.lookup.use-raw-authorization-header | optional | If set to `'true'`, uses the raw value set for the `Authorization` header, without transformation for Basic Authentication (base64, addition of \"Basic \" prefix). If not specified, defaults to `'false'`. |\n| gid.connector.http.source.lookup.request-callback | optional | Specify which `HttpLookupPostRequestCallback` implementation to use. By default, it is set to `slf4j-lookup-logger` corresponding to `Slf4jHttpLookupPostRequestCallback`. |\n| gid.connector.http.source.lookup.connection.timeout | optional | Source table connection timeout. Default - no value. |\n| gid.connector.http.source.lookup.success-codes | optional | Comma separated http codes considered as success response. Use [1-5]XX for groups and '!' character for excluding. |\n| gid.connector.http.source.lookup.retry-codes | optional | Comma separated http codes considered as transient errors. Use [1-5]XX for groups and '!' character for excluding. |\n| gid.connector.http.source.lookup.ignored-response-codes | optional | Comma separated http codes. Content for these responses will be ignored. Use [1-5]XX for groups and '!' character for excluding. Ignored responses togater with `gid.connector.http.source.lookup.success-codes` are considered as successful. |\n| gid.connector.http.source.lookup.retry-strategy.type | optional | Auto retry strategy type: fixed-delay (default) or exponential-delay. |\n| gid.connector.http.source.lookup.retry-strategy.fixed-delay.delay | optional | Fixed-delay interval between retries. Default 1 second. Use with`lookup.max-retries` parameter. |\n| gid.connector.http.source.lookup.retry-strategy.exponential-delay.initial-backoff | optional | Exponential-delay initial delay. Default 1 second. |\n| gid.connector.http.source.lookup.retry-strategy.exponential-delay.max-backoff | optional | Exponential-delay maximum delay. Default 1 minute. Use with `lookup.max-retries` parameter. |\n| gid.connector.http.source.lookup.retry-strategy.exponential-delay.backoff-multiplier | optional | Exponential-delay multiplier. Default value 1.5 |\n| gid.connector.http.source.lookup.continue-on-error | optional | When true, the flow will continue on errors, returning row content. When false (the default) the job ends on errors. |\n| gid.connector.http.source.lookup.proxy.host | optional | Specify the hostname of the proxy. |\n| gid.connector.http.source.lookup.proxy.port | optional | Specify the port of the proxy. |\n| gid.connector.http.source.lookup.proxy.username | optional | Specify the username used for proxy authentication. |\n| gid.connector.http.source.lookup.proxy.password | optional | Specify the password used for proxy authentication. |\n| gid.connector.http.request.query-param-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to query parameters. The parameters are separated by semicolons, such as `param1;param2`. | \n| gid.connector.http.request.body-fields | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The names of the fields that will be mapped to the body. The parameters are separated by semicolons, such as `param1;param2`. | |\n| gid.connector.http.request.url-map | optional | Used for the `GenericJsonAndUrlQueryCreator` query creator. The map of insert names to column names used as url segments. Parses a string as a map of strings. For example if there are table columns called `customerId` and `orderId`, then specifying value `customerId:cid1,orderID:oid` and a url of https://myendpoint/customers/{cid}/orders/{oid} will mean that the url used for the lookup query will dynamically pickup the values for `customerId`, `orderId` and use them in the url. The expected format of the map is: `key1:value1,key2:value2`. |\n\n### HTTP Sink\n\n| Option | Required | Description/Value |\n|---------------------------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| connector | required | Specify what connector to use. For HTTP Sink it should be set to _'http-sink'_. |\n| format | required | Specify what format to use. |\n| url | required | The base URL that should be use for HTTP requests. For example _http://localhost:8080/client_. |\n| gid.connector.http.logging.level | optional | Logging levels for HTTP content. Valid values are `MIN` (the default), `REQRESPONSE` and `MAX`. |\n| insert-method | optional | Specify which HTTP method to use in the request. The value should be set either to `POST` or `PUT`. |\n| sink.batch.max-size | optional | Maximum number of elements that may be passed in a batch to be written downstream. |\n| sink.requests.max-inflight | optional | The maximum number of in flight requests that may exist, if any more in flight requests need to be initiated once the maximum has been reached, then it will be blocked until some have completed. |\n| sink.requests.max-buffered | optional | Maximum number of buffered records before applying backpressure. |\n| sink.flush-buffer.size | optional | The maximum size of a batch of entries that may be sent to the HTTP endpoint measured in bytes. |\n| sink.flush-buffer.timeout | optional | Threshold time in milliseconds for an element to be in a buffer before being flushed. |\n| gid.connector.http.sink.request-callback | optional | Specify which `HttpPostRequestCallback` implementation to use. By default, it is set to `slf4j-logger` corresponding to `Slf4jHttpPostRequestCallback`. |\n| gid.connector.http.sink.error.code | optional | List of HTTP status codes that should be treated as errors by HTTP Sink, separated with comma. |\n| gid.connector.http.sink.error.code.exclude | optional | List of HTTP status codes that should be excluded from the `gid.connector.http.sink.error.code` list, separated with comma. |\n| gid.connector.http.security.cert.server | optional | Path to trusted HTTP server certificate that should be add to connectors key store. More than one path can be specified using `,` as path delimiter. |\n| gid.connector.http.security.cert.client | optional | Path to trusted certificate that should be used by connector's HTTP client for mTLS communication. |\n| gid.connector.http.security.key.client | optional | Path to trusted private key that should be used by connector's HTTP client for mTLS communication. |\n| gid.connector.http.security.cert.server.allowSelfSigned | optional | Accept untrusted certificates for TLS communication. |\n| gid.connector.http.sink.request.timeout | optional | Sets HTTP request timeout in seconds. If not specified, the default value of 30 seconds will be used. |\n| gid.connector.http.sink.writer.thread-pool.size | optional | Sets the size of pool thread for HTTP Sink request processing. Increasing this value would mean that more concurrent requests can be processed in the same time. If not specified, the default value of 1 thread will be used. |\n| gid.connector.http.sink.writer.request.mode | optional | Sets Http Sink request submission mode. Two modes are available to select, `single` and `batch` which is the default mode if option is not specified. |\n| gid.connector.http.sink.request.batch.size | optional | Applicable only for `gid.connector.http.sink.writer.request.mode = batch`. Sets number of individual events/requests that will be submitted as one HTTP request by HTTP sink. The default value is 500 which is same as HTTP Sink `maxBatchSize` |\n\n\n## Lookup Cache \nThe HTTP Client connector can be used in lookup join as a lookup source (also known as a dimension table). \n\nBy default, the lookup cache is not enabled. You can enable it by setting `lookup.cache` to `PARTIAL`.\nThe scope of the cache is per job, so long-running jobs can benefit from this caching.\n\nThe lookup cache is used to improve the performance of temporal joins. By default, the lookup cache is not enabled,\nso all the API requests are sent on the network. When the lookup cache is enabled, Flink looks in the cache first,\nand only sends requests on the network when there is no cached value, then the cache is updated with the returned rows.\nThe oldest rows in this cache are expired when the cache hits the max cached rows `lookup.partial-cache.max-rows`\nor when the row exceeds the max time to live specified by `lookup.partial-cache.expire-after-write`\nor `lookup.partial-cache.expire-after-access`.\n\nBy default, flink caches the empty query result for the primary key. You can toggle this behaviour by setting\n`lookup.partial-cache.cache-missing-key` to false.\n\n\n## Build and deployment\nTo build the project locally you need to have `maven 3` and Java 11+. \u003c/br\u003e\n\nProject build command: `mvn package`. \u003c/br\u003e\nDetailed test report can be found under `target/site/jacoco/index.xml`.\n\n## Demo application\n**Note**: This demo works only for Flink-1.15x.\n\nYou can test this connector using simple mock http server provided with this repository and Flink SQL-client. \nThe mock server can be started from IDE (currently only this way) by running `HttpStubApp::main` method. \nIt will start HTTP server listening on `http://localhost:8080/client`\n\nSteps to follow:\n- Run Mock HTTP server from `HttpStubApp::main` method.\n- Start your Flink cluster, for example as described under https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/try-flink/local_installation/\n- Start Flink SQL Client [6] by calling: `./bin/sql-client.sh -j flink-http-connector-1.0-SNAPSHOT.jar`\n- Execute SQL statements:\nCreate Data Stream source Table:\n```roomsql\nCREATE TABLE Orders (id STRING, id2 STRING, proc_time AS PROCTIME()\n) WITH (\n'connector' = 'datagen', \n'rows-per-second' = '1', \n'fields.id.kind' = 'sequence', \n'fields.id.start' = '1', \n'fields.id.end' = '120', \n'fields.id2.kind' = 'sequence', \n'fields.id2.start' = '2', \n'fields.id2.end' = '120'\n);\n```\n\nCreate Http Connector Lookup Table:\n```roomsql\nCREATE TABLE Customers (\n\tid STRING,\n\tid2 STRING,\n\tmsg STRING,\n\tuuid STRING,\n\tdetails ROW\u003c\n\t isActive BOOLEAN,\n\t nestedDetails ROW\u003c\n\t balance STRING\n\t \u003e\n\t\u003e\n) WITH (\n'connector' = 'rest-lookup',\n'format' = 'json',\n'url' = 'http://localhost:8080/client', \n'asyncPolling' = 'true'\n);\n```\n\nSubmit SQL Select query to join both tables:\n```roomsql\nSELECT o.id, o.id2, c.msg, c.uuid, c.isActive, c.balance FROM Orders AS o JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c ON o.id = c.id AND o.id2 = c.id2;\n```\n\nAs a result, you should see a table with joined records like so:\n![join-result](docs/JoinTable.PNG)\n\nThe `msg` column shows parameters used with REST call for given JOIN record.\n\n## Implementation\n### HTTP Source\nImplementation of an HTTP source connector is based on Flink's `TableFunction` and `AsyncTableFunction` classes. \nTo be more specific we are using a `LookupTableSource`. Unfortunately Flink's new unified source interface [2] cannot be used for this type of source.\nIssue was discussed on Flink's user mailing list - https://lists.apache.org/thread/tx2w1m15zt5qnvt924mmbvr7s8rlyjmw\n\nImplementation of an HTTP Sink is based on Flink's `AsyncSinkBase` introduced in Flink 1.15 [3, 4].\n\n#### Http Response to Table schema mapping\nThe mapping from Http Json Response to SQL table schema is done via Flink's Json Format [5].\n\n## Breaking changes\n- Version 0.10\n - Http Sink submission mode changed from single to batch. From now, body of HTTP POUT/POST request will contain a Json array.\n - Changed API for public HttpSink builder. The `setHttpPostRequestCallback` expects a `PostRequestCallback`\n of generic type [HttpRequest](src/main/java/com/getindata/connectors/http/internal/sink/httpclient/HttpRequest.java)\n instead `HttpSinkRequestEntry`.\n- Version 0.20\n - Http source table parameters: `gid.connector.http.source.lookup.error.code` and `gid.connector.http.source.lookup.error.code.exclude` were removed. These parameters described http status codes which was silently ignored by source lookup table (logged only). it's not recommended to ignore all error response but it's still possible. To do this set all codes as success: `'gid.connector.http.source.lookup.success-codes' = '2XX'` with ignore body from the others responses than 200s: `'gid.connector.http.source.lookup.ignored-response-codes' = '1XX,3XX,4XX,5XX'`. You can still exclude some error codes marking it as transition errors - `gid.connector.http.source.lookup.retry-codes`. Retry-codes have to be excluded from both `success-codes` and `ignored-response-codes`.\n - Added dependency io.github.resilience4j:resilience4j-retry\n\n## TODO\n\n### HTTP TableLookup Source\n- Check other `//TODO`'s.\n\n### HTTP Sink\n- Make `HttpSink` retry the failed requests. Currently, it does not retry those at all, only adds their count to the `numRecordsSendErrors` metric. It should be thoroughly thought over how to do it efficiently and then implemented.\n\n### \n[1] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/sql/queries/joins/#lookup-join\n\u003c/br\u003e\n[2] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/sources/\n\u003c/br\u003e\n[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink\n\u003c/br\u003e\n[4] https://nightlies.apache.org/flink/flink-docs-release-1.15/api/java/org/apache/flink/connector/base/sink/AsyncSinkBase.html\n\u003c/br\u003e\n[5] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/json/\n\u003c/br\u003e\n[6] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/\n\u003c/br\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fflink-http-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetindata%2Fflink-http-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fflink-http-connector/lists"}