{"id":24117196,"url":"https://github.com/infinyon/http-source-connector","last_synced_at":"2025-09-18T06:32:07.534Z","repository":{"id":103498538,"uuid":"593248189","full_name":"infinyon/http-source-connector","owner":"infinyon","description":"Official Infinyon HTTP Source Connector ","archived":false,"fork":false,"pushed_at":"2024-12-30T02:53:17.000Z","size":578,"stargazers_count":7,"open_issues_count":7,"forks_count":7,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-01-06T11:14:04.043Z","etag":null,"topics":["connector","fluvio","http","source"],"latest_commit_sha":null,"homepage":"https://www.fluvio.io/connectors/inbound/http/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/infinyon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-25T15:24:48.000Z","updated_at":"2024-11-01T00:41:09.000Z","dependencies_parsed_at":"2023-12-25T01:25:31.684Z","dependency_job_id":"4334d8e0-165b-44ed-a66e-67c60a00c700","html_url":"https://github.com/infinyon/http-source-connector","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Fhttp-source-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Fhttp-source-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Fhttp-source-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/infinyon%2Fhttp-source-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/infinyon","download_url":"https://codeload.github.com/infinyon/http-source-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233456087,"owners_count":18678963,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["connector","fluvio","http","source"],"created_at":"2025-01-11T07:38:31.339Z","updated_at":"2025-09-18T06:32:02.246Z","avatar_url":"https://github.com/infinyon.png","language":"Shell","readme":"# Fluvio HTTP Inbound Connector\n\nRead HTTP Responses given input HTTP request configuration options and produce them\nto Fluvio topics.\n\nThis connector can be configured to operate in three modes.\n\n- [Polling](#usage-example): Unless otherwise specified, the endpoint will be polled periodically, with the polling interval specified by providing the `interval` config option. Each response will be produced as an individual Fluvio record.\n- [Streaming](#streaming-mode): When the `stream` config option is provided, the HTTP response will be processed as a [data stream](https://en.wikipedia.org/wiki/Chunked_transfer_encoding). A record will be produced to Fluvio every time a `delimiter` segment is encountered, which is set to `\\n` by default.\n- [WebSocket](#websocket-mode): When the provided `endpoint` config option is prefixed with `ws://`, a WebSocket connection will be established, and each incoming message will be produced.\n\nSupports HTTP/1.0, HTTP/1.1, HTTP/2.0 protocols.\n\nSee [docs](https://www.fluvio.io/connectors/inbound/http/) here.\nTutorial for [HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pipeline/).\n\n### Configuration\n| Option           | default                    | type            | description                                                                                |\n|:-----------------|:---------------------------|:----------------|:-------------------------------------------------------------------------------------------|\n| interval         | 10s                        | String          | Interval between each HTTP Request. This is in the form of \"1s\", \"10ms\", \"1m\", \"1ns\", etc. |\n| method           | GET                        | String          | GET, POST, PUT, HEAD                                                                       |\n| endpoint         | -                          | String          | HTTP URL endpoint. Use `ws://` for websocket URLs.                                         |\n| headers          | -                          | Array\\\u003cString\\\u003e | Request header(s) \"Key:Value\" pairs                                                        |\n| body             | -                          | String          | Request body e.g. in POST                                                                  |\n| user-agent       | \"fluvio/http-source 0.1.0\" | String          | Request user-agent                                                                         |\n| output_type      | text                       | String          | `text` = UTF-8 String Output, `json` = UTF-8 JSON Serialized String                        |\n| output_parts     | body                       | String          | `body` = body only, `full` = all status, header and body parts                             |\n| stream           | false                      | bool            | Flag to indicate HTTP streaming mode                                                       |\n| delimiter        | '\\n'                       | String          | Delimiter to separate records when producing from an HTTP streaming endpoint               |\n| websocket_config | {}                         | Object          | WebSocket configuration object. See below.                                                 |\n\n#### Record Type Output\n| Matrix                                                      | Output                                  |\n| :---------------------------------------------------------- | :-------------------------------------- |\n| output_type = text (default), output_parts = body (default) | Only the body of the HTTP Response      |\n| output_type = text (default), output_parts = full           | The full HTTP Response                  |\n| output_type = json, output_parts = body (default)           | Only the \"body\" in JSON struct          |\n| output_type = json, output_parts = full                     | HTTP \"status\", \"body\" and \"header\" JSON |\n\n#### WebSocket Configuration\n| Option                | default | type            | description                                                                                                                                                  |\n|:----------------------|:--------|:----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| subscription_messages | []      | Array\\\u003cString\\\u003e | List of messages to send to the server after connection is established.                                                                                      |\n| ping_interval_ms      | 10000   | int             | Interval in milliseconds to send ping messages to the server.                                                                                                |\n| subscription_message  | -       | String          | (deprecated) Message to send to the server after connection is established. If provided with subscription_messages, subscription_message will be sent first. |\n\n### Usage Example\n\nThis is an example of simple connector config file for polling an endpoint:\n\n```yaml\n# config-example.yaml\napiVersion: 0.1.0\nmeta:\n  version: 0.4.3\n  name: cat-facts\n  type: http-source\n  topic: cat-facts\n  create-topic: true\n  secrets:\n    - name: AUTHORIZATION_TOKEN\nhttp:\n  endpoint: \"https://catfact.ninja/fact\"\n  interval: 10s\n  headers:\n    - \"Authorization: token ${{ secrets.AUTHORIZATION_TOKEN }}\"\n    - \"Cache-Control: no-cache\"\n```\n\nThe produced record in Fluvio topic will be:\n```json\n{\n  \"fact\": \"The biggest wildcat today is the Siberian Tiger. It can be more than 12 feet (3.6 m) long (about the size of a small car) and weigh up to 700 pounds (317 kg).\",\n  \"length\": 158\n}\n```\n### Secrets\n\nFluvio HTTP Source Connector supports Secrets in the `endpoint` and in the `headers` parameters:\n\n```yaml\n# config-example.yaml\napiVersion: 0.1.0\nmeta:\n  version: 0.4.3\n  name: cat-facts\n  type: http-source\n  topic: cat-facts\n  create-topic: true\n  secrets:\n    - name: MY_SECRET_URL\n    - name: MY_AUTHORIZATION_HEADER\nhttp:\n endpoint:\n   secret:\n     name: MY_SECRET_URL\n headers:\n  - \"Authorization: ${{ secrets.MY_AUTHORIZATION_HEADER }}\n interval: 10s\n```\n\n\n### Transformations\nFluvio HTTP Source Connector supports [Transformations](https://www.fluvio.io/docs/concepts/transformations-chain/). Records can be modified before sending to Fluvio topic.\n\nThe previous example can be extended to add extra transformations to outgoing records:\n```yaml\n# config-example.yaml\napiVersion: 0.1.0\nmeta:\n  version: 0.4.3\n  name: cat-facts\n  type: http-source\n  topic: cat-facts\n  create-topic: true\nhttp:\n  endpoint: \"https://catfact.ninja/fact\"\n  interval: 10s\ntransforms:\n  - uses: infinyon/jolt@0.1.0\n    with:\n      spec:\n        - operation: default\n          spec:\n            source: \"http-connector\"\n        - operation: remove\n          spec:\n            length: \"\"\n```\nIn this case, additional transformation will be performed before records are sent to Fluvio topic: field `length` will be removed and\nfield `source` with string value `http-connector` will be added.\n\nNow produced records will have a different shape, for example:\n```json\n{\n  \"fact\": \"A cat has more bones than a human; humans have 206, and the cat - 230.\",\n  \"source\": \"http-connector\"\n}\n```\n\nRead more about [JSON to JSON transformations](https://www.fluvio.io/smartmodules/certified/jolt/).\n\n### Streaming Mode\n\nProvide the `stream` configuration option to enable streaming mode with `delimiter` to determine how the incoming records are separated.\n\n```yaml\n# config-example.yaml\napiVersion: 0.1.0\nmeta:\n  version: 0.4.3\n  name: wiki-updates\n  type: http-source\n  topic: wiki-updates\nhttp:\n  endpoint: \"https://stream.wikimedia.org/v2/stream/recentchange\"\n  method: GET\n  stream: true\n  delimiter: \"\\n\\n\"\n```\n\n### Websocket Mode\nConnect to a websocket endpoint using a `ws://` URL. When reading text messages, they are emitted as equivalent records. Binary messages are initially attempted to be converted into strings.\n\n```yaml\n# config-example.yaml\napiVersion: 0.1.0\nmeta:\n  version: 0.4.3\n  name: websocket-connector\n  type: http-source\n  topic: websocket-updates\nhttp:\n  endpoint: ws://websocket.example/websocket\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinyon%2Fhttp-source-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Finfinyon%2Fhttp-source-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Finfinyon%2Fhttp-source-connector/lists"}