{"id":20087250,"url":"https://github.com/conduitio-labs/conduit-connector-http","last_synced_at":"2025-09-04T00:41:11.556Z","repository":{"id":163055071,"uuid":"638230414","full_name":"conduitio-labs/conduit-connector-http","owner":"conduitio-labs","description":"Conduit connector for HTTP","archived":false,"fork":false,"pushed_at":"2025-08-19T20:51:49.000Z","size":445,"stargazers_count":1,"open_issues_count":1,"forks_count":2,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-08-19T22:28:57.950Z","etag":null,"topics":["conduit","go","golang","http","http-client"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conduitio-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-05-09T10:59:05.000Z","updated_at":"2025-08-19T20:51:52.000Z","dependencies_parsed_at":"2024-01-08T20:02:28.584Z","dependency_job_id":"993830e4-35e6-4684-b037-b6296b8022ae","html_url":"https://github.com/conduitio-labs/conduit-connector-http","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":"ConduitIO/conduit-connector-template","purl":"pkg:github/conduitio-labs/conduit-connector-http","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduitio-labs%2Fconduit-connector-http","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduitio-labs%2Fconduit-connector-http/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduitio-labs%2Fconduit-connector-http/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduitio-labs%2Fconduit-connector-http/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conduitio-labs","download_url":"https://codeload.github.com/conduitio-labs/conduit-connector-http/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduitio-labs%2Fconduit-connector-http/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273534270,"owners_count":25122636,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-03T02:00:09.631Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["conduit","go","golang","http","http-client"],"created_at":"2024-11-13T16:04:43.339Z","updated_at":"2025-09-04T00:41:11.529Z","avatar_url":"https://github.com/conduitio-labs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduit Connector for HTTP\n\n\u003c!-- readmegen:description --\u003e\nConduit HTTP source and destination connectors, they connect to an HTTP URL and send HTTP requests.\u003c!-- /readmegen:description --\u003e\n\n## How to build?\nRun `make build` to build the connector's binary.\n\n## Testing\nRun `make test` to run all the unit tests. \n\n## Source\nThe HTTP source connector pulls data from the HTTP URL every `pollingPeriod`, the source adds the `params` and `headers`\nto the request, and sends it to the URL with the specified `method` from the `Configuration`. The returned data is\nused to create an openCDC record and return it.\n\nNote: when using the `OPTIONS` method, the resulted options will be added to the record's metadata.\n\n### Configuration\n\n\u003c!-- readmegen:source.parameters.yaml --\u003e\n```yaml\nversion: 2.2\npipelines:\n  - id: example\n    status: running\n    connectors:\n      - id: example\n        plugin: \"http\"\n        settings:\n          # Http url to send requests to\n          # Type: string\n          # Required: yes\n          url: \"\"\n          # Http headers to use in the request, comma separated list of :\n          # separated pairs\n          # Type: string\n          # Required: no\n          headers: \"\"\n          # HTTP method to use in the request\n          # Type: string\n          # Required: no\n          method: \"GET\"\n          # parameters to use in the request, use params.* as the config key and\n          # specify its value, ex: set \"params.id\" as \"1\".\n          # Type: string\n          # Required: no\n          params.*: \"\"\n          # how often the connector will get data from the url\n          # Type: duration\n          # Required: no\n          pollingPeriod: \"5m\"\n          # The path to a .js file containing the code to prepare the request\n          # data. The signature of the function needs to be: `function\n          # getRequestData(cfg, previousResponse, position)` where: * `cfg` (a\n          # map) is the connector configuration * `previousResponse` (a map)\n          # contains data from the previous response (if any), returned by\n          # `parseResponse` * `position` (a byte array) contains the starting\n          # position of the connector. The function needs to return a Request\n          # object.\n          # Type: string\n          # Required: no\n          script.getRequestData: \"\"\n          # The path to a .js file containing the code to parse the response.\n          # The signature of the function needs to be: `function\n          # parseResponse(bytes)` where `bytes` are the original response's raw\n          # bytes (i.e. unparsed). The response should be a Response object.\n          # Type: string\n          # Required: no\n          script.parseResponse: \"\"\n          # ValidateConnection sends a HEAD request when opening the connector\n          # to check if the connection works.\n          # Type: bool\n          # Required: no\n          validateConnection: \"true\"\n          # Maximum delay before an incomplete batch is read from the source.\n          # Type: duration\n          # Required: no\n          sdk.batch.delay: \"0\"\n          # Maximum size of batch before it gets read from the source.\n          # Type: int\n          # Required: no\n          sdk.batch.size: \"0\"\n          # Specifies whether to use a schema context name. If set to false, no\n          # schema context name will be used, and schemas will be saved with the\n          # subject name specified in the connector (not safe because of name\n          # conflicts).\n          # Type: bool\n          # Required: no\n          sdk.schema.context.enabled: \"true\"\n          # Schema context name to be used. Used as a prefix for all schema\n          # subject names. If empty, defaults to the connector ID.\n          # Type: string\n          # Required: no\n          sdk.schema.context.name: \"\"\n          # Whether to extract and encode the record key with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.key.enabled: \"true\"\n          # The subject of the key schema. If the record metadata contains the\n          # field \"opencdc.collection\" it is prepended to the subject name and\n          # separated with a dot.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.key.subject: \"key\"\n          # Whether to extract and encode the record payload with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.payload.enabled: \"true\"\n          # The subject of the payload schema. If the record metadata contains\n          # the field \"opencdc.collection\" it is prepended to the subject name\n          # and separated with a dot.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.payload.subject: \"payload\"\n          # The type of the payload schema.\n          # Type: string\n          # Required: no\n          sdk.schema.extract.type: \"avro\"\n```\n\u003c!-- /readmegen:source.parameters.yaml --\u003e\n\n## Destination\n\nThe HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the\n`params` and `headers` to the request, and sends it to the URL with the specified `method` from the `Configuration`. \n\nNote: The request `Body` that will be sent is the value under `record.Payload.After`, if you want to change the format\nof that or manipulate the field in any way, please check our [Builtin Processors Docs](https://conduit.io/docs/processors/builtin/)\n, or check [Standalone Processors Docs](https://conduit.io/docs/processors/standalone/) if you'd like to build your own processor .\n\n### Configuration\n\n\u003c!-- readmegen:destination.parameters.yaml --\u003e\n```yaml\nversion: 2.2\npipelines:\n  - id: example\n    status: running\n    connectors:\n      - id: example\n        plugin: \"http\"\n        settings:\n          # URL is a Go template expression for the URL used in the HTTP\n          # request, using Go [templates](https://pkg.go.dev/text/template). The\n          # value provided to the template is\n          # [opencdc.Record](https://conduit.io/docs/using/opencdc-record), so\n          # the template has access to all its fields (e.g. .Position, .Key,\n          # .Metadata, and so on). We also inject all template functions\n          # provided by [sprig](https://masterminds.github.io/sprig/) to make it\n          # easier to write templates.\n          # Type: string\n          # Required: yes\n          url: \"\"\n          # Http headers to use in the request, comma separated list of :\n          # separated pairs\n          # Type: string\n          # Required: no\n          headers: \"\"\n          # HTTP method to use in the request\n          # Type: string\n          # Required: no\n          method: \"POST\"\n          # parameters to use in the request, use params.* as the config key and\n          # specify its value, ex: set \"params.id\" as \"1\".\n          # Type: string\n          # Required: no\n          params.*: \"\"\n          # ValidateConnection sends a HEAD request when opening the connector\n          # to check if the connection works.\n          # Type: bool\n          # Required: no\n          validateConnection: \"true\"\n          # Maximum delay before an incomplete batch is written to the\n          # destination.\n          # Type: duration\n          # Required: no\n          sdk.batch.delay: \"0\"\n          # Maximum size of batch before it gets written to the destination.\n          # Type: int\n          # Required: no\n          sdk.batch.size: \"0\"\n          # Allow bursts of at most X records (0 or less means that bursts are\n          # not limited). Only takes effect if a rate limit per second is set.\n          # Note that if `sdk.batch.size` is bigger than `sdk.rate.burst`, the\n          # effective batch size will be equal to `sdk.rate.burst`.\n          # Type: int\n          # Required: no\n          sdk.rate.burst: \"0\"\n          # Maximum number of records written per second (0 means no rate\n          # limit).\n          # Type: float\n          # Required: no\n          sdk.rate.perSecond: \"0\"\n          # The format of the output record. See the Conduit documentation for a\n          # full list of supported formats\n          # (https://conduit.io/docs/using/connectors/configuration-parameters/output-format).\n          # Type: string\n          # Required: no\n          sdk.record.format: \"opencdc/json\"\n          # Options to configure the chosen output record format. Options are\n          # normally key=value pairs separated with comma (e.g.\n          # opt1=val2,opt2=val2), except for the `template` record format, where\n          # options are a Go template.\n          # Type: string\n          # Required: no\n          sdk.record.format.options: \"\"\n          # Whether to extract and decode the record key with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.key.enabled: \"true\"\n          # Whether to extract and decode the record payload with a schema.\n          # Type: bool\n          # Required: no\n          sdk.schema.extract.payload.enabled: \"true\"\n```\n\u003c!-- /readmegen:destination.parameters.yaml --\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio-labs%2Fconduit-connector-http","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduitio-labs%2Fconduit-connector-http","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduitio-labs%2Fconduit-connector-http/lists"}