https://github.com/conduitio-labs/conduit-connector-http
Conduit connector for HTTP
https://github.com/conduitio-labs/conduit-connector-http
conduit go golang http http-client
Last synced: 3 months ago
JSON representation
Conduit connector for HTTP
- Host: GitHub
- URL: https://github.com/conduitio-labs/conduit-connector-http
- Owner: conduitio-labs
- License: apache-2.0
- Created: 2023-05-09T10:59:05.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-12T16:25:19.000Z (about 1 year ago)
- Last Synced: 2024-04-12T23:10:05.793Z (about 1 year ago)
- Topics: conduit, go, golang, http, http-client
- Language: Go
- Homepage:
- Size: 122 KB
- Stars: 0
- Watchers: 9
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
README
# Conduit Connector for HTTP
The HTTP connector is a [Conduit](https://github.com/ConduitIO/conduit) plugin. It provides both, a source
and a destination HTTP connectors.
## How to build?
Run `make build` to build the connector's binary.
## Testing
Run `make test` to run all the unit tests.
## Source
The HTTP source connector pulls data from the HTTP URL every `pollingPeriod`, the source adds the `params` and `headers`
to the request, and sends it to the URL with the specified `method` from the `Configuration`. The returned data is
used to create an openCDC record and return it.
Note: when using the `OPTIONS` method, the resulted options will be added to the record's metadata.
### Configuration
name
description
required
default value
example
url
HTTP URL to send requests to.
true
https://example.com/api/v1
method
HTTP method to use in the request, supported methods are (GET,HEAD,OPTIONS).
false
GET
POST
headers
HTTP headers to use in the request, comma separated list of : separated pairs.
false
Authorization:Bearer TOKEN_VALUE,Content-Type:application/xml
params.*
parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1".
false
params.query="foobar"
pollingperiod
how often the connector will get data from the url, formatted as a time.Duration.
false
"5m"
"5m"
script.parseResponse
The path to a .js file containing the code to parse the response.
The signature of the function needs to be:
function parseResponse(bytes)
where bytes is the original response's raw bytes (i.e. unparsed).
The function needs to return a Response object.
false
/path/to/get_request_data.js
An example script can be found in test/get_request_data.js
script.getRequestData
The path to a .js file containing the code to prepare the request data.
The signature of the function needs to be:
function getRequestData(cfg, previousResponse, position)
where:
-
cfg(a map) is the connector configuration -
previousResponse(a map) contains data from the previous response (if any), returned byparseResponse -
position(a byte array) contains the starting position of the connector.
The function needs to return a Request object.
false
/path/to/parse_response.js An example script can be found in
test/parse_response.js
## Destination
The HTTP destination connector pushes data from upstream resources to an HTTP URL via Conduit. the destination adds the
`params` and `headers` to the request, and sends it to the URL with the specified `method` from the `Configuration`.
Note: The request `Body` that will be sent is the value under `record.Payload.After`, if you want to change the format
of that or manipulate the field in any way, please check our [Builtin Processors Docs](https://conduit.io/docs/processors/builtin/)
, or check [Standalone Processors Docs](https://conduit.io/docs/processors/standalone/) if you'd like to build your own processor .
### Configuration
| name | description | required | default value |
|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|---------------|
| `url` | Is a Go template expression for the URL used in the HTTP request, using Go [templates](https://pkg.go.dev/text/template). The value provided to the template is [opencdc.Record](https://conduit.io/docs/features/opencdc-record), so the template has access to all its fields (e.g. .Position, .Key, .Metadata, and so on). We also inject all template functions provided by [sprig](https://masterminds.github.io/sprig/) to make it easier to write templates. | true | |
| `method` | Http method to use in the request, supported methods are (`POST`,`PUT`,`DELETE`,`PATCH`). | false | `POST` |
| `headers` | Http headers to use in the request, comma separated list of : separated pairs. | false | |
| `params.*` | parameters to use in the request, use params.* as the config key and specify its value, ex: set "params.id" as "1". | false | |