https://github.com/aiven-open/http-connector-for-apache-kafka
Apache Kafka Connect sink connector for HTTP
https://github.com/aiven-open/http-connector-for-apache-kafka
http kafka kafka-connect
Last synced: about 1 year ago
JSON representation
Apache Kafka Connect sink connector for HTTP
- Host: GitHub
- URL: https://github.com/aiven-open/http-connector-for-apache-kafka
- Owner: Aiven-Open
- License: apache-2.0
- Created: 2019-10-07T12:38:50.000Z (over 6 years ago)
- Default Branch: main
- Last Pushed: 2025-03-01T09:14:32.000Z (over 1 year ago)
- Last Synced: 2025-03-29T13:03:45.611Z (about 1 year ago)
- Topics: http, kafka, kafka-connect
- Language: Java
- Homepage:
- Size: 712 KB
- Stars: 106
- Watchers: 67
- Forks: 49
- Open Issues: 28
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
README
# Aiven's HTTP Sink Connector for Apache Kafka®
This is a sink [Apache Kafka Connect](https://kafka.apache.org/documentation/#connect) connector that sends Kafka records over HTTP.
## Status
We consider the connector **stable** despite the major version is currently 0.
## How it works
The connector uses the POST HTTP method to deliver records.
The connector supports:
- authorization (static, OAuth2);
- setting HTTP headers;
- batching;
- delivery retries.
An example of configuration (file `connect-http-sink.json`):
```json
{
"name": "http-sink-test",
"config": {
"connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
"topics.regex": "domain_a.*",
"http.authorization.type": "none",
"http.url": "http://httpmockserver:1080",
"batching.enabled": true,
"batch.max.size": 2,
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter"
}
}
```
Then deploy the connector configuration against the Kafka Connect REST API with the following command:
```shell
curl -d @config/connect-http-sink.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors
```
See the [configuration options](docs/sink-connector-config-options.rst) for details.
## Development
The connector supports Java 11 or later. It uses [Gradle](https://gradle.org/) for building and automating tasks.
To build a distribution:
```bash
./gradlew clean check distTar distZip
```
(check `build/distributions/` then).
Integration tests can be run with
```bash
./gradlew integrationTest
```
and require [Docker](https://www.docker.com/) installed.
# Formatting
Checkstyle is used for verifying code formatting.
For misc files Spotless is used.
Both are run on Gradle `check` target. Spotless formatting can be done with
```bash
./gradlew spotlessApply
```
## License
This project is licensed under the [Apache License, Version 2.0](LICENSE).
## Trademark
Apache Kafka, Apache Kafka Connect are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.