https://github.com/jailtonjunior94/kafka-poc
https://github.com/jailtonjunior94/kafka-poc
docker docker-compose golang kafka kafkaconnect
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/jailtonjunior94/kafka-poc
- Owner: JailtonJunior94
- Created: 2023-11-01T18:23:44.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-12-11T23:08:46.000Z (over 1 year ago)
- Last Synced: 2025-01-28T23:25:49.315Z (4 months ago)
- Topics: docker, docker-compose, golang, kafka, kafkaconnect
- Language: Go
- Homepage:
- Size: 121 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Escopo
Criação de um Kafka Connect para enviar notificações com HTTP Sink
[Documentação de Connectors](https://docs.confluent.io/cloud/current/connectors/index.html)
[Documentação do HTTP Sink](https://docs.confluent.io/kafka-connectors/http/current/overview.html)
# Desenho da Solução
# Executando com Docker
- Para executar o projeto local com docker, devemos utilizar os comandos
```
make start
```
- Para parar a execução do projeto
```
make stop
```
# Configurando um novo connector
- Podemos criar um arquivo `.properties` como no exemplo abaixo e fazer upload do mesmo no `control-center`
- Exemplo de arquivo `.properties`
```
name=HttpSink
connector.class=io.confluent.connect.http.HttpSinkConnector
tasks.max=1
value.converter=org.apache.kafka.connect.storage.StringConverter
topics=http-messages
http.api.url=http://api:3000/api/messages
request.method=post
confluent.topic.bootstrap.servers=kafka:9092
confluent.topic.replication.factor=1
reporter.bootstrap.servers=kafka:9092
reporter.result.topic.name=success-responses
reporter.result.topic.replication.factor=1
reporter.error.topic.name=error-responses
reporter.error.topic.replication.factor=1
```
- Podemos também usar a API Rest disponibilizada pelo connectors com os seguintes endpoints
- Listando connectors instalados no Kafka
```
curl --location 'http://localhost:8083/connector-plugins'
```
- Listando connectors criados
```
curl --location 'http://localhost:8083/connectors'
```
- Cadastrando um novo connector
```
curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
"name": "HttpSink",
"config": {
"topics": "http-messages",
"tasks.max": "1",
"connector.class": "io.confluent.connect.http.HttpSinkConnector",
"http.api.url": "http://api:3000/api/messages",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter.schemas.enable": false,
"confluent.topic.bootstrap.servers": "kafka:9092",
"confluent.topic.replication.factor": "1",
"reporter.bootstrap.servers": "kafka:9092",
"reporter.result.topic.name": "success-responses",
"reporter.result.topic.replication.factor": "1",
"reporter.error.topic.name":"error-responses",
"reporter.error.topic.replication.factor":"1"
}
}'
```
- Verificando status do connector cadastrado
```
curl --location 'http://localhost:8083/connectors/HttpSink/status'
```
- Deletando um connector
```
curl --location --request DELETE 'http://localhost:8083/connectors/HttpSink'
```