https://github.com/qdrant/qdrant-kafka
Kafka Sink connector for Qdrant. Stream vector data into a Qdrant collection. Supports named/unnamed dense/sparse vectors.
https://github.com/qdrant/qdrant-kafka
Last synced: 10 months ago
JSON representation
Kafka Sink connector for Qdrant. Stream vector data into a Qdrant collection. Supports named/unnamed dense/sparse vectors.
- Host: GitHub
- URL: https://github.com/qdrant/qdrant-kafka
- Owner: qdrant
- License: apache-2.0
- Created: 2024-06-24T06:31:39.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-27T08:19:10.000Z (over 1 year ago)
- Last Synced: 2025-06-01T17:42:54.559Z (12 months ago)
- Language: Java
- Homepage:
- Size: 93.8 KB
- Stars: 7
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-kafka-connect - qdrant/qdrant-kafka - Qdrant vector database sink connector (Vector Databases / Qdrant)
README
# Qdrant Sink Connector
Use Qdrant as a sink destination in [Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html). Supports streaming dense/sparse/multi vectors into Qdrant collections.
## Usage
> [!IMPORTANT]
> Qdrant collections have to be [created](https://qdrant.tech/documentation/concepts/collections/#create-a-collection) in advance with the appropriate vector dimensions and configurations.
Learn to use the connector with
- [Kafka on Confluent Cloud](https://github.com/qdrant/qdrant-kafka/blob/main/CONFLUENT.md)
- [Self-hosted Kafka](https://github.com/qdrant/qdrant-kafka/blob/main/KAFKA.md)
## Message Formats
This sink connector supports messages with multiple dense/sparse vectors.
_Click each to expand._
Unnamed/Default vector
Reference: [Creating a collection with a default vector](https://qdrant.tech/documentation/concepts/collections/#create-a-collection).
```json
{
"collection_name": "{collection_name}",
"id": 1,
"vector": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8
],
"payload": {
"name": "kafka",
"description": "Kafka is a distributed streaming platform",
"url": "https://kafka.apache.org/"
}
}
```
Named multiple vectors
Reference: [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
```json
{
"collection_name": "{collection_name}",
"id": 1,
"vector": {
"some-dense": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8
],
"some-other-dense": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8
]
},
"payload": {
"name": "kafka",
"description": "Kafka is a distributed streaming platform",
"url": "https://kafka.apache.org/"
}
}
```
Sparse vectors
Reference: [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
```json
{
"collection_name": "{collection_name}",
"id": 1,
"vector": {
"some-sparse": {
"indices": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9
],
"values": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
]
}
},
"payload": {
"name": "kafka",
"description": "Kafka is a distributed streaming platform",
"url": "https://kafka.apache.org/"
}
}
```
Multi-vector
```json
{
"collection_name": "{collection_name}",
"id": 1,
"vector": {
"some-multi": [
[
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
],
[
1.0,
0.9,
0.8,
0.5,
0.4,
0.8,
0.6,
0.4,
0.2,
0.1
]
]
},
"payload": {
"name": "kafka",
"description": "Kafka is a distributed streaming platform",
"url": "https://kafka.apache.org/"
}
}
```
Combination of named dense and sparse vectors
Reference:
- [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
- [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
```json
{
"collection_name": "{collection_name}",
"id": "a10435b5-2a58-427a-a3a0-a5d845b147b7",
"vector": {
"some-other-dense": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8
],
"some-sparse": {
"indices": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9
],
"values": [
0.1,
0.2,
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.9,
1.0
]
}
},
"payload": {
"name": "kafka",
"description": "Kafka is a distributed streaming platform",
"url": "https://kafka.apache.org/"
}
}
```
## LICENSE
Apache 2.0 © [2024](https://github.com/qdrant/qdrant-kafka/blob/main/LICENSE)