Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hoptical/grafana-kafka-datasource
A Grafana datasource plugin for Apache Kafka
https://github.com/hoptical/grafana-kafka-datasource
grafana kafka monitoring
Last synced: about 5 hours ago
JSON representation
A Grafana datasource plugin for Apache Kafka
- Host: GitHub
- URL: https://github.com/hoptical/grafana-kafka-datasource
- Owner: hoptical
- License: apache-2.0
- Created: 2021-10-22T19:40:44.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-09T08:39:46.000Z (over 1 year ago)
- Last Synced: 2024-06-19T05:57:08.589Z (5 months ago)
- Topics: grafana, kafka, monitoring
- Language: TypeScript
- Homepage:
- Size: 4.56 MB
- Stars: 34
- Watchers: 2
- Forks: 12
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Kafka Datasource for Grafana
[](LICENSE)
[](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/ci.yml)
[](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/release.yml)
The Kafka data source plugin allows you to visualize streaming Kafka data from within Grafana.
## Reqirements
- Apache Kafka v0.9+
- Grafana v8.0+
> Note: This is a backend plugin, so the Grafana server should've access to the Kafka broker.
## Getting started
### Installation via grafana-cli tool
Use the grafana-cli tool to install the plugin from the commandline:
```bash
grafana-cli plugins install hamedkarbasi93-kafka-datasource
```
The plugin will be installed into your grafana plugins directory; the default is `/var/lib/grafana/plugins`. [More information on the cli tool](https://grafana.com/docs/grafana/latest/administration/cli/#plugins-commands).
### Installation via zip file
Alternatively, you can manually download the [latest](https://github.com/hoptical/grafana-kafka-datasource/releases/latest) release .zip file and unpack it into your grafana plugins directory; the default is `/var/lib/grafana/plugins`.
## Configure the data source
[Add a data source](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/) by filling in the following fields:
### Basic fields
| Field | Description |
| ----- | -------------------------------------------------- |
| Name | A name for this particular AppDynamics data source |
| Servers | The URL of the Kafka bootstrap servers separated by comma. E.g. `broker1:9092, broker2:9092` |
### Query the Data source
To query the Kafka topic, you have to config the below items in the query editor.
| Field | Description |
| ----- | -------------------------------------------------- |
| Topic | Topic Name |
| Partition | Partition Number |
| Auto offset reset | Starting offset to consume that can be from latest or last 100. |
| Timestamp Mode | Timestamp of the message value to visualize; It can be Now or Message Timestamp
> **Note**: Make sure to enable the `streaming` toggle.
## Known limitations
- The plugin currently does not support any authorization and authentication method.
- The plugin currently does not support TLS.
- Plugin is based on [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go), hence it only supports Linux-based operating systems as discussed in [#6](https://github.com/hoptical/grafana-kafka-datasource/issues/6). However, we're cosidering changing the base package to support all operating systems.
This plugin supports topics publishing very simple JSON formatted messages. Note that only the following structure is supported as of now:
```json
{
"value1": 1.0,
"value2": 2,
"value3": 3.33,
...
}
```
We plan to support more complex JSON data structures, Protobuf and AVRO in the upcoming releases. Contributions are highly encouraged!
## Compiling the data source by yourself
Note: there are build issues with Node v16. Use Node v14. Known working dev environments:
* Ubuntu 20.04 LTS
* node v14.19.3 (npm 6.14.17)
* yarn 1.22.19
* go 1.18
and
* Ubuntu 22.04 LTS
* node v14.20.0 (npm 6.14.17)
* yarn 1.22.19
* go 1.18.1
A data source backend plugin consists of both frontend and backend components.
### Frontend
1. Install dependencies
```bash
yarn install
```
2. Build plugin in development mode or run in watch mode
```bash
yarn dev
```
or
```bash
yarn watch
```
3. Build plugin in production mode
```bash
yarn build
```
### Backend
1. Update [Grafana plugin SDK for Go](https://grafana.com/docs/grafana/latest/developers/plugins/backend/grafana-plugin-sdk-for-go/) dependency to the latest minor version:
```bash
go get -u github.com/grafana/grafana-plugin-sdk-go
go mod tidy
```
2. Build backend plugin binaries for Linux:
```bash
mage build:backend
```
## Contributing
Thank you for considering contributing! If you find an issue or have a better way to do something, feel free to open an issue or a PR.
## License
This repository is open-sourced software licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
## Learn more
- [Build a data source backend plugin tutorial](https://grafana.com/tutorials/build-a-data-source-backend-plugin)
- [Grafana plugin SDK for Go](https://grafana.com/docs/grafana/latest/developers/plugins/backend/grafana-plugin-sdk-for-go/)