Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/benmizrahi/kxplore

Kxplore - Kafka Explorer
https://github.com/benmizrahi/kxplore

apache-k docker-container kafka kafka-cluster kafka-message ksql management ngx ngx-bootstrap typescript ui

Last synced: 9 days ago
JSON representation

Kxplore - Kafka Explorer

Host: GitHub
URL: https://github.com/benmizrahi/kxplore
Owner: benmizrahi
License: apache-2.0
Created: 2018-10-01T21:20:54.000Z (about 6 years ago)
Default Branch: master
Last Pushed: 2020-12-04T14:28:46.000Z (almost 4 years ago)
Last Synced: 2023-06-17T20:36:47.988Z (over 1 year ago)
Topics: apache-k, docker-container, kafka, kafka-cluster, kafka-message, ksql, management, ngx, ngx-bootstrap, typescript, ui
Language: TypeScript
Homepage:
Size: 51.8 MB
Stars: 3
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE

Awesome Lists containing this project

README

## ![alt text](https://raw.githubusercontent.com/benmizrahi/Kxplore/master/app/src/assets/k_logo.png)
## Motivation
Our mission is to create a simple and stable tool that can give us the ability to explore data in our Kafka topics - Kxplore containes is a UI layer over Kafka consumer that gives powefull tools over you're Kafka cluster.

### Features Included
- SQL Like syntax for filter messages
- Javascript expression handling to apply transformation over Kafka Messages
- Export messages to JSON, CSV files.
- Charts view over Data stream

### Getting Started

1. Build from source
2. Docker container

#### 1. Build From Source

##### Prerequisite
- Angular-cli v6.0.x / above
- Nodejs v8.11.x / above
- Typescript 2.9.x / above

##### Building The Source Files:
To get started clone the repository to you're own envierment:

`
git clone https://github.com/benmizrahi/Kxplore.git
cd Kxplore/
`

Install client & server dependencies - the project contains both client (Angular 6) and server (Nodejs)

``
npm run build
``

Build Angular App this command uses ng-cli to compile all client assets and copy sources to the public express folder

`
npm run build:prod:app
`

After this command you should see a dist folder on the root project folder - this folder contains all the Angular build sources that the Express app serves.

Complie all server Typescript files:

`
tsc .
`

The server startup.js takes two envierment varaibles:
1. CONFIG_PATH - you're local path to the configuration file (see config explantion)
2. BASE_DOMAIN_URL - the base domain name (in local mode set to localhost:3000).

Running the app execute example:

``
CONFIG_PATH=/path/to/env.json BASE_DOMAIN_URL=http://localhost:3000 node startup.js
``

Now you can browse the app in http://localhost:3000/

#### 2. Docker Container

The Kxplore project can run also via docker container - to run via docker just pull the image to you're local by running the following:

`
docker pull benmizrahi/kxplore
`

Running the container:

``
docker run -p 3000:3000 -e "CONFIG_PATH=/path/to/env.json" -e "BASE_DOMAIN_URL=http://localhost:4000" kxplore
``

Now you can browse the app in http://localhost:3000/

### Mysql

### Configuration File
The Kxplore configuration file is a simple JSON file with arguments, you can set the PATH to the config file via the ENV parameter: CONFIG_PATH - the preferred location is /etc/kxplore/env.json

** IMPORTENT - you must create kxplore db in the traget mysql you defined in the configuration **

Configuration Example:
```

{
"authConfig":{
"SECRET_KEY":"SECRET_KEY",
"googleConfig":{
"clientID": "YOURE-GOOGLE-CLIENT-ID",
"clientSecret": "YOURE-GOOGLE-CLIENT-SECRET",
"passReqToCallback" : true
},
"superuser":{
"user":"Admin",
"password":"Admin"
}
},
"mysql":{
"host":"127.0.0.1", //MYSQL host
"port":"3307", //MYSQL port
"user":"kxplore",
"password":"kxplore",
"database":"kxplore"
},
"kafkaConfig": {
"messagePool":100, //how match data will be pulled from kafka each interval
"minMessage":20, //min messages that triggers the consumer pool interval
"intervalMs": "1000", // pool interval in ms
"poolCount":10
},
// If planning to scale up the server - you need to configure redis to comunicate between each node
"redis-config":{
"host":"localhost",
"port":"6379"
}
}

```

### Example For Environment Configuration

When adding an environment to Kxplore you must add configuration of how to communicate with Kafka Borkers.
For Example Add the following config to you environment just replace the zookeeper url:

```
{
"groupId": "kxplore-group",
"properties": {
"fromOffset": "true",
"fetch.max.bytes": "52428800",
"max.poll.records": "20",
"auto.offset.reset": "largest",
"fetch.max.wait.ms": "10000",
"enable.auto.commit": "false",
"session.timeout.ms": "10000",
"max.poll.interval.ms": "10000",
"rebalance.backoff.ms": "10000",
"num.consumer.fetchers": "10",
"rebalance.max.retries": "3",
"auto.commit.interval.ms": "10000",
"queued.max.message.chunks": "50",
"refresh.leader.backoff.ms": "10000",
"partition.assignment.strategy": "roundrobin"
},
"getMetadata": true,
"threadCount": 5,
"zookeeperUrl": "host-of-zookeeper:2181"
}
```
## Images:

### Home Page
![Alt text](images/home_page.png?raw=true "Home Page")

### Stream Events

![Alt text](images/stream.png?raw=true "Home Page")

### Filter Stream Data And Display JSONS

![Alt text](images/filter_stream.png?raw=true "Home Page")

### Future Of Kxplore
A few features will be included soon:

1. Aggregation over stream
2. Define charts that updates every X interval
3. Allow to inject SQL scripts