Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dateytanmay/event-catalyst

This repository automates the generation and management of an event catalog. Scripts and configurations handle the extraction of event types from Kibana, anonymization of sensitive information, and dynamic updates to reflect changes in eventNames.json, eventPatterns.json, and consumer.json without manual intervention. Simply run main.
https://github.com/dateytanmay/event-catalyst

automation elasticsearch event eventcatalog eventdrivenarchitecture java kafka kibana node npm springboot

Last synced: 17 days ago
JSON representation

Host: GitHub
URL: https://github.com/dateytanmay/event-catalyst
Owner: dateytanmay
Created: 2024-06-25T11:38:22.000Z (7 months ago)
Default Branch: master
Last Pushed: 2024-06-25T11:39:30.000Z (7 months ago)
Last Synced: 2025-01-16T06:16:53.416Z (17 days ago)
Topics: automation, elasticsearch, event, eventcatalog, eventdrivenarchitecture, java, kafka, kibana, node, npm, springboot
Language: JavaScript
Homepage:
Size: 328 KB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md

Awesome Lists containing this project

README

# Event Catalog
=======
## Introduction

This repository contains scripts and configurations for generating and managing an event catalog automatically. As long as you are running the node server main.js any changes made in `eventNames.json`, `eventPatterns.json`, `consumer.json` will automatically reflect the changes in event catalog without the need to recreate events manually.

## Setup Instructions

### 1. Extract Payload from kibana for each eventType
- Extract all event types from the kibana using this query
POST /eventstore/_search
```{
"size": 0,
"aggs": {
"unique_event_types": {
"terms": {
"field": "type",
"size": 1000000
},
"aggs": {
"example_event": {
"top_hits": {
"size": 1
}
}
}
}
}
}
```
and save it in `event-catalog-automation/json/originalPayload.json` file.

### 2. Extract Event Types
- Extract all event types from the `event-catalog-automation/json/originalPayload.json` and save them in `evemt-catalog-automation/json/eventNames.json` using the python script `eventtype_extractor.py`.

### 3. Anonymize Information
- Run the `PIFaker.js` script located at `event-catalog-automation/js` using the command:

This will fake all the information and create a JSON file named `anonymized_events.json` in `event-catalog-automation/json/autogenerated`.

### 3. Add Event Patterns
- Add all event patterns to the `event-catalog-automation/json/eventPatterns.json` file for each service.
- For example, if a service name is `SampleService`, then it should have all the events like `sample.test`, `sample.test.new`, etc. So, the pattern will be `sample.test`.

### 4. Configure Producer to Consumer Mapping
- Add all producers to the consumer configuration in `event-catalog-automation/json/consumer.json`.
- The producer is the key, and the consumer is a list of consumers for that producer.

### 5. Run the Main Script
- Execute the `main.js` script located at `event-catalog-automation/js` using the command:

### 6. Start Development Server
- Navigate to the `event-catalog/` directory.
- Run the following command to start the development server:

### 7. View the Catalog
- Once the development server is running, you can access the catalog at `localhost:3000`.

## Notes
- Make sure to follow the steps in the specified order for proper setup and execution.
- Ensure that all required dependencies are installed before running the scripts.