https://github.com/morzhanov/go-elk

Go ELK stack (with Filebeat) example.
https://github.com/morzhanov/go-elk

Last synced: 5 days ago
JSON representation

Go ELK stack (with Filebeat) example.

• Host: GitHub
• URL: https://github.com/morzhanov/go-elk
• Owner: morzhanov
• Created: 2021-10-13T23:02:58.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2021-10-17T20:10:38.000Z (about 3 years ago)
• Last Synced: 2024-11-08T00:52:36.287Z (about 2 months ago)
• Language: Go
• Size: 68.4 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Go ELK example

Go ELK stack example with filebeat, metricbeat and ElasticSearch API.



## Features

- ELK stack setup

- Filebeat logs collector setup

- Metricbeat collector setup

- ElasticSearch API usage

## Structure

- `/cmd` - application setup

- `/config` - .env file with environment variables

- `/deploy`

  - `/filebeat` - filebeat.yml file for Filebeat setup 

  - `/metricbeat` - metricbeat.yml file for Metricbeat setup and Metricbeat Dockerfile

  - `docker-compose.yml` - docker-compose file with ELK stack and beats

- `/internal`

  - `/config` - config module based on viper package

  - `/doc` - ES Document interface

  - `/es` - ElasticSearch module for ES API implementation usage

  - `/generator` - generates Documents and sends them to ES using `es` module

  - `/logger` - application logger, creates file transport (for filebeat) and console transport

  - `/metrics` - setups application metrics handling via `expvar` package

  - `/rest` - application REST module

## How it works

1. ES API

   - application generate data with generator and saves them to ES

   - application registers public REST API for ES data manipulation

      - Update doc

      - Delete doc

      - Search docs

2. Filebeat

   - logger sends logs to `/logs` directory

   - filebeat crawl the logs and sends them to ES

3. Metricbeat

   - metrics package registers application metric data

     - go general metric data (memstat, cpu stats, etc.)

     - request count

     - error count

     - docs count

   - metricbeat consumes metrics from `localhost:8080/debug/vars` endpoint and sends them to the ES

4. Kibana - uses for data visualization

### Postman

Application contains `go-elk-example.postman_collection` file which you can import to your Postman app and perform requests to the application REST API.

## Kibana setup

### Create index patterns

Firstly you should create index patterns for ES data:

- `goelkmetr` - Go app metrics data

- `goelklog` - Go app logs data

- `docs` - ES Docs API data

Creating Index patterns:



Index patterns created:



Docs index example data:



### Discover

After index patterns created we could search and filter data on the `Discover` page:



Metrics data on `Discover` page:



### Dashboard

Also, we could create a Kibana Dashboard and visualize ES data:

- error count

- request count

- documents count

- some go stats

- some docs stats