https://github.com/lizozom/docker-elastic-observability-with-apm
Docker compose setup of an observability use case. Reporting metrics and traces from a nodejs app to Elasticsearch using APM Server using APM RUM and APM Node.js agents.
https://github.com/lizozom/docker-elastic-observability-with-apm
apm docker docker-compose elasticsearch express kibana nodejs
Last synced: about 1 year ago
JSON representation
Docker compose setup of an observability use case. Reporting metrics and traces from a nodejs app to Elasticsearch using APM Server using APM RUM and APM Node.js agents.
- Host: GitHub
- URL: https://github.com/lizozom/docker-elastic-observability-with-apm
- Owner: lizozom
- License: mit
- Created: 2022-12-14T08:59:04.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-12-27T17:48:19.000Z (over 3 years ago)
- Last Synced: 2025-06-19T01:42:38.555Z (about 1 year ago)
- Topics: apm, docker, docker-compose, elasticsearch, express, kibana, nodejs
- Language: Dockerfile
- Homepage:
- Size: 110 KB
- Stars: 23
- Watchers: 3
- Forks: 8
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: .github/CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: .github/CODE_OF_CONDUCT.md
- Security: .github/SECURITY.md
Awesome Lists containing this project
README
Observability with Elasticsearch
APM Node.js and APM RUM agents on Docker
# Introduction
Elastic Stack (**ELK**) Docker Composition, with a Node.JS application reporting APM client + server side metrics and traces to APM Server.
Suitable for Demoing and MVPs.
Stack Version: [8.5.3](https://www.elastic.co/blog/whats-new-elastic-8-5-3) 🎉 - Based on [Official Elastic Docker Images](https://www.docker.elastic.co/)
> You can change Elastic Stack version by setting `ELK_VERSION` in `.env` file and rebuild your images. Any version >= 8.0.0 is compatible with this template.
Based on [elastdocker](https://github.com/sherifabdlnaby/elastdocker)
### Main Features 📜
- Node.JS application with Node APM Agent.
- Elasticsearch, configured as a Production Single Node Cluster.
- Security Enabled on Elasticsearch and Kibana.
- APM Server receives *Non secure* connections.
- Use Docker-Compose and `.env` to configure your entire stack parameters.
- Persist Elasticsearch's Keystore and SSL Certifications.
-----
# Requirements
- [Docker 20.05 or higher](https://docs.docker.com/install/)
- [Docker-Compose 1.29 or higher](https://docs.docker.com/compose/install/)
- 4GB RAM (For Windows and MacOS make sure Docker's VM has more than 4GB+ memory.)
# Setup
1. Clone the Repository
```bash
git clone https://github.com/lizozom/docker-elastic-observability-with-apm.git
```
2. Initialize Elasticsearch Keystore and TLS Self-Signed Certificates
```bash
$ make setup
```
> **For Linux's docker hosts only**. By default virtual memory [is not enough](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html) so run the next command as root `sysctl -w vm.max_map_count=262144`
3. Start Elastic Stack
```bash
$ make elk $ docker-compose up -d $ docker compose up -d
```
4. Navigate to the web application at [http://localhost:3000/](http://localhost:3000) to generate APM traces.
5. Visit Kibana at [https://localhost:5601/app/apm](https://localhost:5601/app/apm) to see the APM traces from our `webapp`
Default Username: `elastic`, Password: `changeme`
# Configuration
* Some Configuration are parameterized in the `.env` file.
* `ELASTIC_PASSWORD`, user `elastic`'s password (default: `changeme` _pls_).
* `ELK_VERSION` Elastic Stack Version (default: `8.3.2`)
* `ELASTICSEARCH_HEAP`, how much Elasticsearch allocate from memory (default: 1GB -good for development only-)
* Other configurations which their such as cluster name, and node name, etc.
* Elasticsearch Configuration in `elasticsearch.yml` at `./elasticsearch/config`.
* Kibana Configuration in `kibana.yml` at `./kibana/config`.
### Setting Up Keystore
You can extend the Keystore generation script by adding keys to `./setup/keystore.sh` script. (e.g Add S3 Snapshot Repository Credentials)
To Re-generate Keystore:
```
make keystore
```
### Notes
- ⚠️ Elasticsearch HTTP layer is using SSL, thus mean you need to configure your elasticsearch clients with the `CA` in `secrets/certs/ca/ca.crt`, or configure client to ignore SSL Certificate Verification (e.g `--insecure` in `curl`).
- Makefile is a wrapper around `Docker-Compose` commands, use `make help` to know every command.
- Elasticsearch will save its data to a volume named `elasticsearch-data`
- Elasticsearch Keystore (that contains passwords and credentials) and SSL Certificate are generated in the `./secrets` directory by the setup command.
- Make sure to run `make setup` if you changed `ELASTIC_PASSWORD` and to restart the stack afterwards.
- **Important**: For Linux Users it's recommended to set the following configuration (run as `root`)
```
sysctl -w vm.max_map_count=262144
```
By default, Virtual Memory [is not enough](https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html).
# License
[MIT License](https://raw.githubusercontent.com/sherifabdlnaby/elastdocker/master/LICENSE)
Copyright (c) 2022 Sherif Abdel-Naby
# Contribution
PR(s) are Open and Welcomed.