Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/george-mountain/centralized-log-management
Centralized Log Management - Fastapi, Elastic Search, Logstash, Kibana and Docker.
https://github.com/george-mountain/centralized-log-management
docker elasticsearch elasticstack fastapi kibana kibana-dashboard kibana-visualization logstash loguru
Last synced: about 7 hours ago
JSON representation
Centralized Log Management - Fastapi, Elastic Search, Logstash, Kibana and Docker.
- Host: GitHub
- URL: https://github.com/george-mountain/centralized-log-management
- Owner: george-mountain
- Created: 2024-06-25T03:26:30.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-06-25T04:18:55.000Z (5 months ago)
- Last Synced: 2024-06-25T06:35:45.907Z (5 months ago)
- Topics: docker, elasticsearch, elasticstack, fastapi, kibana, kibana-dashboard, kibana-visualization, logstash, loguru
- Language: Python
- Homepage:
- Size: 27.3 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Centralized Log Management System
This repository showcases how to centralize and manage logs from microservice applications using the Elastic Stack (Logstash, Elasticsearch, and Kibana) and Loguru for logging within a FastAPI app.
## Project Overview
This project is designed for a school management system comprising multiple microservices:
- **Students App**
- **Library Management App**
- **Faculty App**
Each of these microservices can operate independently but are currently integrated using a common API gateway. We use Loguru for logging and the Elastic Stack for centralized log management.
## Key Components
### FastAPI App
The FastAPI app serves as the core of the school management system, integrating the various microservices.
### Elastic Stack
- **Logstash:** Collects and processes logs from the FastAPI app.
- **Elasticsearch:** Stores and indexes logs for efficient searching and analysis.
- **Kibana:** Provides a web interface for visualizing and exploring logs.
### Docker
All applications, including the Elastic Stack, are dockerized for easy deployment and management.
## Prerequisites
To run this project, you need to have Docker installed on your machine.
## Getting Started
### 1. Fork or clone the Repository
```sh
git clone https://github.com/george-mountain/Centralized-Log-Management.git
cd centralized-log-management
```
### 2. Create a `.env` File
Create a `.env` file in the root directory of the project and configure it as shown in the `env-sample` file provided in the repository.
### 3. Update the `.env` File
Your `.env` file should look like this:
```env
ELASTIC_PASSWORD=YOURELASTICPASSWORDHERE #e.g elastic1234
ELASTICSEARCH_USERNAME=YOURELASTICUSERNAMEHERE #e.g elastic
ELASTICSEARCH_PASSWORD=YOURELASTICSEARCHPASSWORDHERE #e.g elastic1234
```
### 4. Run the Application
Use Docker Compose to start all services:
```sh
docker-compose up --build
```
This command will start the FastAPI app, Logstash, Elasticsearch, and Kibana.
## Configuration
### Docker Compose
The `docker-compose.yml` file is configured to:
- Start the Elasticsearch container with security enabled.
- Start the Logstash container to collect logs from the FastAPI app.
- Start the Kibana container to visualize logs.
- Start the FastAPI app for the school management system.
### Logstash Configuration
The Logstash configuration file `logstash.conf` uses environment variables for credentials, ensuring sensitive information is not hard-coded.
### Environment Variables
Environment variables for credentials are stored in the `.env` file, ensuring they are securely managed and can be easily changed without modifying the codebase.
## Accessing Services
### FastAPI App
The FastAPI app will be available at `http://localhost:8000`.
The fastapi endpoints are:
http://localhost:8000/docs#/ --- This is the central application endpoint documentation
http://localhost:8000/school-management/student/docs#/ --- This is the student app endpoints documentation
http://localhost:8000/school-management/library/docs#/ --- This is the library app endpoints documentation
http://localhost:8000/school-management/faculty/docs#/ --- This is the faculty app endpoints documentation.
### Kibana
Kibana will be available at `http://localhost:5601`. Use the credentials configured in your `.env` file to log in.
Note: When you access the above url, you will be promted to login. Use the username and password which you configured
on the .env file for the elastic stack.
### Elasticsearch
Elasticsearch will be available at `http://localhost:9200`. Use the credentials configured in your `.env` file to access it.
For any questions or issues, please open an issue in the repository