{"id":49230287,"url":"https://github.com/apakhbari/log-testing-with-fluent-bit","last_synced_at":"2026-04-24T11:44:04.106Z","repository":{"id":327880283,"uuid":"1109071680","full_name":"apakhbari/log-testing-with-fluent-bit","owner":"apakhbari","description":"A Simple Docker-based log generator for Fluent Bit configurations with GELF output.","archived":false,"fork":false,"pushed_at":"2025-12-07T12:29:32.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-10T04:06:42.115Z","etag":null,"topics":["docker","fluent-bit","graylog"],"latest_commit_sha":null,"homepage":"","language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apakhbari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-03T09:50:10.000Z","updated_at":"2025-12-07T12:29:35.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/apakhbari/log-testing-with-fluent-bit","commit_stats":null,"previous_names":["apakhbari/log-testing-with-fluent-bit"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/apakhbari/log-testing-with-fluent-bit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apakhbari%2Flog-testing-with-fluent-bit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apakhbari%2Flog-testing-with-fluent-bit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apakhbari%2Flog-testing-with-fluent-bit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apakhbari%2Flog-testing-with-fluent-bit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apakhbari","download_url":"https://codeload.github.com/apakhbari/log-testing-with-fluent-bit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apakhbari%2Flog-testing-with-fluent-bit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32222482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T10:26:35.452Z","status":"ssl_error","status_checked_at":"2026-04-24T10:25:27.643Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","fluent-bit","graylog"],"created_at":"2026-04-24T11:44:03.486Z","updated_at":"2026-04-24T11:44:04.092Z","avatar_url":"https://github.com/apakhbari.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# log-testing-with-fluent-bit\n```\n ___      _______  _______  _______  ___   __    _  _______      _     _ /     _______  ___      __   __  _______  __    _  _______         _______  ___   _______ \n|   |    |       ||       ||       ||   | |  |  | ||       |    | | _ | |     |       ||   |    |  | |  ||       ||  |  | ||       |       |  _    ||   | |       |\n|   |    |   _   ||    ___||    ___||   | |   |_| ||    ___|    | || || |     |    ___||   |    |  | |  ||    ___||   |_| ||_     _| ____  | |_|   ||   | |_     _|\n|   |    |  | |  ||   | __ |   | __ |   | |       ||   | __     |       |     |   |___ |   |    |  |_|  ||   |___ |       |  |   |  |____| |       ||   |   |   |  \n|   |___ |  |_|  ||   ||  ||   ||  ||   | |  _    ||   ||  |    |       |     |    ___||   |___ |       ||    ___||  _    |  |   |         |  _   | |   |   |   |  \n|       ||       ||   |_| ||   |_| ||   | | | |   ||   |_| |    |   _   |     |   |    |       ||       ||   |___ | | |   |  |   |         | |_|   ||   |   |   |  \n|_______||_______||_______||_______||___| |_|  |__||_______|    |__| |__|     |___|    |_______||_______||_______||_|  |__|  |___|         |_______||___|   |___|  \n```\n\nA Docker-based log generation and forwarding system for testing Fluent Bit configurations with GELF (Graylog Extended Log Format) output.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Features](#features)\n- [Prerequisites](#prerequisites)\n- [Project Structure](#project-structure)\n- [Configuration](#configuration)\n  - [Docker Compose](#docker-compose)\n  - [Fluent Bit Configuration](#fluent-bit-configuration)\n  - [Lua Script](#lua-script)\n- [Getting Started](#getting-started)\n- [Log Format Examples](#log-format-examples)\n- [Customization](#customization)\n- [Troubleshooting](#troubleshooting)\n- [Acknowledgment](#acknowledgment)\n\n## Overview\n\nThis project provides a complete testing environment for Fluent Bit's GELF output functionality. It consists of two main components:\n\n1. **Log Generator**: A bash-based container that generates various log levels (INFO, DEBUG, WARN, ERROR)\n2. **Fluent Bit**: Collects logs via Docker's fluentd logging driver and forwards them to a GELF endpoint\n\n## Architecture\n\n```mermaid\nflowchart LR\n    A[Log Generator Container] --\u003e|fluentd driver| B[Fluent Bit :24224]\n    B --\u003e|Lua Filter| C[Tag Processing]\n    C --\u003e|Add dividing_name| D[Field Cleanup]\n    D --\u003e|GELF TCP| E[GELF Endpoint\u003cbr/\u003e10.10.21.151:31221]\n    \n    style A fill:#e1f5ff\n    style B fill:#fff4e1\n    style C fill:#f0e1ff\n    style D fill:#f0e1ff\n    style E fill:#e1ffe1\n```\n\n## Features\n\n- **Automatic log generation** with multiple severity levels (INFO, DEBUG, WARN, ERROR)\n- **Docker fluentd logging driver** integration\n- **Custom Lua filtering** for tag extraction and field manipulation\n- **GELF TCP output** for compatibility with Graylog, Logstash, and other GELF receivers\n- **Container name tagging** with custom identifier format\n- **Line ending fixes** for cross-platform compatibility (CRLF → LF)\n\n## Prerequisites\n\n- Docker Engine 20.10+\n- Docker Compose 1.29+\n- Network access to your GELF endpoint (default: `10.10.21.151:31221`)\n\n## Project Structure\n\n```\n.\n├── docker-compose.yml      # Service orchestration\n├── Dockerfile             # Log generator image\n├── fluent-bit.conf        # Fluent Bit configuration\n├── functions.lua          # Lua filter for log processing\n└── log-generator.sh       # Bash script for log generation\n```\n\n## Configuration\n\n### Docker Compose\n\nThe `docker-compose.yml` defines two services:\n\n- **fluent-bit**: Listens on port 24224 (TCP/UDP) for incoming logs\n- **log-generator**: Generates logs and sends them via fluentd driver\n\nBoth services run on a shared `logging` bridge network.\n\n### Fluent Bit Configuration\n\nKey configuration in `fluent-bit.conf`:\n\n| Section | Purpose |\n|---------|---------|\n| **INPUT** | Forward input listening on `0.0.0.0:24224` |\n| **FILTER** | Lua script (`functions.lua`) processes tags and records |\n| **OUTPUT** | GELF TCP output to `10.10.21.151:31221` |\n\n**Important Settings:**\n- `Gelf_Short_Message_Key`: Set to `log` (matches Docker log field)\n- `Mode`: `tcp` for reliable delivery\n\n### Lua Script\n\nThe `functions.lua` script performs three operations:\n\n1. **Tag Parsing**: Extracts container name from `docker.container_name` format\n2. **Custom Field Addition**: Creates `dividing_name` field with format `ubuntu_prod:container_name`\n3. **Field Cleanup**: Removes `container_id` to reduce GELF payload size\n\n## Getting Started\n\n1. **Clone or create the project structure** with all five files\n\n2. **Update GELF endpoint** in `fluent-bit.conf`:\n   ```properties\n   [OUTPUT]\n       Host    YOUR_GELF_HOST\n       Port    YOUR_GELF_PORT\n   ```\n\n3. **Customize server identifier** in `functions.lua`:\n   ```lua\n   record[\"dividing_name\"] = \"YOUR_SERVER_NAME:\" .. container_name\n   ```\n\n4. **Start the services**:\n   ```bash\n   docker-compose up --build\n   ```\n\n5. **Verify logs are flowing**:\n   ```bash\n   docker-compose logs -f fluent-bit\n   ```\n\n6. **Check your GELF receiver** (Graylog, etc.) for incoming messages\n\n## Log Format Examples\n\nThe log generator produces five types of messages in rotation:\n\n```\n[INFO] 2024-12-07 10:15:30 - Processing request #1\n[DEBUG] 2024-12-07 10:15:32 - Debug message 2 - Memory usage: 47%\n[WARN] 2024-12-07 10:15:34 - Warning: High latency detected - 1823ms\n[ERROR] 2024-12-07 10:15:36 - Error processing item 4 - Retrying...\n[INFO] 2024-12-07 10:15:38 - Successfully completed operation 5\n```\n\nEach log entry includes:\n- Severity level\n- Timestamp\n- Descriptive message\n- Counter/random data\n\n## Customization\n\n### Change Log Generation Frequency\n\nEdit `log-generator.sh`:\n```bash\nsleep 2  # Change to desired interval in seconds\n```\n\n### Add More Log Levels\n\nExtend the case statement in `log-generator.sh`:\n```bash\ncase $((counter % 6)) in\n    5)\n        echo \"[CRITICAL] $(date '+%Y-%m-%d %H:%M:%S') - Critical error!\"\n        ;;\nesac\n```\n\n### Modify GELF Fields\n\nIn `functions.lua`, add custom fields:\n```lua\nrecord[\"environment\"] = \"production\"\nrecord[\"application\"] = \"my-app\"\nrecord[\"version\"] = \"1.0.0\"\n```\n\n### Use UDP Instead of TCP\n\nChange in `fluent-bit.conf`:\n```properties\n[OUTPUT]\n    Mode    udp\n```\n\n## Troubleshooting\n\n### Logs Not Appearing in GELF Receiver\n\n1. **Check Fluent Bit logs**:\n   ```bash\n   docker-compose logs fluent-bit\n   ```\n\n2. **Verify network connectivity**:\n   ```bash\n   docker-compose exec fluent-bit ping 10.10.21.151\n   ```\n\n3. **Test GELF endpoint**:\n   ```bash\n   nc -zv 10.10.21.151 31221\n   ```\n\n### \"Exec format error\" or Script Fails\n\nThe Dockerfile includes `dos2unix` to fix line endings. If issues persist:\n```bash\ndos2unix log-generator.sh\ngit add --renormalize .\n```\n\n### Container Exits Immediately\n\nCheck log generator output:\n```bash\ndocker-compose logs log-generator\n```\n\n### Fluent Bit Connection Refused\n\nEnsure the fluentd address matches your Docker network:\n```yaml\nfluentd-address: \"fluent-bit:24224\"  # Use service name, not localhost\n```\n\n## Acknowledgment\n\n### Contributors\n\n* APA 🖖🏻\n\n### Links\n- [go2docs.graylog.org](go2docs.graylog.org)\n\n\n```\n  aaaaaaaaaaaaa  ppppp   ppppppppp     aaaaaaaaaaaaa   \n  a::::::::::::a p::::ppp:::::::::p    a::::::::::::a  \n  aaaaaaaaa:::::ap:::::::::::::::::p   aaaaaaaaa:::::a \n           a::::app::::::ppppp::::::p           a::::a \n    aaaaaaa:::::a p:::::p     p:::::p    aaaaaaa:::::a \n  aa::::::::::::a p:::::p     p:::::p  aa::::::::::::a \n a::::aaaa::::::a p:::::p     p:::::p a::::aaaa::::::a \na::::a    a:::::a p:::::p    p::::::pa::::a    a:::::a \na::::a    a:::::a p:::::ppppp:::::::pa::::a    a:::::a \na:::::aaaa::::::a p::::::::::::::::p a:::::aaaa::::::a \n a::::::::::aa:::ap::::::::::::::pp   a::::::::::aa:::a\n  aaaaaaaaaa  aaaap::::::pppppppp      aaaaaaaaaa  aaaa\n                  p:::::p                              \n                  p:::::p                              \n                 p:::::::p                             \n                 p:::::::p                             \n                 p:::::::p                             \n                 ppppppppp                                                        \n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapakhbari%2Flog-testing-with-fluent-bit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapakhbari%2Flog-testing-with-fluent-bit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapakhbari%2Flog-testing-with-fluent-bit/lists"}