{"id":13587758,"url":"https://github.com/lux4rd0/grafana-loki-syslog-aio","last_synced_at":"2025-04-09T12:08:33.250Z","repository":{"id":45511923,"uuid":"331464302","full_name":"lux4rd0/grafana-loki-syslog-aio","owner":"lux4rd0","description":"Grafana Loki Syslog All In One Syslog Deployable Stack","archived":false,"fork":false,"pushed_at":"2024-06-29T23:38:16.000Z","size":681,"stargazers_count":190,"open_issues_count":8,"forks_count":60,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-04-02T10:13:04.402Z","etag":null,"topics":["cadvisor","docker","grafana","loki","minio","node-exporter","prometheus","promtail","syslog","syslog-ng"],"latest_commit_sha":null,"homepage":"https://labs.lux4rd0.com/2021/01/oldskool-syslog-meets-newskool-loki/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lux4rd0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-20T23:55:02.000Z","updated_at":"2025-03-28T21:58:32.000Z","dependencies_parsed_at":"2024-11-06T06:44:40.190Z","dependency_job_id":null,"html_url":"https://github.com/lux4rd0/grafana-loki-syslog-aio","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lux4rd0%2Fgrafana-loki-syslog-aio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lux4rd0%2Fgrafana-loki-syslog-aio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lux4rd0%2Fgrafana-loki-syslog-aio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lux4rd0%2Fgrafana-loki-syslog-aio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lux4rd0","download_url":"https://codeload.github.com/lux4rd0/grafana-loki-syslog-aio/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248036067,"owners_count":21037092,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cadvisor","docker","grafana","loki","minio","node-exporter","prometheus","promtail","syslog","syslog-ng"],"created_at":"2024-08-01T15:06:20.950Z","updated_at":"2025-04-09T12:08:33.229Z","avatar_url":"https://github.com/lux4rd0.png","language":"Shell","funding_links":[],"categories":["Shell","docker"],"sub_categories":[],"readme":"\n## grafana-loki-syslog-aio\n\n\u003ccenter\u003e\u003cimg src=\"https://github.com/lux4rd0/grafana-loki-syslog-aio/blob/main/loki_syslog_aio.png\"\u003e\u003c/center\u003e\n\n## About The Project\n\nThis Loki Syslog All-In-One example is geared to help you get up and running quickly with a Syslog ingestor and visualization of logs. It uses [Grafana Loki](https://grafana.com/oss/loki/) and Promtail as a receiver for forwarded syslog-ng logs. I wrote an [introductory blog post](https://labs.lux4rd0.com/2021/01/oldskool-syslog-meets-newskool-loki/) about how this AIO project came about as well (pesky intermittent network issues!!) \n\n\u003ccenter\u003e\u003cimg src=\"https://github.com/lux4rd0/grafana-loki-syslog-aio/blob/main/loki_syslog_aio_overview_sized.png\"\u003e\u003c/center\u003e\n\n*Note that this All In One is geared towards getting network traffic from legacy syslog (RFC3164 UDP port 514) into Loki via [syslog-ng](https://www.syslog-ng.com/) and [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/).*\n\nEssentially:\n\n\u003e RFC3164 Network/Compute Devices -\u003e syslog-ng (UDP port 514) -\u003e\n\u003e Promtail (port 1514) -\u003e Loki (port 3100) \u003c- Grafana (port 3000)\n\n## Getting Started\n\nThe project is built around a pre-configured Docker stack of the following:\n\n - [Grafana](https://grafana.com/oss/grafana/)\n - [Grafana Loki](https://grafana.com/oss/loki/) (configured for [MinIO](https://min.io/))\n - [Grafana Promtail](https://grafana.com/docs/loki/latest/clients/promtail/)\n - [syslog-ng](https://www.syslog-ng.com/)\n\nThe stack has been extended to include pre-configured monitoring with:\n\n- [Prometheus](https://grafana.com/oss/prometheus/)\n- [Node-Exporter](https://github.com/prometheus/node_exporter)\n- [cAdvisor](https://github.com/google/cadvisor)\n\nA simple Syslog generator is included based on Vicente Zepeda Mas's [random-logger](https://github.com/chentex/random-logger) project.\n\n## Prerequisites\n\n- [Docker](https://docs.docker.com/install)\n- [Docker Compose](https://docs.docker.com/compose/install)\n\n## Using\n\nThis project is built and tested on Linux CentOS 7. To get started, download the code from this repository and extract it into an empty directory. For example:\n\n    wget https://github.com/lux4rd0/grafana-loki-syslog-aio/archive/main.zip\n    unzip main.zip\n    cd grafana-loki-syslog-aio-main\n    \nFrom that directory, run the docker-compose command:\n\n**Full Example Stack:** Grafana, Loki with s3/MinIO, Promtail, syslog-ng, Prometheus, cAdvisor, node-exporter\n\n    docker-compose -f ./docker-compose.yml up -d\n\nThis will start to download all of the needed application containers and start them up. \n\n*(Optional docker-compose configurations are listed under **Options** below)*\n\n**Grafana Dashboards**\n\nOnce all of the docker containers are started up, point your Web browser to the Grafana page, typically http://hostname:3000/ - with hostname being the name of the server you ran the docker-compose up -d command on. The \"Loki Syslog AIO - Overview\" dashboard is defaulted without having to log in.\n\n*Note: this docker-compose stack is designed to be as easy as possible to deploy and go. Logins have been disabled, and the default user has an admin role. This can be changed to an Editor or Viewer role by changing the Grafana environmental variable in the docker-compose.yml file to:*\n\n    GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer\n    \n**Getting Started With Loki**\n\nHere are some additional resources you might find helpful if you're just getting started with Loki:\n\n- [Getting started with Grafana and Loki in under 4\n   minutes](https://grafana.com/go/webinar/loki-getting-started/)\n- [An (only slightly technical) introduction to Loki](https://grafana.com/blog/2020/05/12/an-only-slightly-technical-introduction-to-loki-the-prometheus-inspired-open-source-logging-system/)\n- [Video tutorial: Effective troubleshooting queries with Grafana\n   Loki](https://grafana.com/blog/2021/01/07/video-tutorial-effective-troubleshooting-queries-with-grafana-loki/)\n\n## Stack Options:\n\nA few other docker-compose files are also available:\n\n**Full Example Stack with Syslog Generator:** Grafana, Loki with s3/MinIO, Promtail, syslog-ng, Prometheus, cAdvisor, node-exporter, Syslog Generator\n\n    docker-compose -f ./docker-compose-with-generator.yml up -d\n\n**Example Stack without monitoring or Syslog generator**: Grafana, Loki with s3/MinIO, Promtail, syslog-ng\n\n    docker-compose -f ./docker-compose-without-monitoring.yml up -d\n\n**Example Stack without MinIO, monitoring, or Syslog generator:** Grafana, Loki with the filesystem, Promtail, syslog-ng\n\n    docker-compose -f ./docker-compose-filesystem.yml up -d\n\nThe *Syslog Generator* configuration will need access to the Internet to do a local docker build from the configurations location in ./generator. It'll provide some named hosts and random INFO, WARN, DEBUG, ERROR logs sent over to syslog-ng/Loki.\n\n\u003ccenter\u003e\u003cimg src=\"https://github.com/lux4rd0/grafana-loki-syslog-aio/blob/main/loki_syslog_aio_overview_generator_sized.png\"\u003e\u003c/center\u003e\n\n## Configuration Review:\n\nThe default Loki storage configuration docker-compose.yml uses S3 storage with MinIO. If you want to use the filesystem instead, use the different docker-compose configurations listed above or change the configuration directly. An example would be:\n\n    volumes:\n    - ./config/loki-config-filesystem.ym:/etc/loki/loki-config.yml:ro\n\n**Changing MinIO Keys**\n\nThe MinIO configurations default the Access Key and Secret Key at startup. If you want to change them, you'll need to update two files:\n\n./docker-compose.yml\n\n      MINIO_ACCESS_KEY: minio123\n      MINIO_SECRET_KEY: minio456\n      \n./config/loki-config-s3.yml\n\n     aws:\n      s3: s3://minio123:minio456@minio.:9000/loki\n\n## Changed Default Configurations In syslog-ng and Promtail\n\nTo set this example All In One project up, the following configurations have been added to the docker-compose.yml. If you already have syslog-ng running on your deployment server - make similar changes below and comment out the docker container stanza.\n\n#### SYSLOG-NG CONFIGURATION (docker container listens on port 514)\n\n**# syslog-ng.conf**\n\n    source s_local {\n        internal();\n    };\n    \n    source s_network {\n        default-network-drivers(\n        );\n    };\n    \n    destination d_loki {\n        syslog(\"promtail\" transport(\"tcp\") port(\"1514\"));\n    };\n    \n    log {\n            source(s_local);\n            source(s_network);\n            destination(d_loki);\n    };\n\n\u003e Note: the above \"`promtail`\" configuration for `destination d_loki` is\n\u003e the *hostname* where Promtail is running. Is this example, it happens\n\u003e to be the Promtail *docker container* name that I configured for the\n\u003e All-In-One example.\n\n#### PROMTAIL CONFIGURATION (docker container listens on port 1514)\n\n **# promtail-config.yml**\n\n    server:\n      http_listen_port: 9080\n      grpc_listen_port: 0\n    \n    positions:\n      filename: /tmp/positions.yaml\n    \n    clients:\n      - url: http://loki:3100/loki/api/v1/push\n    \n    scrape_configs:\n    \n    - job_name: syslog\n      syslog:\n        listen_address: 0.0.0.0:1514\n        idle_timeout: 60s\n        label_structured_data: yes\n        labels:\n          job: \"syslog\"\n      relabel_configs:\n        - source_labels: ['__syslog_message_hostname']\n          target_label: 'host'\n\n## Contributing\n\nContributions make the open source community such a fantastic place to learn, inspire, and create. Any contributions you make are greatly appreciated.\n\n- Fork the Project\n- Create your Feature Branch (git checkout -b feature/AmazingFeature)\n- Commit your Changes (git commit -m 'Add some AmazingFeature')\n- Push to the Branch (git push origin feature/AmazingFeature)\n- Open a Pull Request\n\n## Contact\n\nDave Schmid - [@lux4rd0](https://twitter.com/lux4rd0) - dave@pulpfree.org\n\nProject Link: https://github.com/lux4rd0/grafana-loki-syslog-aio\n\n## Acknowledgements\n\n- Grafana Labs - https://grafana.com/\n- Grafana Loki - https://grafana.com/oss/loki/\n- Grafana - https://grafana.com/oss/grafana/\n- syslog-ng - https://www.syslog-ng.com/\n- Random Logger - https://github.com/chentex/random-logger\n- Grafana Dashboard Community (Performance Overviews) - https://grafana.com/grafana/dashboards\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flux4rd0%2Fgrafana-loki-syslog-aio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flux4rd0%2Fgrafana-loki-syslog-aio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flux4rd0%2Fgrafana-loki-syslog-aio/lists"}