{"id":25409473,"url":"https://github.com/utibeabasi6/fluentd-logging","last_synced_at":"2026-04-09T11:33:54.904Z","repository":{"id":131673001,"uuid":"468841439","full_name":"utibeabasi6/fluentd-logging","owner":"utibeabasi6","description":"Centralised logging with fluentd","archived":false,"fork":false,"pushed_at":"2022-03-11T17:32:00.000Z","size":2824,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-13T20:53:49.203Z","etag":null,"topics":["docker","docker-compose","elasticsearch","flask","fluentd","kafka","kibana","s3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/utibeabasi6.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-11T17:25:35.000Z","updated_at":"2022-03-11T17:34:07.000Z","dependencies_parsed_at":"2023-06-25T22:11:55.882Z","dependency_job_id":null,"html_url":"https://github.com/utibeabasi6/fluentd-logging","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utibeabasi6%2Ffluentd-logging","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utibeabasi6%2Ffluentd-logging/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utibeabasi6%2Ffluentd-logging/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/utibeabasi6%2Ffluentd-logging/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/utibeabasi6","download_url":"https://codeload.github.com/utibeabasi6/fluentd-logging/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248782281,"owners_count":21160716,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","elasticsearch","flask","fluentd","kafka","kibana","s3"],"created_at":"2025-02-16T08:28:04.784Z","updated_at":"2025-12-30T23:05:04.170Z","avatar_url":"https://github.com/utibeabasi6.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Centralized logging with Fluentd\n\nThis project demonstrates how to implement centralised logging with Fluentd. Fluentd collects logs from the application and pushes them to Elastic search, Kafka and AWS S3\n\n## Prerequisites\n\nTo follow along with this demo, here are a few things you need to have installed:\n\n- Docker and Docker Compose: You can install these [here](https://docs.docker.com/get-docker/)\n\n- You will also need to create a slack bot. Follow the instructions [here](https://slack.com/help/articles/115005265703-Create-a-bot-for-your-workspace)\n\n## How to run\n\n- In the docker-compose.yaml file, replace the environment variable for the slack bot token with the token of your own bot\n![token](images/image1.png)\n\n- In the conf/fluent.conf file, enter your aws access key, secret key, s3 bucket name and aws region to configure fluentd to push logs to the s3 bucket specified\n\n![token](images/image2.png)\n\n- A `Makefile` has been provided to make setup easy so just run the command `make up` to build and run all the services. Note this might take a few minutes so be patient until all the services have fully initialized and you see logs consistent with what is shown below\n\n![token](images/image3.png)\n\n- Next, navigate to `http://localhost:5601` in your browser to view the Elastic search UI\n\n![token](images/image4.png)\n\n- In the side menu, Navigate to the `Discover` section and wait until elastic search picks up the logs from fluentd so you can create an index.\n\n![token](images/image5.png)\n\n- If no data shows up, keep refreshing and wait a bit for elastic search to pich up the logs\n\n![token](images/image10.png)\n\n![token](images/image11.png)\n\n- Create an index pattern called `fluentd*` and then head back to the `Discover` section to view the logs collected by fluentd\n\n![token](images/image6.png)\n\n![token](images/image7.png)\n\n- Now, in a new tab, navigate to `http://localhost:5000` to view the demo application. You will notice a big red button that says `BREAK THE APP!`. Click this and it will shutdown the flask server. This is so we can push error logs to Kafka which in turn pushes it to a Slack channel\n\n![token](images/image8.png)\n\n- Give it a few seconds and you should see your logs appear in the slack channel\n\n![token](images/image9.png)\n\n- Finally, kill the running process with `CTRL + C` and run the command `make down` to remove all the containers created by docker compose.\n\n## Hire me\n\nLooking for a DevOps engineer / SRE to join your team? Reach out to me by email at [utibeabasiumanah6@gmail.com](mailto://utibeabasiumanah6@gmail.com) or by any of the channels [here](https://linktr.ee/Utibeabasi6)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futibeabasi6%2Ffluentd-logging","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Futibeabasi6%2Ffluentd-logging","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futibeabasi6%2Ffluentd-logging/lists"}