{"id":19586322,"url":"https://github.com/tbobm/trail","last_synced_at":"2026-04-28T12:36:18.134Z","repository":{"id":53195092,"uuid":"262347880","full_name":"tbobm/trail","owner":"tbobm","description":"Simple and scalable Docker container \"tail -f\". Centralize logs to a single web page. Built using ansible, docker and elastic tools.","archived":false,"fork":false,"pushed_at":"2021-04-01T12:54:09.000Z","size":5607,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-01-09T05:48:35.548Z","etag":null,"topics":["ansible","docker","elk","exchange","filebeat","frontend","logging","logs","logstash","mongodb","rabbitmq","school"],"latest_commit_sha":null,"homepage":"https://github.com/tbobm/trail","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tbobm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-08T14:40:48.000Z","updated_at":"2021-11-26T10:03:28.000Z","dependencies_parsed_at":"2022-09-10T03:53:21.214Z","dependency_job_id":null,"html_url":"https://github.com/tbobm/trail","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbobm%2Ftrail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbobm%2Ftrail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbobm%2Ftrail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tbobm%2Ftrail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tbobm","download_url":"https://codeload.github.com/tbobm/trail/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240858576,"owners_count":19868998,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","docker","elk","exchange","filebeat","frontend","logging","logs","logstash","mongodb","rabbitmq","school"],"created_at":"2024-11-11T07:59:24.080Z","updated_at":"2026-04-28T12:36:18.081Z","avatar_url":"https://github.com/tbobm.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Trail\n\n_School project built during the 168h \"codecamp\"._\n\n## Goal\n\nCreate a painless web tail for containers based on logstash / fluentbit\n\nSee [the subject](./index.html) for further description.\n\n## Usage\n\n### Start the project\n\nIn order to bring the whole stack up:\n\n```\ndocker-compose up -d\n```\n\n### Deploying agents\n\nThe ansible role `trail` will deploy a docker-enabled filebeat container with the required permissions (read capability on the docker socket and container directories).\n\nAn [example playbook](./playbook.yml) demonstrates a basic usage of this role. The `LOGSTASH_TARGET` is required, as it is used in the [filebeat config file](./roles/trail/templates/filebeat.yml.j2).\n\nThe `trail` role will:\n  1. Template the filebeat configuration in `/etc/filebeat.yml`\n  2. Ensure `pip` is available on the remote host\n  3. Ensure the python docker package is available on the remote host and install it if necessary\n  4. Run a container named `filebeat_agent` on the remote host\n\n### Running the frontend\n\nBy running `docker-compose up -d`, you will have the api on your local port `3000` and the frontend on the port `8080`.\n\n## Architecture\n\n[![](https://mermaid.ink/img/eyJjb2RlIjoiZ3JhcGggTFJcbiAgc3ViZ3JhcGggYmFja2VuZFxuICAgIEMoW0NvbnRhaW5lcl0pIC0tPnxsb2cgdG8gc3Rkb3V0L3N0ZGVycnwgQltkb2NrZXIganNvbiBsb2cgZHJpdmVyXVxuICAgIEIgLS0-fG1vdW50ZWQgbG9nIGRpcmVjdG9yaWVzfCBGW0ZpbGViZWF0XSAtLT58ZmlsZWJlYXQgaW5wdXR8IExbL0xvZ3N0YXNoL11cbiAgICBMIC0tPnxhbXFwfCBSWy9SYWJiaXRNUS9dXG4gIGVuZFxuICBzdWJncmFwaCBmcm9udGVuZFxuICAgIHN1YmdyYXBoIEFQSVxuICAgICAgUiAtLS18YW1xcHwgQVtBUEldXG4gICAgZW5kXG4gICAgQSAtLS18d3N8QzFbQ2xpZW50MV1cbiAgICBBIC0tLXx3c3xDMltDbGllbnQyXVxuICAgIEEgLS0tfHdzfEMzW0NsaWVudDNdXG4gIGVuZFxuIiwibWVybWFpZCI6eyJ0aGVtZSI6ImRhcmsifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiZ3JhcGggTFJcbiAgc3ViZ3JhcGggYmFja2VuZFxuICAgIEMoW0NvbnRhaW5lcl0pIC0tPnxsb2cgdG8gc3Rkb3V0L3N0ZGVycnwgQltkb2NrZXIganNvbiBsb2cgZHJpdmVyXVxuICAgIEIgLS0-fG1vdW50ZWQgbG9nIGRpcmVjdG9yaWVzfCBGW0ZpbGViZWF0XSAtLT58ZmlsZWJlYXQgaW5wdXR8IExbL0xvZ3N0YXNoL11cbiAgICBMIC0tPnxhbXFwfCBSWy9SYWJiaXRNUS9dXG4gIGVuZFxuICBzdWJncmFwaCBmcm9udGVuZFxuICAgIHN1YmdyYXBoIEFQSVxuICAgICAgUiAtLS18YW1xcHwgQVtBUEldXG4gICAgZW5kXG4gICAgQSAtLS18d3N8QzFbQ2xpZW50MV1cbiAgICBBIC0tLXx3c3xDMltDbGllbnQyXVxuICAgIEEgLS0tfHdzfEMzW0NsaWVudDNdXG4gIGVuZFxuIiwibWVybWFpZCI6eyJ0aGVtZSI6ImRhcmsifSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)\n\n## Example projects\n\nA [Vagrantfile](./Vagrantfile) is provided to experiment with `trail`. It will create 5 VMs with multiple docker-based example projects (`wordpress`, `rails`, `django`, a failing `nginx` configuration and a `flog` container).\n\n### Configuration\n\nInstall the required `ansible` role (docker installation and configuration) by running the following:\n```bash\nansible-galaxy install -r requirements.yml\n```\n\nThe `LOGSTASH_TARGET` variable in [the provisioning playbook](./provisioning.playbook.yml) is supposed to be modified to suit your needs.\n\n### Running the example projects\n\nIn order to use the provisioning part of the [Vagrantfile](./Vagrantfile), simply bring the VMs up and Vagrant will automatically run the corresponding playbook based on the node name.\n\n```bash\nvagrant up\n```\n\n## Backend\n\n### Stack\n\nname | role\n------|------\nfilebeat | log collection - fetch logs from containers, either by mapping the docker sockets/directories to filebeat, or deploy as a kubernetes pod\nlogstash | centralize, parse and enrich logs\nrabbitmq | access point for logs through the [topic exchange](https://www.rabbitmq.com/tutorials/amqp-concepts.html#exchange-topic) called `logs`\nflog | open source log generation tool\nmongodb | short term storage (6h)\n\n### Running the backend only\n\nI highly advise running each items one after the other, to ensure everything is running smooth.\n\n1. `docker-compose up -d broker`\n2. `docker-compose up -d logstash`\n3. `docker-compose up -d filebeat`\n4. `docker-compose up -d storage`\n5. `docker-compose up log-generator` (in order to have it in the foreground)\n\n### How to consume logs from the broker\n\nThe Exchange is automatically created by Logstash at startup time, based on the \n\n- Connect to RabbitMQ\n- Create a Queue (_NOTE: define if queue have to be exclusive and/or durable_)\n- Bind your Queue to the Exchange, specify if needed a *routing key*\n- Consume the messages from your Queue\n\n### Short term log retention\n\nThe logstash pipeline will, in addition to forwarding the logs in RabbitMQ, store the logs in MongoDB.\n\nAs seen in the [mongoscripts configuration directory](./mongoscripts/logs.js), an unoptimized index based on the `@timestamp` field will enable the deletion of each log after 6 hours.\n\nQuerying these logs through the frontend was not implemented, a helper script [search.py](./search.py) was added instead as a lightweight replacement.\n\n_Note: Implementing this in the frontend should be quite quick_\n\n#### Example usage\n\n```bash\n# install pymongo\npip install --user -r requirements.txt\n```\n\n```bash\nexport MONGODB_URI=mongodb://localhost:27017/trail  # This script uses the environment variable MONGODB_URI to connect to MongoDB\npython search.py term\n# ...\n# \u003c@timestamp\u003e, \u003cmessage\u003e\n```\n\n## Logs\n\n### Schema\n\nSee the [log schema](./log-schema.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbobm%2Ftrail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftbobm%2Ftrail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftbobm%2Ftrail/lists"}