{"id":19557209,"url":"https://github.com/gingray/fluentd_tail_elasticsearch","last_synced_at":"2026-04-29T00:01:38.176Z","repository":{"id":151847146,"uuid":"141888621","full_name":"gingray/fluentd_tail_elasticsearch","owner":"gingray","description":"Config for fluentd with in_tail input and output is elasticsearch","archived":false,"fork":false,"pushed_at":"2018-07-22T11:36:28.000Z","size":3,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-02-19T04:30:27.931Z","etag":null,"topics":["elasticsearch","fluentd","json","logs","tail"],"latest_commit_sha":null,"homepage":null,"language":"Dockerfile","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gingray.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-22T11:07:26.000Z","updated_at":"2018-07-22T11:37:56.000Z","dependencies_parsed_at":"2023-05-12T07:45:16.682Z","dependency_job_id":null,"html_url":"https://github.com/gingray/fluentd_tail_elasticsearch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gingray/fluentd_tail_elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Ffluentd_tail_elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Ffluentd_tail_elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Ffluentd_tail_elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Ffluentd_tail_elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gingray","download_url":"https://codeload.github.com/gingray/fluentd_tail_elasticsearch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gingray%2Ffluentd_tail_elasticsearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32404340,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T19:38:08.556Z","status":"ssl_error","status_checked_at":"2026-04-28T19:37:55.688Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","fluentd","json","logs","tail"],"created_at":"2024-11-11T04:40:51.362Z","updated_at":"2026-04-29T00:01:38.162Z","avatar_url":"https://github.com/gingray.png","language":"Dockerfile","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fluentd config with Elasticsearch\nBasically this config solve one problem when you have a lot of log files and you want to push them into ElasticSearch.\nIf you google a while you see that you need an `tail` plugn with `elasticsearch` plugin. There Docker Compose config to handle this\n\nAssumes:\n\n1. ElasticSearch available on host machine on port 9200 (you can wrap in to a same network but you need to remove network mode and put your own network tag)\n2. All path assume that you have user `app` like `/home/app` change it to you own\n3. No auto restat for container because assume that its one time task\n\nInformation about paths:\n` /home/app/logs:/fluentd/log/` path where logs are stored on your machine\n`/home/app/flunetd/conf:/fluentd/etc/` path where config for fluentd is stored\n`/home/app/flunetd/buffer:/fluentd/buffer` path where file buffer need to be store\n`/home/app/fluentd/pos:/fluentd/pos` path where position of reading files need to be stored for plugin in_tail\n\n\nBit more information about config\n\n```\n\u003csource\u003e\n  @type tail #plugin type for read from files\n  path /fluentd/log/*.log #assume that you need push all files in directory with ext *.log\n  tag fluentd #tag for fluentd \n  format json #assume that you logs in valid json format each line json object\n  time_key time #assume that your time key is field time in json object \n  time_format %Y-%m-%dT%H:%M:%S%z #time format in time key\n  read_from_head true #mean that all files will be readed form begining\n\u003c/source\u003e\n\n\u003cfilter **\u003e #maybe you don't need this filter because he do one thing its keep time key in record basically you can remove it \n  @type record_transformer\n  enable_ruby\n  \u003crecord\u003e\n    time ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}\n    from_file true\n  \u003c/record\u003e\n\u003c/filter\u003e\n\n\u003cmatch **\u003e #elasticsearch plugin config\n    @type elasticsearch\n    host 127.0.0.1\n    port 9200\n    logstash_format true #keeplogstash format\n    include_tag_key true \n    logstash_prefix fluentd-files\n    logstash_dateformat %Y\n    tag_key @log_name\n\n    time_key time #time key for record\n    time_key_format %Y-%m-%dT%H:%M:%S%z #format that was set above\n    reconnect_on_error true #reconnect each time on error\n    reload_on_failure true \n    reload_connections false\n    request_timeout 120s #it set 120s becuase if your elasticsearch server is slow \n    include_timestamp true\n    \u003cbuffer\u003e #thats important part because by default using memorry buffer and it will overflow very fast if you have enough logs\n      @type file\n      flush_interval 20s\n      retry_type periodic\n      retry_forever true\n      retry_wait 10s\n      chunk_limit_size 4Mb\n      queue_limit_length 4096\n      total_limit_size 60Gb\n      path /fluentd/buffer/elastic.buff\n    \u003c/buffer\u003e\n\u003c/match\u003e\n```\nPS\n\nI've spent a lot of time to find right config buts maybe I was just so lucky. I'm not sure thats all of these working properly or most efficiently if you find something that maybe improve please point me about that. Thank you.\n\n* [Fluentd in_tail](https://docs.fluentd.org/v1.0/articles/in_tail)\n* [Fluentd ElastciSearch](https://docs.fluentd.org/v1.0/articles/out_elasticsearch)\n* [Fluentd ElastciSearch repo](https://github.com/uken/fluent-plugin-elasticsearch)\n* [Fluentd Buffers](https://docs.fluentd.org/v1.0/articles/buffer-section)\n* [Fluentd Buffer Overview](https://docs.fluentd.org/v1.0/articles/buffer-plugin-overview)\n* [Fluentd Buffer File](https://docs.fluentd.org/v1.0/articles/buf_file)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgingray%2Ffluentd_tail_elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgingray%2Ffluentd_tail_elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgingray%2Ffluentd_tail_elasticsearch/lists"}