{"id":16307092,"url":"https://github.com/fujiwara/fluent-agent-hydra","last_synced_at":"2025-06-27T10:36:57.464Z","repository":{"id":19230286,"uuid":"22464816","full_name":"fujiwara/fluent-agent-hydra","owner":"fujiwara","description":"A Fluentd log agent.","archived":false,"fork":false,"pushed_at":"2024-01-16T11:41:34.000Z","size":3883,"stargazers_count":111,"open_issues_count":4,"forks_count":19,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-01T22:10:15.848Z","etag":null,"topics":["fluentd","go","golang","kayac"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fujiwara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-31T10:03:10.000Z","updated_at":"2024-11-12T11:37:42.000Z","dependencies_parsed_at":"2024-06-20T10:23:29.479Z","dependency_job_id":"5b5cbef8-a9d7-4fb0-bfcd-d1f5de379f54","html_url":"https://github.com/fujiwara/fluent-agent-hydra","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/fujiwara/fluent-agent-hydra","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2Ffluent-agent-hydra","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2Ffluent-agent-hydra/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2Ffluent-agent-hydra/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2Ffluent-agent-hydra/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fujiwara","download_url":"https://codeload.github.com/fujiwara/fluent-agent-hydra/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2Ffluent-agent-hydra/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262240813,"owners_count":23280588,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","go","golang","kayac"],"created_at":"2024-10-10T21:12:28.294Z","updated_at":"2025-06-27T10:36:57.443Z","avatar_url":"https://github.com/fujiwara.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# fluent-agent-hydra\n\nA Fluentd log agent.\n\n[![](https://github.com/fujiwara/fluent-agent-hydra/workflows/Go/badge.svg)](https://github.com/fujiwara/fluent-agent-hydra/actions?query=workflow%3AGo+branch%3Amaster)\n[![](https://github.com/fujiwara/fluent-agent-hydra/workflows/Release/badge.svg)](https://github.com/fujiwara/fluent-agent-hydra/actions?query=workflow%3ARelease+branch%3Amaster)\n\nThis agent is inspired by [fluent-agent-lite](https://github.com/tagomoris/fluent-agent-lite).\n\n## Features\n\n- Tailing log files (like in_tail)\n  - enable to handle multiple files in a single process.\n  - parse JSON or LTSV format.\n- Forwarding messages to external fluentd (like out_forward)\n  - multiple fluentd server can be used. When primary server is down, messages will sent to secondary server.\n  - if config.ServerRoundRobin = true, select one server from all servers by round robin.\n- Receiving a fluentd's forward protocol messages via TCP (like in_forward)\n  - includes simplified on-memory queue.\n- Stats monitor httpd server\n  - serve an agent stats by JSON format.\n- Supports sub-second time\n  - Supported by Fluentd 0.14 or later. When you use Fluentd \u003c= 0.12 as forwarded servers, fluentd will not accept records including sub-second time.\n\n## Installation\n\n[Binary releases](https://github.com/fujiwara/fluent-agent-hydra/releases)\n\nor\n\n```\ngo get github.com/fujiwara/fluent-agent-hydra/cmd/fluent-agent-hydra/\n```\n\n## Usage\n\n### Using command line arguments\n\n```\nfluent-agent-hydra [options] TAG TARGET_FILE PRIMARY_SERVER SECONDARY_SERVER\n```\n\nOptions\n\n* -f=\"message\": fieldname of fluentd log message attribute (DEFAULT: message)\n* -monitor=\"127.0.0.1:24223\": monitor httpd daemon address (DEFAULT: -)\n\nUsage example\n\n```\nfluent-agent-hydra -f msg tagname /path/to/foo.log fluentd.example.com:24224 127.0.0.1:24224\n```\n\n* Field name: \"msg\"\n* Filename: \"/path/to/foo.log\"\n* Primary server: \"fluentd.example.com:24224\"\n* Secondary server: \"127.0.0.1:24224\"\n\n### Using configuration file\n\n```\nfluent-agent-hydra -c /path/to/config.toml\n```\n\nA example of config.toml\n\n```toml\n# global settings\nTagPrefix = \"nginx\"       # \"nginx.access\", \"nginx.error\"\nFieldName = \"message\"     # default \"message\"\nReadBufferSize = 1048576  # default 64KB.\nServerRoundRobin = true   # default false\nSubSecondTime = true      # default false. for Fluentd 0.14 or later only\n\n# tailing log file (in_tail)\n[[Logs]]\nFile = \"/var/log/nginx/access.log\"\nTag = \"access\"\n# parse as ltsv format. (see http://ltsv.org/)\n# Format = \"None\"(default) | \"LTSV\" | \"JSON\" | \"Regexp\"\nFormat = \"LTSV\"\n\n# If Format is \"Regexp\", Regexp directive is required.\n# Regexp = \"(your regexp string)\" | \"apache\" | \"nginx\" | \"syslog\"\n\n# convert column data type\n# 'column1_name:type,column2_name:type'\n# type = \"interger\" | \"float\" | \"bool\" | otherwise as string\nTypes = \"reqtime:float,size:integer,apptime:float,status:integer\"\n\n# parse a time string in log lines, and set it as record's timestamp\nTimeParse = true      # default false\nTimeKey = \"timestamp\" # default \"time\"\n\n# TimeFormat is passed to Golang's time.Parse().\n# http://golang.org/pkg/time/#Parse\n# default time.RFC3339 == \"2006-01-02T15:04:05Z07:00\"\n# \"apache\" | \"nginx\" | \"syslog\" | \"unix\" is also available\nTimeFormat = \"02/Jan/2006:15:04:05 Z0700\"\n\n[[Logs]]\nFile = \"/var/log/nginx/error.log\"\nTag = \"error\"\n\n# forwarding fluentd server (out_forward)\n[[Servers]]\nHost = \"fluentd.example.com\"\nPort = 24224\n\n[[Servers]]\nHost = \"fluentd-backup.example.com\"\nPort = 24224\n\n# receive fluentd forward protocol daemon (in_forward)\n[Receiver]\nPort = 24224\n\n# stats monitor http daemon\n[Monitor]\nHost = \"localhost\"\nPort = 24223\n```\n\n### About special conversion behavior for numerical value\n\nWhen the `Format` is JSON, fluent-agent-hydra treats a numerical value as float64 even if its type is integer.\nFor treating a numerical value as integer, set a column data type integer with the `Types`.\n\n```toml\nTypes = \"column_name:integer\"\n```\n\nIts type is converted to int64.\n\n## Stats monitor\n\nFor enabling stats monitor, specify command line option `-m host:port` or `[Monitor]` section in config file.\n\n### Hydra application stats\n\n`curl -s [Monitor.Host]:[Monitor.Port]/ | jq .`\n\nAn example response.\n\n```json\n{\n  \"receiver\": {\n    \"buffered\": 0,\n    \"disposed\": 0,\n    \"messages\": 123,\n    \"max_buffer_messages\": 1048576,\n    \"current_connections\": 1,\n    \"total_connections\": 10,\n    \"address\": \"[::]:24224\"\n  },\n  \"servers\": [\n    {\n      \"error\": \"\",\n      \"alive\": true,\n      \"address\": \"fluentd.example.com:24224\"\n    },\n    {\n      \"error\": \"[2014-08-18 18:25:28.965066394 +0900 JST] dial tcp 192.168.1.11:24224: connection refused\",\n      \"alive\": false,\n      \"address\": \"fluentd-backup.example.com:24224\"\n    }\n  ],\n  \"files\": {\n    \"/var/log/nginx/error.log\": {\n      \"error\": \"\",\n      \"position\": 95039,\n      \"tag\": \"nginx.error\"\n    },\n    \"/var/log/nginx/access.log\": {\n      \"error\": \"\",\n      \"position\": 112093,\n      \"tag\": \"nginx.access\"\n    }\n  },\n  \"sent\": {\n    \"nginx.error\": {\n      \"bytes\": 2578,\n      \"messages\": 8\n    },\n    \"nginx.access\": {\n      \"bytes\": 44996,\n      \"messages\": 109\n    }\n  }\n}\n```\n\n### system stats\n\n`curl -s [Monitor.Host]:[Monitor.Port]/system | jq .`\n\nAn example response.\n\n```json\n{\n  \"time\": 1417748153556699400,\n  \"go_version\": \"go1.3\",\n  \"go_os\": \"darwin\",\n  \"go_arch\": \"amd64\",\n  \"cpu_num\": 4,\n  \"goroutine_num\": 17,\n  \"gomaxprocs\": 1,\n  \"cgo_call_num\": 48,\n  \"memory_alloc\": 551840,\n  \"memory_total_alloc\": 17886960,\n  \"memory_sys\": 5310712,\n  \"memory_lookups\": 321,\n  \"memory_mallocs\": 4645,\n  \"memory_frees\": 3622,\n  \"memory_stack\": 131072,\n  \"heap_alloc\": 551840,\n  \"heap_sys\": 2097152,\n  \"heap_idle\": 1253376,\n  \"heap_inuse\": 843776,\n  \"heap_released\": 0,\n  \"heap_objects\": 1023,\n  \"gc_next\": 1083088,\n  \"gc_last\": 1417748153454501600,\n  \"gc_num\": 34,\n  \"gc_per_second\": 0.966939666110773,\n  \"gc_pause_per_second\": 0.641048,\n  \"gc_pause\": [\n    0.2991,\n    0.341948\n  ]\n}\n```\n\n## Benchmark\n\nSee [benchmark/README](benchmark/README.md) .\n\n## Thanks to\n\n* `fluent/fluent.go, utils.go` imported and modified from [github.com/t-k/fluent-logger-golang](https://github.com/t-k/fluent-logger-golang).\n* `fluent/server.go` imported and modified from [github.com/moriyoshi/ik](https://github.com/moriyoshi/ik/).\n\n## Author\n\nFujiwara Shunichiro \u003cfujiwara.shunichiro@gmail.com\u003e\n\n## Licence\n\nCopyright 2014 Fujiwara Shunichiro. / KAYAC Inc.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffujiwara%2Ffluent-agent-hydra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffujiwara%2Ffluent-agent-hydra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffujiwara%2Ffluent-agent-hydra/lists"}