{"id":18150055,"url":"https://github.com/sysulq/gofluent","last_synced_at":"2025-09-25T22:29:12.936Z","repository":{"id":21611021,"uuid":"24931326","full_name":"sysulq/gofluent","owner":"sysulq","description":"(Not Maintained) Something acting like fluentd rewritten in Go.","archived":false,"fork":false,"pushed_at":"2015-08-14T15:27:10.000Z","size":11761,"stargazers_count":181,"open_issues_count":0,"forks_count":63,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-03-28T13:21:33.358Z","etag":null,"topics":["fluentd","go","httpsqs","mongodb","tail"],"latest_commit_sha":null,"homepage":"http://hnlq715.github.io/gofluent","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sysulq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-08T08:26:39.000Z","updated_at":"2024-07-30T09:24:12.000Z","dependencies_parsed_at":"2022-07-16T14:18:25.051Z","dependency_job_id":null,"html_url":"https://github.com/sysulq/gofluent","commit_stats":null,"previous_names":["sysulq/gofluent","hnlq715/gofluent"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fgofluent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fgofluent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fgofluent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sysulq%2Fgofluent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sysulq","download_url":"https://codeload.github.com/sysulq/gofluent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248988977,"owners_count":21194507,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fluentd","go","httpsqs","mongodb","tail"],"created_at":"2024-11-02T00:07:20.266Z","updated_at":"2025-09-25T22:29:12.822Z","avatar_url":"https://github.com/sysulq.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"gofluent\n========\n[![wercker status](https://app.wercker.com/status/1908429c90a6d7bd437a63210fb1a97f/s \"wercker status\")](https://app.wercker.com/project/bykey/1908429c90a6d7bd437a63210fb1a97f)\n\nThis program is something acting like fluentd rewritten in Go.\n\nTable of Contents\n=================\n\n* [Introduction](#introduction)\n* [Architecture](#architecture)\n* [Implementation](#implementation)\n\t* [Overview](#overview)\n\t* [Data flow](#data-flow)\n* [Plugins](#plugins)\n\t* [Tail Input Plugin](#tail-input-plugin)\n\t* [Httpsqs Output Plugin](#httpsqs-output-plugin)\n\t* [Stdout Output Plugin](#stdout-output-plugin)\n\t* [Mongodb Output Plugin](#mongodb-output-plugin)\n\nIntroduction\n============\n\nFluentd is originally written in CRuby, which has too many dependecies.\n\nI hope fluentd to be simpler and cleaner, as its main feature is simplicity and rubostness.\n\nArchitecture\n============\n\n```\n    +---------+     +---------+     +---------+     +---------+\n    | server1 |     | server2 |     | server3 |     | serverN |\n    |---------|     |---------|     |---------|     |---------|\n    |         |     |         |     |         |     |         |\n    |---------|     |---------|     |---------|     |---------|\n    |gofluent |     |gofluent |     |gofluent |     |gofluent |\n    +---------+     +---------+     +---------+     +---------+\n        |               |               |               |\n         -----------------------------------------------\n                                |\n                                | HTTP POST\n                                V\n                        +-----------------+\n                        |                 |\n                        |      Httpmq     |\n                        |                 | \n                        +-----------------+\n                                |\n                                | HTTP GET\n                                V \n                        +-----------------+                 +-----------------+\n                        |                 |                 |                 |\n                        |   Preprocessor  | --------------\u003e |     Storage     |\n                        |                 |                 |                 | \n                        +-----------------+                 +-----------------+\n```\n\nImplementation\n==============\nOverview\n--------\n\n```\nInput -\u003e Router -\u003e Output\n```\nData flow\n---------\n\n```\n                        -------\u003c-------- \n                        |               |\n                        V               | generate pool\n       InputRunner.inputRecycleChan     | recycling\n            |           |               |     ^   \n            |            -------\u003e-------       \\ \n            |               ^           ^       \\\n    InputRunner.inChan      |           |        \\\n            |               |           |         \\\n            |               |           |          \\\n    consume |               |           |           \\\n            V               |           |            \\\n          Input(Router.inChan) ----\u003e  Router ----\u003e (Router.outChan)Output.inChan\n\n```\n\nPlugins\n=======\n\nTail Input Plugin\n-----------------\nThe in_tail input plugin allows gofluent to read events from the tail of text files. Its behavior is similar to the tail -F command.\n\nExample Configuration\n\nin_tail is included in gofluent’s core. No additional installation process is required.\n```\n\u003csource\u003e\n  type tail\n  path /var/log/httpd-access.log\n  pos_file /var/log/httpd-access.log.pos\n  tag apache.access\n  format /^(?P\u003chost\u003e[^ ]*) [^ ]* (?P\u003cuser\u003e[^ ]*) \\[(?P\u003ctime\u003e[^\\]]*)\\] \"(?P\u003cmethod\u003e\\S+)(?: +(?P\u003cpath\u003e[^ ]*) +\\S*)?\" (?P\u003ccode\u003e[^ ]*) (?P\u003csize\u003e[^ ]*)(?: \"(?P\u003creferer\u003e[^\\\"]*)\" \"(?P\u003cagent\u003e[^\\\"]*)\")?$/\n\u003c/source\u003e\n```\n*type (required)*\nThe value must be tail.\n\n*tag (required)*\nThe tag of the event.\n\n*path (required)*\nThe paths to read.\n\n*format (required)*\nThe format of the log. It is the name of a template or regexp surrounded by ‘/’.\nThe regexp must have at least one named capture (?P\\\u003cNAME\\\u003ePATTERN).\n\nThe following templates are supported:\n- regexp\n- json\nOne JSON map, per line. This is the most straight forward format :).\n```\nformat json\n```\n*pos_file (highly recommended)*\nThis parameter is highly recommended. gofluent will record the position it last read into this file.\n```\npos_file /var/log/access.log.pos\n```\n\n*sync_interval*\nThe sync interval of pos file, default is 2s.\n\nHttpsqs Output Plugin\n---------------------\nThe out_httpsqs output plugin allows gofluent to send data to httpsqs mq.\n\nExample Configuration\n\nout_httpsqs is included in gofluent’s core. No additional installation process is required.\n```\n\u003cmatch httpsqs.**\u003e\n  type httpsqs\n  host localhost\n  port 1218\n  flush_interval 10\n\u003c/match\u003e\n```\n*type (required)*\nThe value must be httpsqs.\n\n*host (required)*\nThe output target host ip.\n\n*port (required)*\nThe output target host port.\n\n*auth (highly recommended)*\nThe auth password for httpsqs.\n\n*flush_interval*\nThe flush interval for sending data to httpsqs.\n\n*gzip*\nThe gzip switch, default is on.\n\nForward Output Plugin\n---------------------\nThe out_forward output plugin allows gofluent to forward events to another gofluent.\n\nExample Configuration\n\nout_forward is included in gofluent’s core. No additional installation process is required.\n```\n\u003cmatch forward.**\u003e\n  type forward\n  host localhost\n  port 1218\n  flush_interval 10\n\u003c/match\u003e\n```\n*type (required)*\nThe value must be forward.\n\n*host (required)*\nThe output target host ip.\n\n*port (required)*\nThe output target host port.\n\n*flush_interval*\nThe flush interval for sending data to httpsqs.\n\n*connect_timeout*\nThe connect timeout value.\n\n*sync_interval*\nThe sync interval of metadata file, default is 2s.\n\n*buffer_path(required)*\nThe disk buffer path for output plugin, default is /tmp/test.\n\n*buffer_queue_limit*\nThe queue limit of disk buffer, default is 64M.\n\n*buffer_chunk_limit*\nThe chunk limit of disk buffer to forward, default is 8M.\n\nStdout Output Plugin\n--------------------\nThe out_stdout output plugin allows gofluent to print events to stdout.\n\nExample Configuration\n\nout_stdout is included in gofluent’s core. No additional installation process is required.\n```\n\u003cmatch stdout.**\u003e\n  type stdout\n\u003c/match\u003e\n```\n*type (required)*\nThe value must be stdout.\n\nMongodb Output Plugin\n---------------------\nThe out_mongodb output plugin allows gofluent to send message to mongodb.\n\nExample Configuration\n\nout_mongodb is included in gofluent's core. No additional installation process is required.\n```\n\u003cmatch mongodb.**\u003e\n  type mongodb\n  host localhost\n  port 27017\n  capped on\n  capped_size 1024\n  database test\n  collection test\n  user test\n  password test\n\u003c/match\u003e\n```\n*type(required)* \nThe value must be mongodb.\n\n*host (required)*\nThe output target host ip.\n\n*port (required)*\nThe output target host port.\n\n*capped (highly recommended)*\nTo create a capped collection.\n\n*capped_size*\nSpecify the actual size(MB) of the capped collection.\n\n*database(required)*\nThe name of the database to be created or used.\n\n*collection(required)*\nThe name of the collection to be created or used.\n\n*user(highly recommended)*\nThe user name to login the database.\n\n*password(highly recommended)*\nThe password to login the database.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysulq%2Fgofluent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsysulq%2Fgofluent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsysulq%2Fgofluent/lists"}