{"id":13481361,"url":"https://github.com/childe/gohangout","last_synced_at":"2025-05-14T12:09:02.276Z","repository":{"id":28622739,"uuid":"118875954","full_name":"childe/gohangout","owner":"childe","description":"使用 golang 模仿的 Logstash。用于消费 Kafka 数据，处理后写入 ES、Clickhouse 等。","archived":false,"fork":false,"pushed_at":"2025-04-09T09:06:36.000Z","size":780,"stargazers_count":1062,"open_issues_count":5,"forks_count":246,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-04-11T04:58:12.910Z","etag":null,"topics":["elasticsearch","golang","kafka","logstash"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/childe.png","metadata":{"files":{"readme":"README-EN.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-25T07:04:02.000Z","updated_at":"2025-04-09T09:06:23.000Z","dependencies_parsed_at":"2023-12-02T05:27:59.387Z","dependency_job_id":"386cf35f-8c2e-4019-8a8f-1dc219d2681a","html_url":"https://github.com/childe/gohangout","commit_stats":{"total_commits":847,"total_committers":25,"mean_commits":33.88,"dds":0.358913813459268,"last_synced_commit":"3d8a1af5b700a2ccf5bb5f066a746070ea3d7019"},"previous_names":[],"tags_count":115,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/childe%2Fgohangout","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/childe%2Fgohangout/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/childe%2Fgohangout/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/childe%2Fgohangout/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/childe","download_url":"https://codeload.github.com/childe/gohangout/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248345273,"owners_count":21088244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","golang","kafka","logstash"],"created_at":"2024-07-31T17:00:51.207Z","updated_at":"2025-04-11T04:58:21.886Z","avatar_url":"https://github.com/childe.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"Gohangout is an application to do data transport. It consumes data from `input plugin` such as kafka or tcp/udp , and do data transforms using `filter plugin`, and then emit data to `output plugin`, such as Elasticsearch or Clickhouse.\n\n## Install\n\nWe could build it from source code , or download binary app.\n\n### build from source code\n\njust clone code and run make\n\n```\nmake\n```\n\nIt is recommended to compile it with CGO disabled if you want to run it in docker.\n\n```\nGOOS=linux GOARCH=amd64 CGO_ENABLED=0 make\n```\n\n### download binary\n\n[https://github.com/childe/gohangout/releases](https://github.com/childe/gohangout/releases) \n\n### go get\n\n```\ngo get github.com/childe/gohangout\n```\n\n### Third party plugin\n\n- Exmaples for developing 3th party plugin [gohangout-plugin-examples](https://github.com/childe/gohangout-plugin-examples)\n- [Kafka Input using Saramp](https://github.com/DukeAnn/gohangout-input-kafka_sarama)\n- [Kafka Input using kafka-go](https://github.com/huangjacky/gohangout-input-kafkago)\n- [Redis Input](https://github.com/childe/gohangout-input-redis)\n- [Split Filter](https://github.com/childe/gohangout-plugin-examples/tree/master/gohangout-filter-split) Split one message to multi\n- [File Output](https://github.com/childe/gohangout-plugin-examples/tree/master/gohangout-file-output) file output\n\n## Run\n\n```\ngohangout --config config.yml\n```\n\n### log\n\nGohangout use klog.\n\nuse `-v n` to set log level. \n\nI usually set n to 5. You can set it to 10 or 20 to see more detailed log.\n\n### multi threads\n\n--worker 4 (default 1)\n\nabove args make gohangout use 4 goroutines to process data.  Notice: one thread to consume from input, and then have 4 goroutines to do filter and output.\n\n### reload\n\n--reload\n\nabove args enables reload. Gohangout will relaod config file when it changes.\n\n`kill -USER $pid` also triggers reload.\n\n## Simple Config Example\n\n```\ninputs:\n    - Kafka:\n        topic:\n            weblog: 1 # One Kafka consumer thread\n        codec: json\n        consumer_settings:\n            bootstrap.servers: \"10.0.0.100:9092\"\n            group.id: gohangout.weblog\nfilters:\n    - Grok:\n        src: message\n        match:\n            - '^(?P\u003clogtime\u003e\\S+) (?P\u003cname\u003e\\w+) (?P\u003cstatus\u003e\\d+)$'\n            - '^(?P\u003clogtime\u003e\\S+) (?P\u003cstatus\u003e\\d+) (?P\u003cloglevel\u003e\\w+)$'\n        remove_fields: ['message']\n    - Date:\n        location: 'UTC'\n        src: logtime\n        formats:\n            - 'RFC3339'\n        remove_fields: [\"logtime\"]\noutputs:\n    - Elasticsearch:\n        hosts:\n            - 'http://admin:password@127.0.0.1:9200'\n        index: 'web-%{appid}-%{+2006-01-02}'\n        index_type: \"logs\"\n        bulk_actions: 5000\n        bulk_size: 20\n        flush_interval: 60\n```\n\n## Value Render Protocol\n\nsome exmaples and explanation below\n\n```\nfields:\n    logtime: '%{date} {%time}'\n    type: 'weblog'\n    hostname: '[host]'\n    name: '{{.firstname}}.{{.lastname}}'\n    name2: '$.name'\n    city: '[geo][cityname]'\n    '[a][b]': '[stored][message]'\n```\n\n### format 1 JSONPATH\n\nGohangout use JsonPath to render value if it begins witch `$.`\n\n```\n$.store.book[0].title\n\n$['store']['book'][0]['title']\n\n$.store.book[(@.length-1)].title\n\n$.store.book[?(@.price \u003c 10)].title\n```\n\nMore usage and examples: [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/)\n\n### format2 [X][Y]\n\n**Not recommended, please use format 1**\n\n`city: '[geo][cityname]'` equals to `$.geo.cityname` . It must be strictly `[X][Y]`, in other words, there can not be any other words in front of `[X][Y]` or after it.\n\n### format 3 {{XXX}}\n\nGohangout will render value using [Golang Template]((https://golang.org/pkg/text/template/)). It could contains other words before or after {{XXX}}, such as `name: 'my name is {{.firstname}}.{{.lastname}}'`\n\nOne example you may use: We get a time-type field with `Date` filter, and then render a string with customed format.\n\n```\nAdd:\n  fields:\n    ts: '{{ .ts.Format \"2006.01.02\" }}'\n```\n\n### format 4 %{XXX}\n\nit itherits from [logstash](https://www.elastic.co/logstash)\n\nfor example, render index name in Elasticsearch output: `web-%{appid}-%{+2006-01-02}` .\n\n## Input\n\nAll settings in below plugins could be checked in [Chinese doc](https://github.com/childe/gohangout/blob/master/README.md#input). \n\nSetting and explanation in English doc will be added later.\n\n- Stdin\n- TCP\n- Kafka\n\n## Output\n\n- Stdout\n- TCP\n- Elasticsearch\n- Kafka\n- ClickHouse\n\n## Filter\n\n### general settings\n\n#### if\n\nsyntax example\n\n```\nDrop:\n    if:\n      - 'EQ(name,\"childe\")'\n      - 'Before(-24h) || After(24h)'\n```\n\nRelationship between conditions is **AND**, if passes only if all conditions pass.\n\nmore complicated example using bool operator: `Exist(a) \u0026\u0026 (!Exist(b) || !Exist(c))`\n\nAll functions supported for now:\n\n**NOtice**: value in EQ/IN functions must be quoted by \" , because the value could be a number or a string.  User must tell Gohangout whether it is a string or a number.  \nvalue in other functions could be quoted by \" or not , gohangout will treat it as string.\n\n- `Exist(user,name)` if [user][name] exists\n\n- `EQ(user,age,20)` `EQ($.user.age,20)` if [user][age] exists and equal to 20\n\n- `EQ(user,age,\"20\")` `EQ($.user.age,\"20\")` if [user][age] exists and equal to \"20\" (string)\n\n- `IN(tags,\"app\")` `IN($.tags,\"app\")` tags is a list or do not pass. if \"app\" contained in the list\n\n- `HasPrefix(user,name,liu)` `HasPrefix($.user.name,\"liu\")`\n\n- `HasSuffix(user,name,jia)` `HasSuffix($.user.name,\"jia\")`\n\n- `Contains(user,name,jia)` `Contains($.user.name,\"jia\")`\n\n- `Match(user,name,^liu.*a$)` `Match($.user.name,\"^liu.*a$\")`\n\n- `Random(20)` return true with 5% probability\n\n- `Before(24h)`  *@timestamp* field exists and it must be a Time type\n- `After(-24h)`  *@timestamp* field exists and it must be a Time type\n\n#### add_fields\n\nexample:\n\n```\nGrok:\n    src: message\n    match:\n        - '^(?P\u003clogtime\u003e\\S+) (?P\u003cname\u003e\\w+) (?P\u003cstatus\u003e\\d+)$'\n        - '^(?P\u003clogtime\u003e\\S+) (?P\u003cstatus\u003e\\d+) (?P\u003cloglevel\u003e\\w+)$'\n    remove_fields: ['message']\n    add_fields:\n      grok_result: 'ok'\n```\n\nFields could be added if the filter process the event successfully. And it is ignored if filter failed to process the event.\n\n#### remove_fields\n\n```\nGrok:\n    src: message\n    match:\n        - '^(?P\u003clogtime\u003e\\S+) (?P\u003cname\u003e\\w+) (?P\u003cstatus\u003e\\d+)$'\n        - '^(?P\u003clogtime\u003e\\S+) (?P\u003cstatus\u003e\\d+) (?P\u003cloglevel\u003e\\w+)$'\n    remove_fields: ['message']\n    add_fields:\n      grok_result: 'ok'\n```\n\nremove some fields if the filter process the event successfully. And it is ignored if filter failed to process the event. \n\n### Filter Plugins\n\n- Add\n- Convert\n- Date # it convert one string-type field to Time-type field\n- Drop\n- Filters\n- Grok\n- IPIP\n- KV\n- Lowercase\n- Remove\n- Rename\n- Split\n- Translate\n- Uppercase\n- Replace\n- URLDecode\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchilde%2Fgohangout","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchilde%2Fgohangout","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchilde%2Fgohangout/lists"}