{"id":16682502,"url":"https://github.com/shulhan/haminer","last_synced_at":"2025-03-17T00:32:44.358Z","repository":{"id":56796014,"uuid":"127550543","full_name":"shuLhan/haminer","owner":"shuLhan","description":"[mirror] Library and program to parse and forward HAProxy HTTP logs","archived":false,"fork":false,"pushed_at":"2024-09-07T18:22:43.000Z","size":175,"stargazers_count":23,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-02-27T15:28:49.176Z","etag":null,"topics":["go","haproxy","influxdb","log","miner"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shuLhan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-31T16:34:29.000Z","updated_at":"2024-11-20T06:04:46.000Z","dependencies_parsed_at":"2024-06-19T05:16:26.330Z","dependency_job_id":"577c7314-2319-41a3-9991-c8e29fd107b3","html_url":"https://github.com/shuLhan/haminer","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shuLhan%2Fhaminer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shuLhan%2Fhaminer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shuLhan%2Fhaminer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shuLhan%2Fhaminer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shuLhan","download_url":"https://codeload.github.com/shuLhan/haminer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243835942,"owners_count":20355611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","haproxy","influxdb","log","miner"],"created_at":"2024-10-12T14:07:43.441Z","updated_at":"2025-03-17T00:32:44.352Z","avatar_url":"https://github.com/shuLhan.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# haminer\n\nLibrary and program to parse and forward HAProxy logs.\n\nThe HTTP logs is HTTP request that received by HAProxy frontend and forwarded\nto backend.\nIn default format, it looks like these (split into multi lines, for\nreadability):\n\n```\n\u003c158\u003eSep  4 17:08:47 haproxy[109530]: 185.83.144.103:46376\n  [04/Sep/2022:17:08:47.264] www~ be_kilabit/kilabit-0.0/0/1/2/3 200 89 - -\n  ---- 5/5/0/0/0 0/0 \"GET / HTTP/1.1\"\n```\n\nSee\n[HTTP log format documentation](https://www.haproxy.com/documentation/hapee/1-8r1/onepage/#8.2.3)\nfor more information.\n\nCurrently, there are supported database where haminer can forward the\nparsed log: Influxdb, Questdb, and Postgresql.\nHaminer support Influxdb v1 and v2.\n\n```\n +---------+  UDP  +---------+      +-----------+\n | HAProxy |------\u003e| haminer |-----\u003e| Influxdb  |\n +---------+       +---------+      | / Questdb |\n                                    +-----------+\n```\n\nIn Influxdb, the log are stored as measurement called `haproxy`.\nIn Questdb, the log are stored as table called `haproxy`.\n\nThe following fields are stored as tags (in Influxdb) or symbol (in Questdb):\nhost, server, backend, frontend, http_method, http_url, http_query,\nhttp_proto, http_status, term_state, client_ip, client_port.\n\nAnd the following fields are stored as fields (in Influxdb) or values (in\nQuestdb): time_req, time_wait, time_connect, time_rsp, time_all,\nconn_active, conn_frontend, conn_backend, conn_server, conn_retries,\nqueue_server, queue_backend, bytes_read.\n\nOnce the log has been accumulated, we can query the data.\nFor example, with Questdb we can count each visited URL using the following\nquery,\n\n```\nselect backend, http_url, count(*) as visit from 'haproxy'\ngroup by backend, http_url\norder by visit desc;\n```\n\n## Installation\n\n### Building from source\n\nRequirements,\n\n- [Go](https://golang.org) for building the source code\n- [git](https://git-scm.com) for downloading the source code\n\nGet the source code using git,\n\n```\n$ git clone https://git.sr.ht/~shulhan/haminer\n$ cd haminer\n$ make\n```\n\nThe binary name is `haminer` build in the current directory.\n\n\n### Pre-build package\n\nThe Arch Linux package is available at build.kilabit.info.\nAdd the following repository to your `pacman.conf`,\n\n```\n[build.kilabit.info]\nServer = https://build.kilabit.info/aur\n```\n\nTo install it,\n\n\t$ sudo pacman -Sy --noconfirm haminer-git\n\n\n## Configuration\n\nhaminer by default will load it's config from `/etc/haminer.conf`, if not\nspecified when running the program.\n\nSee\n[haminer.conf](https://git.sr.ht/~shulhan/haminer/tree/main/item/cmd/haminer/haminer.conf)\nfor an example of possible configuration and their explanation.\n\n\n### Forwarders\n\nCurrently, there are several database where haminer can forward the parsed\nlog: Influxdb,  Questdb, and Postgresql.\nHaminer support Influxdb v1 and v2.\n\n#### Influxdb v1\n\nFor v1, you need to create the user and database first,\n\n```\n$ influx\n\u003e CREATE USER \"haminer\" WITH PASSWORD 'haminer'\n\u003e CREATE DATABASE haminer\n\u003e GRANT ALL ON haminer TO haminer\n```\n\nExample of forwarder configuration,\n\n```\n[forwarder \"influxd\"]\nversion = v1\nurl = http://127.0.0.1:8086\nbucket = haminer\nuser = haminer\npassword  = haminer\n```\n\n#### Influxdb v2\n\nFor v2,\n\n```\n$ sudo influx bucket create \\\n\t--name haminer \\\n\t--retention 30d\n```\n\nFor v2, the example configuration is\n\n```\n[forwarder \"influxd\"]\nversion = v2\nurl = http://127.0.0.1:8086\norg = $org\nbucket = haminer\ntoken = $token\n```\n\n#### Questdb\n\nFor Questdb the configuration is quite simple,\n\n```\n[forwarder \"questdb\"]\nurl = udp://127.0.0.1:9009\n```\n\nWe did not need to create the table, Questdb will handled that automatically.\n\n#### Postgresql\n\nFor Postgresql, you need to create the user and database first, for example,\n\n```\npostgres$ psql\npostgres=\u003e CREATE ROLE haminer PASSWORD 'haminer' CREATEDB INHERIT LOGIN;\npostgres=\u003e CREATE DATABASE haminer OWNER haminer;\npostgres=\u003e \\q\n```\n\nThe configuration only need the Data Source Name (DSN),\n\n```\n[forwarder \"postgresql\"]\nurl = postgres://\u003cuser\u003e:\u003cpass\u003e@\u003chost\u003e/\u003cdatabase\u003e?sslmode=\u003crequire|verify-full|verify-ca|disable\u003e\n```\n\n\n## Deployment\n\nCopy configuration from `$SOURCE/cmd/haminer/haminer/conf` to\n`/etc/haminer.conf`\n\nUpdate haminer configuration in `/etc/haminer.conf`.\nFor example,\n\n```\n[haminer]\nlisten = 127.0.0.1:5140\n\n...\n```\n\nAdd one or more provider to the configuration as the example above.\n\nUpdate HAProxy config to forward log to UDP port other than rsyslog.\nFor example,\n\n```\nglobal\n\t...\n\tlog 127.0.0.1:5140 local3\n\t...\n```\n\nThen reload or restart HAProxy.\n\nRun the haminer program,\n\n```\n$ haminer\n```\n\nor use a\n[systemd service](https://git.sr.ht/~shulhan/haminer/tree/main/item/cmd/haminer/haminer.service).\n\n```\n$ sudo systemctl enable haminer\n$ sudo systemctl start  haminer\n```\n\n\n##  License\n\n```\nhaminer - Library and program to parse and forward HAProxy logs.\nCopyright (C) 2018-2024 M. Shulhan \u003cms@kilabit.info\u003e\n\nThis program is free software: you can redistribute it and/or modify it under\nthe terms of the GNU General Public License as published by the Free Software\nFoundation, either version 3 of the License, or any later version.\n\nThis program is distributed in the hope that it will be useful, but WITHOUT\nANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS\nFOR A PARTICULAR PURPOSE.  See the GNU General Public License for more\ndetails.\n\nYou should have received a copy of the GNU General Public License along with\nthis program.\nIf not, see \u003chttp://www.gnu.org/licenses/\u003e.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshulhan%2Fhaminer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshulhan%2Fhaminer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshulhan%2Fhaminer/lists"}