{"id":28567008,"url":"https://github.com/altinity/clicktail","last_synced_at":"2025-06-10T15:41:00.073Z","repository":{"id":57535349,"uuid":"120750954","full_name":"Altinity/clicktail","owner":"Altinity","description":null,"archived":false,"fork":false,"pushed_at":"2019-10-01T23:08:15.000Z","size":5782,"stargazers_count":281,"open_issues_count":10,"forks_count":58,"subscribers_count":15,"default_branch":"master","last_synced_at":"2024-06-21T03:02:33.019Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Altinity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-08T11:09:43.000Z","updated_at":"2024-04-29T10:30:58.000Z","dependencies_parsed_at":"2022-09-26T18:21:54.161Z","dependency_job_id":null,"html_url":"https://github.com/Altinity/clicktail","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclicktail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclicktail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclicktail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclicktail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Altinity","download_url":"https://codeload.github.com/Altinity/clicktail/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Altinity%2Fclicktail/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259104370,"owners_count":22805866,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-10T15:40:23.601Z","updated_at":"2025-06-10T15:41:00.043Z","avatar_url":"https://github.com/Altinity.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clicktail\n\n`clicktail` is [Altinity](https://altinity.com)'s agent for ingesting log file data into ClickHouse and making it available for exploration. Its favorite format is **JSON**, but understands how to parse a range of other well-known log formats.\n\n`clicktail` is based on `honeytail` tool see [here](https://github.com/honeycombio/honeytail) and [its documentation](https://honeycomb.io/docs/send-data/agent/)\n\n## Supported Parsers\n\n`clicktail` supports reading files from `STDIN` as well as from a file on disk.\n\nOur complete list of parsers can be found in the [`parsers/` directory](parsers/), but as of this writing, `clicktail` will support parsing logs generated by:\n\n- [ArangoDB](parsers/arangodb/)\n- [MongoDB](parsers/mongodb/)\n- [MySQL](parsers/mysql/)\n- [PostgreSQL](parsers/postgresql/)\n- [nginx](parsers/nginx/)\n- [regex](parsers/regex/)\n- [mysqlaudit](parsers/mysqlaudit/)\n\n## Installation\n\nThere are three installation options for clicktail:\n+ From package repository\n+ From downloaded package\n+ From source\n\n#### Install from `Apt` package repository:\n\n```\ncurl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.deb.sh | bash\n```\nif you want to force `os` and `dist` of your system, use the following command:\n\n```\ncurl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.deb.sh | os=ubuntu dist=xenial bash\n```\n\nThen install clicktail itself:\n```\napt-get install clicktail\n```\n\n#### Install from individual `dep` package\n\nDownload and install the package from here: https://packagecloud.io/Altinity/clickhouse\n\n#### Install from source:\n\n```\ngo get github.com/Altinity/clicktail\n```\n\nto install to a specific path:\n\n```\nGOPATH=/usr/local go get github.com/Altinity/clicktail\n```\n\n...and make it\n\n```\nGOPATH=/usr/local go build\nGOPATH=/usr/local go install\n```\n\nthe binary will install to `/usr/local/bin/clicktail`\n\n## Configuration\n\nClicktail supports command line options as well as configuration file. In fact the file is not picked up by default when you are running `clicktail` from CLI so one should explicitly specify it with `-c` option.\n\nUse `clicktail.conf` file to manage options. There are section of the file called `Required Options` which should be set in the first place.\n\n#### Config Example\n\n`/etc/clicktail/clicktail.conf`\n```\n[Application Options]\nAPIHost = http://localhost:8123/\n...\n[Required Options]\nParserName = mysql\nLogFiles = /var/log/mysql/mariadb-slow.log\nDataset = clicktail.mysql_slow_log\n```\n\n#### Extra options for MySQL parser\n\nThere are useful options that could be passed along with mysql slow log entries but its not logged within the file itself, i.e. hostname of actual server. To get this data one can specify MySQL server connection details in config file.\n\n```\n...\n[MySQL Parser Options]\nHost = localhost:3306   ; or @unix(/var/run/mysqld/mysqld.sock)\nUser = username\nPass = userpass\n```\n\n## Usage\n\nMake sure ClickHouse server has proper schema created. See `schema` folder for `CREATE DATABASE` and `CREATE TABLE` statements.\n\nOr do the following:\n\nCreate DB:\n```\ncat schema/db.sql | clickhouse-client --multiline\n```\n\nCreate Table for MySQL slow logs:\n```\ncat schema/mysql.sql | clickhouse-client --multiline\n```\n\nOnce schema is prepared you can run binary from CLI with MySQL parser:\n```\nclicktail --dataset='clicktail.mysql_slow_log' --parser=mysql --file=/var/log/mysql/mysql-slow.log\n```\n\nOr with Nginx parser:\n\n```\nclicktail -p nginx -f /var/log/nginx/access.log -d clicktail.nginx_log --nginx.conf=/etc/nginx/nginx.conf --nginx.format=combined\n```\n\nAfter you done with checking out your configuration options, you will need to store them in `clicktail.conf` in order to run `clicktail` as a service just like that:\n\n```\nservice clicktail start\n```\n\n#### Retroactive logs loading\n\nIf you want to load files you already have into clicktail. You can use the same call as mentioned above but with extra parameter `--backfill`\n\n```\nclicktail --dataset='clicktail.mysql_slow_log' --parser=mysql --file=/var/log/mysql/mysql-slow.log --backfill\n```\n...this will load `mysql-slow.log` file into ClickTail and end the process.\n\n## ClickHouse Setup\n\nClicktail is required ClickHouse to be accessible as a target server. So you should have ClickHouse server installed.\n\nFollow these steps in order to install ClickHouse server and make it work with Clicktail.\n\nOnce you installed Clickhouse you whould probably need it to be open to outside connections. In order to do so update `/etc/clickhouse-server/config.xml` to add the following line:\n\n`\u003clisten_host\u003e0.0.0.0\u003c/listen_host\u003e`\n\njust after: `\u003clisten_host\u003e127.0.0.1\u003c/listen_host\u003e`\n\nAlso make sure ClickHouse port (which is `8123` by default) is open with your firewall.\n\n\nFor more advanced usage, options, and the ability to scrub or drop specific fields, see [documentation](https://honeycomb.io/docs/send-data/agent).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclicktail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faltinity%2Fclicktail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faltinity%2Fclicktail/lists"}