Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/kak-tus/loghouse-acceptor
Accept and write logs to Clickhouse in LogHouse compatible format
https://github.com/kak-tus/loghouse-acceptor
Last synced: about 1 month ago
JSON representation
Accept and write logs to Clickhouse in LogHouse compatible format
- Host: GitHub
- URL: https://github.com/kak-tus/loghouse-acceptor
- Owner: kak-tus
- License: mit
- Created: 2018-03-23T23:24:32.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-11-05T20:20:38.000Z (about 5 years ago)
- Last Synced: 2024-06-20T00:35:06.477Z (6 months ago)
- Language: Go
- Homepage:
- Size: 376 KB
- Stars: 20
- Watchers: 3
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: Changes
- License: LICENSE
Awesome Lists containing this project
README
# loghouse-acceptor
Accept and write logs to ClickHouse in Loghouse compatible format.
Daemon accepts log only in syslog RFC5424 format from rsyslog. Later may be will be added
some additinal protocols support.
## Daemon configuration
Configuration is possible with environment variables
### CLICKHOUSE_ADDR
ClickHouse address
```
CLICKHOUSE_ADDR=127.0.0.1:9000
```
### ACC_PERIOD
Period in seconds to send data to ClickHouse if batch size is not reached.
```
ACC_PERIOD=60
```
### ACC_BATCH
Batch size to send to ClickHouse immediately.
```
ACC_BATCH=10000
```
### ACC_TABLE_TYPE
Loghouse table type. In most cases you need basic table type - default to Loghouse. But you can use other type - extended, with some additional vendored columns.
```
ACC_TABLE_TYPE=basic
```
### ACC_PARTITION_TYPE
Configurable partitions periods - daily or hourly. Loghouse supports both types.
```
ACC_PARTITION_TYPE=hourly
```
### ACC_SHARD_TYPE
Sharding type. If you didn't use shards and replicas set this to default value "basic". In other case see "Sharding and replication in ClickHouse" section.
```
ACC_SHARD_TYPE=basic
```
### ACC_TABLE_CLUSTER
Cluster name in ClickHouse. See "Sharding and replication in ClickHouse" section.
```
ACC_TABLE_CLUSTER=my-cluster
```
## Rsyslog configuration
Configure rsyslog to resend logs to daemon
```
action(type="omfwd" Target="127.0.0.1" Port="3333" Protocol="tcp" KeepAlive="on" Template="RSYSLOG_SyslogProtocol23Format")
```
## Running
```
docker run --rm -it -p 3333:3333 -e ClickHouse_ADDR=127.0.0.1:9000 kaktuss/loghouse-acceptor
```
## Sharding and replication in ClickHouse
It is possible to use and create sharded and replicated partitions. See [replication](https://ClickHouse.yandex/docs/en/operations/table_engines/replication/) and [distributed](https://ClickHouse.yandex/docs/en/operations/table_engines/distributed/#table_engines-distributed) in ClickHouse docs.
To use this feature with loghouse-acceptor you need to configure two macros in ClickHouse.
```
...
02
example05-02-1.yandex.ru
...
```
Then configure cluster in ClickHouse
```
...
```
Then set ACC_SHARD_TYPE variable to basic_sharded and ACC_TABLE_CLUSTER to your cluster name.
```
ACC_SHARD_TYPE=basic_sharded
ACC_TABLE_CLUSTER=my-cluster
```