{"id":19273600,"url":"https://github.com/zikwall/grower","last_synced_at":"2026-05-12T20:45:01.602Z","repository":{"id":50488016,"uuid":"519231184","full_name":"zikwall/grower","owner":"zikwall","description":"Grower is a pure Go tool-set for writing Nginx logs to the Clickhouse columnar database.","archived":false,"fork":false,"pushed_at":"2022-10-03T20:16:01.000Z","size":106,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-05T14:24:25.450Z","etag":null,"topics":["clickhouse","golang","nginx"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zikwall.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-29T13:47:07.000Z","updated_at":"2022-09-19T18:08:11.000Z","dependencies_parsed_at":"2022-09-15T01:02:57.978Z","dependency_job_id":null,"html_url":"https://github.com/zikwall/grower","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fgrower/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zikwall","download_url":"https://codeload.github.com/zikwall/grower/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240378869,"owners_count":19792039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","golang","nginx"],"created_at":"2024-11-09T20:43:35.860Z","updated_at":"2026-05-12T20:45:01.574Z","avatar_url":"https://github.com/zikwall.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![build](https://github.com/zikwall/grower/workflows/build_and_tests/badge.svg)](https://github.com/zikwall/clickhouse-buffer/v4/actions)\n[![build](https://github.com/zikwall/grower/workflows/golangci_lint/badge.svg)](https://github.com/zikwall/clickhouse-buffer/v4/actions)\n\n\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eGrower\u003c/h1\u003e\n  \u003ch5\u003eAn easy-to-use, powerful and productive tool that allows you to write Nginx logs to a Clickhouse columnar database.\u003c/h5\u003e\n\u003c/div\u003e\n\n**Features:**\n\n- **No dependencies**: it works like a regular binary file or in docker\n- Supports for **multiple versions** of content delivery:\n  - [x] **FileLog** read/write/rotate support\n  - [x] **SysLog** protocol (tcp, udp, unix) support\n  - [x] **FileBuf** gRPC client and server\n  - [x] **KafkaLog** FileLog Client-Server + Kafka as Broker\n  - [ ] **RabbitLog** FileLog Client-Server + RabbitMQ as Broker\n  - [ ] **RedisLog** FileLog Client-Server + Redis as Broker\n  - [ ] **NATSLog** FileLog Client-Server + NATS as Broker\n- **Fully customizable**: \n  - timeouts and runtime limitations (buffer sizes, flush intervals, retries configuration),\n  - schema \u0026 log formats\n  - support **all native nginx attributes** and ability to add your **own fields**\n  - **multithreading** support and customizable\n- **Completely Type Safe**: native support for protection types\n\n**TODO:**\n\n- prometheus metrics and dashboard configuration\n- saving corrupted files for manual processing\n- possibility of log native compression\n- native support for more data types\n- native support for complex data types such as:\n  - Geo: `GeoIPRegion(ip)`, `GeoIPCity(ip)`, `GeoIPAS(ip)`\n  - JSON: `JSONStringField(field_name, json_string_field)`, `JSONUInt64Field(field_name, json_string_field)`\n  - RegExp: `RegExp('/\\(?([0-9]{3})\\)?([ .-]?)([0-9]{3})\\2([0-9]{4})/', target_field)` - example get phone number from string\n  - Cast: `toUInt32(GeoIPAS(ip))`\n\n### How to use it?\n\n### Configuration\n\nis very simple and clear, see [sample.yaml configuration file](./sample_test.yaml)\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003eExample:\u003c/b\u003e\u003c/summary\u003e\n\n```yaml\nnginx:\n  log_type: csv\n  log_time_format: '02/Jan/2006:15:04:05 -0700'\n  log_time_rewrite: true\n  log_custom_casts_enable: true\n  log_custom_casts:\n    custom_field: Integer\n    custom_time_field: Datetime\n  log_format: '$remote_addr - $remote_user [$time_local] \"$request\" $status $bytes_sent $request_time \"$request_method\" \"$http_referer\" \"$http_user_agent\" $https $custom_field \u003c$custom_time_field\u003e'\n  log_remove_hyphen: true\nscheme:\n  logs_table: only_tests.access_log\n  columns:\n    remote_addr: remote_addr\n    remote_user: remote_user\n    time_local: time_local\n    request: request\n    status: status\n    bytes_sent: bytes_sent\n    request_time: request_time\n    request_method: request_method\n    http_referer: http_referer\n    http_user_agent: http_user_agent\n    https: https\n    custom_field: custom_field\n    custom_time_field: custom_time_field\n```\n\u003c/details\u003e\n\n### FileLog\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/filelog/main.go  \\\n    --config-file ./sample_test.yaml \\\n    --bind-address 0.0.0.0:3000 \\\n    --logs-dir /var/log/nginx \\\n    --source-log-file access.log \\\n    --scrape-interval '10s' \\\n    --backup-files 5 \\\n    --backup-file-max-age '1m0s' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9000' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9001' \\\n    --clickhouse-user default \\\n    --clickhouse-database default \\\n    --clickhouse-password '' \\\n    --buffer-size 10000 \\\n    --buffer-flush-interval 5000 \\\n    --write-timeout '0m30s' \\\n    --parallelism 5 \\\n    --debug \\\n    --auto-create-target-from-scratch \\\n    --enable-rotating \\\n    --skip-nginx-reopen \\\n    --run-http-server \\\n    --run-rotating-at-startup\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -v /usr/share/config/:/config/ \\\n   -e CONFIG_FILE='/config/sample_test.yaml' \\\n   -e BIND_ADDRESS='0.0.0.0:3004' \\\n   -e SOURCE_LOG_FILE='access.log' \\\n   -e LOGS_DIR='/var/log/nginx' \\\n   -e SCRAPE_INTERVAL='1m0s' \\\n   -e BACKUP_FILES=5 \\\n   -e BACKUP_FILE_MAX_AGE='5m0s' \\\n   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \\\n   -e CLICKHOUSE_USER='default' \\\n   -e CLICKHOUSE_PASSWORD = '' \\\n   -e CLICKHOUSE_DATABASE='default' \\\n   -e BUFFER_FLUSH_INTERVAL=2000 \\\n   -e BUFFER_SIZE=5000 \\\n   -e WRITE_TIMEOUT='0m30s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e AUTO_CREATE_TARGET_FROM_SCRATCH \\\n   -e ENABLE_ROTATING \\\n   -e SKIP_NGINX_REOPEN \\\n   -e RUN_ROTATING_AT_STARTUP \\\n   -e DEBUG=true \\\n   --name grower-syslog qwx1337/grower-filelog:latest\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-filelog:latest -f ./cmd/filelog/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/filelog/main.go --help`\n\n### SysLog\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/syslog/main.go  \\\n    --config-file ./sample_test.yaml \\\n    --bind-address 0.0.0.0:3000 \\\n    --syslog-unix-socket /tmp/syslog.sock \\\n    --syslog-udp-address 0.0.0.0:3011 \\\n    --syslog-tcp-address 0.0.0.0:3012 \\\n    --listeners 'unix' \\\n    --listeners 'tcp' \\\n    --listeners 'udp' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9000' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9001' \\\n    --clickhouse-user default \\\n    --clickhouse-database default \\\n    --clickhouse-password '' \\\n    --buffer-size 5000 \\\n    --buffer-flush-interval 2000 \\\n    --write-timeout '0m30s' \\\n    --parallelism 5 \\\n    --run-http-server \\\n    --debug\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -v /usr/share/config/:/config/ \\\n   -e CONFIG_FILE='/config/sample_test.yaml' \\\n   -e BIND_ADDRESS='0.0.0.0:3004' \\\n   -e SYSLOG_UNIX_SOCKET='/tmp/syslog.sock' \\\n   -e SYSLOG_UDP_ADDRESS='0.0.0.0:3011' \\\n   -e SYSLOG_TCP_ADDRESS='0.0.0.0:3012' \\\n   -e LISTENERS='unix,tcp,udp' \\\n   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \\\n   -e CLICKHOUSE_USER='default' \\\n   -e CLICKHOUSE_PASSWORD = '' \\\n   -e CLICKHOUSE_DATABASE='default' \\\n   -e BUFFER_FLUSH_INTERVAL=2000 \\\n   -e BUFFER_SIZE=5000 \\\n   -e WRITE_TIMEOUT='0m30s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e DEBUG=true \\\n   --name grower-syslog qwx1337/grower-syslog:latest\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-syslog:latest -f ./cmd/syslog/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/syslog/main.go --help`\n\n### KafkaLog: Kafka buffer client and server\n\n**Client side:**\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eClient\u003c/b\u003e Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/kafkalog/client/main.go  \\\n    --kafka-brokers xxx.xx.xx.xx:9092 \\\n    --kafka-brokers xxx.xx.xx.xx:9093 \\\n    --kafka-topic example2 \\\n    --kafka-balancer least_bytes \\\n    --kafka-write-timeout '0m5s' \\\n    --kafka-async \\\n    --logs-dir /var/log/nginx \\\n    --source-log-file access.log \\\n    --scrape-interval '10s' \\\n    --backup-files 5 \\\n    --backup-file-max-age '1m0s' \\\n    --parallelism 5 \\\n    --debug \\\n    --auto-create-target-from-scratch \\\n    --enable-rotating \\\n    --skip-nginx-reopen \\\n    --run-rotating-at-startup \\\n    --run-http-server\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eClient\u003c/b\u003e container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -e KAFKA_BROKERS='xxx.xx.xx.xx:9092,xxx.xx.xx.xx:9093' \\\n   -e KAFKA_TOPIC='example2' \\\n   -e KAFKA_BALANCER='least_bytes' \\\n   -e KAFKA_WRITE_TIMEOUT='0m5s' \\\n   -e KAFKA_ASYNC=true \\\n   -e BIND_ADDRESS='0.0.0.0:3004' \\\n   -e GRPC_BIND_ADDRESS='0.0.0.0:3003' \\\n   -e SOURCE_LOG_FILE='access.log' \\\n   -e LOGS_DIR='/var/log/nginx' \\\n   -e SCRAPE_INTERVAL='1m0s' \\\n   -e BACKUP_FILES=5 \\\n   -e BACKUP_FILE_MAX_AGE='5m0s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e AUTO_CREATE_TARGET_FROM_SCRATCH \\\n   -e ENABLE_ROTATING \\\n   -e SKIP_NGINX_REOPEN \\\n   -e RUN_ROTATING_AT_STARTUP \\\n   -e DEBUG=true \\\n   -e RUN_HTTP_SERVER=true \\\n   --name grower-filebuf-client qwx1337/grower-kafkalog-client:latest\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build \u003cb\u003eClient\u003c/b\u003e Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-kafkalog-client:latest -f ./cmd/kafkalog/client/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/kafkalog/client/main.go --help`\n\n**Server side::**\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eServer\u003c/b\u003e Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/kafkalog/server/main.go  \\\n    --config-file ./sample_test.yaml \\\n    --kafka-brokers xxx.xx.xx.xx:9092 \\\n    --kafka-brokers xxx.xx.xx.xx:9093 \\\n    --kafka-topic example2 \\\n    --kafka-group test1 \\\n    --clickhouse-host 'xxx.xx.xx.xx:9000' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9001' \\\n    --clickhouse-user default \\\n    --clickhouse-database default \\\n    --clickhouse-password '' \\\n    --buffer-size 10000 \\\n    --buffer-flush-interval 5000 \\\n    --write-timeout '0m30s' \\\n    --parallelism 5 \\\n    --run-http-server \\\n    --debug\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eServer\u003c/b\u003e container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -v /usr/share/config/:/config/ \\\n   -e CONFIG_FILE='/config/sample_test.yaml' \\\n   -e KAFKA_BROKERS='xxx.xx.xx.xx:9092,xxx.xx.xx.xx:9093' \\\n   -e KAFKA_TOPIC='example2' \\\n   -e KAFKA_GROUP='test1' \\\n   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \\\n   -e CLICKHOUSE_USER='default' \\\n   -e CLICKHOUSE_PASSWORD = '' \\\n   -e CLICKHOUSE_DATABASE='default' \\\n   -e BUFFER_FLUSH_INTERVAL=2000 \\\n   -e BUFFER_SIZE=5000 \\\n   -e WRITE_TIMEOUT='0m30s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e DEBUG=true \\\n   --name grower-kafkalog-server qwx1337/grower-kafkalog-server:latest\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build \u003cb\u003eServer\u003c/b\u003e Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-kafkalog-server:latest -f ./cmd/kafkalog/server/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/kafkalog/server/main.go --help`\n\n### Kafka Quick Help\n\n`docker exec -it \u003ccontainer\u003e sh -c \"\u003ccommand\u003e\"`, example for _bitnami/kafka_:\n\n- **create:** `/opt/bitnami/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 6 --topic example2`\n- **list:** `/opt/bitnami/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092`\n- **delete:** `/opt/bitnami/kafka/bin/kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic example2`\n- **read all:** `/opt/bitnami/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic example2 --from-beginning`\n\n\u003cdetails\u003e\n  \u003csummary\u003eSample docker-compose for Apache Kafka:\u003c/summary\u003e\n\n```yaml\n# development version, do not use in production\n# change plaintext to ssl\n# add certs\nversion: '3.3'\n\nservices:\n  kafka:\n    image: bitnami/kafka:latest\n    ports:\n      - \"9092:9092\"\n    volumes:\n      - \"/volumes/kafka/data:/bitnami\"\n      - \"/volumes/kafka/logs:/tmp/kafka_logs\"\n    environment:\n      - ALLOW_PLAINTEXT_LISTENER=yes\n      - KAFKA_CFG_BROKER_ID=0\n      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092\n      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://xxx.xx.xx.xx:9092\n      - KAFKA_ZOOKEEPER_PROTOCOL=PLAINTEXT\n      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true\n      - KAFKA_CFG_ZOOKEEPER_CONNECT=xxx.xx.xx.xx:2181\n      - KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MS=6000\n      # - KAFKA_ZOOKEEPER_TLS_TRUSTSTORE_FILE=/bitnami/kafka/config/certs/zookeeper.truststore.jks\n      # - KAFKA_CFG_SSL_KEYSTORE_LOCATION=/opt/bitnami/kafka/conf/certs/server.keystore.jks\n      # - KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/opt/bitnami/kafka/conf/certs/server.truststore.jks\n      - KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS=300000\n      - KAFKA_CFG_LOG_RETENTION_HOURS=24\n      - KAFKA_CFG_LOG_DIRS=/volumes/tmp/kafka_logs\n```\n\u003c/details\u003e\n\n### FileBuf: gRPC client and server\n\n**Client side:**\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eClient\u003c/b\u003e Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/filegrpc/client/main.go  \\\n    --bind-address 0.0.0.0:3000 \\\n    --grpc-conn-address 0.0.0.0:3003 \\\n    --logs-dir /var/log/nginx \\\n    --source-log-file access.log \\\n    --scrape-interval '10s' \\\n    --backup-files 5 \\\n    --backup-file-max-age '1m0s' \\\n    --parallelism 5 \\\n    --debug \\\n    --auto-create-target-from-scratch \\\n    --enable-rotating \\\n    --skip-nginx-reopen \\\n    --run-rotating-at-startup \\\n    --run-http-server\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eClient\u003c/b\u003e container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -e BIND_ADDRESS='0.0.0.0:3004' \\\n   -e GRPC_BIND_ADDRESS='0.0.0.0:3003' \\\n   -e SOURCE_LOG_FILE='access.log' \\\n   -e LOGS_DIR='/var/log/nginx' \\\n   -e SCRAPE_INTERVAL='1m0s' \\\n   -e BACKUP_FILES=5 \\\n   -e BACKUP_FILE_MAX_AGE='5m0s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e AUTO_CREATE_TARGET_FROM_SCRATCH \\\n   -e ENABLE_ROTATING \\\n   -e SKIP_NGINX_REOPEN \\\n   -e RUN_ROTATING_AT_STARTUP \\\n   -e DEBUG=true \\\n   -e RUN_HTTP_SERVER=true \\\n   --name grower-filebuf-client qwx1337/grower-filebuf-client:latest\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build \u003cb\u003eClient\u003c/b\u003e Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-filebuf-client:latest -f ./cmd/fileclient/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/filegrpc/client/main.go --help`\n\n**Server side::**\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eServer\u003c/b\u003e Go native binary:\u003c/summary\u003e\n\n```shell\ngo run ./cmd/filegrpc/server/main.go  \\\n    --config-file ./sample_test.yaml \\\n    --bind-address 0.0.0.0:3000 \\\n    --grpc-bind-address 0.0.0.0:3003 \\\n    --clickhouse-host 'xxx.xx.xx.xx:9000' \\\n    --clickhouse-host 'xxx.xx.xx.xx:9001' \\\n    --clickhouse-user default \\\n    --clickhouse-database default \\\n    --clickhouse-password '' \\\n    --buffer-size 10000 \\\n    --buffer-flush-interval 5000 \\\n    --write-timeout '0m30s' \\\n    --parallelism 5 \\\n    --run-http-server \\\n    --debug\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun \u003cb\u003eServer\u003c/b\u003e container in Docker:\u003c/summary\u003e\n\n```shell\ndocker run -d --net=host \\\n   -v /usr/share/config/:/config/ \\\n   -e CONFIG_FILE='/config/sample_test.yaml' \\\n   -e BIND_ADDRESS='0.0.0.0:3004' \\\n   -e GRPC_BIND_ADDRESS='0.0.0.0:3003' \\\n   -e CLICKHOUSE_HOST='xxx.xx.xx.xx:9000,xxx.xx.xx.xx:9001' \\\n   -e CLICKHOUSE_USER='default' \\\n   -e CLICKHOUSE_PASSWORD = '' \\\n   -e CLICKHOUSE_DATABASE='default' \\\n   -e BUFFER_FLUSH_INTERVAL=2000 \\\n   -e BUFFER_SIZE=5000 \\\n   -e WRITE_TIMEOUT='0m30s' \\\n   -e PARALLELISM=5 \\\n   -e RUN_HTTP_SERVER=true \\\n   -e DEBUG=true \\\n   --name grower-filebuf-server qwx1337/grower-filebuf-server:latest\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eRun build \u003cb\u003eServer\u003c/b\u003e Docker image:\u003c/summary\u003e\n\n```shell\n#!/bin/bash\n\ndocker build -t qwx1337/grower-filebuf-server:latest -f ./cmd/fileserver/Dockerfile .\n```\n\u003c/details\u003e\n\n**For more information:**\n\n`$ go run ./cmd/filegrpc/server/main.go --help`\n\n### Recommendations and Notes\n\n1. Only for big data (200k\u003e) or fast line-by-line processing (10k/s\u003e):\n   - Use a buffer size between 1000 and 5000 if you want less CPU utilization, but this **will increase processing time**.\n   - Use a buffer size in the range of 5000, 10000 and more if you want to process data faster, but this **will increase CPU and memory utilization**.\n2. You can find the most advantageous option in your case yourself by simply configuring the following arguments: (`--buffer-size`, `--buffer-flush-interval`)\n3. Easiest way to use it is SysLog, at the same time optimal, because there are no sharp spikes in load when reading a file, as in the case of FileLog\n4. Remember that delivery to **SysLog via UDP is not guaranteed**\n5. FileLog is convenient, because there is no need to raise separate servers, as in the case of SysLog and FileBuf\n6. FileBuf is as load-efficient as SysLog, but with reliable content delivery.\n7. At the moment FileBuf **does not support automatic reconnection** of streams\n8. You can configure an acceptable multithreading for yourself through the argument `--parallelism`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fgrower","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzikwall%2Fgrower","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fgrower/lists"}