{"id":19273647,"url":"https://github.com/zikwall/clickhouse-statistic","last_synced_at":"2025-10-13T17:08:51.888Z","repository":{"id":104958104,"uuid":"284002920","full_name":"zikwall/clickhouse-statistic","owner":"zikwall","description":"🐾 Clickhouse and Apache Kafka Docker - Your mini assistant in future problems :)","archived":false,"fork":false,"pushed_at":"2020-08-27T13:30:06.000Z","size":81,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-10-05T11:50:49.829Z","etag":null,"topics":["clickhouse","cluster","docker","docker-compose","docker-container","documentation","go","golang","kafka","zookeeper"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zikwall.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-31T10:04:20.000Z","updated_at":"2024-07-29T12:42:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"390637c3-81fd-444a-a492-3a903b555140","html_url":"https://github.com/zikwall/clickhouse-statistic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zikwall/clickhouse-statistic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fclickhouse-statistic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fclickhouse-statistic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fclickhouse-statistic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fclickhouse-statistic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zikwall","download_url":"https://codeload.github.com/zikwall/clickhouse-statistic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zikwall%2Fclickhouse-statistic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016283,"owners_count":26085827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","cluster","docker","docker-compose","docker-container","documentation","go","golang","kafka","zookeeper"],"created_at":"2024-11-09T20:43:43.393Z","updated_at":"2025-10-13T17:08:51.825Z","avatar_url":"https://github.com/zikwall.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eClickhouse container statistic project\u003c/h1\u003e\n  \u003ch4\u003eYour mini assistant in future problems :)\u003c/h4\u003e\n  \u003ch5\u003eThis guide contains step by step instructions with working examples of deploying Clickhouse, Apache Kafka, Zookeeper in Docker\u003c/h5\u003e\n\u003c/div\u003e\n\n### Getting Started\n\n- [x] `$ mkdir -p /shared/ch/{clickhouse,zookeeper,kafka}`\n- [x] First step create clickhouse network `$ docker network create clickhouse-net`\n- [x] You can check result of prev. step `$ docker network ls`\n- [x] If OK, then run `$ docker-compose up -d`\n\n\u003cdetails\u003e\n  \u003csummary\u003eOutput\u003c/summary\u003e\n  \n  ```shell script\n    msi@msi clickhouse-compose # docker-compose up -d\n    Starting clickhouse-zookeeper ... done\n    Recreating clickhouse-04      ... done\n    Recreating clickhouse-05      ... done\n    Recreating clickhouse-01      ... done\n    Recreating clickhouse-02      ... done\n    Recreating clickhouse-06      ... done\n    Recreating clickhouse-03      ... done\n  ```\n\u003c/details\u003e\n\n- [x] Again check `$ docker container ls -a`\n\n\u003cdetails\u003e\n  \u003csummary\u003eOutput\u003c/summary\u003e\n  \n  ```shell script\n    CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS                                                            NAMES\n    442a79a43f3a        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9003-\u003e9000/tcp                       clickhouse-03\n    f5279aec0e37        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9006-\u003e9000/tcp                       clickhouse-06\n    3a783ee75502        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9002-\u003e9000/tcp                       clickhouse-02\n    ace4df988157        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9001-\u003e9000/tcp                       clickhouse-01\n    a40ac11a5194        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9005-\u003e9000/tcp                       clickhouse-05\n    23495201a490        yandex/clickhouse-server   \"/entrypoint.sh\"         2 minutes ago       Up 2 minutes        8123/tcp, 9009/tcp, 0.0.0.0:9004-\u003e9000/tcp                       clickhouse-04\n    8de765edf713        zookeeper                  \"/docker-entrypoint.…\"   4 minutes ago       Up 2 minutes        2888/tcp, 3888/tcp, 0.0.0.0:2181-2182-\u003e2181-2182/tcp, 8080/tcp   clickhouse-zookeeper\n    ... other own containers\n  ```\n\u003c/details\u003e\n\n- [x] For stopping all containers `$ docker-compose stop`\n\n### Connect to one of cluster server\n\n- [x] `$ make cluster-client ch=01`\n- [x] Check: `SELECT * FROM system.clusters;`\n\n```shell script\nclickhouse-01 :) SELECT * FROM system.clusters;\n\nSELECT *\nFROM system.clusters\n\n┌─cluster─────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name─────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐\n│ cluster_1                   │         1 │            1 │           1 │ clickhouse-01 │ 172.19.0.6   │ 9000 │        1 │ default │                  │            0 │                       0 │\n│ cluster_1                   │         1 │            1 │           2 │ clickhouse-06 │ 172.19.0.7   │ 9000 │        0 │ default │                  │            0 │                       0 │\n│ cluster_1                   │         2 │            1 │           1 │ clickhouse-02 │ 172.19.0.8   │ 9000 │        0 │ default │                  │            0 │                       0 │\n│ cluster_1                   │         2 │            1 │           2 │ clickhouse-03 │ 172.19.0.4   │ 9000 │        0 │ default │                  │            0 │                       0 │\n│ cluster_1                   │         3 │            1 │           1 │ clickhouse-04 │ 172.19.0.3   │ 9000 │        0 │ default │                  │            0 │                       0 │\n│ cluster_1                   │         3 │            1 │           2 │ clickhouse-05 │ 172.19.0.5   │ 9000 │        0 │ default │                  │            0 │                       0 │\n│ test_shard_localhost        │         1 │            1 │           1 │ localhost     │ 127.0.0.1    │ 9000 │        1 │ default │                  │            0 │                       0 │\n│ test_shard_localhost_secure │         1 │            1 │           1 │ localhost     │ 127.0.0.1    │ 9440 │        0 │ default │                  │            0 │                       0 │\n└─────────────────────────────┴───────────┴──────────────┴─────────────┴───────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘\n```\n\n### Create first data\n\n- [x] Use `.sql` files for create tables from folder [example/database](./example/database)\n\n### How work with Apache Kafka in Docker\n\n- [x] Create new topic\n\n```shell script\ndocker exec -t clickhouse-kafka \\\n  kafka-topics.sh \\\n    --bootstrap-server :9092 \\\n    --create \\\n    --topic ClickhouseTopic \\\n    --partitions 3 \\\n    --replication-factor 1\n```\n\n- [x] Print out the topics\n\n```shell script\ndocker exec -t clickhouse-kafka \\\n  kafka-topics.sh \\\n    --bootstrap-server :9092 \\\n    --list\n```\n\n- [x] Describe\n\n```shell script\ndocker exec -t clickhouse-kafka \\\n  kafka-topics.sh \\\n    --bootstrap-server :9092 \\\n    --describe \\\n    --topic MyTopic1\n```\n\n- [x] Run Kafka console consumer (run in another console)\n\n```shell script\ndocker exec -t clickhouse-kafka \\\n  kafka-console-consumer.sh \\\n    --bootstrap-server :9092 \\\n    --group my-group \\\n    --topic MyTopic1\n```\n\n- [x] Run Kafka console producer\n\n```shell script\ndocker exec -it clickhouse-kafka \\\n  kafka-console-producer.sh \\\n    --broker-list :9092 \\\n    --topic MyTopic1\n```\n\n- [x] Get count messages of topic\n\n```shell script\ndocker exec -it clickhouse-kafka \\\n  kafka-run-class.sh \\\n  kafka.tools.GetOffsetShell \\\n    --broker-list :9092 \\\n    --topic MyTopic1 \\\n    --time -1\n```\n\n- [x] Drop topic\n\n```shell script\ndocker exec -it clickhouse-kafka \\\n  kafka-topics.sh \\\n    --bootstrap-server :2181 \\\n    --topic MyTopic1 \\\n    --delete\n```\n\n- [x] Drop groups\n\n```shell script\ndocker exec -it clickhouse-kafka \\\n  kafka-consumer-groups.sh \\\n    --bootstrap-server localhost:9092 \\\n    --delete \\\n    --group my-group\n```\n\n- [x] State consumer\n\n```shell script\ndocker exec -it clickhouse-kafka \\\n  kafka-consumer-groups.sh \\\n    --bootstrap-server localhost:9092 \\\n    --describe \\\n    --state \\\n    --group my-group\n```\n\n**after put messages in producer console \u0026\u0026 to see messages printed out in second terminal, where run Kafka CLI consumer**\n\n- [x] Show full logs fro Kafka run: `$ docker logs -f clickhouse-kafka`\n- [x] [Kafkacat](https://github.com/edenhill/kafkacat)\n\n### How wotk with Zookeeper\n\n- [x] Connect container `$ docker exec -it clickhouse-zookeeper bash`\n- [x] Connect server `$ bin/zkCli.sh -server 127.0.0.1:2181`\n- [x] List root `$ ls /`\n\n**Output**\n\n```shell script\n[admin, brokers, clickhouse, cluster, config, consumers, controller, controller_epoch, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]\n```\n\n- [x] `$ ls brokers/` after `$ ls brokers/topics`\n\n**Output**\n\n```shell script\n[MyTopic, MyTopic1, __consumer_offsets]\n```\n\n- [x] `$ ls /consumers`\n\n### Tests\n\n- [x] First add next line `127.0.0.1    clickhouse-kafka` in `/etc/hosts`, for DEV, why? [1](https://ealebed.github.io/posts/2018/docker-%D1%81%D0%BE%D0%B2%D0%B5%D1%82-28-%D0%BA%D0%B0%D0%BA-%D0%B8%D1%81%D0%BF%D1%80%D0%B0%D0%B2%D0%B8%D1%82%D1%8C-%D0%BE%D1%88%D0%B8%D0%B1%D0%BA%D1%83-connection-reset-by-peer/), [2](https://github.com/grafana/metrictank/issues/1286), [3](https://github.com/wurstmeister/kafka-docker/issues/424)\n- [x] Create topic `ClickhouseTopic` if already is not created\n- [x] Run consumer in another terminal\n- [x] Run app from example/backend `$ go run .`\n\n\u003cdetails\u003e\n  \u003csummary\u003eOutput in Go terminal\u003c/summary\u003e\n  \n  ```shell script\n      Send message to broker: user 23, time 2020-08-04 13:58:14\n      Send message to broker: user 16, time 2020-08-04 13:58:15\n      Send message to broker: user 29, time 2020-08-04 13:58:16\n      Send message to broker: user 11, time 2020-08-04 13:58:17\n      Send message to broker: user 22, time 2020-08-04 13:58:18\n      Send message to broker: user 25, time 2020-08-04 13:58:19\n      Send message to broker: user 15, time 2020-08-04 13:58:20\n      Send message to broker: user 20, time 2020-08-04 13:58:21\n      Send message to broker: user 17, time 2020-08-04 13:58:22\n      message at topic/partition/offset MyTopic/0/189:  = {\"user_id\":23,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:14\"}\n      message at topic/partition/offset MyTopic/0/190:  = {\"user_id\":16,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:15\"}\n      message at topic/partition/offset MyTopic/0/191:  = {\"user_id\":29,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:16\"}\n      message at topic/partition/offset MyTopic/0/192:  = {\"user_id\":11,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:17\"}\n      message at topic/partition/offset MyTopic/0/193:  = {\"user_id\":22,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:18\"}\n      message at topic/partition/offset MyTopic/0/194:  = {\"user_id\":25,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:19\"}\n      message at topic/partition/offset MyTopic/0/195:  = {\"user_id\":15,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:20\"}\n      message at topic/partition/offset MyTopic/0/196:  = {\"user_id\":20,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:21\"}\n      message at topic/partition/offset MyTopic/0/197:  = {\"user_id\":17,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:22\"}\n      Send message to broker: user 19, time 2020-08-04 13:58:23\n      Send message to broker: user 18, time 2020-08-04 13:58:24\n      Send message to broker: user 28, time 2020-08-04 13:58:25\n  ```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n  \u003csummary\u003eOutput in consumer terminal\u003c/summary\u003e\n  \n  ```shell script\n      {\"user_id\":19,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:23\"}\n      {\"user_id\":18,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:24\"}\n      {\"user_id\":28,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:25\"}\n      {\"user_id\":14,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:26\"}\n      {\"user_id\":13,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:27\"}\n      {\"user_id\":14,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:28\"}\n      {\"user_id\":17,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:29\"}\n      {\"user_id\":22,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:30\"}\n      {\"user_id\":22,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:31\"}\n      {\"user_id\":26,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:32\"}\n      {\"user_id\":29,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:33\"}\n      {\"user_id\":10,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:34\"}\n      {\"user_id\":16,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:35\"}\n      {\"user_id\":21,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:36\"}\n      {\"user_id\":26,\"app\":\"\",\"host\":\"\",\"event\":\"\",\"ip\":\"\",\"guid\":\"\",\"created_at\":\"2020-08-04 13:58:37\"}\n  ```\n\u003c/details\u003e\n\n- [x] Another terminal `make cluster-client ch=01` for connect `ch-01` server\n- [x] `SELECT * from main;`\n\n**Output**\n\n```shell script\nclickhouse-01 :) select * from main;\n\n┌─user_id─┬─app─┬─host─┬─event─┬─ip─┬─guid─┬──────────created_at─┐\n│      10 │     │      │       │    │      │ 2020-08-04 15:48:12 │\n│      29 │     │      │       │    │      │ 2020-08-04 15:48:13 │\n│      24 │     │      │       │    │      │ 2020-08-04 15:48:14 │\n│      14 │     │      │       │    │      │ 2020-08-04 15:48:15 │\n│      10 │     │      │       │    │      │ 2020-08-04 15:48:16 │\n│      21 │     │      │       │    │      │ 2020-08-04 15:48:17 │\n│      11 │     │      │       │    │      │ 2020-08-04 15:48:18 │\n│      27 │     │      │       │    │      │ 2020-08-04 15:48:19 │\n└─────────┴─────┴──────┴───────┴────┴──────┴─────────────────────┘\n```\n\n### Cluster\n\n- [x] Create `main`, `queue` and `mainconsumer` tables each hosts `ch-`: 01, 02, 03, 04, 05\n    - you can use `$ bin/create-replica.sh 03`\n- [x] Create distributed table on last host `ch-06` from `example/database/distributed.sql`\n    - or use `$ bin/create-distributed.sh 06`\n- [x] connect to `ch-06`\n    - `$ make cluster-client ch=06`\n- [x] `SELECT COUNT() FROM main_distributed`;\n\n**Output**\n\n```shell script\nclickhouse-06 :) select count() from main_distributed;\n\nSELECT count()\nFROM main_distributed\n\n┌─count()─┐\n│    1016 │\n└─────────┘\n```\n\n- [x] Check connect one of server, example `ch-01`, `ch-02` and `SELECT count(*) from main;`\n\n**Output**\n\n```shell script\nclickhouse-01 :) select count() from main;\n\nSELECT count()\nFROM main\n\n┌─count()─┐\n│     945 │\n└─────────┘\n```\n\n**Output**\n\n```shell script\nclickhouse-02 :) select count() from main;\n\nSELECT count()\nFROM main\n\n┌─count()─┐\n│      71 │\n└─────────┘\n```\n\nHappy! ^_^\n\n### Following manuals\n\n- [x] [Hard manual](https://github.com/zikwall/clickhouse-docs)\n- [x] [Advanced manual](./ADVANCED.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fclickhouse-statistic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzikwall%2Fclickhouse-statistic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzikwall%2Fclickhouse-statistic/lists"}