{"id":19381777,"url":"https://github.com/taosdata/go-demo-kafka","last_synced_at":"2025-04-23T20:32:00.974Z","repository":{"id":88771235,"uuid":"380268290","full_name":"taosdata/go-demo-kafka","owner":"taosdata","description":"TDengine Demo Series - Consume Messages from Kafka","archived":true,"fork":false,"pushed_at":"2021-07-16T09:38:55.000Z","size":161,"stargazers_count":10,"open_issues_count":0,"forks_count":0,"subscribers_count":37,"default_branch":"main","last_synced_at":"2025-03-24T12:35:35.647Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taosdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-25T14:45:30.000Z","updated_at":"2024-11-15T07:21:50.000Z","dependencies_parsed_at":"2023-03-13T18:18:49.078Z","dependency_job_id":null,"html_url":"https://github.com/taosdata/go-demo-kafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taosdata%2Fgo-demo-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taosdata%2Fgo-demo-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taosdata%2Fgo-demo-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taosdata%2Fgo-demo-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taosdata","download_url":"https://codeload.github.com/taosdata/go-demo-kafka/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250509727,"owners_count":21442486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T09:18:21.214Z","updated_at":"2025-04-23T20:32:00.966Z","avatar_url":"https://github.com/taosdata.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TDengine Demo Series - Consume Messages from Kafka\n\n## A Simple Start\n\n### Kafka Setup\n\n```sh\ndocker-compose up -d kafka\n```\n\n### Create Topic\n\n```sh\ndocker-compose exec kafka \\\n  kafka-topics.sh --create \\\n    --zookeeper zookeeper:2181 \\\n    --replication-factor 1 \\\n    --partitions 10 \\\n    --topic test\n# check topic list\ndocker-compose exec kafka \\\n  kafka-topics.sh --list \\\n  --zookeeper zookeeper:2181\n```\n\n### Producer and Consumer Test using internal client\n\n```sh\ndocker-compose exec kafka \\\n  kafka-console-producer.sh --broker-list kafka:9092 --topic test\ndocker-compose exec kafka \\\n  kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning\n```\n\nAlter partitions\n\n```sh\ndocker-compose exec kafka \\\n  kafka-topics.sh --bootstrap-server kafka:9092 --alter --topic test --partitions 10\n```\n\n![internal-client-test](assets/internal-client-test.png)\n\n### Go Producer and Consumer\n\n*NOTE: may only works on ubuntu because of the runtime image.*\n\nFirst we start tdengine in docker-compose\n\n```sh\ndocker-compose up -d tdengine\n```\n\nCreate test database\n\n```sh\ndocker-compose exec tdengine taos -s \"create database if not exists test\"\n```\n\nBuild go producer and consumer.\n\n```sh\ngo build cmd/producer/producer.go \ngo build cmd/consumer/consumer.go\n```\n\nRun consumer.\n\n```sh\ndocker-compose run --rm consumer\n```\n\nRun producer.\n\n```sh\ndocker-compose run --rm producer\n```\n\nThe results in console:\n\n![go-produce-consume](assets/go-produce-consume.png)\n\nTo use multiple consumer, it only requires your topic and messages support multitple paritions. When it's ready, just start multiple consumer service, use docker-compose, it will be:\n\n```sh\n$ docker-compose up -d --scale consumer=2 consumer\nCreating go-demo-kafka_consumer_1 ... done\nCreating go-demo-kafka_consumer_2 ... done\n```\n\nThe consumers will consume different partitions automatically.\n\n## Design Goals\n\n- Produce messages from kafka, and consume them to TDengine\n- Works as multi-producer-multi-consumer mode\n- Avoid redundant consuming\n- Insert into TDengine batch by batch\n- Modular development\n- Easy to config\n- Easy to use as an project template\n\n## Code Details\n\n*We use \u003chttps://github.com/golang-standards/project-layout\u003e for modular development. Make your decision as you prefer.*\n\nWe have developed some reusable modules for the use case:\n\n- [Queue](pkg/queue): A batch queue could handle message in batch.\n- [TaosEncoder](pkg/utils/tosql.go): Inteface and helpers for build sql easily.\n- [Codec](pkg/utils/codec.go): Interface and helpers for message encoding/decoding.\n- [MQ](pkg/kafka/common/common.go): Abstraction for message queue, eg. kafka.\n- [MQ - Kafka](pkg/kafka/kafka/): Kafka implementations of MQ.\n\nFor demo use case, we have implemented an demo message object and Codec/TaosEncoder interfaces in [internal/record](internal/record.go). Implement as you need for real use.\n\nA message simulator is implemented in [internal/config.go](internal/config.go).\n\nThen, producer and consumer commands are avalible at [cmd/](cmd/) directory:\n\n- producer: [cmd/producer/producer.go](cmd/producer/producer.go)\n- consumer: [cmd/consumer/consumer.go](cmd/consumer/consumer.go)\n\n### Demo Message Details\n\nDemo message object:\n\n```go\n// Record is the structual message object detail, provide both Codec and TaosEncoder interface.\ntype Record struct {\n\tDeviceID  string // for table name\n\tTimestamp time.Time\n\tCols      []interface{}\n}\n```\n\nTaosEncoder interface\n\n```go\ntype TaosEncoder interface {\n\tTaosDatabase() string\n\tTaosSTable() string\n\tTaosTable() string\n\tTaosTags() []interface{}\n\tTaosCols() []string\n\tTaosValues() []interface{}\n}\n```\n\nThe implementation, check comments for each method for the definition.\n\n```go\n// TaosEncoder implementations\n\n// If this is setted, sql will use db.table for tablename\nfunc (r Record) TaosDatabase() string {\n\treturn \"\"\n}\n\n// Auto create table using stable and tags\nfunc (r Record) TaosSTable() string {\n\treturn \"stb\"\n}\n\n// tags must be setted with TaosSTable\nfunc (r Record) TaosTags() []interface{} {\n\tvar tags []interface{}\n\ttags = append(tags, r.DeviceID)\n\treturn tags\n}\n\n// Dynamic device id as table name\nfunc (r Record) TaosTable() string {\n\treturn r.DeviceID\n}\n\n// Use specific column names as you need\nfunc (r Record) TaosCols() []string {\n\tvar tags []string\n\treturn tags\n}\n\n// Values\nfunc (r Record) TaosValues() []interface{} {\n\tvar values []interface{}\n\tvalues = append(values, r.Timestamp)\n\tvalues = append(values, r.Cols...)\n\treturn values\n}\n```\n\nConsumer will use `ToTaosBatchInsertSql` method in [pkg/utils](pkg/utils/) package to generate sql in batch. You can see more details in [pkg/utils/tosql.go](pkg/utils/tosql.go).\n\n`Codec` implementations, `Partition` id is provided in this interface.\n\n```go\n// Interface for message\ntype Codec interface {\n\tCodecMethod() CodecMethodEnum\n\tPartition() int32\n}\n\n// Change max partitions as you need.\nconst MAX_PARTITIONS = 10\n\n// Codec interface\n\n// Encoding method\nfunc (r Record) CodecMethod() utils.CodecMethodEnum {\n\treturn utils.MessagePack\n}\n\n// Show how to set partition for an message\nfunc (r Record) Partition() int32 {\n\th := fnv.New32a()\n\th.Write([]byte(r.DeviceID))\n\treturn int32(h.Sum32() % MAX_PARTITIONS)\n}\n```\n\n### Demo Consumer Details\n\nWe'll use all the above to build an feature-rich consumer.\n\n- Everything is configurable.\n- Stateless: zero-config for scaling.\n- Clear code.\n- Multithreading.\n- Gracefully teminating.\n\nHere is the implementation.\n\n#### 1. List all options\n\n```go\nvar kafkaOptions options.Kafka\nvar topic string = \"test\"\nvar consumedRecords int64\nvar log = logrus.New()\nvar qOption *queue.Option = queue.DefaultOption().SetMaxQueueSize(10000).SetMaxBatchSize(100)\nvar q *queue.Queue\nvar workers = 4\nvar taosuri = \"root:taosdata/tcp(tdengine:6030)/test\"\nvar consumedPartions = make([]int, record.MAX_PARTITIONS)\n```\n\nConfig with flag:\n\n```go\n// Option parsing\nqOption.FlagInit()\nkafkaOptions.Init()\nflag.StringVar(\u0026topic, \"topic\", topic, \"kafka topic to consume\")\nflag.StringVar(\u0026taosuri, \"taosuri\", topic, \"TDengine uri\")\nflag.IntVar(\u0026workers, \"worker\", workers, \"worker for TDengine insertion\")\nflag.Parse()\n```\n\n#### 2. Setup batch queue\n\n```go\nq = queue.NewWithOption(qOption)\n```\n\n#### 3. Prepare logger\n\n```go\n// var log = logrus.New()\nlogger := log.WithFields(logrus.Fields{\n  \"topic\": topic,\n})\n```\n\n#### 4. Subscribe kafka with topic\n\nSome event handler:\n\n```go\n// OnConnect hanlder\nfunc onConnect() {\n\tlog.Println(\"connected\")\n}\n\n// OnConnectionLost handler\nfunc onConnectionLost(err error) {\n\tlog.Println(\"connection lost\")\n}\n// Consume message and push to batch queue\nfunc msgHandler(message *sarama.ConsumerMessage) (err error) {\n\tvar r record.Record\n\t// for partition messages count in each consumer, you can ignore it\n\tconsumedPartions[message.Partition] += 1\n\terr = utils.FromKafkaBytes(message.Value, \u0026r)\n\tif err != nil {\n\t\tlog.Errorf(\"Partition:%d Offset:%d Key:%v Err: %v\\n\", message.Partition, message.Offset, message.Key, err)\n\t\treturn err\n\t}\n\t_, err = q.Enqueue(r)\n\treturn err\n}\n```\n\nSubscribe in main:\n\n```go\n// Subscribe kafka\nmq := kafka.Run(\u0026kafkaOptions, logger, common.Options{OnConnect: onConnect, OnConnectionLost: onConnectionLost})\nhandler, err := mq.QueueSubscribe(topic, topic, msgHandler)\nif err != nil {\n  log.Error(\"subscribe error: \", err)\n  os.Exit(1)\n}\n```\n\n#### 5. Consume messages batch by batch with multithread workers.\n\nHere, we implment an function called `TaosConsume`:\n\n```go\n// Parallel insert use goroutines\nfunc TaosConsume(q *queue.Queue, taos *sql.DB, workers int) {\n\t// Setup workers for batch queue\n\tfor i := 0; i \u003c workers; i++ {\n\t\tgo func(q *queue.Queue) {\n\t\t\tfor {\n\t\t\t\truntime.Gosched()\n\t\t\t\trecords, err := q.Dequeue()\n\t\t\t\tif err != nil {\n\t\t\t\t\ttime.Sleep(200 * time.Millisecond)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tsql, err := utils.ToTaosBatchInsertSql(records)\n\t\t\t\tif err != nil {\n\t\t\t\t\tlog.Errorf(\"cannot build sql with records: %v\", err)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\truntime.Gosched()\n\t\t\t\t// fmt.Println(sql)\n\t\t\t\tres, err := taos.Exec(sql)\n\t\t\t\tif err != nil {\n\t\t\t\t\tlog.Errorf(\"exec query error: %v, the sql command is:\\n%s\\n\", err, sql)\n\t\t\t\t}\n\n\t\t\t\truntime.Gosched()\n\t\t\t\tn, _ := res.RowsAffected()\n\t\t\t\tatomic.AddInt64(\u0026consumedRecords, n)\n\t\t\t}\n\t\t}(q)\n\t}\n\t// Setup a watcher goroutine for profiling\n\tgo func() {\n\t\tvar last int64\n\t\tfor {\n\t\t\tconsumed := atomic.LoadInt64(\u0026consumedRecords)\n\t\t\tif consumed == 0 {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\ttime.Sleep(1 * time.Second)\n\t\t\tif consumed-last != 0 {\n\t\t\t\tlog.Infof(\"consumed %d records (%d in last second), consumed partitions:%v\\n\",\n\t\t\t\t\tconsumed, consumed-last, consumedPartions)\n\t\t\t}\n\t\t\tlast = consumed\n\t\t}\n\n\t}()\n}\n```\n\nCall this func in main:\n\n```go\n// Consume message object queue to TDengine batch by batch\nToasConsume(q, taos, workers)\n```\n\n#### 6. Listen to the os signals to hold\n\n```go\nc := make(chan os.Signal, 2)\nsignal.Notify(c, os.Interrupt, syscall.SIGTERM)\n\u003c-c\n```\n\n#### 7. Gracefully teminated\n\n```go\n// 1. stop the queue to not recived data any more\nq.Pause()\n// 2. stop subscribe handler\nhandler.Unsubscribe()\n// 3. wait until all consumed messages and their jobs are finished\nfor {\n\tif q.IsEmpty() {\n\t\tlog.Info(\"all job in the queue is finished\")\n\t\tbreak\n\t}\n\ttime.Sleep(100 * time.Millisecond)\n}\n// 4. stop the message queue\nmq.Stop()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaosdata%2Fgo-demo-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaosdata%2Fgo-demo-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaosdata%2Fgo-demo-kafka/lists"}