{"id":23919215,"url":"https://github.com/zkfmapf123/kafka-pattern","last_synced_at":"2026-05-18T05:11:36.777Z","repository":{"id":264265396,"uuid":"885669009","full_name":"zkfmapf123/kafka-pattern","owner":"zkfmapf123","description":"kafka 패턴 및 방법론","archived":false,"fork":false,"pushed_at":"2025-02-23T05:51:05.000Z","size":1395,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-23T06:27:26.272Z","etag":null,"topics":["golang","kafak","nodejs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zkfmapf123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-09T04:33:18.000Z","updated_at":"2025-02-23T05:51:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"12a16e08-53bc-4ad6-9f89-67838d7a880e","html_url":"https://github.com/zkfmapf123/kafka-pattern","commit_stats":null,"previous_names":["zkfmapf123/kafka-in-go","zkfmapf123/kafka-pattern"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkfmapf123%2Fkafka-pattern","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkfmapf123%2Fkafka-pattern/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkfmapf123%2Fkafka-pattern/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zkfmapf123%2Fkafka-pattern/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zkfmapf123","download_url":"https://codeload.github.com/zkfmapf123/kafka-pattern/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240378874,"owners_count":19792039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","kafak","nodejs"],"created_at":"2025-01-05T14:38:03.721Z","updated_at":"2026-05-18T05:11:31.732Z","avatar_url":"https://github.com/zkfmapf123.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Essential\n\n## Folder Architecture\n\n```sh\n## simple\n|- producer \t\t\t\t## producer 코드 (nodejs)\n|- consumer\t\t\t\t## consumer 코드 (golang)\n\n## advacned\n|- producer-lambda\t\t\t## producer 코드 람다 코드 (javascript)\n|- consumer-batch-listener\t\t## consumer Batch 코드 (golang)\n\n## cdc / outbox pattern\n|- producer-cdc\t\t\t\t## producer cdc 코드 (javascript + mysql)\n|- producer-outbox\t\t\t## producer outbox 코드 (javascript + mysql)\n|- consumer-mysql\t\t\t## consumer mysql 코드 (golang + mysql)\n```\n\n## Exec\n\n```sh\n\t## pub-sub\n\tmake up\n\n\t## cdc up\n\tmake cdc-up\n\n\t## outbox up\n\tmake outbox-up\n```\n\n## Infra / Container\n\n- [kafka docker-compose.yml 코드](./infra/docker-compose.yml)\n- [api docker-compose.yml 코드](./docker-compose.yml)\n\n## Producer Code (nodejs)\n\n```javascript\nimport { Kafka, Partitioners } from \"kafkajs\";\nimport { getConfig } from \"./env.confg.js\";\n\nclass KafkaConfig {\n  _kafkaConn = undefined;\n\n  constructor(clientId, brokers) {\n    if (!this._kafkaConn) {\n      console.log(\"kafak connect...\");\n      this._kafkaConn = new Kafka({\n        clientId,\n        brokers: brokers.split(\",\"),\n        // requestTimeout: 10000,\n        // retry: 5,\n      });\n    } else {\n      console.log(\"kafka is already connect...\");\n    }\n  }\n\n  async isExistTopic(topic) {\n    const admin = this._kafkaConn.admin();\n\n    try {\n      await admin.connect();\n\n      const topics = await admin.listTopics();\n      if (topics.includes(topic)) {\n        return true;\n      }\n\n      return false;\n    } catch (e) {\n      console.error(e);\n    }\n  }\n\n  async producer(value) {\n    const topic = getConfig().KAFKA_TOPIC;\n    const p = await this._kafkaConn.producer({\n      createPartitioner: Partitioners.LegacyPartitioner,\n    });\n\n    try {\n      // const isExistsTopic = await this.isExistTopic(topic);\n      // console.log(isExistsTopic);\n\n      await p.connect();\n      await p.send({\n        topic,\n        messages: [{ key: value.id, value: JSON.stringify(value) }],\n      });\n    } catch (e) {\n      console.error(e);\n    } finally {\n      await p.disconnect();\n    }\n  }\n}\n\nexport const kafkaConn = new KafkaConfig(\n  getConfig().KAFKA_CLUSTER_NAME,\n  getConfig().KAFKA_BROKERS\n);\n```\n\n## Consumer Code (Golang)\n\n```golang\nfunc NewKafka() kafkaConn {\n\n\tkafkaBrokers := strings.Split(KAFKA_BROKERS, \",\")\n\n\tconfig := sarama.NewConfig()\n\tconfig.Consumer.Return.Errors = true\n\tconfig.Version = sarama.V3_6_0_0\n\n\tconsumer,err := sarama.NewConsumer(kafkaBrokers, config)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\treturn kafkaConn{\n\t\tconsumer : consumer,\n\t}\n}\n\nfunc (k kafkaConn) Consume(topic string)  {\n\n\tpartitionList, err := k.consumer.Partitions(topic)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\tfor _, partition := range partitionList {\n\t\tpConsumer, err := k.consumer.ConsumePartition(topic, partition, sarama.OffsetNewest)\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\n\t\tgo func(pc sarama.PartitionConsumer) {\n\t\t\tfor msg := range pc.Messages() {\n\t\t\t\tfmt.Println(\"[consume] topic : \",topic, string(msg.Value))\n\t\t\t}\n\t\t}(pConsumer)\n\t}\n}\n\nfunc (k kafkaConn) Close() error {\n\tlog.Panicln(\"Closing kafka consumer...\")\n\treturn k.consumer.Close()\n}\n```\n\n## Consumer 성능개선 (BatchListener)\n\n```golang\nfunc NewKafka() kafkaConn {\n\n\tkafkaBrokers := strings.Split(KAFKA_BROKERS, \",\")\n\n\tconfig := sarama.NewConfig()\n\tconfig.Consumer.Return.Errors = true\n\tconfig.Version = sarama.V3_6_0_0\n\n\tconfig.Consumer.Offsets.AutoCommit.Enable = false\n\tconfig.Consumer.Offsets.Initial = sarama.OffsetNewest // 최신 메시지부터 수동커밋...\n\n\tconsumerGroup, err := sarama.NewConsumerGroup(kafkaBrokers, KAFKA_CONSUMER_GROUP, config)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\treturn kafkaConn{\n\t\tconsumer: consumerGroup,\n\t}\n}\n\ntype BatchListener struct{}\n\n// Cleanup implements sarama.ConsumerGroupHandler.\nfunc (b *BatchListener) Cleanup(sarama.ConsumerGroupSession) error {\n\tlog.Panicln(\"[Cleanup] 파티션 재할당\")\n\treturn nil\n}\n\n// Setup implements sarama.ConsumerGroupHandler.\nfunc (b *BatchListener) Setup(sarama.ConsumerGroupSession) error {\n\tlog.Println(\"[Setup] 파티션 할당\")\n\treturn nil\n}\n\nfunc messageProcessor(session sarama.ConsumerGroupSession, batch []*sarama.ConsumerMessage) {\n\tfor _, msg := range batch {\n\t\tlog.Println(\"Topic : \", msg.Topic, \"Values : \", string(msg.Value))\n\t\tsession.MarkMessage(msg, \"\")\n\t}\n\n\tlog.Println(\"Commit... \", len(batch))\n}\n\n// ConsumeClaim implements sarama.ConsumerGroupHandler.\nvar (\n\t_BATCH_SIZE = os.Getenv(\"BATCH_SIZE\")\n)\n\nfunc (b *BatchListener) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {\n\n\tBATCH_SIZE ,_ := strconv.Atoi(_BATCH_SIZE)\n\tBATCH_TIMEOUT := 5 * time.Second\n\n\n\tvar batch []*sarama.ConsumerMessage\n\tbatchTimer := time.NewTimer(BATCH_TIMEOUT)\n\n\tfor {\n\t\tselect {\n\t\t\tcase msg := \u003c- claim.Messages():\n\t\t\t\tbatch = append(batch, msg)\n\t\t\t\tif len(batch) \u003e= BATCH_SIZE {\n\t\t\t\t\tmessageProcessor(session, batch)\n\t\t\t\t\tbatch = nil\n\t\t\t\t\tbatchTimer.Reset(BATCH_TIMEOUT)\n\t\t\t\t}\n\n\t\t\tcase \u003c-batchTimer.C:\n\t\t\t\tif len(batch) \u003e0 {\n\t\t\t\t\tmessageProcessor(session, batch)\n\t\t\t\t\tbatch = nil\n\t\t\t\t}\n\t\t\t\tbatchTimer.Reset(BATCH_TIMEOUT)\n\t\t}\n\t}\n}\n\nfunc (k kafkaConn) ConsumeBatch() {\n\n\ttopics := strings.Split(KAFKA_TOPICS, \",\")\n\tgo func() {\n\n\t\tfor {\n\t\t\tif err := k.consumer.Consume(context.TODO(), topics, \u0026BatchListener{}); err != nil {\n\t\t\t\tpanic(err)\n\t\t\t}\n\t\t}\n\t}()\n}\n\nfunc (k kafkaConn) Close() error {\n\tlog.Panicln(\"Closing kafka consumer...\")\n\treturn k.consumer.Close()\n}\n```\n\n## Consumer\n\n![simple consumer](./public/consumer.drawio.png)\n\n## Consumer (Batch Listener)\n\n![batch listener](./public/batch-consumer.drawio.png)\n\n## Pub / Sub Application\n\n![app](./public/app.drawio.png)\n\n## Producer use lambda\n\n![1](./public/1.png)\n\n[Lambda Code](./producer-lambda/main.go)\n\n## CDC Pattern (\\*\\*)\n\n- 단점\n  - 주기적으로 DB에서 값을 읽어오기때문에 -\u003e 변경사항이 잦다면 -\u003e 인지하기 어려움 (이슈 발생 여지 존재)\n\n![cdc](./public/cdc.drawio.png)\n\n## Outbox Pattern (\\*\\*)\n\n![outbox](./public/outbox.drawio.png)\n\n- 특징\n  - outbox라는 별도의 테이블을 활용 (별도의 Queue로 대체)\n  - DB에 테이블 CRUD 후, outbox라는 테이블에 구성\n  - 두개의 테이블 CRUD 후 transaction... (별도의 모듈이 필요할듯함)\n- 장점\n  - 데이터를 outbox라는 테이블에 immutable하게 사용함 -\u003e 매번의 변경사항을 확인할 수 있음\n  - 나름 이상적인 패턴임...\n- 단점\n  - 좀더 구현하기가 복잡할 수 있음 (별도의 테이블을 구성..., immutable 하게 구성해야 함) -\u003e 주기적으로 데이터를 지워줘야 함 (스토리지)\n\n## Consumer 측에서 발생할수 있는 이슈\n\n### Exception\n\n- JsonProcessingTimeoutException\n  - 원인 : Json 파싱 실패\n  - 해결 : 컨슈머 측에서는 해결 불가 (이미 파싱이 불가한거라... 코드수정) -\u003e Not Retryable\n- TimeoutException\n  - 원인 : DB를 쓴다거나, API 요청을 한다던가, 브로커가 안된다던가 (사실 너무 많음)\n  - 해결 : \u003cb\u003e주로 일시적인 문제일 가능성이 높음\u003c/b\u003e - Retryable\n\n### 해결방법 (Retry)\n\n- Retry\n  - 너무 자주 하면 서버에 부하가 많이 온다.\n  - Fixed ( 1초 대기 -\u003e 1초 대기 -\u003e 1초 대기) = x만큼 대기\n  - Exponential ( 2초대기 -\u003e 4초대기 -\u003e 8초대기) = x \\* 2 만큼 대기 (서버부하를 줄일 수 있음...)\n\n![retry](./public/retry.png)\n\n```go\n// Exponential\nfunc mustRetry(retryCount int, maxRetryCount int, backoff int) {\n\n\t// exit\n\tif retryCount == maxRetryCount {\n\t\tlog.Fatalf(\"Retry 최대한도... %d\", retryCount)\n\t}\n\n\tretryBackoffDuration := retryCount * backoff\n\tlog.Printf(\"재시도 횟수 : %d 재시도 시간 : %ds\",retryCount, retryBackoffDuration )\n\ttime.Sleep(time.Duration(retryBackoffDuration))\n}\n```\n\n- Dead Letter Topic\n  - Consume이 실패한 메시지를 새로운 토픽에다가 메시지를 넘긴다.\n  - Dead Letter와 비슷함... (SQS)\n\n## Issue\n\n- EC2에 구성된 카프카 local에서 접근 시, Connection Error\n\n```json\n\n// Error\n [cause]: KafkaJSConnectionError: Connection error:\n      at Socket.onError (/Users/idong-gyu/dev/study/kafka-in-go/producer/node_modules/kafkajs/src/network/connection.js:210:23)\n      at Socket.emit (node:events:518:28)\n      at emitErrorNT (node:internal/streams/destroy:169:8)\n      at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {\n\n// 해결방법\n\n// KAFAK_CFG_ADVERTISED_LISTENERS 의 EXTERNAL 주소가 localhost로 되어있었음\n - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka1:19092,EXTERNAL://localhost:9092\n\n// Public IP 주소로 바꿔주자...\n - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka1:19092,EXTERNAL://\u003cEC2 퍼블릭 IP\u003e:9092\n```\n\n### Cosnumer의 과도한 리밸런싱을 줄이는 방법\n\n- Consumer Group ID가 리밸런싱에 미치는 영향\n\n  - 고정된 Group ID를 사용하면 리밸런싱을 최소화 할 수 있음\n  - static membership\n\n```sh\n\tgroup.instance.id=consumer-*\n```\n\n    - Session Timeout 값 조정\n    - 너무짧으면 Consumer가 일시적으로 끊겨서 리밸런싱 발생\n\n```sh\n# default 45s (이를 늘리면 불필요한 리밸런싱을 줄일 수 있음)\nsession.timeout.ms=60000 # 60s\n```\n\n    - Auto Offset Reset\n    - latest보다는 earlest가 예측가능성을 높여서 리밸런싱을 최소화 할 수 있음\n\n```sh\nauto.offset.reset=earliest\n```\n\n    - reblance protocol 설정\n    - RangeAssignor, StickyAssignor, CooperativeStickyAssignor 중 **CooperativeStickyAssignor**를 사용하면 최소한의 리밸런싱만 발생함\n\n```sh\npartition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor\n```\n\n✅ Consumer가 재시작해도 리밸런싱 없음\n✅ 일시적인 네트워크 장애에도 세션 유지\n✅ Kubernetes/Docker 환경에서 안정적으로 운영 가능\n\n- 즉, Static Membership을 사용하면, consumer가 일단 남아있어 리밸런싱이 발생하지 않음... (consumer-batch-listener에 구성되어있음)\n\n```golang\nfunc NewKafka() kafkaConn {\n\n\tkafkaBrokers := strings.Split(KAFKA_BROKERS, \",\")\n\n\tconfig := sarama.NewConfig()\n\tconfig.Consumer.Return.Errors = true\n\tconfig.Version = sarama.V3_6_0_0\n\n\tgroupId := \"\"\n\thostname, _ := os.Hostname()\n\n\t// static membership\n\tif IS_STATIC_MEMBERSHIP == \"true\" {\n\t\tgroupId = fmt.Sprintf(\"consumer-%s\", hostname)\n\t}else{\n\t\tgroupId = hostname\n\t}\n\n\tconfig.Consumer.Offsets.AutoCommit.Enable = false\n\tconfig.Consumer.Offsets.Initial = sarama.OffsetNewest // 최신 메시지부터 수동커밋...\n\n\t// static membership\n\t// config.Consumer.Group.InstanceId = \"consumer-\"\n\n\n\tconsumerGroup, err := sarama.NewConsumerGroup(kafkaBrokers, groupId, config)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\n\treturn kafkaConn{\n\t\tconsumer: consumerGroup,\n\t}\n}\n```\n\n## Reference\n\n- \u003ca href=\"https://kafka.js.org/docs/getting-started\"\u003eKafakjs\u003c/a\u003e\n\n```\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzkfmapf123%2Fkafka-pattern","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzkfmapf123%2Fkafka-pattern","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzkfmapf123%2Fkafka-pattern/lists"}