{"id":13697621,"url":"https://github.com/nmaquet/kasper","last_synced_at":"2025-12-15T23:29:58.920Z","repository":{"id":55067708,"uuid":"81270985","full_name":"nmaquet/kasper","owner":"nmaquet","description":"Kasper is a lightweight library for processing Kafka topics.","archived":false,"fork":false,"pushed_at":"2017-09-16T07:10:15.000Z","size":295,"stargazers_count":439,"open_issues_count":1,"forks_count":24,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-04-14T12:12:22.677Z","etag":null,"topics":["golang-library","kafka","stream-processing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nmaquet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-02-08T00:53:02.000Z","updated_at":"2024-03-04T16:05:35.000Z","dependencies_parsed_at":"2022-08-14T11:00:26.816Z","dependency_job_id":null,"html_url":"https://github.com/nmaquet/kasper","commit_stats":null,"previous_names":["movio/kasper"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmaquet%2Fkasper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmaquet%2Fkasper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmaquet%2Fkasper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nmaquet%2Fkasper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nmaquet","download_url":"https://codeload.github.com/nmaquet/kasper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252252561,"owners_count":21718765,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang-library","kafka","stream-processing"],"created_at":"2024-08-02T18:01:00.864Z","updated_at":"2025-12-15T23:29:58.884Z","avatar_url":"https://github.com/nmaquet.png","language":"Go","funding_links":[],"categories":["Libraries","Go"],"sub_categories":["Kafka"],"readme":"kasper\n======\n\n[![GoDoc](https://godoc.org/github.com/movio/kasper?status.svg)](https://godoc.org/github.com/movio/kasper)\n[![Build Status](https://travis-ci.org/movio/kasper.svg?branch=master)](https://travis-ci.org/movio/kasper)\n[![Go Report Card](https://goreportcard.com/badge/github.com/movio/kasper)](https://goreportcard.com/report/github.com/movio/kasper)\n[![Coverage Status](https://coveralls.io/repos/github/movio/kasper/badge.svg?branch=master)](https://coveralls.io/github/movio/kasper?branch=master)\n\n*This project is no longer in active development.*\n\nFor an introduction to Kasper and the motivation behind it, you can [read our introductory blog post](https://movio.co/blog/Kasper-process-library/).\n\nKasper is a lightweight library for processing Kafka topics.\nIt is heavily inspired by Apache Samza (See http://samza.apache.org).\nKasper processes Kafka messages in small batches and is designed to work with centralized key-value stores such as Redis,\nCassandra or Elasticsearch for maintaining state during processing. Kasper is a good fit for high-throughput applications\n(\u003e 10k messages per second) that can tolerate a moderate amount of processing latency (~1000ms).\nPlease note that Kasper is designed for idempotent processing of at-least-once semantics streams.\nIf you require exactly-once semantics or need to perform non-idempotent operations, Kasper is likely not a good choice.\n\n## Step 1 - Create a sarama Client\n\nKasper uses Shopify's excellent sarama library (see https://github.com/Shopify/sarama)\nfor consuming and producing messages to Kafka. All Kasper application must begin with instantiating a sarama Client.\nChoose the parameters in sarama.Config carefully; the performance, reliability, and correctness\nof your application are all highly sensitive to these settings.\nWe recommend setting sarama.Config.Producer.RequiredAcks to WaitForAll.\n\n\tsaramaConfig := sarama.NewConfig()\n\tsaramaConfig.Producer.RequiredAcks = sarama.WaitForAll\n\tclient, err := sarama.NewClient([]string{\"kafka-broker.local:9092\"}, saramaConfig)\n\n## Step 2 - create a Config\n\nTopicProcessorName is used for logging, labeling metrics, and is used as a suffix to the Kafka consumer group.\nInputTopics and InputPartitions are the lists of topics and partitions to consume.\nPlease note that Kasper currently does not support consuming topics with differing numbers of partitions.\nThis limitation can be alleviated by manually adding an extra fan-out step in your processing pipeline\nto a new topic with the desired number of partitions.\n\n\tconfig := \u0026kasper.Config{\n\t\tTopicProcessorName:    \"twitter-reach\",\n\t\tClient:                client,\n\t\tInputTopics:           []string{\"tweets\", \"twitter-followers\"},\n\t\tInputPartitions:       []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11},\n\t\tBatchSize:             10000,\n\t\tBatchWaitDuration:     5 * time.Second,\n\t\tLogger:                kasper.NewJSONLogger(\"twitter-reach-0\", false),\n\t\tMetricsProvider:       kasper.NewPrometheus(\"twitter-reach-0\"),\n\t\tMetricsUpdateInterval: 60 * time.Second,\n\t}\n\nKasper is instrumented with a number of useful metrics so we\nrecommend setting MetricsProvider for production applications. Kasper includes an implementation for collecting\nmetrics in Prometheus and adapting the interface to other tools should be easy.\n\n## Step 3 - Create a MessageProcessor per input partition\n\nYou need to create a `map[int]MessageProcessor`. The MessageProcessor instances can safely be shared across partitions.\nEach MessageProcessor must implement a single function:\n\n\tfunc (*TweetProcessor) Process(messages []*sarama.ConsumerMessage, sender Sender) error {\n\t\t// process messages here\n\t}\n\nAll messages for the input topics on the specified partition will be passed to the appropriate MessageProcessor instance.\nThis is useful for implementing partition-wise joins of different topics.\nThe Sender instance must be used to produce messages to output topics. Messages passed to Sender are not sent directly but are collected in an array instead.\nWhen Process returns, the messages are sent to Kafka and Kasper waits for the configured number of acks.\nWhen all messages have been successfully produced, Kasper updates the consumer offsets of the input partitions and\nresumes processing. If Process returns a non-nil error value, Kasper stops all processing.\n\n## Step 4 - Create a TopicProcessor\n\nTo start processing messages, call TopicProcessor.RunLoop(). Kasper does not spawn any goroutines and runs a single-threaded\nevent loop instead. RunLoop() will block the current goroutine and will run forever until an error occurs or until\nClose() is called.\nFor parallel processing, run multiple TopicProcessor instances in different goroutines or processes\n(the input partitions cannot overlap). You should set Config.TopicProcessorName to the same value on\nall instances in order to easily scale the processing up or down.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnmaquet%2Fkasper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnmaquet%2Fkasper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnmaquet%2Fkasper/lists"}