{"id":13811898,"url":"https://github.com/harlow/kinesis-consumer","last_synced_at":"2025-05-14T01:09:55.937Z","repository":{"id":19023367,"uuid":"22246793","full_name":"harlow/kinesis-consumer","owner":"harlow","description":"Golang library for consuming Kinesis stream data","archived":false,"fork":false,"pushed_at":"2024-12-07T19:54:45.000Z","size":1022,"stargazers_count":275,"open_issues_count":18,"forks_count":95,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-04-03T16:25:31.787Z","etag":null,"topics":["go","golang","golang-kinesis-connector","kinesis","kinesis-consumer","stream"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harlow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-25T06:03:41.000Z","updated_at":"2025-03-28T05:37:44.000Z","dependencies_parsed_at":"2023-12-15T01:59:50.441Z","dependency_job_id":"904df79a-297c-4ff1-a182-cce8bac01b6b","html_url":"https://github.com/harlow/kinesis-consumer","commit_stats":{"total_commits":199,"total_committers":37,"mean_commits":5.378378378378378,"dds":0.3919597989949749,"last_synced_commit":"baf8258298362f90b75a3f0e4674683fdabe1c5b"},"previous_names":["harlow/kinesis-connectors"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlow%2Fkinesis-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlow%2Fkinesis-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlow%2Fkinesis-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlow%2Fkinesis-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harlow","download_url":"https://codeload.github.com/harlow/kinesis-consumer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248287922,"owners_count":21078810,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","golang-kinesis-connector","kinesis","kinesis-consumer","stream"],"created_at":"2024-08-04T04:00:38.648Z","updated_at":"2025-04-10T20:04:05.436Z","avatar_url":"https://github.com/harlow.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# Golang Kinesis Consumer\n\n![technology Go](https://img.shields.io/badge/technology-go-blue.svg) [![Build Status](https://travis-ci.com/harlow/kinesis-consumer.svg?branch=master)](https://travis-ci.com/harlow/kinesis-consumer) [![GoDoc](https://godoc.org/github.com/harlow/kinesis-consumer?status.svg)](https://godoc.org/github.com/harlow/kinesis-consumer) [![GoReportCard](https://goreportcard.com/badge/github.com/harlow/kinesis-consumer)](https://goreportcard.com/report/harlow/kinesis-consumer)\n\nKinesis consumer applications written in Go. This library is intended to be a lightweight wrapper around the Kinesis API to read records, save checkpoints (with swappable backends), and gracefully recover from service timeouts/errors.\n\n__Alternate serverless options:__\n\n* [Kinesis to Firehose](http://docs.aws.amazon.com/firehose/latest/dev/writing-with-kinesis-streams.html) can be used to archive data directly to S3, Redshift, or Elasticsearch without running a consumer application.\n\n* [Process Kinesis Streams with Golang and AWS Lambda](https://medium.com/@harlow/processing-kinesis-streams-w-aws-lambda-and-golang-264efc8f979a) for serverless processing and checkpoint management.\n\n## Installation\n\nGet the package source:\n\n    $ go get github.com/harlow/kinesis-consumer\n\nNote: This repo now requires the AWS SDK V2 package. If you are still using\nAWS SDK V1 then use: https://github.com/harlow/kinesis-consumer/releases/tag/v0.3.5\n\n## Overview\n\nThe consumer leverages a handler func that accepts a Kinesis record. The `Scan` method will consume all shards concurrently and call the callback func as it receives records from the stream.\n\n_Important 1: The `Scan` func will also poll the stream to check for new shards, it will automatically start consuming new shards added to the stream._\n\n_Important 2: The default Log, Counter, and Checkpoint are no-op which means no logs, counts, or checkpoints will be emitted when scanning the stream. See the options below to override these defaults._\n\n```go\nimport(\n\t// ...\n\n\tconsumer \"github.com/harlow/kinesis-consumer\"\n)\n\nfunc main() {\n\tvar stream = flag.String(\"stream\", \"\", \"Stream name\")\n\tflag.Parse()\n\n\t// consumer\n\tc, err := consumer.New(*stream)\n\tif err != nil {\n\t\tlog.Fatalf(\"consumer error: %v\", err)\n\t}\n\n\t// start scan\n\terr = c.Scan(context.TODO(), func(r *consumer.Record) error {\n\t\tfmt.Println(string(r.Data))\n\t\treturn nil // continue scanning\n\t})\n\tif err != nil {\n\t\tlog.Fatalf(\"scan error: %v\", err)\n\t}\n\n\t// Note: If you need to aggregate based on a specific shard\n\t// the `ScanShard` function should be used instead.\n}\n```\n\n## ScanFunc\n\nScanFunc is the type of the function called for each message read\nfrom the stream. The record argument contains the original record\nreturned from the AWS Kinesis library.\n\n```go\ntype ScanFunc func(r *Record) error\n```\n\nIf an error is returned, scanning stops. The sole exception is when the\nfunction returns the special value SkipCheckpoint.\n\n```go\n// continue scanning\nreturn nil\n\n// continue scanning, skip checkpoint\nreturn consumer.SkipCheckpoint\n\n// stop scanning, return error\nreturn errors.New(\"my error, exit all scans\")\n```\n\nUse context cancel to signal the scan to exit without error. For example if we wanted to gracefully exit the scan on interrupt.\n\n```go\n// trap SIGINT, wait to trigger shutdown\nsignals := make(chan os.Signal, 1)\nsignal.Notify(signals, os.Interrupt)\n\n// context with cancel\nctx, cancel := context.WithCancel(context.Background())\n\ngo func() {\n\t\u003c-signals\n\tcancel() // call cancellation\n}()\n\nerr := c.Scan(ctx, func(r *consumer.Record) error {\n\tfmt.Println(string(r.Data))\n\treturn nil // continue scanning\n})\n```\n\n## Options\n\nThe consumer allows the following optional overrides.\n\n### Store\n\nTo record the progress of the consumer in the stream (checkpoint) we use a storage layer to persist the last sequence number the consumer has read from a particular shard. The boolean value ErrSkipCheckpoint of consumer.ScanError determines if checkpoint will be activated. ScanError is returned by the record processing callback.\n\nThis will allow consumers to re-launch and pick up at the position in the stream where they left off.\n\nThe uniq identifier for a consumer is `[appName, streamName, shardID]`\n\n\u003cimg width=\"722\" alt=\"kinesis-checkpoints\" src=\"https://user-images.githubusercontent.com/739782/33085867-d8336122-ce9a-11e7-8c8a-a8afeb09dff1.png\"\u003e\n\nNote: The default storage is in-memory (no-op). Which means the scan will not persist any state and the consumer will start from the beginning of the stream each time it is re-started.\n\nThe consumer accepts a `WithStore` option to set the storage layer:\n\n```go\nc, err := consumer.New(*stream, consumer.WithStore(db))\nif err != nil {\n\tlog.Log(\"consumer error: %v\", err)\n}\n```\n\nTo persist scan progress choose one of the following storage layers:\n\n#### Redis\n\nThe Redis checkpoint requires App Name, and Stream Name:\n\n```go\nimport store \"github.com/harlow/kinesis-consumer/store/redis\"\n\n// redis checkpoint\ndb, err := store.New(appName)\nif err != nil {\n\tlog.Fatalf(\"new checkpoint error: %v\", err)\n}\n```\n\n#### DynamoDB\n\nThe DynamoDB checkpoint requires Table Name, App Name, and Stream Name:\n\n```go\nimport store \"github.com/harlow/kinesis-consumer/store/ddb\"\n\n// ddb checkpoint\ndb, err := store.New(appName, tableName)\nif err != nil {\n\tlog.Fatalf(\"new checkpoint error: %v\", err)\n}\n\n// Override the Kinesis if any needs on session (e.g. assume role)\nmyDynamoDbClient := dynamodb.New(session.New(aws.NewConfig()))\n\n// For versions of AWS sdk that fixed config being picked up properly, the example of\n// setting region should work.\n//    myDynamoDbClient := dynamodb.New(session.New(aws.NewConfig()), \u0026aws.Config{\n//        Region: aws.String(\"us-west-2\"),\n//    })\n\ndb, err := store.New(*app, *table, checkpoint.WithDynamoClient(myDynamoDbClient))\nif err != nil {\n  log.Fatalf(\"new checkpoint error: %v\", err)\n}\n\n// Or we can provide your own Retryer to customize what triggers a retry inside checkpoint\n// See code in examples\n// ck, err := checkpoint.New(*app, *table, checkpoint.WithDynamoClient(myDynamoDbClient), checkpoint.WithRetryer(\u0026MyRetryer{}))\n```\n\nTo leverage the DDB checkpoint we'll also need to create a table:\n\n```\nPartition key: namespace\nSort key: shard_id\n```\n\n\u003cimg width=\"727\" alt=\"screen shot 2017-11-22 at 7 59 36 pm\" src=\"https://user-images.githubusercontent.com/739782/33158557-b90e4228-cfbf-11e7-9a99-73b56a446f5f.png\"\u003e\n\n#### Postgres\n\nThe Postgres checkpoint requires Table Name, App Name, Stream Name and ConnectionString:\n\n```go\nimport store \"github.com/harlow/kinesis-consumer/store/postgres\"\n\n// postgres checkpoint\ndb, err := store.New(app, table, connStr)\nif err != nil {\n  log.Fatalf(\"new checkpoint error: %v\", err)\n}\n\n```\n\nTo leverage the Postgres checkpoint we'll also need to create a table:\n\n```sql\nCREATE TABLE kinesis_consumer (\n\tnamespace text NOT NULL,\n\tshard_id text NOT NULL,\n\tsequence_number numeric NOT NULL,\n\tCONSTRAINT kinesis_consumer_pk PRIMARY KEY (namespace, shard_id)\n);\n```\n\nThe table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity.\n\n#### Mysql\n\nThe Mysql checkpoint requires Table Name, App Name, Stream Name and ConnectionString (just like the Postgres checkpoint!):\n\n```go\nimport store \"github.com/harlow/kinesis-consumer/store/mysql\"\n\n// mysql checkpoint\ndb, err := store.New(app, table, connStr)\nif err != nil {\n  log.Fatalf(\"new checkpoint error: %v\", err)\n}\n\n```\n\nTo leverage the Mysql checkpoint we'll also need to create a table:\n\n```sql\nCREATE TABLE kinesis_consumer (\n\tnamespace varchar(255) NOT NULL,\n\tshard_id varchar(255) NOT NULL,\n\tsequence_number numeric(65,0) NOT NULL,\n\tCONSTRAINT kinesis_consumer_pk PRIMARY KEY (namespace, shard_id)\n);\n```\n\nThe table name has to be the same that you specify when creating the checkpoint. The primary key composed by namespace and shard_id is mandatory in order to the checkpoint run without issues and also to ensure data integrity.\n\n### Kinesis Client\n\nOverride the Kinesis client if there is any special config needed:\n\n```go\n// client\nclient := kinesis.New(session.NewSession(aws.NewConfig()))\n\n// consumer\nc, err := consumer.New(streamName, consumer.WithClient(client))\n```\n\n### Metrics\n\nAdd optional counter for exposing counts for checkpoints and records processed:\n\n```go\n// counter\ncounter := expvar.NewMap(\"counters\")\n\n// consumer\nc, err := consumer.New(streamName, consumer.WithCounter(counter))\n```\n\nThe [expvar package](https://golang.org/pkg/expvar/) will display consumer counts:\n\n```json\n\"counters\": {\n  \"checkpoints\": 3,\n  \"records\": 13005\n},\n```\n\n### Consumer starting point\n\nKinesis allows consumers to specify where on the stream they'd like to start consuming from. The default in this library is `LATEST` (Start reading just after the most recent record in the shard).\n\nThis can be adjusted by using the `WithShardIteratorType` option in the library:\n\n```go\n// override starting place on stream to use TRIM_HORIZON\nc, err := consumer.New(\n  *stream,\n  consumer.WithShardIteratorType(kinesis.ShardIteratorTypeTrimHorizon)\n)\n```\n\n[See AWS Docs for more options.](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html)\n\n### Logging\n\nLogging supports the basic built-in logging library or use third party external one, so long as\nit implements the Logger interface.\n\nFor example, to use the builtin logging package, we wrap it with myLogger structure.\n\n```go\n// A myLogger provides a minimalistic logger satisfying the Logger interface.\ntype myLogger struct {\n\tlogger *log.Logger\n}\n\n// Log logs the parameters to the stdlib logger. See log.Println.\nfunc (l *myLogger) Log(args ...interface{}) {\n\tl.logger.Println(args...)\n}\n```\n\nThe package defaults to `ioutil.Discard` so swallow all logs. This can be customized with the preferred logging strategy:\n\n```go\n// logger\nlogger := \u0026myLogger{\n\tlogger: log.New(os.Stdout, \"consumer-example: \", log.LstdFlags),\n}\n\n// consumer\nc, err := consumer.New(streamName, consumer.WithLogger(logger))\n```\n\nTo use a more complicated logging library, e.g. apex log\n\n```go\ntype myLogger struct {\n\tlogger *log.Logger\n}\n\nfunc (l *myLogger) Log(args ...interface{}) {\n\tl.logger.Infof(\"producer\", args...)\n}\n\nfunc main() {\n\tlog := \u0026myLogger{\n\t\tlogger: alog.Logger{\n\t\t\tHandler: text.New(os.Stderr),\n\t\t\tLevel:   alog.DebugLevel,\n\t\t},\n\t}\n```\n\n# Examples\n\nThere are examples of producer and comsumer in the `/examples` directory. These should help give end-to-end examples of setting up consumers with different checkpoint strategies.\n\nThe examples run locally against [Kinesis Lite](https://github.com/mhart/kinesalite).\n\n\t$ kinesalite \u0026\n\nProduce data to the stream:\n\n\t$ cat examples/producer/users.txt  | go run examples/producer/main.go --stream myStream\n\nConsume data from the stream:\n\n\t$ go run examples/consumer/main.go --stream myStream\n\n## Contributing\n\nPlease see [CONTRIBUTING.md] for more information. Thank you, [contributors]!\n\n[LICENSE]: /MIT-LICENSE\n[CONTRIBUTING.md]: /CONTRIBUTING.md\n\n## License\n\nCopyright (c) 2015 Harlow Ward. It is free software, and may\nbe redistributed under the terms specified in the [LICENSE] file.\n\n[contributors]: https://github.com/harlow/kinesis-connectors/graphs/contributors\n\n\u003e [www.hward.com](http://www.hward.com) \u0026nbsp;\u0026middot;\u0026nbsp;\n\u003e GitHub [@harlow](https://github.com/harlow) \u0026nbsp;\u0026middot;\u0026nbsp;\n\u003e Twitter [@harlow_ward](https://twitter.com/harlow_ward)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharlow%2Fkinesis-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharlow%2Fkinesis-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharlow%2Fkinesis-consumer/lists"}