{"id":19078342,"url":"https://github.com/kickstarter/telekinesis","last_synced_at":"2025-04-30T04:48:40.653Z","repository":{"id":22795617,"uuid":"26141961","full_name":"kickstarter/telekinesis","owner":"kickstarter","description":":zap: A JRuby Kinesis Client :zap:","archived":false,"fork":false,"pushed_at":"2022-08-10T19:03:57.000Z","size":55,"stargazers_count":21,"open_issues_count":2,"forks_count":9,"subscribers_count":137,"default_branch":"master","last_synced_at":"2025-04-30T04:48:32.358Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kickstarter.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-03T22:31:00.000Z","updated_at":"2023-11-11T09:43:26.000Z","dependencies_parsed_at":"2022-08-09T02:00:04.699Z","dependency_job_id":null,"html_url":"https://github.com/kickstarter/telekinesis","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kickstarter%2Ftelekinesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kickstarter%2Ftelekinesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kickstarter%2Ftelekinesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kickstarter%2Ftelekinesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kickstarter","download_url":"https://codeload.github.com/kickstarter/telekinesis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251644827,"owners_count":21620630,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T02:08:32.427Z","updated_at":"2025-04-30T04:48:40.632Z","avatar_url":"https://github.com/kickstarter.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Looking for Maintainers\n\nWe're not actively maintaining this project. If you're interested in maintaining it, please post a comment on [this issue](https://github.com/kickstarter/telekinesis/issues/22).\n\n## Table of Contents\n\n- [Telekinesis](#telekinesis)\n  - [Requirements](#requirements)\n  - [Installing](#installing)\n  - [Producers](#producers)\n    - [SyncProducer](#syncproducer)\n    - [AsyncProducer](#asyncproducer)\n  - [Consumers](#consumers)\n    - [KCL](#kcl)\n      - [Client State](#client-state)\n      - [Errors while processing records](#errors-while-processing-records)\n      - [Checkpoints and `INITIAL_POSITION_IN_STREAM`](#checkpoints-and-initial_position_in_stream)\n  - [Java client logging](#java-client-logging)\n  - [](#)\n- [Building](#building)\n  - [Prerequisites](#prerequisites)\n  - [Build](#build)\n- [Testing](#testing)\n- [License](#license)\n\n# Telekinesis\n\nTelekinesis is a high-level client for Amazon Kinesis.\n\nThe library provides a high-throughput asynchronous producer and wraps the\n[Kinesis Client Library](https://github.com/awslabs/amazon-kinesis-client) to\nprovide an easy interface for writing consumers.\n\n## Requirements\n\nTelekinesis runs on JRuby 1.7.x or later, with at least Java 6.\n\nIf you want to build from source, you need to have Apache Maven installed.\n\n## Installing\n\n```\ngem install telekinesis\n```\n\n## Producers\n\nTelekinesis includes two high-level\n[Producers](http://docs.aws.amazon.com/kinesis/latest/dev/amazon-kinesis-producers.html).\n\nTelekinesis assumes that records are `[key, value]` pairs of strings. The key\n*must* be a string as enforced by Kinesis itself. Keys are used by the service\nto partition data into shards. Values can be any old blob of data, but for\nsimplicity, Telekinesis expects strings.\n\nBoth keys and values should respect any Kinesis\n[limits](http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html).\nand all of the [restrictions](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html)\nin the PutRecords API documentation.\n\n### SyncProducer\n\nThe `SyncProducer` sends data to Kinesis every time `put` or `put_all`\nis called. These calls will block until the call to Kinesis returns.\n\n\n```ruby\nrequire 'telekinesis'\n\nproducer = Telekinesis::Producer::SyncProducer.create(\n  stream: 'my stream',\n  credentials: {\n    access_key_id: 'foo',\n    secret_access_key: 'bar'\n  }\n)\n```\n\nCalls to `put` send a single record at a time to Kinesis, where calls to\n`put_all` can send up to 500 records at a time, which is the Kinesis service\nlimit.  If more than 500 records are passed to `put_all` they're grouped into\nbatches and sent.\n\n\u003e NOTE: To send fewer records to Kinesis at a time when using `put_all`, you\n\u003e can adjust the `:send_size` parameter in the `create` method.\n\nUsing `put_all` over `put` is recommended if you have any way to batch your\ndata. Since Kinesis has an HTTP API and often has high latency, it tends to make\nsense to try and increase throughput as much as possible by batching data.\n\n```ruby\n# file is an instance of File containing CSV data that looks like:\n#\n#   \"some,very,important,data,with,a,partition_key\"\n#\nlines = file.lines.map do |line|\n  key = line.split(/,/).last\n  data = line\n  [key, data]\nend\n\n# One record at a time\nlines.each do |key, data|\n  producer.put(key, data)\nend\n\n# Manually control your batches\nlines.each_slice(200) do |batch|\n  producer.put_all(batch)\nend\n\n# Go hog wild\nproducer.put_all(lines.to_a)\n```\n\nWhen something goes wrong and the Kinesis client throws an exception, it bubbles\nup as a `Telekinesis::Aws::KinesisError` with the underlying exception accessible\nas the `cause` field.\n\nWhen some of (but maybe not all of) the records passed to `put_all` cause\nproblems, they're returned as an array of\n`[key, value, error_code, error_message]` tuples.\n\n### AsyncProducer\n\nThe `AsyncProducer` queues events interally and uses background threads to send\ndata to Kinesis.  Data is sent when a batch reaches the Kinesis limit of 500,\nwhen the producer's timeout is reached, or when the producer is shut down.\n\n\u003e NOTE: You can configure the size at which a batch is sent by passing the\n\u003e `:send_size` parameter to create. The producer's internal timeout can be\n\u003e set by using the `:send_every_ms` parameter.\n\nThe API for the `AsyncProducer` is looks similar to the `SyncProducer`. However,\nall `put` and `put_all` calls return immediately. Both `put` and `put_all`\nreturn `true` if the producer enqueued the data for sending later, and `false`\nif the producer is not accepting data for any reason. If the producer's internal\nqueue fill up, calls to `put` and `put_all` will block.\n\nSince sending (and therefore failures) happen in a different thread, you can\nprovide an `AsyncProducer` with a failure handler that's called whenever\nsomething bad happens.\n\n```ruby\nrequire 'telekinesis'\n\nclass MyFailureHandler\n  def on_record_failure(kv_pairs_and_errors)\n    items = kv_pairs_and_errors.map do |k, v, code, message|\n      maybe_log_error(code, message)\n      [k, v]\n    end\n    save_for_later(items)\n  end\n\n  def on_kinesis_error(err, items)\n    log_exception(err.cause)\n    save_for_later(items)\n  end\nend\n\nproducer = Telekinesis::Producer::AsyncProducer.create(\n  stream: 'my stream',\n  failure_handler: MyFailureHandler.new,\n  send_every_ms: 1500,\n  credentials: {\n    access_key_id: 'foo',\n    secret_access_key: 'bar'\n  }\n)\n```\n\n## Consumers\n\n### KCL\n\n`Telekinesis::Consumer::KCL` is a wrapper around Amazon's [Kinesis Client\nLibrary (also called the KCL)](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html#kinesis-record-processor-overview-kcl).\n\nEach KCL instance is part of a group of consumers that make up an\n_application_. An application can be running on any number of hosts in any\nnumber of processes.  Consumers identify themself uniquely within an\napplication by specifying a `worker_id`.\n\nAll of the consumers within an application attempt to distribute work evenly\nbetween themselves by coordinating through a DynamoDB table. This coordination\nensures that a single consumer processes each shard, and that if one consumer\nfails for any reason, another consumer can pick up from the point at which it\nlast checkpointed.\n\nThis is all part of the official AWS library! Telekinesis just makes it easier\nto use from JRuby.\n\nEach client has to know how to process all the data it's\nretreiving from Kinesis. That's done by creating a [record\nprocessor](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-implementation-interface-java)\nand telling a `KCL` how to create a processor when it becomes\nresponsible for a shard.\n\nWe highly recommend reading the [official\ndocs](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html#kinesis-record-processor-implementation-interface-java)\non implementing the `IRecordProcessor` interface before you continue.\n\n\u003e NOTE: Since `initialize` is a reserved method, Telekinesis takes care of\n\u003e calling your `init` method whenever the KCL calls `IRecordProcessor`'s\n\u003e `initialize` method.\n\n\u003e NOTE: Make sure you read the Kinesis Record Processor documentation carefully.\n\u003e Failures, checkpoints, and shutting require some attention. More on that later.\n\nAfter it is created, a record processor is initialized with the ID of the shard\nit's processing, and handed an enumerable of\n[Records](http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/kinesis/AmazonKinesisClient.html) and a checkpointer (see below) every time the consumer detects new data to\nprocess.\n\nDefining and creating a simple processor might look like:\n\n```ruby\nrequire 'telekinesis'\n\nclass MyProcessor\n  def init(init_input)\n    @shard_id = init_input.shard_id\n    $stderr.puts \"Started processing #{@shard_id}\"\n  end\n\n  def process_records(process_records_input)\n    process_records_input.records.each do\n      |r| puts \"key=#{r.partition_key} value=#{String.from_java_bytes(r.data.array)}\"\n    end\n  end\n\n  def shutdown\n    $stderr.puts \"Shutting down #{@shard_id}\"\n  end\nend\n\nworker = Telekinesis::Consumer::KCL.new(stream: 'some-events', app: 'example') do\n  MyProcessor.new\nend\n\nworker.run\n```\n\nTo make defining record processors easier, Telekinesis comes with a `Block`\nprocessor that lets you use a block to specify your `process_records` method.\nUse this if you don't need to do any explicit startup or shutdown in a record\nprocessor.\n\n```ruby\nrequire 'telekinesis'\n\nworker = Telekinesis::Consumer::KCL.new(stream: 'some-events', app: 'example') do\n  Telekinesis::Consumer::Block.new do |records, checkpointer, millis_behind|\n    records.each {|r| puts \"key=#{r.partition_key} value=#{String.from_java_bytes(r.data.array)}\" }\n  end\nend\n\nworker.run\n```\n\nOnce you get into building a client application, you'll probably want\nto know about some of the following advanced tips and tricks.\n\n#### Client State\n\nEach KCL Application gets its own DynamoDB table that stores all of this state.\nThe `:application` name is used as the DynamoDB table name, so beware of\nnamespace collisions if you use DynamoDB on its own. Altering or reseting any\nof this state involves manually altering the application's Dynamo table.\n\n#### Errors while processing records\n\nWhen a call to `process_records` fails, the KCL expects you to handle the\nfailure and try to reprocess. If you let an exception escape, it happily moves\non to the next batch of records from Kinesis and will let you checkpoint further\non down the road.\n\nFrom the [official docs](http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-implementation-app-java.html):\n\n\u003e The KCL relies on processRecords to handle any exceptions that arise from\n\u003e processing the data records. If an exception is thrown from processRecords,\n\u003e the KCL skips over the data records that were passed prior to the exception;\n\u003e that is, these records are not re-sent to the record processor that threw the\n\u003e exception or to any other record processor in the application.\n\nThe moral of the story is that you should be absolutely sure you catch any\nexceptions that get thrown in your `process_records` implementation. If you\ndon't, you can (silently) drop data on the floor.\n\nIf something terrible happens and you can't attempt to re-read the list of\nrecords and re-do whatever work you needed to do in process records, we've been\nadvised by the Kinesis team that killing the entire JVM that's running the\nworker is the safest thing to do. On restart, the consumer (or another consumer\nin the application group) will pick up the orphaned shards and attempt to\nrestart from the last available checkpoint.\n\n#### Checkpoints and `INITIAL_POSITION_IN_STREAM`\n\nThe second object passed to `process_records` is a checkpointer. This can be\nused to checkpoint all records that have been passed to the processor so far\n(by just calling `checkpointer.checkpoint`) or up to a particular sequence\nnumber (by calling `checkpointer.checkpoint(record.sequence_number)`).\n\nWhile a `KCL` consumer can be initialized with an `:initial_position_in_stream`\noption, any existing checkpoint for a shard will take precedent over that\nvalue. Furthermore, any existing STATE in DynamoDB will take precedent, so if\nyou start a consumer with `initial_position_in_stream: 'LATEST'` and then\nrestart with `initial_position_in_stream: 'TRIM_HORIZON'` you still end up\nstarting from `LATEST`.\n\n## Java client logging\n\nThe AWS Java SDK can be extremely noisy and hard to control, since it logs\nthrough `java.util.logging`.\n\nTelekinesis comes with a shim that can silence all of that logging or redirect\nit to a Ruby Logger of your choice. This isn't fine-grained control - you're\ncapturing or disabling ALL logging from any Java dependency that uses\n`java.util.logging` - so use it with care.\n\nTo entirely disable logging:\n\n```ruby\nTelekinesis::Logging.disable_java_logging\n```\n\nTo capture all logging and send it through a Ruby logger:\n\n```ruby\nTelekinesis::Logging.capture_java_logging(Logger.new($stderr))\n```\n\n----\n\n# Building\n\n## Prerequisites\n\n* JRuby 1.7.9 or later.\n* Apache Maven\n\n## Build\n\nInstall JRuby 1.7.9 or later, for example with `rbenv` you would:\n\n```\n$ rbenv install jruby-1.7.9\n```\n\nInstall Bundler and required Gems.\n\n```\n$ gem install bundler\n$ bundle install\n```\n\nInstall Apache Maven.\n\nOn Ubuntu or related use:\n\n```\n$ sudo apt-get install maven\n```\n\nThe easiest method on OSX is via `brew`.\n\n```\n$ sudo brew install maven\n```\n\nEnsure `JAVA_HOME` is set on OSX.\n\nEnsure your `JAVA_HOME` environment variable is set. In Bash for example\nadd the following to `~/.bash_profile`.\n\n```\nexport JAVA_HOME=$(/usr/libexec/java_home)\n```\n\nThen run:\n\n```\n$ source ~/.bash_profile\n```\n\nBuild the Java shim and jar.\n\n```\n$ rake ext:build\n```\n\nThe `rake ext:build` task builds the Java shim and packages all of the required Java\nclasses into a single jar. Since bytecode is portable, the JAR is shipped with\nthe built gem.\n\nBuild the Gem.\n\nUse the `rake gem:build` task to build the complete gem, uberjar and all.\n\n```\n$ rake gem:build\n```\n\n# Testing\n\nTelekinesis comes with a small set of unit tests. Run those with plain ol'\n`rake test`.\n\n\u003e NOTE: The Java extension *must* be built and installed before you can run\n\u003e unit tests.\n\nIntegration tests coming soon.\n\n\n# License\n\nCopyright Kickstarter, PBC.\n\nReleased under an [MIT License](http://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkickstarter%2Ftelekinesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkickstarter%2Ftelekinesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkickstarter%2Ftelekinesis/lists"}