{"id":18489694,"url":"https://github.com/abicky/ruby-kafka-ec2","last_synced_at":"2025-05-13T23:34:54.356Z","repository":{"id":53749775,"uuid":"273992047","full_name":"abicky/ruby-kafka-ec2","owner":"abicky","description":"An extension of ruby-kafka for EC2","archived":false,"fork":false,"pushed_at":"2022-03-29T17:46:01.000Z","size":49,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T06:37:58.297Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abicky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-21T22:17:45.000Z","updated_at":"2022-03-29T17:43:19.000Z","dependencies_parsed_at":"2022-09-19T09:41:22.184Z","dependency_job_id":null,"html_url":"https://github.com/abicky/ruby-kafka-ec2","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Fruby-kafka-ec2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Fruby-kafka-ec2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Fruby-kafka-ec2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abicky%2Fruby-kafka-ec2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abicky","download_url":"https://codeload.github.com/abicky/ruby-kafka-ec2/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254043217,"owners_count":22004912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T12:57:34.731Z","updated_at":"2025-05-13T23:34:54.291Z","avatar_url":"https://github.com/abicky.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ruby-kafka-ec2\n\n![](https://github.com/abicky/ruby-kafka-ec2/workflows/CI/badge.svg?branch=master)\n\nruby-kafka-ec2 is an extension of ruby-kafka that provides useful features for EC2 like Kafka::EC2::MixedInstanceAssignmentStrategy.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'ruby-kafka-ec2'\n```\n\nAnd then execute:\n\n    $ bundle install\n\nOr install it yourself as:\n\n    $ gem install ruby-kafka-ec2\n\n## Usage\n\n### Kafka::EC2::MixedInstanceAssignmentStrategy\n\n`Kafka::EC2::MixedInstanceAssignmentStrategy` is an assignor for auto-scaling groups with mixed instance policies. The throughputs of consumers usually depend on instance families and availability zones. For example, if your application writes data to a database, the throughputs of consumers running on the same availability zone as that of the writer DB instance is higher.\n\nTo assign more partitions to consumers with high throughputs, you have to initialize `Kafka::EC2::MixedInstanceAssignmentStrategy` first like below:\n\n```ruby\nrequire \"aws-sdk-rds\"\nrequire \"kafka\"\nrequire \"kafka/ec2\"\n\nrds = Aws::RDS::Client.new(region: \"ap-northeast-1\")\nassignment_strategy = Kafka::EC2::MixedInstanceAssignmentStrategy.new(\n  instance_family_weights: {\n    \"r4\" =\u003e 1.00,\n    \"r5\" =\u003e 1.20,\n    \"m5\" =\u003e 1.35,\n    \"c5\" =\u003e 1.50,\n  },\n  availability_zone_weights: -\u003e() {\n    db_cluster = rds.describe_db_clusters(filters: [\n      { name: \"db-cluster-id\", values: [ENV[\"RDS_CLUSTER\"]] },\n    ]).db_clusters.first\n    db_instance_id = db_cluster.db_cluster_members.find { |m| m.is_cluster_writer }.db_instance_identifier\n    db_instance = rds.describe_db_instances(filters: [\n      { name: \"db-cluster-id\", values: [ENV[\"RDS_CLUSTER\"]] },\n      { name: \"db-instance-id\", values: [db_instance_id] },\n    ]).db_instances.first\n\n    if db_instance.availability_zone == \"ap-northeast-1a\"\n      {\n        \"ap-northeast-1a\" =\u003e 1,\n        \"ap-northeast-1c\" =\u003e 0.25,\n      }\n    else\n      {\n        \"ap-northeast-1a\" =\u003e 0.25,\n        \"ap-northeast-1c\" =\u003e 1,\n      }\n    end\n  },\n)\n```\n\nIn the preceding example, consumers running on c5 instances will have 1.5x as many partitions compared to consumers running on r4 instances. In a similar way, if the writer DB instance is in ap-northeast-1a, consumers in ap-northeast-1a will have 4x as many partitions compared to consumers in ap-northeast-1c.\n\nYou can use `Kafka::EC2::MixedInstanceAssignmentStrategy` by specifying it to `Kafka#consumer`:\n\n\n```ruby\nconsumer = kafka.consumer(group_id: ENV[\"KAFKA_CONSUMER_GROUP_ID\"], assignment_strategy: assignment_strategy)\n```\n\nYou can also specify weights for each combination of availability zones and instance families:\n\n```ruby\nassignment_strategy = Kafka::EC2::MixedInstanceAssignmentStrategy.new(\n  weights: -\u003e() {\n    db_cluster = rds.describe_db_clusters(filters: [\n      { name: \"db-cluster-id\", values: [ENV[\"RDS_CLUSTER\"]] },\n    ]).db_clusters.first\n    db_instance_id = db_cluster.db_cluster_members.find { |m| m.is_cluster_writer }.db_instance_identifier\n    db_instance = rds.describe_db_instances(filters: [\n      { name: \"db-cluster-id\", values: [ENV[\"RDS_CLUSTER\"]] },\n      { name: \"db-instance-id\", values: [db_instance_id] },\n    ]).db_instances.first\n\n    weights_for_writer_az = {\n      \"r4\" =\u003e 1.00,\n      \"r5\" =\u003e 1.20,\n      \"m5\" =\u003e 1.35,\n      \"c5\" =\u003e 1.50,\n    }\n    weights_for_other_az = {\n      \"r4\" =\u003e 0.40,\n      \"r5\" =\u003e 0.70,\n      \"m5\" =\u003e 0.80,\n      \"c5\" =\u003e 1.00,\n    }\n    if db_instance.availability_zone == \"ap-northeast-1a\"\n      {\n        \"ap-northeast-1a\" =\u003e weights_for_writer_az,\n        \"ap-northeast-1c\" =\u003e weights_for_other_az,\n      }\n    else\n      {\n        \"ap-northeast-1a\" =\u003e weights_for_other_az,\n        \"ap-northeast-1c\" =\u003e weights_for_writer_az,,\n      }\n    end\n  },\n)\n```\n\nThe strategy also has the option `partition_weights`. This is useful when the topic has some skewed partitions. Suppose the partition with ID 0 of the topic \"foo\" receives twice as many records as other partitions. To reduce the number of partitions assigned to the consumer that consumes the partition with ID 0, specify `partition_weights` like below:\n\n```ruby\nassignment_strategy = Kafka::EC2::MixedInstanceAssignmentStrategy.new(\n  partition_weights: {\n    \"foo\" =\u003e {\n      0 =\u003e 2,\n    },\n  }\n)\n```\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/abicky/ruby-kafka-ec2.\n\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabicky%2Fruby-kafka-ec2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabicky%2Fruby-kafka-ec2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabicky%2Fruby-kafka-ec2/lists"}