{"id":15405635,"url":"https://github.com/krisleech/axe","last_synced_at":"2025-10-23T19:05:35.352Z","repository":{"id":66255547,"uuid":"45972278","full_name":"krisleech/axe","owner":"krisleech","description":"A small stream processing framework for routing Kafka topics to parallelised Ruby objects.","archived":false,"fork":false,"pushed_at":"2015-12-19T02:21:07.000Z","size":63,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-17T04:44:45.830Z","etag":null,"topics":["event-sourcing","kafka"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/krisleech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-11T09:29:10.000Z","updated_at":"2017-07-17T10:06:11.000Z","dependencies_parsed_at":"2023-02-20T18:45:19.607Z","dependency_job_id":null,"html_url":"https://github.com/krisleech/axe","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krisleech%2Faxe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krisleech%2Faxe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krisleech%2Faxe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/krisleech%2Faxe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/krisleech","download_url":"https://codeload.github.com/krisleech/axe/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245359248,"owners_count":20602322,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-sourcing","kafka"],"created_at":"2024-10-01T16:17:37.487Z","updated_at":"2025-10-23T19:05:30.319Z","avatar_url":"https://github.com/krisleech.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Axe (Alpha)\n\nA small stream processing framework for routing Kafka topics to parallelized Ruby objects.\n\n* pluggable deseriazation of message payload (inc. JSON and AVRO)\n* plugable payload compression / encryption (inc. GZIP and Snappy)\n* pluggable consumer offset management (inc. memory and disk)\n* process per handler for memory isolation\n* preforking for reduced memory consumption\n\n## Usage\n\n### Handlers\n\nA handler is an object which responds to `#call(message)` or a lambda.\n\n```ruby\nclass MyHandler\n  def call(message)\n    puts message\n  end\nend\n```\n\nEach handler can be mapped to one Kafka topic. Messages are received in time\norder.\n\nIf an exception occurs the handler will be stopped and the error reported so it can be fixed before further\nmessages are consumed.\n\nEach Handler is run in its own preforked process.\n\n### Configure an Application\n\n```ruby\napp = Axe::App.new(logger: Logger.new(...),\n                   exception_handler: lambda { |exception, consumer| ... })\n```\n\n* `logger` - a [Logger](http://ruby-doc.org/stdlib-2.2.3/libdoc/logger/rdoc/Logger.html) instance\n* `exception_handler` - any callable object. This is typically used to generate a notification. There is no need to re-raise the error as the handler will be stopped so the exception can be fixed before proceeding.\n\nAlso see: [examples/app.rb](https://github.com/krisleech/axe/tree/master/examples).\n\n### Registering Handlers\n\n```ruby\napp.register(id:      \"my_handler\",\n             topic:   \"test\",\n             handler: MyHandler.new,\n             parser:  Axe::App::JsonParser.new,\n             logger:  Logger.new(...),\n             retries: 3,\n             delay:   5)\n```\n\n* `id` - an identifier, it is what the offsets are keyed on. Therefore\n  changing it will mean the handler will get events from offset 0. Likewise\n  reusing a previously used offset will mean the handler will start from the\n  offset stored for that id.\n* `topic` - the Kafka topic.\n* `handler` - an object which responds to `#call(message)`.\n* `parser` - an object which responds to `#call(message)`, it will parse the\n  message before passing it to the handler.\n    Included handlers are:\n    * `JSON`\n    * `Avro`\n    * `Gzip`\n    * `Snappy`\n    * `Default` (no parsing)\n* `logger` - the logger for this handler, if not specified the default\n  application logger is used. (optional)\n* `delay` - the number of seconds to pause between batches of messages (optional)\n* `retries` - the number of times the handler will be retried when an error\n  occurs. (optional)\n\nIf you need to use multiple parsers, e.g. for decompression, you can pass\nan array of parsers:\n\n```ruby\napp.register(...,\n             parser:  [Axe::App::GzipParser.new, Axe::App::JsonParser.new])\n```\n\nThe same technique can be used for decryption too.\n\n### Starting the Application\n\n```ruby\napp.start\n```\n\nThis will block until stopped. If all handlers are stopped, e.g. due to exceptions, then the application will stop.\n\nAxe does not handler demonization, pid files, restart etc. You can either use Ruby to do this or use something like initd.\n\nIf a connection to Kafka can not be established Axe will keep retrying until it becomes available.\n\nOften topics are auto-created when the first message is published. If a topic does not exist Axe will print a warning to the log file and continue to keep trying until the topic is created.\n\nThe application process name will be `axe [master]`. Each child process with be named by its id, e.g. `axe [my_handler]`.\n\n### Stopping the Application\n\nTo stop the app gracefully send a TERM signal, e.g. Ctrl-C if the process is not demonized, to the master process.\n\nThis will ensure that any messages currently being processed are allowed to finish.\n\nFailure to gracefully shutdown could result in a handlers getting their last\nmessage for a second time.\n\nThere is no need to send signals to the child processes.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\n### Specs\n\n```\nbundle exec rspec\n```\n\nRun on code change:\n\n```\nls **/*.rb | entr -c bundle exec rspec\n```\n\nYou can run the sample app and have it restart on code change with:\n\n```\nrerun ruby app.rb\n```\n\n### Writing parsers\n\nA parser is any object which responds to `#call(messages)` where `message` is a\nString. It must return the parsed message, this can be any Ruby object,\ntypically it will be a Hash.\n\nTake a look at the [build in parsers](https://github.com/krisleech/axe/tree/master/lib/axe/app/parsers) for examples.\n\n### Writing offsets stores\n\nAn offset store is an object which responds to `#[]` and `#[]=`. An in-memory\noffset store could be implemented using a Hash.\n\n```\napp = Axe::App.new(offset_store: Hash.new)\n```\n\nTake a look at the [build in stores](https://github.com/krisleech/axe/tree/master/lib/axe/app/offset_stores)\nfor further examples.\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at [https://github.com/krisleech/axe](https://github.com/krisleech/axe). This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct.\n\n## Releasing\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrisleech%2Faxe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkrisleech%2Faxe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkrisleech%2Faxe/lists"}