{"id":41313706,"url":"https://github.com/socrata/eurybates","last_synced_at":"2026-01-23T05:27:54.340Z","repository":{"id":33909921,"uuid":"45071134","full_name":"socrata/eurybates","owner":"socrata","description":"Consumer \u0026 producer library for many-to-many communication over ActiveMQ or Kafka","archived":false,"fork":false,"pushed_at":"2025-04-09T23:34:24.000Z","size":156,"stargazers_count":2,"open_issues_count":5,"forks_count":3,"subscribers_count":35,"default_branch":"main","last_synced_at":"2025-04-10T00:29:13.447Z","etag":null,"topics":["core-platform","engineering"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/socrata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-27T21:27:42.000Z","updated_at":"2023-01-27T10:15:08.000Z","dependencies_parsed_at":"2024-05-07T21:40:47.933Z","dependency_job_id":"47682a98-858e-4859-ba9a-3331bbae76b4","html_url":"https://github.com/socrata/eurybates","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/socrata/eurybates","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socrata%2Feurybates","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socrata%2Feurybates/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socrata%2Feurybates/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socrata%2Feurybates/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/socrata","download_url":"https://codeload.github.com/socrata/eurybates/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/socrata%2Feurybates/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28680693,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-23T04:33:33.518Z","status":"ssl_error","status_checked_at":"2026-01-23T04:33:30.433Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["core-platform","engineering"],"created_at":"2026-01-23T05:27:54.273Z","updated_at":"2026-01-23T05:27:54.332Z","avatar_url":"https://github.com/socrata.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Eurybates\n\n\u003c!---\nAt the top of the file there should be a short introduction and/ or overview that explains **what** the project is. This description should match descriptions added for package managers (Gemspec, package.json, etc.)\n--\u003e\n\nEurybates is a library for producing to and consuming from queues. There is one queue per consumer (named `eurybates.\u003cservice-name\u003e`), and producers write to each consumer's queue*, after getting the list of consumer queues from Zookeeper. Each consumer receives all messages, but the consumer library discards all events which don't have the tag which you specify when you initialize the consumer (a simple string). This was judged to be more efficient than consumers somehow registering their tags with the queue server.\n\nWhen a new service which consumes from Eurybates is deployed, its name must be added to Zookeeper under the path `/eurybates/services`. Producers then pick up the new service and start sending to its queue. (See section \"Adding a queue to ActiveMQ\" below)\n\nEurybates supports Kafka and ActiveMQ, but as of writing we use it with ActiveMQ at Socrata. It's configured via a properties object and a client-application-defined Source ID.\n\n\n## Installation\n\n\u003c!---\nProvide code examples and explanations of how to get the project.\n--\u003e\nAdd Eurybates to your list of library dependencies\n\n```\nlibraryDependencies ++= Seq(\n    \"com.socrata\" %% \"eurybates\" % EURYBATES_VERSION\n)\n```\n\n## Build\n\n\u003c!---\nProvide an example of how to build this project.\n\n```\n# Command(s) for how to build your project.\n```\n--\u003e\n\nTraditional SBT Tasks\n\n```\nsbt compile test package\n```\n\n## Code Example\n\n\u003c!---\nShow what the library does as concisely as possible, developers should be able to figure out **how** your project solves their\nproblem by looking at the code example. Make sure the API you are showing off is obvious, and that your code is short and concise.\n--\u003e\n\nEurybates can be used to produce or consume messages.  To do either requires the following properties configuration.  All configuration\n  is rooted with the prefix of *eurybates*.\n\n```\neurybates.*\n```\n\nConfiguration differs based off different components.\n\n* Producer\n* Consumer (Planned)\n\n### Configuring Producer Type\n\n1. Picking a Producer type(s).  The available options are activemq, kafka, local_service, and noop.\n\n```\neurybates.producers = activemq | kafka | local_service | noop\n```\n\nor\n\nFor Multi-Plexing Producers\n```\neurybates.producers = activemq,kafka\n```\n\n### Configuring Kafka\n\nConfiguring Kafka Requires a broker list Properties.  The broker list is a comma separated of Kafka broker host:port.  Each\n  broker must be apart of the same Cluster.\n```\neurybates.kafka.broker_list = kafka-1:9092,kafka-2:9092\n```\n\n### Configuring ActiveMQ\n\n```\neurybates.activemq.connection_string = tcp://activemq-1:PORT\n```\n\n#### Adding a queue to ActiveMQ\n\n1. Find and ssh into a zookeeper node (for example, `knife search node 'role:*zookeeper* AND environment:*staging*'`)\n\n2.\n```bash\nsudo su\n/opt/zookeeper/zookeeper-3.4.6/bin/zkCli.sh\ncreate /eurybates/services/new-queue-name \"new-queue-name\"\n```\n\n### Service Definition\n\nYou could use any consumer library capable of consuming messages from JMS (for activemq) or Kafka.  We are looking\n to modernize Eurybates to use a better streaming models. Possible options for consuming Kafka Messages.\n\n* Kafka Clients: [Maven Library](http://search.maven.org/#artifactdetails%7Corg.apache.kafka%7Ckafka-clients%7C0.8.2.2%7Cjar)\n* Scala Kafka: https://github.com/stealthly/scala-kafka\n* Reactive Kafka: https://github.com/softwaremill/reactive-kafka\n* Eurybates itself\n\nEurybates enforces some interesting requirements and design decisions that can make it initially confusing.  Eurybates\nalso makes some assumptions about how you want to commit offsets as well.  Currently, Eurybates automatically\ncommits offsets every 10 seconds.  This means you have no direct control over when you tell Kafka you have recieved a\nmessage.  If this does not fit your use case do NOT use Eurybates at this time.  Please use one of the above options.\n\nIf you decide to use Eurybates, in order to consume messages you must define a class that extends *com.socrata.eurybates.Service*.\n\n```\nclass FooBarService extends Service {\n\n  override def messageReceived(message: Message): Unit = {\n    // Process the message however you like\n    System.out.println(s\"Yay I Received a message: $message\")\n  }\n\n}\n```\n\nThe service serves as a callback and a consumer must defined in your container class, object, or trait like so.\n\n```\nobject FooBarWrapper {\n\n  def consumer() = new KafkaServiceConsumer(\"kafka-broker-1:9062\",\n    \"some-source-id\",\n    Executors.newFixedThreadPool(1),\n    (sn: ServiceName, s: String, t: Throwable) =\u003e {},\n    Map((Name, new FooBarService())))\n\n}\n```\n\n\u003c!---\n## API Reference\n\nDepending on the size of the project, if it is small and simple enough the reference docs can be added to the README.\nFor medium size to larger projects it is important to at least provide a link to where the API reference docs live.\n--\u003e\n\n## Tests\n\nTODO Document how to incorporate Eurybates into your test.\n\n\u003c!---\nDescribe and show how to run the tests with code examples.\n\n```\n# Command(s) for how to test your project.\n```\n--\u003e\n\nNone at this time... Yikes.\n\n## Contributors\n\n\u003c!---\nLet people know how they can dive into the project, include important links to things like issue trackers, irc, twitter accounts if applicable.\n--\u003e\n\nSocrata Engineering Members\n* Robert Macomber\n* Andrew Gall\n* Michael Hotan\n\n## License\n\n\u003c!---\nA short snippet describing the license (MIT, Apache, etc.)\n--\u003e\n\nApache Version 2.0 License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocrata%2Feurybates","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsocrata%2Feurybates","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsocrata%2Feurybates/lists"}