{"id":13485928,"url":"https://github.com/zalando/nakadi","last_synced_at":"2025-10-04T05:31:35.764Z","repository":{"id":37549863,"uuid":"44234368","full_name":"zalando/nakadi","owner":"zalando","description":"A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues","archived":true,"fork":false,"pushed_at":"2024-04-11T12:06:35.000Z","size":15424,"stargazers_count":955,"open_issues_count":46,"forks_count":292,"subscribers_count":44,"default_branch":"master","last_synced_at":"2024-09-29T23:04:08.691Z","etag":null,"topics":["apis","data-infrastructure","event-bus","java","java-8","kafka","microservices","postgresql","restful"],"latest_commit_sha":null,"homepage":"https://nakadi.io","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zalando.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-14T08:30:17.000Z","updated_at":"2024-09-23T18:14:50.000Z","dependencies_parsed_at":"2023-02-19T09:31:32.316Z","dependency_job_id":"ebd95b77-bb79-4166-b69b-2e69623191c3","html_url":"https://github.com/zalando/nakadi","commit_stats":null,"previous_names":[],"tags_count":298,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando%2Fnakadi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando%2Fnakadi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando%2Fnakadi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zalando%2Fnakadi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zalando","download_url":"https://codeload.github.com/zalando/nakadi/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235222543,"owners_count":18955327,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apis","data-infrastructure","event-bus","java","java-8","kafka","microservices","postgresql","restful"],"created_at":"2024-07-31T18:00:34.390Z","updated_at":"2025-10-04T05:31:29.577Z","avatar_url":"https://github.com/zalando.png","language":"Java","funding_links":[],"categories":["Java","Data Ingestion","项目","Projects","进程间通信","Messaging"],"sub_categories":["消息传递","Messaging"],"readme":"## [Nakadi Event Broker](https://zalando.github.io/nakadi/)\n\n**Nakadi is no longer developed in the open as we've not seen usage outside of Zalando for many years. **\nIf you are a Zalando employee, please use Zalando's internal repository for contributing.\n\n## ARCHIVED\n\n[![Build Status](https://travis-ci.org/zalando/nakadi.svg?branch=master)](https://travis-ci.org/zalando/nakadi)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/785ccd4ab5e34867b760a8b07c3b62f1)](https://www.codacy.com/app/aruha/nakadi?utm_source=www.github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=zalando/nakadi\u0026amp;utm_campaign=Badge_Grade)\n\nNakadi is a distributed event bus broker that implements a RESTful API abstraction on top of Kafka-like queues, which can be used to send, receive, and analyze streaming data in real time, in a reliable and highly available manner.\n\nOne of the most prominent use cases of Nakadi is to decouple micro-services by building data streams between producers and consumers.\n\nMain users of nakadi are developers and analysts. Nakadi provides features like REST based integration, multi consumer, ordered delivery, interactive UI, fully managed, security, ensuring data quality, abstraction of big data technology, and push model based consumption.\n\nNakadi is in active developement and is currently in production inside Zalando as the backbone of our microservices sending millions of events daily with a throughput of more than hundreds gigabytes per second. In one line, Nakadi is a **high-scalability data-stream for enterprise engineering teams**.  \n\n![Nakadi Deployment Diagram](docs/img/NakadiDeploymentDiagram.png)\n\nMore detailed information can be found on our [website](http://zalando.github.io/nakadi/).\n\n### Project goal\n\nThe goal of Nakadi (**ნაკადი** means *stream* in Georgian) is to provide an event broker infrastructure to:\n\n- Abstract event delivery via a secured [RESTful API](https://zalando.github.io/nakadi/manual.html#nakadi-event-bus-api).\n\n    This allows microservices teams to maintain service boundaries, and not directly depend on any specific message broker technology.\n    Access can be managed individually for every queue and secured using *OAuth* and custom authorization plugins.\n\n- Enable convenient development of event-driven applications and asynchronous microservices.\n\n    Event types can be defined with [Event type schemas](https://zalando.github.io/nakadi/manual.html#using_event-types)\n    and managed via a registry. All events will be validated against the schema before publishing.\n    This guarantees data quality and consistency for consumers.    \n\n- Efficient low latency event delivery.\n\n    Once a publisher sends an event using a simple [HTTP POST](https://zalando.github.io/nakadi/manual.html#using_producing-events),\n    consumers can be pushed to via a [streaming](https://zalando.github.io/nakadi/manual.html#using_consuming-events-lola)\n    HTTP connection, allowing near real-time event processing.\n    The consumer connection has keepalive controls and support for managing stream offsets using\n    [subscriptions](https://zalando.github.io/nakadi/manual.html#using_consuming-events-hila).\n\n### Development status\n\n- Nakadi is high-load production ready.\n- Zalando uses Nakadi as its central Event Bus Service.\n- Nakadi reliably handles the traffic from thousands event types with the throughput of more than hundreds gigabytes per second.\n- The project is in active development.  \n\n### Presentations\n\n- [Watch the talk on Nakadi at Fosdem](https://archive.fosdem.org/2018/schedule/event/nakadi/)\n- [Background on the Zalando Microservices platform](https://www.youtube.com/watch?v=gEeHZwjwehs)\n\n#### Features\n\n* Stream:    \n    * REST abstraction over Kafka-like queues.\n    * CRUD for event types.\n    * Event batch publishing.\n    * Low-level interface (**deprecated**).\n        * manual client side partition management is needed\n        * no support of commits\n    * High-level interface (Subscription API).\n        * automatic redistribution of partitions between consuming clients\n        * commits should be issued to move server-side cursors\n* Schema:    \n    * Schema registry.\n    * Several event type categories (Undefined, Business, Data Change).\n    * Several partitioning strategies (Random, Hash, User defined).\n    * Event enrichment strategies.\n    * Schema evolution.\n    * Events validation using an event type schema.\n* Security:          \n    * OAuth2 authentication.\n    * Per-event type authorization.\n    * Blacklist of users and applications.    \n* Operations:    \n    * [STUPS](https://stups.io/) platform compatible.    \n    * [ZMON](https://zmon.io/) monitoring compatible.\n    * SLO monitoring.\n    * Timelines:\n        * this allows transparently switch production and consumption to different cluster (tier, region, AZ) without\n        moving actual data and any service degradation.\n        * opens the possibility for implementation of other streaming technologies and engines besides Kafka\n        (like Amazon Kinesis or Google Cloud Pub/Sub)\n\nRead more about latest development on the [releases page](https://github.com/zalando/nakadi/releases).\n\n#### Additional features that we plan to cover in the future are:\n\n* Support for different streaming technologies and engines. Nakadi currently uses [Apache Kafka](http://kafka.apache.org/)\n    as its broker, but other providers (such as Kinesis) will be possible.     \n* Filtering of events for subscribing consumers.\n* Store old published events forever using transparent fall back backup shortages like AWS S3.\n* Separate the internal schema register to standalone service.\n* Use additional schema formats and protocols like Avro, protobuf and [others](https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats).\n\n#### Related projects\n\nThe [zalando-nakadi](https://github.com/zalando-nakadi/) organisation contains many useful related projects\nlike\n\n* [Use Nakadi with a client in your language of choice](https://nakadi.io/manual.html#using_clients),\n* SDK,\n* [Web UI](https://github.com/zalando-incubator/nakadi-ui),\n* DevOps tools, and more.\n\n\n\n## How to contribute to Nakadi\n\nRead our [contribution guidelines](CONTRIBUTING.md) on how to submit issues and pull requests, then get Nakadi up and running locally using Docker:\n\n### Dependencies\n\nThe Nakadi server is a Java 8 [Spring Boot](https://projects.spring.io/spring-boot/) application.\nIt uses [Kafka 1.1.1](https://kafka.apache.org/11/documentation.html) as its broker and\n [PostgreSQL 9.5](https://www.postgresql.org/docs/9.5/static/release-9-5.html) as its supporting database.\n\nNakadi requires recent versions of docker and docker-compose. In\nparticular, docker-compose \u003e= v1.7.0 is required. See [Install Docker\nCompose](https://docs.docker.com/compose/install/) for information on\ninstalling the most recent docker-compose version.\n\nThe project is built with [Gradle](http://gradle.org).\nThe `./gradlew` [wrapper script](http://www.gradle.org/docs/current/userguide/gradle_wrapper.html) will bootstrap\nthe right Gradle version if it's not already installed.\n\n### Install\n\nTo get the source, clone the git repository.\n```sh\ngit clone https://github.com/zalando/nakadi.git\n```\n### Building\n\nThe gradle setup is fairly standard, the main tasks are:\n\n- `./gradlew build`: run a build and test\n- `./gradlew clean`: clean down the build\n\nSome other useful tasks are:\n\n- `./gradlew startNakadi`: build Nakadi and start docker-compose services: nakadi, postgresql, zookeeper and kafka\n- `./gradlew stopNakadi`: shutdown docker-compose services\n- `./gradlew startStorages`: start docker-compose services: postgres, zookeeper and kafka (useful for development purposes)\n- `./gradlew fullAcceptanceTest`: start Nakadi configured for acceptance tests and run acceptance tests\n\nFor working with an IDE, the `eclipse` IDE task is available and you'll be able to import the `build.gradle` into Intellij IDEA directly.\n\n### Running a Server\n\n**Note**: Nakadi Docker for ARM processors is available at [here](./docker-arm)\n\nFrom the project's home directory you can start Nakadi via Gradle:\n\n```sh\n./gradlew startNakadi\n```\n\nThis will build the project and run docker compose with 4 services:\n\n- Nakadi (8080)\n- PostgreSQL (5432)\n- Kafka (9092)\n- Zookeeper (2181)\n\nTo stop the running Nakadi server:\n\n```sh\n./gradlew stopNakadi\n```\n\n\n## Using Nakadi and its API\n\nPlease read the [manual](https://zalando.github.io/nakadi/manual.html) for the full API usage details.\n\n### Creating Event Types\n\nThe Nakadi API allows the publishing and consuming of _events_ over HTTP.\nTo do this the producer must register an _event type_ with the Nakadi schema\nregistry.\n\nThis example shows a minimalistic `undefined` category event type with a wildcard schema:\n\n```sh\ncurl -v -XPOST http://localhost:8080/event-types -H \"Content-type: application/json\" -d '{\n  \"name\": \"order.ORDER_RECEIVED\",\n  \"owning_application\": \"order-service\",\n  \"category\": \"undefined\",\n  \"schema\": {\n    \"type\": \"json_schema\",\n    \"schema\": \"{ \\\"additionalProperties\\\": true }\"\n  }\n}'\n```\n**Note:** This is not a recommended category and schema. It should be used only for testing.\n\nYou can read more about this in the [manual](https://zalando.github.io/nakadi/manual.html#using_event-types).\n\n### Consuming Events\n\nYou can open a stream for an event type via the `events` sub-resource:\n\n```sh\ncurl -v http://localhost:8080/event-types/order.ORDER_RECEIVED/events\n\n\nHTTP/1.1 200 OK\n\n{\"cursor\":{\"partition\":\"0\",\"offset\":\"82376-000087231\"},\"events\":[{\"order_number\": \"ORDER_001\"}]}\n{\"cursor\":{\"partition\":\"0\",\"offset\":\"82376-000087232\"}}\n{\"cursor\":{\"partition\":\"0\",\"offset\":\"82376-000087232\"},\"events\":[{\"order_number\": \"ORDER_002\"}]}\n{\"cursor\":{\"partition\":\"0\",\"offset\":\"82376-000087233\"},\"events\":[{\"order_number\": \"ORDER_003\"}]}\n```\nYou will see the events when you publish them from another console for example.\nThe records without `events` field are `Keep Alive` messages.\n\n**Note:** the [low-level API](https://zalando.github.io/nakadi/manual.html#using_consuming-events-lola) should be\nused only for debugging. It is not recommended for production systems.\nFor production systems, please use the [Subscriptions API](https://zalando.github.io/nakadi/manual.html#using_consuming-events-hila).\n\n### Publishing Events\n\nEvents for an event type can be published by posting to its \"events\" collection:\n\n```sh\ncurl -v -XPOST http://localhost:8080/event-types/order.ORDER_RECEIVED/events \\\n -H \"Content-type: application/json\" \\\n -d '[{\n    \"order_number\": \"24873243241\"\n  }, {\n    \"order_number\": \"24873243242\"\n  }]'\n\n\nHTTP/1.1 200 OK  \n```\n\nRead more in the [manual](https://zalando.github.io/nakadi/manual.html#using_producing-events).\n\n## Contributing\n\nNakadi accepts contributions from the open-source community.\n\nPlease read [`CONTRIBUTING.md`](CONTRIBUTING.md).\n\nPlease also note our [`CODE_OF_CONDUCT.md`](CODE_OF_CONDUCT.md).\n\n## Contact\n\nThis [email address](MAINTAINERS) serves as the main contact address for this project.\n\nBug reports and feature requests are more likely to be addressed\nif posted as [issues](https://github.com/zalando/nakadi/issues) here on GitHub.\n\n## License\n\nPlease read the full [`LICENSE`](LICENSE)\n\nThe MIT License (MIT) Copyright © 2015 Zalando SE, https://tech.zalando.com\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated\ndocumentation files (the “Software”), to deal in the Software without restriction, including without limitation\nthe rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and\nto permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions\nof the Software.\n\nTHE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED\nTO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\nTHE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,\nTORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalando%2Fnakadi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzalando%2Fnakadi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzalando%2Fnakadi/lists"}