Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/linkedin/brooklin

An extensible distributed system for reliable nearline data streaming at scale
https://github.com/linkedin/brooklin

change-data-capture data-streaming distributed-systems java kafka kafka-mirror-maker linkedin scalability

Last synced: about 1 month ago
JSON representation

An extensible distributed system for reliable nearline data streaming at scale

Awesome Lists containing this project

README 

        
= Brooklin



image:https://img.shields.io/github/license/linkedin/brooklin.svg[link="https://github.com/linkedin/brooklin/blob/master/LICENSE"]

image:https://img.shields.io/badge/bintray-v5.1.0-blue.svg?style=popout[link="https://bintray.com/linkedin/maven/brooklin"]

image:https://img.shields.io/gitter/room/linkedin/kafka.svg?style=popout[link="https://gitter.im/linkedin/brooklin"]

image:https://img.shields.io/github/last-commit/linkedin/brooklin.svg?style=popout[link="https://github.com/linkedin/brooklin/commits/master"]

image:https://img.shields.io/github/issues/linkedin/brooklin/bug.svg?color=orange?style=popout[link="https://github.com/linkedin/brooklin/labels/bug"]



image::images/brooklin-overview.svg[Brooklin Overview]

{empty} +

Brooklin is a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale. Designed for multitenancy, Brooklin can simultaneously power hundreds of data pipelines across different systems and can easily be extended to support new sources and destinations.



== Distinguishing features



* *Extensible for any source and destination*

+

** Brooklin offers a flexible API that can be extended to support a wide variety of source and destination systems. It is not confined to single type of source or destination system.

** Source and destination systems can be freely mixed and matched. They do not have to be the same.



* *Scalable*

** Brooklin supports creating an arbitrary number of data streams that are processed concurrently and independently such that errors in one stream are isolated from the rest.

** Brooklin supports partitioned data streams throughout its core implementation and APIs.

** Brooklin can be deployed to a cluster of machines (scale out) to support as many data streams as desired.



* *Easy to operate and manage*

** Brooklin exposes a REST endpoint for managing data streams, that offers a rich set of operations on them in addition to CRUD (e.g. `pause` and `resume`).

** Brooklin also exposes a diagnostics REST endpoint that enables on-demand querying of a data stream's status.



* *Battle-tested at scale with Kafka*

** While it is not limited to any particular system, Brooklin provides capabilities for reading/writing massive amounts of data to/from Kafka with high reliability at scale. You can learn more about this in the <> section.



* *Supports Change Data Capture with bootstrap*

** Brooklin supports propagating https://en.wikipedia.org/wiki/Change_data_capture[Change Data Capture] events from data stores, e.g. RDBMS, KV stores ... etc.

** Brooklin also supports streaming a snapshot of the existing data before propagating change events.



== Use cases

=== Mirroring Kafka clusters



* *Multitenancy*

+

A single Brooklin cluster can be used to mirror data across several Kafka clusters.



* *Fault isolation across topic partitions*

+

One bad partition will not affect an entire Kafka topic. Mirroring will continue for all the other healthy partitions.



* *Whitelisting topics using regular expressions*

+

Select the topics to mirror using regular expression patterns against their names.



* *Pausing and resuming individual partitions*

+

Through its https://github.com/linkedin/brooklin/wiki/Brooklin-Architecture#rest-endpoints[Datastream Management Service (DMS)], Brooklin exposes REST APIs that allow finer control over replication pipelines, like being able to pause and resume individual partitions of a Kafka topic.



> Check out https://github.com/linkedin/brooklin/wiki/mirroring-kafka-clusters[Mirroring Kafka Clusters] to learn more about using Brooklin to mirror Kafka clusters.



=== Change Data Capture

* Brooklin supports propagating https://en.wikipedia.org/wiki/Change_data_capture[Change Data Capture] events from data stores, e.g. RDBMS, KV stores ... etc.

* Brooklin supports bootstrapping data from a datastore, i.e. streaming a snapshot of the existing data before any change events.

* MySQL support is currently under development.



=== Stream processing bridge

* Brooklin can be used as the underlying streaming infrastructure feeding data to Stream Processing systems, e.g. http://samza.apache.org/[Apache Samza], https://storm.apache.org/[Apache Storm], https://spark.apache.org/[Apache Spark], https://flink.apache.org/[Apache Flink].



== Trying out Brooklin

Feel free to check out our https://github.com/linkedin/brooklin/wiki/test-driving-brooklin[step-by-step tutorials] for running Brooklin locally in a few example scenarios.



== Documentation

https://github.com/linkedin/Brooklin/wiki[Brooklin Wiki Pages]



== Community

* Join our https://gitter.im/linkedin/brooklin[Brooklin chat room on Gitter]

* File a bug or request features using https://github.com/linkedin/Brooklin/issues[GitHub issues]



== Contributing

* https://github.com/linkedin/Brooklin/wiki/Developer-Guide[Developer Guide]

* https://github.com/linkedin/brooklin/blob/master/CODE_OF_CONDUCT.md[Code of Conduct]



== License

Copyright (c) LinkedIn Corporation. All rights reserved.

Licensed under the https://github.com/linkedin/brooklin/blob/master/LICENSE[BSD 2-Clause] License.