https://github.com/zicat/tributary

tributary
https://github.com/zicat/tributary

big-data flume java kafka

Last synced: about 2 months ago
JSON representation

tributary

• Host: GitHub
• URL: https://github.com/zicat/tributary
• Owner: zicat
• Created: 2023-03-30T13:10:54.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2024-10-17T05:57:30.000Z (3 months ago)
• Last Synced: 2024-10-19T06:30:38.515Z (3 months ago)
• Topics: big-data, flume, java, kafka
• Language: Java
• Homepage:
• Size: 2.48 MB
• Stars: 5
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Welcome to Tributary

Tributary is a reliable, stateless, fault-tolerance service for efficiently collecting and moving

huge amounts of

records. It has a simple and flexible architecture based on streaming records flows.

![image](doc/picture/tributary.png)

# Why Choose Tributary

Tributary is designed to solve the reliability and isolation with multiple sinks consuming one

channel.

For example, sink data to HDFS and Kafka from the same source for batching and

streaming computing requirement.

Persistently receiving and forwarding data to external systems while

achieving mutual fault isolation is a highly challenging task.

The current mainstream solution, such as [Apache Flume](https://flume.apache.org/), adopts a write

amplification strategy to address isolation issues. Specifically, the received data is stored in

multiple channels, and each sink consumes a separate channel.

Tributary adopts a read amplification to address isolation issues and supports multiple sinks

consuming the same channel based on GroupOffset.

The benefits of reading amplification not only ensure fault isolation, but also greatly reduce the

overhead, especially when the data volume and number of sinks increase.

## Documentation

[Tributary User Guide](doc/user_guide.md)

[Tributary Design Guide](doc/tributary_design_guide.md)