https://github.com/Netflix/suro

Netflix's distributed Data Pipeline
https://github.com/Netflix/suro

Last synced: 9 days ago
JSON representation

Netflix's distributed Data Pipeline

• Host: GitHub
• URL: https://github.com/Netflix/suro
• Owner: Netflix
• License: apache-2.0
• Created: 2013-03-20T21:02:32.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2023-04-10T10:22:36.000Z (over 1 year ago)
• Last Synced: 2024-08-01T17:31:29.376Z (3 months ago)
• Language: Java
• Homepage:
• Size: 4.18 MB
• Stars: 792
• Watchers: 513
• Forks: 170
• Open Issues: 31
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-streaming - suro - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. (Table of Contents / Data Pipeline)
• awesome-streaming - suro - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. (Table of Contents / Data Pipeline)

README

        
# Suro: Netflix's Data Pipeline

Suro is a data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. It has the following features:

- It is distributed and can be horizontally scaled.

- It supports streaming data flow, large number of connections, and high throughput.

- It allows dynamically dispatching events to different locations with flexible dispatching rules.

- It has a simple and flexible architecture to allow users to add additional data destinations.

- It fits well into NetflixOSS ecosystem

- It is a best-effort data pipeline with support of flexible retries and store-and-forward to minimize message loss

Learn more about Suro on the Suro Wiki and the Netflix TechBlog post where Suro was introduced.

## Master Build Status



## Pull Request Build Status



Build

-----

NetflixGraph is built via Gradle (www.gradle.org). To build from the command line:

    ./gradlew build

See the `build.gradle` file for other gradle targets, like `distTar`, `distZip`, `installApp`, and `runServer`.

Running the server

------------------

You can run the server locally by just running `./gradlew runServer`.

More more advanced usage you may wish to run `./gradlew installApp` and then:

	cd suro-server

	java -cp "build/install/suro-server/lib/*" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json

To enable basic logging you can downloaded `slf4j-simple-1.7.7.jar` and copy it into `suro-server` then run:

	cd suro-server

	java -cp "build/install/suro-server/lib/*:slf4j-simple-1.7.7.jar" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json

Support

-----

We will use the Google Group, Suro Users, to discuss issues: https://groups.google.com/forum/#!forum/suro-users