https://github.com/Netflix/suro
Netflix's distributed Data Pipeline
https://github.com/Netflix/suro
Last synced: 9 months ago
JSON representation
Netflix's distributed Data Pipeline
- Host: GitHub
- URL: https://github.com/Netflix/suro
- Owner: Netflix
- License: apache-2.0
- Created: 2013-03-20T21:02:32.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2023-04-10T10:22:36.000Z (over 2 years ago)
- Last Synced: 2024-08-01T17:31:29.376Z (over 1 year ago)
- Language: Java
- Homepage:
- Size: 4.18 MB
- Stars: 792
- Watchers: 513
- Forks: 170
- Open Issues: 31
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-streaming - suro - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. (Table of Contents / Data Pipeline)
- awesome-bigdata - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- awesome-hadoop - Suro - Netflix's distributed Data Pipeline (Data Ingestion and Integration)
- awesome-streaming - suro - data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. (Table of Contents / Data Pipeline)
- useful-java-links - Suro: Netflix's Data Pipeline - 2.0) , [](https://github.com/Netflix/suro).  (II. Databases, search engines, big data and machine learning / 6. Working with messy data)
- awesome-bigdata - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- awesome-hadoop - Suro - Netflix's distributed Data Pipeline (Data Ingestion and Integration)
- fucking-awesome-bigdata - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- awesome-bigdata - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- A-curated-list-of-awesome-big-data-frameworks-ressources-and-other-awesomeness.- - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- data-engineering-collection - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
- awesome-java - Suro
- awesome-bigdata - Netflix Suro - log agregattor like Storm and Samza based on Chukwa. (Data Ingestion)
README
# Suro: Netflix's Data Pipeline
Suro is a data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. It has the following features:
- It is distributed and can be horizontally scaled.
- It supports streaming data flow, large number of connections, and high throughput.
- It allows dynamically dispatching events to different locations with flexible dispatching rules.
- It has a simple and flexible architecture to allow users to add additional data destinations.
- It fits well into NetflixOSS ecosystem
- It is a best-effort data pipeline with support of flexible retries and store-and-forward to minimize message loss
Learn more about Suro on the Suro Wiki and the Netflix TechBlog post where Suro was introduced.
## Master Build Status
## Pull Request Build Status
Build
-----
NetflixGraph is built via Gradle (www.gradle.org). To build from the command line:
./gradlew build
See the `build.gradle` file for other gradle targets, like `distTar`, `distZip`, `installApp`, and `runServer`.
Running the server
------------------
You can run the server locally by just running `./gradlew runServer`.
More more advanced usage you may wish to run `./gradlew installApp` and then:
cd suro-server
java -cp "build/install/suro-server/lib/*" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json
To enable basic logging you can downloaded `slf4j-simple-1.7.7.jar` and copy it into `suro-server` then run:
cd suro-server
java -cp "build/install/suro-server/lib/*:slf4j-simple-1.7.7.jar" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json
Support
-----
We will use the Google Group, Suro Users, to discuss issues: https://groups.google.com/forum/#!forum/suro-users