Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/onyx-platform/onyx
Distributed, masterless, high performance, fault tolerant data processing
https://github.com/onyx-platform/onyx
batch clojure data distributed streaming
Last synced: about 1 month ago
JSON representation
Distributed, masterless, high performance, fault tolerant data processing
- Host: GitHub
- URL: https://github.com/onyx-platform/onyx
- Owner: onyx-platform
- License: epl-1.0
- Archived: true
- Created: 2013-12-02T01:21:14.000Z (almost 11 years ago)
- Default Branch: 0.14.x
- Last Pushed: 2019-08-31T10:58:07.000Z (about 5 years ago)
- Last Synced: 2024-09-21T22:33:21.909Z (about 2 months ago)
- Topics: batch, clojure, data, distributed, streaming
- Language: Clojure
- Homepage: http://www.onyxplatform.org
- Size: 16.2 MB
- Stars: 2,050
- Watchers: 122
- Forks: 204
- Open Issues: 84
-
Metadata Files:
- Readme: README.md
- Changelog: changes.md
- License: LICENSE
Awesome Lists containing this project
- awesome-streaming - Onyx - Distributed, masterless, high performance, fault tolerant data processing. (Table of Contents / Streaming Engine)
- awesome-kafka-in-clojure - onyx-platform/onyx
- awesome-streaming - Onyx - Distributed, masterless, high performance, fault tolerant data processing. (Table of Contents / Streaming Engine)
README
#  Onyx
[](https://gitter.im/onyx-platform/onyx?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
### What is it?
- a masterless, cloud scale, fault tolerant, high performance distributed computation system
- batch and stream hybrid processing model
- exposes an information model for the description and construction of distributed workflows
- Competes against Storm, Flink, Cascading, Cascalog, Spark, Map/Reduce, Sqoop, etc
- written in pure Clojure
### What would I use this for?
- Realtime event stream processing
- CQRS
- Continuous computation
- Extract, transform, load
- Data transformation à la map-reduce
- Data ingestion and storage medium transfer
- Data cleaning
### Installation
Available on Clojars:
```
[org.onyxplatform/onyx "0.14.6"]
```
### Changelog
Changelog can be found at [changes.md](changes.md).
### Quick Lookup Doc
A searchable set of documentation for the Onyx data model is [available](http://www.onyxplatform.org/docs/cheat-sheet/latest/).
### Project Template
A project template can be found at [onyx-template](https://github.com/onyx-platform/onyx-template).
### Plugins and Libraries
### Plugin Template
We provide a plugin template for use in building new plugins. This can be found at [onyx-plugin](https://github.com/onyx-platform/onyx-plugin).
#### Plugin Use
To use the supported plugins, please use version coordinates such as
`[org.onyxplatform/onyx-amazon-sqs "0.14.6.SNAPSHOT.0"]`, and read
the READMEs on the 0.14.x branches linked above.
### Build Status
Component | `release`| `unstable`
----------|--------|----------
[onyx core](https://github.com/onyx-platform/onyx)| [](https://circleci.com/gh/onyx-platform/onyx/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx/tree/master)
[onyx-local-rt](https://github.com/onyx-platform/onyx-local-rt)| [](https://circleci.com/gh/onyx-platform/onyx-local-rt/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-local-rt/tree/master)
[onyx-kafka](https://github.com/onyx-platform/onyx-kafka)| [](https://circleci.com/gh/onyx-platform/onyx-kafka/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-kafka/tree/master)
[onyx-datomic](https://github.com/onyx-platform/onyx-datomic)| [](https://circleci.com/gh/onyx-platform/onyx-datomic/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-datomic/tree/master)
[onyx-redis](https://github.com/onyx-platform/onyx-redis)| [](https://circleci.com/gh/onyx-platform/onyx-redis/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-redis/tree/master)
[onyx-sql](https://github.com/onyx-platform/onyx-sql)| [](https://circleci.com/gh/onyx-platform/onyx-sql/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-sql/tree/master)
[onyx-bookkeeper](https://github.com/onyx-platform/onyx-bookkeeper)| [](https://circleci.com/gh/onyx-platform/onyx-bookkeeper/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-bookkeeper/tree/master)
[onyx-amazon-sqs](https://github.com/onyx-platform/onyx-amazon-sqs)| [](https://circleci.com/gh/onyx-platform/onyx-amazon-sqs/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-amazon-sqs/tree/master)
[onyx-amazon-s3](https://github.com/onyx-platform/onyx-amazon-s3)| [](https://circleci.com/gh/onyx-platform/onyx-amazon-s3/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-amazon-s3/tree/master)
[onyx-http](https://github.com/onyx-platform/onyx-http)| [](https://circleci.com/gh/onyx-platform/onyx-http/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-http/tree/master)
[learn-onyx](https://github.com/onyx-platform/learn-onyx)| [](https://circleci.com/gh/onyx-platform/learn-onyx/tree/answers) | `-`
[onyx-examples](https://github.com/onyx-platform/onyx-examples)| [](https://circleci.com/gh/onyx-platform/onyx-examples/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-examples/tree/master)
[onyx-peer-http-query](https://github.com/onyx-platform/onyx-peer-http-query)| [](https://circleci.com/gh/onyx-platform/onyx-peer-http-query/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-peer-http-query/tree/master)
[lib-onyx](https://github.com/onyx-platform/lib-onyx)| [](https://circleci.com/gh/onyx-platform/lib-onyx/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/lib-onyx/tree/master)
[onyx-plugin](https://github.com/onyx-platform/onyx-plugin)| [](https://circleci.com/gh/onyx-platform/onyx-plugin/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-plugin/tree/master)
[onyx-template](https://github.com/onyx-platform/onyx-template)| [](https://circleci.com/gh/onyx-platform/onyx-template/tree/0.14.x) | [](https://circleci.com/gh/onyx-platform/onyx-template/tree/master)
- `release`: stable, released content
- `unstable`: unreleased content
#### Unsupported plugins
Some plugins are currently unsupported in onyx 0.14.x. These are:
- [`onyx-durable-queue`](https://github.com/onyx-platform/onyx-durable-queue)
- [`onyx-elasticsearch`](https://github.com/onyx-platform/onyx-elasticsearch)
- [`onyx-kafka-0.8`](https://github.com/onyx-platform/onyx-kafka-0.8)
### Companies Running Onyx in Production
[](https://lockedon.com)
### Quick Start Guide
Feeling impatient? Hit the ground running ASAP with the [onyx-starter repo](https://github.com/onyx-platform/onyx-starter) and [walkthrough](https://github.com/onyx-platform/onyx-starter/blob/master/WALKTHROUGH.md). You can also boot into preloaded a Leiningen [application template](https://github.com/onyx-platform/onyx-template).
### User Guide 0.14.6
- [User Guide Table of Contents](http://www.onyxplatform.org/docs)
- [API docs](http://www.onyxplatform.org/docs/api/latest)
- [Cheat Sheet](http://www.onyxplatform.org/docs/cheat-sheet/latest)
### Developer's Guide 0.14.6
- [Branch Policy](doc/developers-guide/branch-policy.md)
- [Release Checklist](doc/developers-guide/release-checklist.md)
- [Deployment Process](doc/developers-guide/deployment-process.md)
### API Docs 0.14.6
Code level API documentation [can be found here](http://www.onyxplatform.org/docs/api/0.14.6).
### Official plugin listing
Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. `onyx-kafka` version `0.14.6.0-SNAPSHOT` goes with `onyx` version `0.14.6`). Fixes to plugins can be applied using a 4th versioning identifier (e.g. `0.14.6.1-SNAPSHOT`).
- [`onyx-core-async`](doc/user-guide/core-async-plugin.adoc)
- [`onyx-kafka`](https://github.com/onyx-platform/onyx-kafka)
- [`onyx-kafka-0.8`](https://github.com/onyx-platform/onyx-kafka-0.8)
- [`onyx-datomic`](https://github.com/onyx-platform/onyx-datomic)
- [`onyx-redis`](https://github.com/onyx-platform/onyx-redis)
- [`onyx-sql`](https://github.com/onyx-platform/onyx-sql)
- [`onyx-bookkeeper`](https://github.com/onyx-platform/onyx-bookkeeper)
- [`onyx-seq`](src/onyx/plugin/seq.cljc)
- [`onyx-durable-queue`](https://github.com/onyx-platform/onyx-durable-queue)
- [`onyx-elasticsearch`](https://github.com/onyx-platform/onyx-elasticsearch)
- [`onyx-http`](https://github.com/onyx-platform/onyx-http)
- [`onyx-amazon-sqs`](https://github.com/onyx-platform/onyx-amazon-sqs)
- [`onyx-amazon-s3`](https://github.com/onyx-platform/onyx-amazon-s3)
Generate plugin templates through Leiningen with [`onyx-plugin`](https://github.com/onyx-platform/onyx-plugin).
### 3rd Party plugin listing
Unofficial plugins have not been vetted.
- [`onyx-rethink`](https://github.com/cddr/onyx-rethink)
### Need help?
Check out the [Onyx Google Group](https://groups.google.com/forum/#!forum/onyx-user).
### Want the logo?
Feel free to use it anywhere. You can find [a few different versions here](https://github.com/onyx-platform/onyx/tree/0.14.x/doc/images/logo).
### Running the tests
A simple `lein test` will run the full suite for Onyx core.
#### Contributor list
- [Michael Drogalis](https://github.com/MichaelDrogalis)
- [Lucas Bradstreet](https://github.com/lbradstreet)
- [Owen Jones](https://github.com/owengalenjones)
- [Bruce Durling](https://github.com/otfrom)
- [Malcolm Sparks](https://github.com/malcolmsparks)
- [Bryce Blanton](https://github.com/bblanton)
- [David Rupp](https://github.com/davidrupp)
- [sbennett33](https://github.com/sbennett33)
- [Tyler van Hensbergen](https://github.com/tvanhens)
- [David Leatherman](https://github.com/leathekd)
- [Daniel Compton](https://github.com/danielcompton)
- [Jeff Rose](https://github.com/rosejn)
- [Ole Krüger](https://github.com/dignati)
- [Juho Teperi](https://github.com/Deraen)
- [Nicolas Ha](https://github.com/nha)
- [Andrew Meredith](https://github.com/kendru)
- [Bridget Hillyer](https://github.com/bridgethillyer)
- [Ivan Mushketyk](https://github.com/mushketyk)
- [Jochen Rau](https://github.com/jocrau)
- [Tienson Qin](https://github.com/tiensonqin)
- [Roman Volosovskyi](https://github.com/rasom)
- [Vijay Kiran](https://github.com/vijaykiran)
- [Paul Kehrer](https://github.com/reaperhulk)
- [Scott Bennett](https://github.com/sbennett33)
- [Nathan Todd.stone](https://github.com/nathants)
- [Mariusz Jachimowicz](https://github.com/mariusz-jachimowicz-83)
- [Jason Bell](https://github.com/jasebell)
#### Acknowledgements
Some code has been incorporated from the following projects:
- [Riemann](https://github.com/aphyr/riemann)
- [zookeeper-clj](https://github.com/liebke/zookeeper-clj)
### License
Copyright © 2017 Michael Drogalis
Distributed under the Eclipse Public License, the same as Clojure.