{"id":19345951,"url":"https://github.com/tolitius/highlander","last_synced_at":"2025-06-26T16:37:04.371Z","repository":{"id":7588195,"uuid":"8944199","full_name":"tolitius/highlander","owner":"tolitius","description":"soft landing for high speed data","archived":false,"fork":false,"pushed_at":"2017-03-21T20:22:24.000Z","size":4928,"stargazers_count":45,"open_issues_count":1,"forks_count":2,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-23T04:44:36.776Z","etag":null,"topics":["clojure","netty","performance","redis","zeromq"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tolitius.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-22T03:31:12.000Z","updated_at":"2025-02-14T17:56:47.000Z","dependencies_parsed_at":"2022-09-21T16:52:49.681Z","dependency_job_id":null,"html_url":"https://github.com/tolitius/highlander","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tolitius/highlander","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fhighlander","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fhighlander/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fhighlander/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fhighlander/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tolitius","download_url":"https://codeload.github.com/tolitius/highlander/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tolitius%2Fhighlander/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262103245,"owners_count":23259418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","netty","performance","redis","zeromq"],"created_at":"2024-11-10T04:08:31.983Z","updated_at":"2025-06-26T16:37:04.332Z","avatar_url":"https://github.com/tolitius.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Highlander\n\nHighlander is taking on the \"lightings\" of data at high speeds and stores this data with the speed that a data store feels comfortable with. \n\n## Main Ingridients\n\n```\n[Non Blocking I/O] =\u003e [Queue] =\u003e [Data Store]\n```\n\nwhere \"Non blocking I/O\", \"Queue\" and \"Data Store\" pieces are pluggable.\n\n![Java NIO =\u003e ZeroMQ =\u003e Redis](https://github.com/tolitius/highlander/blob/master/doc/highlander.baseline.gif?raw=true)\n\n#### Defaults\nBy default Highlander runs on \"[Netty](http://netty.io/)\" =\u003e \"[ZeroMQ](http://www.zeromq.org/)\" =\u003e \"[Redis](http://redis.io/)\"\n\nThere are some built in pieces though that can be used instead of default config:\n\n* \"Netty\" can be swapped with plain \"Java NIO\" (in case of a few connections and higher throughput demand)\n* \"ZeroMQ\" can be swapped with a [Single Write Principle Queue](http://mechanical-sympathy.blogspot.com/2011/09/single-writer-principle.html)\n* \"Redis\" can be swapped with \"any other\" data store\n\n###### _TODO: since a data store (store [thing]) and a queue (produce/consume) are just functions, give an example on how to plug \"your own\"_\n\n## Current Throughput\n \nBenchmark numbers are all relative to many things (hardware, network, clients, other things running, etc..), but to set a rough baseline.. here we go.\n\n### Throughput with a Single Client\n\nAn average throughput of a _single_ connection to Highlander with defaults: e.g. `Netty 4.0.24.Final`, `ZeroMQ 4.0.5` (used via JZMQ) and `Redis 2.8.17`, sending 100 byte messages to `127.0.0.1` is **435,000 messages per second**.\n\nThis does not mean `435K` messages per second are read from the network and persisted in a data store. The idea is to have queue(s) absorb the load, while landing these messages \"comfortably\" to the data store. Hence the concept of a `queue depth` (an absorb bucket) shown in examples below.\n\nWith `--server nio` (a simple Java NIO server), a _single_ connection speed is **`395K`** messages per second. `Netty` is a default, preferred choice due to its flexibility and ability to scale not only in throughput, but also in a number of connections.\n\n### Throughput with Multiple Clients\n\nBy default a built in [bench.streamer](https://github.com/tolitius/highlander/blob/master/src/bench/streamer.clj) will use 5 clients to send data to highlander. Since Netty likes concurrency, it takes it without a hiccup at an average speed of **1.2 million** 100 byte messages **a second**:\n\n```bash\nNov 24, 2014 12:01:23 PM clojure.tools.logging$eval306$fn__310 invoke\nINFO:\ntotal throughput (ALL partitions): 1254800.0 msg/s\n```\n\n_(MacBook Pro 2.3 GHz i7 16GB)_\n\n## Usage\n\n#### ZeroMQ \u0026 Java Bindings\n\nSince \"ZeroMQ\" is a default queuing mechanism, and Highlander is JVM (Clojure), ZeroMQ [libraries](http://www.zeromq.org/intro:get-the-software) and [Java bindings](http://www.zeromq.org/bindings:java) need to be installed, in order for Highlander to run.\n\nAfter ZeroMQ and JZMQ are installed, depending on a version of ZeroMQ (let's say it's `${zeromq.version}`), a zeromq jar can be installed to local maven repo. The `zmq.jar` should live in `/usr/local/share/java/`, after JZMQ is installed. Just copy it somewhere, rename it to have a `${zeromq.version}` (e.g. `zmq-${zeromq.version}.jar`), and: \n\n```bash\nmvn install:install-file -Dfile=./zmq-${zeromq.version}.jar -DgroupId=zmq -DartifactId=zmq -Dversion=${zeromq.version} -Dpackaging=jar\n```\n### Running Highlander\n\nRunning it from sources is really simple (that's how we roll in the Clojure Universe):\n\n```bash\n$ lein run\n```\n\nOnce it is up and running, you should see:\n```\nINFO: highlander is ready for the lightning {:host 0.0.0.0 :port 4242 }\n```\n\nand of course the usage:\n```bash\nNov 24, 2014 12:11:59 PM clojure.tools.logging$eval306$fn__310 invoke\nINFO:  Switches             Default             Desc\n --------             -------             ----\n -h, --host           0.0.0.0             start on this hostname\n -p, --port           4242                listen on this port\n -zq, --zqueue        inproc://zhulk.ipc  use this zmq queue\n -q, --queue          zmq                 queue type [e.g. zmq, swpq]\n -qc, --qcapacity     33554432            queue capacity. used for JVM queues\n -s, --server         netty               server type [e.g. netty, nio]\n -fl, --fixed-length  100                 fixed length messages\n -mi, --monterval     5                   queue monitor interval\n```\n\n### Swapping Qs\n\nAs you can see from the usage in order to use a Single Write Principle queue instead of ZeroMQ, you can do:\n\n```\n$ lein run -q swpq\n```\n\nwhich will reflect that on the startup:\n\n```\n...\nINFO: [swpq]: single write principle queue is ready to roll\n...\n```\n\n### Swapping NIO\n\nUsing Java NIO instead of Netty can be done via a command line param 's' (or '--server'):\n\n```\n$ lein run -s nio\n```\n\n### Peeking Inside the Q\n\nWhile a \"Data Store\" allows you to easily peek inside, queues are different. \nFor example ZeroMQ does not allow you to do that, hence Highlander has a Q monitor built in\nthat gives basic throughput visibility as the data streams in:\n\n```bash\n---------------------------------------------\nqueue  [inproc://zhulk.ipc:29]\n---------------------------------------------\n       message rate: 445253.2 msg/s\n      current depth: 8359612\n pass through total: 8619778\n```\n\n### Exporting LD_LIBRARY_PATH\n\nBefore running Highlander, `LD_LIBRARY_PATH` needs to be exported to let it know where to find ZeroMQ C++/Java libs:\n```bash\nexport LD_LIBRARY_PATH=/usr/local/lib\n```\n\n\n### Q Pusher\n\nThere is a bench playground that has a basic Java NIO streamer, as well as a Q Pusher, which allows to take network (Netty/NIO) and Data Store out of the picture to just focus on Q numbers. It can be run as:\n\n```bash\n$ lein run -m bench.qpusher\n```\n\n```bash\nINFO: Usage:\n\n Switches          Default         Desc\n --------          -------         ----\n -zq, --zqueue     inproc://zhulk  use this zmq queue\n -q, --queue       zmq             queue type (e.g. zmq, swpq)\n -n, --number      100000000       number of things\n -qc, --qcapacity  33554432        queue capacity. used for JVM queues\n -mi, --monterval  5               queue monitor interval\n```\n\nFor example here is a default ZeroMQ rate with a single queue pushing 2.8 million messages a second:\n\n```bash\n---------------------------------------------\nqueue  [inproc://zhulk:1]\n---------------------------------------------\n       message rate: 2850446.8 msg/s\n      current depth: 27747444\n pass through total: 27755805\n```\n\nWhile ZeroMQ is great and it is default, the Q Pusher can be told to work with any other queue implementaion. \nFor example here is how to tell a Q Pusher to work with a built in Single Write Principle Queue, \nwhich is a pure JVM queue implementation:\n\n```\n$ lein run -m bench.qpusher -q swpq\n```\n\n```\n       message rate: 167536.0 msg/s\n      current depth: 40987563\n pass through total: 41007837\n```\n\n### Bench Streamer\n\nHighlander also has a NIO load simulation [streamer](https://github.com/tolitius/highlander/blob/master/src/bench/streamer.clj) that can be run on the same or different host and stream messages to a Highlander instance.\n\nSince this is a simulation streamer, it needs to be as close to the true load as possible, hence it streams \"pure truth\" in a form of a 100 byte question:\n\n```java\npublic static final String ULTIMATE_TRUTH = \n  \"[Did you know that the Answer to the Ultimate Question of Life, the Universe, and Everything is 42?]\";\n```\n\nthat is where a \"stream of truth\" comes from.\n\n#### Usage\n\n```bash\n Switches                Default                                                                                               Desc\n --------                -------                                                                                               ----\n -h, --host              0.0.0.0                                                                                               start on this hostname\n -p, --port              4242                                                                                                  listen on this port\n -c, --clients           5                                                                                                     number of clients\n -n, --number-of-things  200000000                                                                                             number of things to stream\n -t, --thing             [Did you know that the Answer to the Ultimate Question of Life, the Universe, and Everything is 42?]  a thing/message to send\n```\ne.g.\n```bash\n$ lein run -m bench.streamer -h 10.32.2.1 -c 10\n```\n\nBy default `lein run -m bench.streamer` will stream with 5 (clients) * 200 million of \"100 byte\" truths to a Highlander instance. \n\nJust as with Highlander, before running bench streamer, `LD_LIBRARY_PATH` needs to be exported to let it know where to find ZeroMQ C++/Java libs:\n```bash\nexport LD_LIBRARY_PATH=/usr/local/lib\n```\n\n## License\n\nCopyright © 2014 tolitius\n\nDistributed under the Eclipse Public License, the same as Clojure.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolitius%2Fhighlander","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftolitius%2Fhighlander","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftolitius%2Fhighlander/lists"}