{"id":13683385,"url":"https://github.com/jmakov/dragonflybot","last_synced_at":"2025-03-20T23:19:47.606Z","repository":{"id":176643378,"uuid":"655069457","full_name":"jmakov/dragonflybot","owner":"jmakov","description":"A vertically scalable stream processing framework focusing on low latency, helping you scale and consume financial data feeds.","archived":false,"fork":false,"pushed_at":"2023-07-10T21:47:24.000Z","size":109,"stargazers_count":54,"open_issues_count":0,"forks_count":9,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-20T22:34:30.391Z","etag":null,"topics":["algo-trading","algo-trading-software","arbitrage","crypto","cryptocurrency","hft","hft-trading","high-frequency-trading","low-latency","rust","trading"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jmakov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-17T19:38:16.000Z","updated_at":"2024-12-13T13:07:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"cfb656a0-b5e9-4a05-a515-cf0b07a8ecca","html_url":"https://github.com/jmakov/dragonflybot","commit_stats":null,"previous_names":["jmakov/dragonflybot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmakov%2Fdragonflybot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmakov%2Fdragonflybot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmakov%2Fdragonflybot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jmakov%2Fdragonflybot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jmakov","download_url":"https://codeload.github.com/jmakov/dragonflybot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244706543,"owners_count":20496571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algo-trading","algo-trading-software","arbitrage","crypto","cryptocurrency","hft","hft-trading","high-frequency-trading","low-latency","rust","trading"],"created_at":"2024-08-02T13:02:09.279Z","updated_at":"2025-03-20T23:19:47.568Z","avatar_url":"https://github.com/jmakov.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# DragonflyBot\nA vertically scalable stream processing framework focusing on low latency, helping you scale\nand consume financial data feeds.\n\n## Design\nThe framework consists of 5 layers where each layer can be easily extended and can have its own\nbuilder.\n\n### Client/protocol layer\nHere different clients (supporting different protocols) can be defined e.g. we can have a \nbuilder for WebSockets clients and a different builder for FIX clients.\n\n### Feed subscriber layer\nNow that we have a connected client, each client/protocol in general require different subscription\napproaches e.g. a FIX connection might require different methods than a WebSocket connection. We\ncan simply define the requirements or extend them on this layer. \n\n### Feed listener layer\nFeed listeners are already subscribed to feeds (i.e. they require an active subscriber) and \ndeal only with processing/responding to the data e.g. forward the message only if top of the order\nbook has changed.\n\n### Feed listener aggregator layer\nIn practice, we connect to multiple feeds/venues. If we want to get a world view of all, we need to \nconsume what the feed listeners are emitting in one place/worker. Doing this we can e.g. consume\nall the order book messages emitted (and filtered) from feed listeners and construct top best bid,\nbest offer (BBO) of all order books (constructing BBO world view).\n\n### Service layer\nThe final layer - the place where you define your services which consume from feed listener \naggregators and serve your subscribers e.g. a bot could subscribe to your service to get a BBO\nworld view.\n\n## Allowed topologies\nSince the layers are connected only with message passing queues, we can quickly change topologies\nfrom simple ones\n```mermaid\ngraph TD;\n    trading_bot--\u003egRPC_server;\n    \n    gRPC_server--\u003efeed_listener_aggregator;\n    feed_listener_aggregator--\u003efeed_listener1;\n    feed_listener_aggregator--\u003efeed_listener2;\n    feed_listener_aggregator--\u003efeed_listener3;\n\n    feed_listener1--\u003efeed_subscriber1;\n    feed_listener2--\u003efeed_subscriber2;\n    feed_listener3--\u003efeed_subscriber3;\n\n    feed_subscriber1--\u003eclient1/protocol1;\n    feed_subscriber2--\u003eclient2/protocol2;\n    feed_subscriber3--\u003eclient3/protocol1;\n```\nto more complex ones\n```mermaid\ngraph TD;\n    trading_bot--\u003egRPC_server;\n    legacy_customer--\u003eFIX_server;\n    \n    gRPC_server--\u003e|stream BBO| feed_listener_aggregator1;\n    FIX_server--\u003e|stream BBO| feed_listener_aggregator1;\n    FIX_server--\u003e|stream aggregated liquidity| feed_listener_aggregator2;\n    FIX_server--\u003e|liquidate big order| execution_engine\n\n    feed_listener_aggregator1--\u003efeed_listener1;\n    feed_listener_aggregator1--\u003efeed_listener2;\n    feed_listener_aggregator1--\u003efeed_listener3;\n\n    feed_listener_aggregator2--\u003efeed_listener1;\n    feed_listener_aggregator2--\u003efeed_listener2;\n    feed_listener_aggregator2--\u003efeed_listener3;\n\n    feed_listener1--\u003efeed_subscriber1;\n    feed_listener2--\u003efeed_subscriber2;\n    feed_listener3--\u003efeed_subscriber3;\n\n    feed_subscriber1--\u003eclient1/protocol1;\n    feed_subscriber2--\u003eclient2/protocol2;\n    feed_subscriber3--\u003eclient3/protocol1;\n```\nsimply by changing the type of the queue (MPSC to e.g. SPMC) and introducing transformers if needed.\n\n## Performance considerations\n### WebSocket client\nBased on [WS benchmarks](https://github.com/nurmohammed840/web-socket-benchmark),\n[websocket.rs](https://github.com/nurmohammed840/websocket.rs) provides the fastest WS client. However, \nTLS doesn't seem to be supported. The second fastest - `fastwebsockets` supports TLS and reading\nsingle frames. So if we're only interested in top of the book, we could only read a part of the whole\nmessage i.e. read only the frames needed to get top N BBO. \n\n### JSON document parsing\nSince we're only interested in top level access (getting bids/asks), we can go with property based \nparsing libs which are [up to 10x faster than libs which parse the whole JSON](https://github.com/AnnikaCodes/rust-json-parsing-benchmarks).\nUsing property based parsing we also implement functionality of JSON stream parsers, parsing only the number of\nconsecutive values needed i.e. for getting top 10 of the book, we parse only 10 values.\n\n### Alternative allocators\n`tikv-jemallocator` is used for improving the performance of allocations.\n\n### Other opportunities\nFor small vectors `SmallVec` could be used.\n\n\n## Usage\nTo run an example where we aggregate order books and publish top 10 via a gRPC server:\n```shell\n# start the gRPC server in the background\ncargo run --bin dragonflybot-grpc-server -- --instrument-name ethbtc\u0026\n \n# run the client\ncargo run --bin dragonflybot-grpc-client\n\n# Docker\nDOCKER_BUILDKIT=1 docker build -t dragonflybot:latest .\n\n# run the server in the background\ndocker run \\\n  --name dragonflybot_grpc_server \\\n  --rm \\\n  --user=\"$(id -u):$(id -u)\" \\\n  --group-add=\"$(id -u)\" \\\n  -p 127.0.0.1:50051:50051 \\\n  dragonflybot:latest \\\n  dragonflybot-grpc-server --instrument-name ethbtc \u0026\n \n# run the gRPC client\ndocker run \\\n  --name dragonflybot_grpc_client \\\n  --rm \\\n  --user=\"$(id -u):$(id -u)\" \\\n  --group-add=\"$(id -u)\" \\\n  --net=\"host\" \\\n  dragonflybot:latest \\\n  dragonflybot-grpc-client\n  \n# to stop containers\ndocker stop dragonflybot_grpc_client \\\n  \u0026\u0026 docker stop dragonflybot_grpc_server\n```\n\n## Developing\nFor developing a multi stage, multi branch Dockerfile, supporting Rust build cache via `cargo-chef` is available.\n```shell\n# run tests\ncargo test\n\nDOCKER_BUILDKIT=1 docker build --build-arg \"BUILD_PROFILE=dev\" -t dragonflybot_dev:latest .\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmakov%2Fdragonflybot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjmakov%2Fdragonflybot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjmakov%2Fdragonflybot/lists"}