{"id":13593344,"url":"https://github.com/gvolpe/trading","last_synced_at":"2025-05-15T10:06:39.422Z","repository":{"id":37097558,"uuid":"407614833","full_name":"gvolpe/trading","owner":"gvolpe","description":"💱 Trading application written in Scala 3 that showcases an Event-Driven Architecture (EDA) and Functional Programming (FP)","archived":false,"fork":false,"pushed_at":"2025-05-04T00:25:26.000Z","size":24701,"stargazers_count":636,"open_issues_count":16,"forks_count":74,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-10T03:46:38.278Z","etag":null,"topics":["apache-kafka","apache-pulsar","cats-effect","eda","elm","elm-architecture","elm-lang","event-driven-architecture","fp","fs2","functional-programming","grafana","open-tracing","prometheus","scala","scala3","web-sockets","websockets"],"latest_commit_sha":null,"homepage":"https://leanpub.com/feda","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gvolpe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-09-17T16:47:16.000Z","updated_at":"2025-04-29T22:57:13.000Z","dependencies_parsed_at":"2023-02-19T06:01:14.786Z","dependency_job_id":"efa4a03a-e607-4065-960d-dc1be0f38564","html_url":"https://github.com/gvolpe/trading","commit_stats":{"total_commits":586,"total_committers":8,"mean_commits":73.25,"dds":"0.33959044368600677","last_synced_commit":"925cbdcb83f51ac93791ba024ff6ed9f670124a5"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gvolpe%2Ftrading","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gvolpe%2Ftrading/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gvolpe%2Ftrading/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gvolpe%2Ftrading/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gvolpe","download_url":"https://codeload.github.com/gvolpe/trading/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254319719,"owners_count":22051073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","apache-pulsar","cats-effect","eda","elm","elm-architecture","elm-lang","event-driven-architecture","fp","fs2","functional-programming","grafana","open-tracing","prometheus","scala","scala3","web-sockets","websockets"],"created_at":"2024-08-01T16:01:19.448Z","updated_at":"2025-05-15T10:06:34.401Z","avatar_url":"https://github.com/gvolpe.png","language":"Scala","funding_links":[],"categories":["scala","Scala"],"sub_categories":[],"readme":"trading\n=======\n\n[![CI Elm](https://github.com/gvolpe/trading/workflows/Elm/badge.svg)](https://github.com/gvolpe/trading/actions)\n[![CI Scala](https://github.com/gvolpe/trading/workflows/Scala/badge.svg)](https://github.com/gvolpe/trading/actions)\n[![CI Tyrian](https://github.com/gvolpe/trading/workflows/Tyrian/badge.svg)](https://github.com/gvolpe/trading/actions)\n[![CI Registry](https://github.com/gvolpe/trading/actions/workflows/ci-images.yml/badge.svg)](https://github.com/gvolpe/trading/actions/workflows/ci-images.yml)\n[![CI Smokey](https://github.com/gvolpe/trading/actions/workflows/ci-smokey.yml/badge.svg?branch=main)](https://github.com/gvolpe/trading/actions/workflows/ci-smokey.yml)\n\nReference application developed in the [Functional event-driven architecture: Powered by Scala 3](https://leanpub.com/feda) book.\n\n## Table of contents\n\n* [Web App](#web-app)\n   * [ScalaJS](#scalajs)\n* [Overview](#overview)\n* [Requirements](#requirements)\n* [Services](#services)\n   * [Lib](#lib)\n   * [Domain](#domain)\n   * [Core](#core)\n   * [Feed](#feed)\n   * [Forecasts](#forecasts)\n   * [Processor](#processor)\n   * [Snapshots](#snapshots)\n   * [Alerts](#alerts)\n   * [WS Server](#ws-server)\n   * [Tracing](#tracing)\n   * [Tests](#tests)\n   * [X Demo](#x-demo)\n   * [X QA](#x-qa)\n* [Monitoring](#monitoring)\n* [Topic compaction](#topic-compaction)\n\n## Web App\n\nThe web application allows users to subscribe/unsubscribe to/from symbol alerts such as `EURUSD`, which are emitted in real-time via Web Sockets.\n\n![client](./imgs/webapp.png)\n\nIt is written in [Elm](https://elm-lang.org/) and can be built as follows.\n\n```shell\n$ nix build .#elm-webapp\n$ open result/index.html # or specify browser\n```\n\nThere's also a development shell handy for local development.\n\n```shell\n$ nix develop .#elm\n$ cd web-app\n$ elm make src/Main.elm --output=Main.js\n$ open index.html # or specify browser\n```\n\nIf Nix is not your jam, you can install Elm by following the [official instructions](https://guide.elm-lang.org/install/elm.html) and then compile as usual.\n\n```shell\n$ cd web-app\n$ elm make src/Main.elm --output=Main.js\n$ xdg-open index.html # or specify browser\n```\n\n### ScalaJS\n\nThere is also a replica of the Elm application written in Scala using the [Tyrian](https://tyrian.indigoengine.io/) framework that can be built as follows.\n\n```console\n$ sbt 'webapp/fullLinkJS'\n```\n\nYou can then run it via Nix as shown below (it requires [flakes](https://nixos.wiki/wiki/Flakes)).\n\n```console\n$ nix run .#tyrian-webapp\nUsing cache dir: /home/gvolpe/workspace/trading/modules/ws-client/nix-parcel-cache\nServer running at http://localhost:1234\n✨ Built in 7ms\n```\n\nNOTICE: The `nix run` command will create a directory for the Parcel cache, which needs write permissions.\n\nWe use `fullLinkJS` to create a fully optimized JS file. However, we can use `fastLinkJS` for faster iterations.\n\nFor such cases, it may be more convenient to use `yarn` directly.\n\n```console\n$ nix develop .#tyrian\n$ cd modules/ws-client\n$ yarn install\n$ yarn build\n$ yarn start\nyarn run v1.22.17\nparcel index.html --no-cache --dist-dir dist --log-level info\nServer running at http://localhost:1234\n✨ Built in 1.82s\n```\n\nHowever, this is not fully reproducible and can't be guaranteed this will work in the future.\n\nWithout Nix, you need to install `yarn` and `parcel`, and use `yarn` as shown above.\n\n## Overview\n\nHere's an overview of all the components of the system.\n\n![overview](./imgs/overview.png)\n\n- Dotted lines: Pulsar messages such as commands and events.\n- Bold lines: read and writes from / to external components (Redis, Postgres, etc).\n\n## Requirements\n\nThe back-end application is structured as a mono-repo, and it requires both Apache Pulsar and Redis up and running. To make things easier, you can use the provided `docker-compose.yml` file.\n\n### Build JDK image\n\nThe `docker-compose` file depends on declared services to be published on the local docker server. All docker builds are handled within the `build.sbt` using `sbt-native-packager`. \n\nTo build all of the service images, we have a few options. \n\nThe first one via the given [Dockerfile](./modules/Dockerfile).\n\n```shell \n$ docker build -t jdk17-curl modules/\n```\n\nThe second one via Nix, from where we can build a slim image also based on `openjdk:17-slim-buster`.\n\n```console \n$ nix build .#slimDocker -o result-jre\n$ docker load -i result-jre\n```\n\nThe third one also via Nix, though building a layered image based on the same JDK we use for development.\n\n```console \n$ nix build .#docker -o result-jre\n$ docker load -i result-jre\n```\n\nThe main difference between these three options is the resulting image size.\n\n```console\n$ docker images | rg jdk17\njdk17-curl                    latest               0ed94a723ce3   10 months ago   422MB\njdk17-curl-nix                latest               c28f54e42c21   52 years ago    557MB\njdk17-curl-slim               latest               dbe24e7a7163   52 years ago    465MB\n```\n\nAny image is valid. Feel free to pick your preferred method.\n\nNOTE: As of September 2022, the Docker image resulting from `nix build .#docker` is no longer compatible with `sbt-native-packager`, so either go for `nix build` (defaults to the slim image) or build it directly via Docker with the given Dockerfile.\n\n### Build service images\n\nOnce the base `jdk17-curl` image has been built, we can proceed with building all our services' images.\n\n```shell \n$ sbt docker:publishLocal\n```\n\n### Run dependencies: Redis, Kafka, etc\n\n```shell\n$ docker-compose up -d pulsar redis\n```\n\n![pulsar](./imgs/pulsar.png)\n\nTo run the Kafka Demo (see more below in [X Demo](#x-demo)), `kafka.yml` should be used instead.\n\n```shell\n$ docker-compose -f kafka.yml up\n```\n\n### Running application\n\nIf we don't specify any arguments, then all the containers will be started, including all our services (except `feed`), Prometheus, Grafana, and Pulsar Manager.\n\n```console\n$ docker-compose up\nCreating network \"trading_app\" with the default driver\nCreating trading_pulsar_1 ... done\nCreating trading_redis_1  ... done\nCreating trading_ws-server_1      ... done\nCreating trading_pulsar-manager_1 ... done\nCreating trading_alerts_1         ... done\nCreating trading_processor_1      ... done\nCreating trading_snapshots_1      ... done\nCreating trading_forecasts_1      ... done\nCreating trading_tracing_1        ... done\nCreating trading_prometheus_1     ... done\nCreating trading_grafana_1        ... done\n```\n\nIt is recommended to run the `feed` service directly from `sbt` whenever necessary, which publishes random data to the topics where other services are consuming messages from.\n\n## Services\n\nThe back-end application consists of 9 modules, from which 5 are deployable applications, and 3 are just shared modules. There's also a demo module and a web application.\n\n```\nmodules\n├── alerts\n├── core\n├── domain\n├── feed\n├── forecasts\n├── it\n├── lib\n├── processor\n├── snapshots\n├── tracing\n├── ws-client\n├── ws-server\n├── x-demo\n└── x-qa\n```\n\n![backend](./imgs/dev.png)\n\n### Lib\n\nCapability traits such as `Logger`, `Time`, `GenUUID`, and potential library abstractions such as `Consumer` and `Producer`, which abstract over different implementations such as Kafka and Pulsar.\n\n### Domain\n\nCommands, events, state, and all business-related data modeling.\n\n### Core\n\nCore functionality that needs to be shared across different modules such as snapshots, `AppTopic`, and `TradeEngine`.\n\n### Feed\n\nGenerates random `TradeCommand`s and `ForecastCommand`s followed by publishing them to the corresponding topics. In the absence of real input data, this random feed puts the entire system to work.\n\n### Forecasts\n\nRegisters new authors and forecasts, while calculating the author's reputation.\n\n### Processor\n\nThe brain of the trading application. It consumes `TradeCommand`s, processes them to generate a `TradeState` and emitting `TradeEvent`s via the `trading-events` topic.\n\n### Snapshots\n\nIt consumes `TradeEvent`s and recreates the `TradeState` that is persisted as a snapshot, running as a single instance in fail-over mode.\n\n### Alerts\n\nThe alerts engine consumes `TradeEvent`s and emits `Alert` messages such as `Buy`, `StrongBuy` or `Sell` via the `trading-alerts` topic, according to the configured parameters.\n\n### WS Server\n\nIt consumes `Alert` messages and sends them over Web Sockets whenever there's an active subscription for the alert.\n\n### Tracing\n\nA decentralized application that hooks up on multiple topics and creates traces via the Open Tracing protocol, using the Natchez library and Honeycomb.\n\n![tracing](./imgs/tracer.png)\n\n### Tests\n\nAll unit tests can be executed via `sbt test`. There's also a small suite of integration tests that can be executed via `sbt it/test` (it requires Redis to be up).\n\n### X Demo\n\nIt contains all the standalone examples shown in the book. It also showcases both `KafkaDemo` and `MemDemo` programs that use the same `Consumer` and `Producer` abstractions defined in the `lib` module. \n\n#### Pulsar CDC\n\nTo run the Pulsar CDC Demo, you need Postgres and Pulsar (make sure no other instances are running). Before running them, we need to download the connector NAR file.\n\n```shell\n$ mkdir -p pulsarconf/connectors \u0026\u0026 cd pulsarconf/connectors\n$ wget https://archive.apache.org/dist/pulsar/pulsar-2.10.1/connectors/pulsar-io-debezium-postgres-2.10.1.nar\n$ docker-compose -f pulsar-cdc.yml up\n```\n\nOnce both instances are up and healthy, we can run the Pulsar Debezium connector.\n\n```console\n$ docker-compose exec -T pulsar bin/pulsar-admin source localrun --source-config-file /pulsar/conf/debezium-pg.yaml\n```\n\nYou should see this in the logs.\n\n```console\nSnapshot step 3 - Locking captured tables [public.authors]\n```\n\n### X QA\n\nIt contains the `smokey` project that models the smoke test for trading.\n\n## Monitoring\n\nJVM stats are provided for every service via Prometheus and Grafana.\n\n![grafana](./imgs/grafana.png)\n\n## Topic compaction\n\nTwo Pulsar topics can be compacted to speed-up reads on startup, corresponding to `Alert` and `TradeEvent.Switch`.\n\nTo compact a topic on demand (useful for manual testing), run these commands.\n\n```console\n$ docker-compose exec pulsar bin/pulsar-admin topics compact persistent://public/default/trading-alerts\nTopic compaction requested for persistent://public/default/trading-alerts.\n$ docker-compose exec pulsar bin/pulsar-admin topics compact persistent://public/default/trading-switch-events\nTopic compaction requested for persistent://public/default/trading-switch-events\n```\n\nIn production, one would configure topic compaction to be triggered automatically at the namespace level when certain threshold is reached. For example, to trigger compaction when the backlog reaches 10MB:\n\n```console\n$ docker-compose exec pulsar bin/pulsar-admin namespaces set-compaction-threshold --threshold 10M public/default\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgvolpe%2Ftrading","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgvolpe%2Ftrading","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgvolpe%2Ftrading/lists"}