{"id":17982987,"url":"https://github.com/minghsu0107/saga-example","last_synced_at":"2025-03-25T19:31:51.307Z","repository":{"id":41580384,"uuid":"345357954","full_name":"minghsu0107/saga-example","owner":"minghsu0107","description":"Orchestration-based saga implementation dealing with distributed transactions in a microservice architecture.","archived":false,"fork":false,"pushed_at":"2022-10-17T08:53:27.000Z","size":692,"stargazers_count":39,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-20T17:38:44.635Z","etag":null,"topics":["go","microservices","observibility","orchestration","saga","transaction"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/minghsu0107.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-07T13:46:47.000Z","updated_at":"2025-03-16T14:30:11.000Z","dependencies_parsed_at":"2022-08-10T02:50:43.338Z","dependency_job_id":null,"html_url":"https://github.com/minghsu0107/saga-example","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Fsaga-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Fsaga-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Fsaga-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/minghsu0107%2Fsaga-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/minghsu0107","download_url":"https://codeload.github.com/minghsu0107/saga-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245530323,"owners_count":20630530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","microservices","observibility","orchestration","saga","transaction"],"created_at":"2024-10-29T18:15:47.133Z","updated_at":"2025-03-25T19:31:48.170Z","avatar_url":"https://github.com/minghsu0107.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Saga Example\nSaga pattern is a failure management pattern that helps establish consistency in distributed applications, and coordinates transactions between multiple microservices to maintain data consistency. This project shows an orchestration-based saga implementation.\n\nRelated repositories (10K+ LOC):\n- https://github.com/minghsu0107/saga-purchase\n- https://github.com/minghsu0107/saga-account\n- https://github.com/minghsu0107/saga-product\n- https://github.com/minghsu0107/saga-pb\n\nAn all-in-one docker-compose deployment is provided, which includes the following components:\n- Traefik - edge proxy that is responsible for external traffic routing and internal grpc load-balancing. \n- [Account service](https://github.com/minghsu0107/saga-account) - service that handles login, sigup, authentication, and token management.\n- [Purchase service](https://github.com/minghsu0107/saga-purchase) - service that creates purchase and streams results of each saga step.\n- [Transaction services](https://github.com/minghsu0107/saga-product)\n  - Product service - service that creates and checks products; updates product inventories.\n  - Order service - service that creates and queries orders.\n  - Payment service - service that creates and queries payments.\n  - Orchestrator service - **stateless** saga orchestrator.\n- Local databases\n  - Account database (MySQL 8.0)\n  - Product database (MySQL 8.0)\n  - Payment database (MySQL 8.0)\n  - Order database (MySQL 8.0)\n- Six-node redis cluster \n  - As an in-memory cache for account, product, order, and payment.\n  - As the bloom/cuckoo filter for preventing cache penetration (using [Redis Bloom](https://oss.redis.com/redisbloom/)).\n    - Possible issue: as we always check the bloom/cuckoo filter before querying database, we heavily reply on the data consistence between Redis and MySQL. If any update to filter fails or the filter is evicted by Redis, however, the state of data will be inconsistent among cache and storage, causing false-positive queries.\n    - Solution: decouple bloom filter querying into another service, and use message broker (such as Kafka) to handle all data modification events in order to ensure at-least-once delievery (TODO).\n  - As distributed locks for preventing cache avalanche\n  - As a pub/sub for local cache invalidation.\n  - As a streaming platform for obtaining real-time purchase result.\n- Observibility\n  - Prometheus - pulling metrics from all services.\n  - Jaeger - preserving and querying tracing spans accross service boundaries.\n- NATS Streaming - message broker for saga commands and events.\n\nThe following diagram shows a brief overview of the architecture.\n\n\u003cimg width=\"1251\" alt=\"image\" src=\"https://user-images.githubusercontent.com/50090692/154828478-750e3f87-ab4b-4eb4-b8df-590a6b81ef28.png\"\u003e\n\nThis diagram omits cache data flow, bloom filters, and local databases.\n## Usage\nTo run all services locally via docker-compose v1, execute:\n```bash\n./run.sh run\n```\nThis will bootsrap all services as well as their replicas in Docker containers.\n\nTo stop all services, execute:\n```bash\n./run.sh stop\n```\n### Account Service\nFirst, we need to signup a new user:\n```bash\ncurl -X POST localhost/api/account/auth/signup \\\n    --data '{\"password\":\"abcd5432\",\"firstname\":\"ming\",\"lastname\":\"hsu\",\"email\":\"ming@ming.com\",\"address\":\"taipei\",\"phone_number\":\"1234567\"}'\n```\nUser account login:\n```bash\ncurl -X POST localhost/api/account/auth/login \\\n    --data '{\"email\":\"ming@ming.com\",\"password\":\"abcd5432\"}'\n```\nThis will return a new token pair (refresh token + access token). We should provide the access token in the `Authorization` header for those APIs with authentication.\n\nWe could obtain a new token pair by refreshing with the refresh token:\n```bash\ncurl -X POST localhost/api/account/auth/refresh \\\n    --data '{\"refresh_token\":\"\u003crefresh_token\u003e\"}'\n```\nGet user personal information:\n```bash\ncurl localhost/api/account/info/person -H \"Authorization: bearer \u003caccess_token\u003e\"\n```\nUpdate user personal information:\n```bash\ncurl -X PUT localhost/api/account/info/person -H \"Authorization: bearer \u003caccess_token\u003e\" \\\n    --data '{\"firstname\":\"newfirst\",\"lastname\":\"newlast\",\"email\":\"ming3@ming.com\"}'\n```\nGet user shipping information:\n```bash\ncurl localhost/api/account/info/shipping -H \"Authorization: bearer \u003caccess_token\u003e\"\n```\nUpdate user shipping information:\n```bash\ncurl -X PUT localhost/api/account/info/shipping -H \"Authorization: bearer \u003caccess_token\u003e\" \\\n    --data '{\"address\":\"japan\",\"phone_number\":\"54321\"}'\n```\n### Product Service\nNext, let's create some new products:\n```bash\ncurl -X POST localhost/api/product \\\n     --data '{\"name\": \"product1\",\"description\":\"first product\",\"brand_name\":\"mingbrand\",\"price\":100,\"inventory\":1000}'\ncurl -X POST localhost/api/product \\\n     --data '{\"name\": \"product2\",\"description\":\"second product\",\"brand_name\":\"mingbrand\",\"price\":100,\"inventory\":10}'\n```\nThe API will return the ID of the created product.\n\nList all products with pagination:\n```bash\ncurl \"localhost/api/products?offset=0\u0026size=100\"\n```\nThis will return a list of product catalog, including its ID, name, price, and current inventory.\n\nGet product details:\n```bash\ncurl \"localhost/api/product/\u003cproduct_id\u003e\"\n```\nThis will return the name, description, brand, price and cached inventory of the queried product.\n### Purchase Service\nHere comes the core part. We are going to create a new purchase, which sends a new purchase event to the saga orchestrator and triggers distributed transactions. It will return the ID of the new purchase when success.\n```bash\ncurl -X POST localhost/api/purchase -H \"Authorization: bearer \u003caccess_token\u003e\" \\\n    --data '{\"purchase_items\":[{\"product_id\":\u003cproduct_id\u003e,\"amount\":1}],\"payment\":{\"currency_code\":\"NT\"}}'\n```\n\nAfter creating a purchase, we can subscribe to `/api/purchase/result` to receive **realtime transaction results**. The purchase service pushes results using [server-sent events (SSE)](https://developer.mozilla.org/zh-TW/docs/Web/API/Server-sent_events/Using_server-sent_events). The following code example shows how to subscribe to server-sent events using Javascript. We will use [this library](https://github.com/Yaffle/EventSource) to send SSE request with `Authorization` header.\n\n```javascript\nvar script = document.createElement('script');script.src = \"https://unpkg.com/event-source-polyfill@1.0.9/src/eventsource.js\";document.getElementsByTagName('head')[0].appendChild(script);\nvar es = new EventSourcePolyfill('http://localhost/api/purchase/result', {\n  headers: {\n    'Authorization': 'bearer \u003caccess_token\u003e'\n  },\n});\nvar listener = function (event) {\n  var data = JSON.stringify(event.data);\n  console.log(data);\n};\nes.addEventListener(\"data\", listener);\n```\n\nIf the subscription is successful, we would receive realtime results like the following:\n\n\u003cimg width=\"1470\" alt=\"image\" src=\"https://user-images.githubusercontent.com/50090692/151049964-b4752356-ca9f-4e90-ae85-247e538778b9.png\"\u003e\n\n### Order Service\n\nNext, we could check whether our order is successfully created:\n```bash \ncurl \"localhost/api/order/\u003cpayment_id\u003e\"  -H \"Authorization: bearer \u003caccess_token\u003e\"\n```\n### Payment Service\nFinally, we could check whether our payment is successfully created:\n```bash\ncurl \"localhost/api/payment/\u003cpayment_id\u003e\"  -H \"Authorization: bearer \u003caccess_token\u003e\"\n```\n## Observability\nAll services could be configured to expose Prometheus metrics and send tracing spans. By default, all services have their Prometheus metric endpoints exposed on port `8080`. As for distributed tracing, we could simply enable it by setting environment variable `JAEGER_URL` to the Jaeger collection endpoint.\n\nTo check whether all services are alive, visit Prometheus at `http://localhost:9090/targets`.\n\n\u003cimg width=\"1792\" alt=\"image\" src=\"https://user-images.githubusercontent.com/50090692/150729950-aea0687a-ee0f-41f1-8220-a9febcf20a72.png\"\u003e\n\nVisit the Jaeger web UI at `http://localhost:16686`. We can check all tracing spans of our API calling chains, starting from Traefik. For example, the following figure shows a request that queries `/api/order/\u003corder_id\u003e`. We can see that once order service receives the request, it authenticates the request first by calling `auth.AuthService.Auth`, a gRPC authentication API provided by account service. If the authentication is successful, order service will continue processing the request. To obtain a complete order, order service will ask product service for details of purchased products through another gRPC call `product.ProductService.GetProducts`.\n\n\u003cimg width=\"1792\" alt=\"image\" src=\"https://user-images.githubusercontent.com/50090692/153759779-b60c1086-35dd-4b08-890b-6b925b0f9374.png\"\u003e\n\nLet's see a more complex example. This figure shows how transaction services interact with each other after we create a new purchase. The authentication process is similar to the previous example. After purchase service authenticates the request successfully, it publishes a `CreatePurchaseCmd` event to the message broker. Orchestrator service will then receive the event and start saga transactions. The following diagram show all related traces in a single purchase, including traces of streaming results and Redis operations.\n\n\u003cimg width=\"1792\" alt=\"image\" src=\"https://user-images.githubusercontent.com/50090692/153759967-43e2d4c4-83cf-4cad-a94d-3446b3b0c442.png\"\u003e\n\nEach transaction service adds the current span context to the event before publishing it. When a subscriber receives a new event, it extracts the span context from the event payload. This extracted span then becomes the parent span of the current span. By doing this, we could generate a full pub/sub calling chain across all transactions. \n\nIn addition, Jaeger will create service topologies for our spans. The following figure shows the topology when a client creates a new purchase.\n\n![](https://i.imgur.com/gJHzNMN.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminghsu0107%2Fsaga-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fminghsu0107%2Fsaga-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fminghsu0107%2Fsaga-example/lists"}