{"id":15074363,"url":"https://github.com/rsachdeva/drive-deposits","last_synced_at":"2025-04-15T20:09:36.201Z","repository":{"id":255746310,"uuid":"852430518","full_name":"rsachdeva/drive-deposits","owner":"rsachdeva","description":"DriveDeposits: Microservices-based Financial System with Delta Growth. Backend: Rust, Async Rust, Tokio, gRPC (Tonic 0.12.x), REST (Axum 0.8.x with Path Segment Updated), Amazon Web Services AWS SDK for Rust, Lambda, DynamoDB, EventBridge, API Gateway,  AWS SAM, Docker, Kubernetes. Architecture: server-based and serverless microservice","archived":false,"fork":false,"pushed_at":"2025-02-28T19:16:15.000Z","size":2254,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-15T20:09:26.427Z","etag":null,"topics":["async-rust","aws-sam","aws-sdk-rust","axum","backend","cloudformation","docker","grpc","kubernetes","localstack","microservice","microservices","rest","rust","rustlang","serverless","tokio","tokio-rs","tonic","tower"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rsachdeva.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-04T19:41:55.000Z","updated_at":"2025-02-28T19:16:19.000Z","dependencies_parsed_at":"2024-10-13T06:00:42.432Z","dependency_job_id":"fffc34ee-ebef-4640-ad5a-30871d81f50e","html_url":"https://github.com/rsachdeva/drive-deposits","commit_stats":{"total_commits":19,"total_committers":1,"mean_commits":19.0,"dds":0.0,"last_synced_commit":"b8196edb4edb10341e60811d804b508daf8c27f2"},"previous_names":["rsachdeva/drive-deposits"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsachdeva%2Fdrive-deposits","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsachdeva%2Fdrive-deposits/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsachdeva%2Fdrive-deposits/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rsachdeva%2Fdrive-deposits/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rsachdeva","download_url":"https://codeload.github.com/rsachdeva/drive-deposits/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249145303,"owners_count":21219966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async-rust","aws-sam","aws-sdk-rust","axum","backend","cloudformation","docker","grpc","kubernetes","localstack","microservice","microservices","rest","rust","rustlang","serverless","tokio","tokio-rs","tonic","tower"],"created_at":"2024-09-25T03:32:33.784Z","updated_at":"2025-04-15T20:09:36.192Z","avatar_url":"https://github.com/rsachdeva.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"## DriveDeposits: A Robust Financial Microservices System with Delta Growth Analysis, Powered by a Scalable Rust Backend.\n\n[![CI Pipeline](https://github.com/rsachdeva/drive-deposits/actions/workflows/rust.yml/badge.svg)](https://github.com/rsachdeva/drive-deposits/actions/workflows/rust.yml)\n\n### Server-based \u0026 Serverless: Utilizes request-response blocking and sends events using async Rust for non-blocking, event-driven temporal decoupling, supporting both synchronous and asynchronous operations.\n\n\u003cimg src=\"DriveDeposits-Rust.png\" alt=\"DriveDeposits\" width=\"256\" height=\"256\" style=\"border-radius: 10%; vertical-align: middle;\"\u003e\n\n## Table of Contents\n\n- [Member crates in workspace](#member-crates-in-workspace)\n- [DriveDeposits System Design and Microservices Architecture: Watch Ingestion and Query Workflows](#drivedeposits-system-design-and-microservices-architecture-watch-ingestion-and-query-workflows)\n    - [System Design](#system-design)\n    - [Workflow Videos](#workflow-videos)\n        - [Ingestion Workflow Videos](#ingestion-workflow-videos)\n            - [Valid Portfolio Request](#valid-portfolio-request)\n            - [Portfolio request and response](#portfolio-request-and-response)\n        - [Query Workflow Videos](#query-workflow-videos)\n            - [Query API Retrieves a list of portfolios based on the delta growth criteria](#query-api-retrieves-a-list-of-portfolios-based-on-the-delta-growth-criteria)\n            - [Query API Retrieves a list of banks for a portfolio based on the delta growth criteria](#query-api-retrieves-a-list-of-banks-for-a-portfolio-based-on-the-delta-growth-criteria)\n            - [Query API Retrieves a list of deposits for a portfolio based on the delta growth criteria](#query-api-retrieves-a-list-of-deposits-for-a-portfolio-based-on-the-delta-growth-criteria)\n            - [Query API Retrieves a list of deposits for a portfolio based on the maturity date criteria](#query-api-retrieves-a-list-of-deposits-for-a-portfolio-based-on-the-maturity-date-criteria)\n- [Domain Driven Terminology](#domain-driven-terminology)\n- [DriveDeposits: Architectural Pillars](#drivedeposits-architectural-pillars)\n- [Synchronous Microservices Components](#synchronous-microservices-components)\n- [Asynchronous Microservices Components](#asynchronous-microservices-components)\n- [Bridging Synchronous and Asynchronous Components In DriveDeposits Microservices](#bridging-synchronous-and-asynchronous-components-in-drivedeposits-microservices)\n- [Deployment of Microservices](#deployment-of-microservices)\n    - [Serverless: AWS using SAM which uses CloudFormation under the hood](#serverless-aws-using-sam-which-uses-cloudformation-under-the-hood)\n    - [Server-based: Run the REST and gRPC servers Natively, With Docker Compose And Kubernetes!](#server-based-run-the-rest-and-grpc-servers-natively-with-docker-compose-and-kubernetes)\n    - [Test microservices integration](#test-microservices-integration)\n    - [Test AWS Lambda microservice directly](#test-aws-lambda-microservice-directly)\n    - [Data population](#data-population)\n    - [Querying with custom domain](#querying-with-custom-domain)\n- [Running Tests](#running-tests)\n    - [Integration tests](#integration-tests)\n    - [Unit and Integration tests](#unit-and-integration-tests)\n    - [End-to-End tests](#end-to-end-tests)\n    - [Hybrid Integration Testing Tool](#hybrid-integration-testing-tool)\n        - [Efficient Command for Synchronous Calculation Flow and Asynchronous AWS Serverless Event Testing](#efficient-command-for-synchronous-calculation-flow-and-asynchronous-aws-serverless-event-testing)\n        - [Alias](#alias)\n- [Development Tool: cargo lambda](#development-tool-cargo-lambda)\n- [Development Tool: LocalStack](#development-tool-localstack)\n- [Clean And Build](#clean-and-build)\n- [Configurations for DriveDeposits](#configurations-for-drivedeposits)\n\n### Member crates in workspace\n\nSee cargo workspace members:\n[Cargo.toml](Cargo.toml)\n\n#### Type-specific Crates and Their Purpose\n\nThe workspace includes several crates with the \"-types\" suffix that handle domain models, type definitions, conversions,\nand business logic, with each crate maintaining focused dependencies for its specific purpose:\n\n* **drive-deposits-cal-types**: Core calculation domain models and business logic. The protocol-specific gRPC and REST\n  implementations reside in drive-deposits-grpc-server and drive-deposits-rest-gateway-server respectively. While this\n  crate sends events to EventBridge, the EventBridge implementation lives in drive-deposits-event-source, enabling\n  drive-deposits-cal-types to maintain focused dependencies.\n* **drive-deposits-lambda-db-types**: DynamoDB item types and conversions. Converts to aws_sdk_dynamodb::types::\n  AttributeValue for the writer, while actual persistence happens in drive-deposits-logs-lambda-target. The conversion\n  to query response types occurs here, which drive-deposits-lambda-dynamodb-reader lambda uses for sending JSON query\n  responses.\n* **drive-deposits-rest-types**: REST API request/response types and validations\n* **drive-deposits-proto-grpc-types**: gRPC protocol buffer generated types\n\nThis separation provides **Clean Architecture and Maintainability**: Each type crate has a clear, single responsibility.\n\n#### Naming: why keeping prefix drive-deposits\n\nUsing the \"drive-deposits-\" prefix for crate names clearly distinguishes these as separate crates within\nthe workspace, not just modules within a single crate. This distinction is crucial for understanding the project\nstructure and for managing dependencies. It also allows for more flexibility in terms of versioning and publishing each\ncrate independently if needed. This naming convention effectively communicates the relationship between the crates while\nmaintaining their individual identities within the Rust ecosystem.\n\n#### Summary of the Responsibilities for crates drive-deposits-logs-lambda-target and drive-deposits-lambda-dynamodb-reader in Workspace\n\n###### drive-deposits-logs-lambda-target:\n\n* Triggered by EventBridge\n* Handles log groups based on event rules\n* Writes data to DynamoDB\n* Error handling when adding items in DynamoDB indicates the source of the error and the level context in which it\n  occurred.\n* Error logs can be seen in CloudWatch Logs\n\n###### drive-deposits-lambda-dynamodb-reader:\n\n* Responsible for querying data from DynamoDB\n* Uses Axum in Lambda\n* Exposed through an API Gateway\n* Error handling when reading items from DynamoDB and creating Response for Query requests indicates the source of the\n  error and the level context in which it occurred.\n* Error logs can be seen in CloudWatch Logs\n\n[Back to Table of Contents](#table-of-contents)\n\n### DriveDeposits System Design and Microservices Architecture: Watch Ingestion and Query Workflows\n\n#### System Design\n\n![DriveDeposits Design](DriveDeposits.drawio.svg)\n\n#### Workflow Videos\n\n#### Ingestion Workflow Videos\n\n##### Valid Portfolio Request\n\n##### [![Valid portfolio request](media/valid-portfolio-request.png)](https://vimeo.com/1042543651)\n\n##### Portfolio request and response\n\n##### [![Portfolio request and response](media/portfolio-request-and-response.png)](https://vimeo.com/1042543661)\n\n#### Query Workflow Videos\n\n#### [Query API Retrieves a list of portfolios based on the delta growth criteria](https://vimeo.com/1042543613)\n\n#### [Query API Retrieves a list of banks for a portfolio based on the delta growth criteria](https://vimeo.com/1042543640)\n\n#### [Query API Retrieves a list of deposits for a portfolio based on the delta growth criteria](https://vimeo.com/1042543632)\n\n#### [Query API Retrieves a list of deposits for a portfolio based on the maturity date criteria](https://vimeo.com/1042543620)\n\n[Back to Table of Contents](#table-of-contents)\n\n### Domain Driven Terminology:\n\nThe following terms are consistently used throughout the DriveDeposits project, forming the core vocabulary of our\ndomain-specific language in both design and development:\n\n* **Portfolio:** A collection of investments, such as stocks, bonds, and mutual funds, owned by an individual or an\n  organization.\n* **Bank:** A financial institution that accepts deposits for investment. In this context, \"bank\" refers to any\n  financial institution.\n* **Deposit:** A sum of money placed in a bank account or other investment vehicle.\n* **Levels (Portfolio, Bank, Deposit):**  DriveDeposits allows query of data at different levels - portfolios, banks,\n  or deposits - to make informed financial decisions.\n* **Delta:**  Represents the growth over a user-specified period, as specified\n  within the Portfolio\n  Request. It\n  measures investment\n  performance at\n  various levels (portfolios, banks, deposits). See\n  this [PortfolioRequest](drive-deposits-rest-gateway-server/data/portfolio_request_valid.json) example:\n\n  ```json\n  {\n      \"new_delta\": {\n        \"period\": \"1\",\n        \"period_unit\": \"Month\"\n      }\n  }\n  ```\n\n    * **Delta Growth:** The increase in value calculated in Portfolio Response. See JSON Path as\n      outcome.delta.growth.\n      This fluctuation is calculated at the portfolio, bank, and deposit levels.\n      See [PortfolioResponse](drive-deposits-rest-gateway-server/data/portfolio_response_for_valid.json) at the Deposit\n      Level for\n      example:\n      ```json\n      {\n        \"uuid\": \"eb8ea161-c461-4b1a-8f7c-7b845ba5bcbc\",\n        \"account\": \"1235N\",\n        \"account_type\": \"BrokerageCertificateOfDeposit\",\n        \"apy\": \"2.4\",\n        \"years\": \"7\",\n        \"outcome\": {\n          \"delta\": {\n            \"period\": \"1\",\n            \"period_unit\": \"Month\",\n            \"growth\": \"21.68\"\n          },\n          \"maturity\": {\n            \"amount\": \"10990\",\n            \"interest\": \"1846.32\",\n            \"total\": \"12836.32\"\n          },\n          \"errors\": []\n        },\n        \"outcome_with_dates\": {\n          \"start_date_in_bank_tz\": \"2024-02-16\",\n          \"maturity_date_in_bank_tz\": \"2031-02-14\",\n          \"errors\": []\n        }\n      } \n      ```\n      And at the Bank Level:\n      ```json\n      {\n        \"outcome\": {\n          \"delta\": {\n            \"period\": \"1\",\n            \"period_unit\": \"Month\",\n            \"growth\": \"246.16\"\n          },\n          \"maturity\": {\n            \"amount\": \"71100\",\n            \"interest\": \"7765.94\",\n            \"total\": \"78865.94\"\n          },\n          \"errors\": []\n        }\n      }\n      ```\n      And at the Portfolio Level:\n      ```json\n      {\n        \"outcome\": {\n          \"delta\": {\n          \"period\": \"1\",\n          \"period_unit\": \"Month\",\n          \"growth\": \"367.76\"\n          },\n          \"maturity\": {\n          \"amount\": \"108580.50\",\n          \"interest\": \"24462.92\",\n          \"total\": \"133043.42\"\n          },\n          \"errors\": []\n          }\n      }\n      ```\n\n\n* **Sorting Capabilities with Top K Based on Delta Growth:** DriveDeposits allows sorting based on delta growth by\n  portfolio, bank and deposits level and maturity date at deposits level retrieving the top 'k' items (where 'k' is the\n  number defined by the user in the query) in ascending or descending\n  order. For example:\n\n```curl\ncurl '{{aws_api_gateway_host}}/by-level-for-portfolios/delta-growth?order=asc\u0026top_k=10' \\\n    | jq\n```\n\n```curl\ncurl '{{aws_api_gateway_host}}/portfolios/{{aws_portfolio_uuid}}/by-level-for-banks/delta-growth/?order=asc\u0026top_k=10' \\\n        | jq\n```\n\n```curl\ncurl '{{aws_api_gateway_host}}/portfolios/{{aws_portfolio_uuid}}/by-level-for-deposits/maturity-date?order=asc\u0026top_k=2' \\\n        | jq\n```\n\nThe sort key in DynamoDB combines delta growth and created_at timestamp as a composite value, ensuring consistent\nordering through DynamoDB's native sorting capabilities. When delta growth values match exactly between portfolios, the\ncreated_at portion of the sort key determines their relative position in ascending or descending order.\n\n* **Maturity date:** The date when a deposit or investment reaches its full value or the end of its term.\n* **Sorting capabilities with top_k based on maturity date:** DriveDeposits allows sorting by maturity date,\n  retrieving the top 'k' deposits (where 'k' is the number defined by the user in the query) in ascending or descending\n  order. For\n  example:\n\n```curl\ncurl '{{aws_api_gateway_host}}/portfolios/{{aws_portfolio_uuid}}/by-level-for-deposits/maturity-date?order=asc\u0026top_k=2' \\\n        | jq\n```\n\n[Back to Table of Contents](#table-of-contents)\n\n### DriveDeposits: Architectural Pillars\n\n* **Microservices Architecture:** DriveDeposits is built on a robust microservices foundation, combining both\n  server-based and serverless components for maximum flexibility and scalability.\n* **Server-based Microservices:** Utilizing gRPC (Tonic) and REST (Axum) for high-performance, inter-service\n  communication and external API access.\n* **Serverless Microservices:** Leveraging AWS Lambda for auto-scaling, event-driven processing of financial\n  calculations and data management. This includes a sophisticated query system using AWS API Gateway + Lambda API\n  allowing efficient querying of portfolio, bank, and deposit data at various levels.\n* **Event-Driven Architecture:** Built on AWS EventBridge and EventBus, ensuring seamless communication and real-time\n  data processing for dynamic financial calculations.\n* **Rust-Powered Performance:** Benefiting from Rust's renowned speed, safety, and concurrency features for a highly\n  efficient and reliable system.\n* **AWS Rust SDK Integration:** Seamless interaction with AWS services, including Lambda runtime for serverless\n  execution and DynamoDB for scalable data storage and retrieval.\n* **Tokio-Powered Concurrency:** Efficient management of asynchronous tasks, including event processing with AWS\n  EventBridge.\n* **SAM Deployment:** Serverless components deployed using AWS Serverless Application Model (SAM), streamlining the\n  deployment of Lambda functions, DynamoDB, and EventBridge rules.\n\n**DriveDeposits offers a powerful and flexible microservices solution for:**\n\n* **Financial institutions** modernizing their calculation infrastructure.\n* **FinTech companies** seeking a scalable, reliable platform for real-time financial data processing.\n* **Developers** building high-performance, innovative financial applications.\n\n**Experience the future of microservices-based financial calculations with DriveDeposits!**\nDocumentation for Drive Deposits is a work in progress. More details will be added.\n\n[Back to Table of Contents](#table-of-contents)\n\n### Synchronous Microservices Components\n\n- Using ***Axum***:\n    - Serverless Microservice: AWS Lambda Reader (routes for querying data)\n        - With AWS API Gateway\n    - Server-Based Microservice: REST Gateway\n\n- Using ***Tonic***\n    - Server-based Microservice: gRPC Server\n\n[Back to Table of Contents](#table-of-contents)\n\n### Asynchronous Microservices Components\n\n- using ***Tokio***\n    - Asynchronous Tasks to Send Events to EventBridge\n\n- Using the ***AWS SDK for Rust*** for:\n    - Serverless Microservice: AWS Lambda Writer\n    - AWS DynamoDB Integration\n    - AWS EventBridge Integration\n\n[Back to Table of Contents](#table-of-contents)\n\n### Bridging Synchronous and Asynchronous Components In DriveDeposits Microservices\n\nDriveDeposits is a cutting-edge financial calculation platform built on a robust microservices architecture that\ncombines synchronous and asynchronous components. The synchronous gRPC server microservice utilizes Tokio to\nasynchronously send events to the EventBridge service, seamlessly integrating with the asynchronous microservices\ncomponents of the system.\n\nThe gRPC server microservice spawns asynchronous Tokio tasks to send events to EventBridge, effectively bridging the\nsynchronous and asynchronous parts of the architecture.\n\nThe Lambda Reader with Axum Endpoints component serves as a serverless microservice API built with AWS Lambda and the\nAxum web framework in Rust. This microservice provides a set of endpoints to query and retrieve financial data from a\nDynamoDB table. The endpoints are designed to fetch data at different levels of granularity, such as portfolios, banks,\nand deposits, based on specific criteria like delta growth or maturity date.\n\nThe main functionalities of this microservice include:\n\n- **Portfolio Level Endpoint**: Retrieves a list of portfolios based on the delta growth criteria.\n- **Bank Level Endpoint**: Fetches a list of banks for a given portfolio UUID, sorted by the delta growth criteria.\n- **Deposit Level Endpoints**:\n    - **Delta Growth**: Retrieves a list of deposits for a given portfolio UUID, sorted by the delta growth criteria.\n    - **Maturity Date**: Fetches a list of deposits for a given portfolio UUID, sorted by the maturity date.\n\nThis serverless microservice interacts with an AWS DynamoDB table to read and query the required data. It utilizes the\n`aws-sdk-rust` crate to communicate with the DynamoDB service. The Axum web framework is used to define the API routes\nand handle HTTP requests and responses, making it a highly efficient and scalable component of the DriveDeposits\nmicroservices ecosystem.\n\n[Back to Table of Contents](#table-of-contents)\n\n### Deployment of Microservices\n\n#### Serverless: AWS using SAM which uses CloudFormation under the hood\n\nThis project uses SAM (Serverless Application Model) for deploying the following AWS resources that support our\nmicroservices architecture:\n\n    * EventBridge\n    * EventBus\n    * Targets (including CloudWatch Log group)\n    * Lambda functions (for serverless microservices)\n    * DynamoDB table\n\nThe SAM-based deployment streamlines the process of setting up and managing these AWS services for the DriveDeposits\nmicroservices application.\n\nThis project also utilizes the [Justfile](https://github.com/casey/just) for managing project-level recipes and\nmicroservices deployment tasks.\n\n- **Deploy all serverless microservices and AWS resources**\n\n`just deploy-drive-deposits-dynamodb-queries`\n\nThis command calls dependent recipes to deploy the entire microservices ecosystem, including the event bus, event rules\nwith lambda targets, and lambda functions for queries.\n\n- **Delete all serverless microservices and AWS resources**\n\n`just deployed-delete-drive-deposits-event-bus`\n\nThis command calls dependent recipes to delete all deployed microservices and resources, including event rules, lambda\ntargets, event bus, and query-related resources.\n\n- **Deploy event bridge with event rules and target lambda microservices components**\n\n`just deploy-drive-deposits-event-rules`\n\nThis command deploys the event-driven components of our microservices architecture, including EventBridge, EventBus, and\nrelated resources.\n\n- **Deploy Lambda Function with API Gateway and DynamoDB Reader microservices components**\n\n`just deploy-drive-deposits-dynamodb-queries-only`\n\n#### Server-based: Run the REST and gRPC servers Natively, With Docker Compose And Kubernetes!\n\n- **Natively** (without Docker)\n\n`just run-drive-deposits-grpc-server`\n\n`just run-drive-deposits-rest-grpc-gateway-server`\n\n- **Docker Compose**\n  Start Docker Desktop first.\n\nThen run:\n\n`just compose-up-grpc-server`\n\n`just compose-up-rest-server`\n\n- **Kubernetes** (It uses local images to show k8s for local)\n  Install and start Colima with Kubernetes enabled:\n\n```bash\nbrew install colima\ncolima start --cpu 2 --memory 4 --kubernetes\n```\n\nWhen you run colima start --kubernetes, Colima automatically:\n\nCreates a new Docker context named \"colima\"\nCreates a new Kubernetes context\nSets both contexts as current/active\nUpdates ~/.docker/config.json for Docker context\nUpdates ~/.kube/config for Kubernetes context\n\nList all available Docker and Kubernetes contexts:\n\n```bash\ndocker context list\nkubectl config get-contexts\n```\n\nVerify your Docker and Kubernetes context:\n\n  ```bash\n  docker context show\n  kubectl config current-context\n  ```\n\nDocker also stores context information in ~/.docker/config.json and Kubernetes stores context information in ~\n/.kube/config.\n\nYou can always switch back to Colima's context when needed using:\n\n  ```bash\n  docker context use colima\n  kubectl config use-context colima\n  ```\n\nInstall Helm if not already installed:\n\n  ```bash\n  brew install helm\n  ```\n\nAdd and update the nginx-ingress Helm repository:\n\n  ```bash\n  helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\n  helm repo update\n  helm repo list\n  ```\n\nInstall the nginx-ingress controller:\n\n  ```bash\n  helm install ingress-nginx ingress-nginx/ingress-nginx\n  ```\n\nMonitor the nginx ingress controller logs:\n\n  ```bash\n  kubectl logs -l app.kubernetes.io/name=ingress-nginx -f\n  ```\n\nOnly if using ingress controller:\nAdd the domain to your /etc/hosts:\n\n  ```bash\n  echo \"127.0.0.1 api.drivedeposits.local\" | sudo tee -a /etc/hosts\n  ```\n\nCreate AWS credentials secret for the gRPC server:\n\n```bash\nkubectl create secret generic aws-credentials \\\n  --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \\\n  --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \\\n  --from-literal=AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION\n```\n\nTo verify the secret:\n\n```bash\nkubectl get secret aws-credentials -o json\n```\n\nDeploy the services:\n\n  ```bash\n  just k8s-grpc-server\n  just k8s-rest-server\n  ```\n\nThe REST API will now be accessible at http://api.drivedeposits.local\n\n#### Test microservices integration\n\nIn justfile set\n\nfor native and docker-compose:\n\n```justfile\nrest_gateway_server_host := \"http://localhost:3000\"\n```\n\nAnd\n\nfor k8s with ingress:\n\n```justfile\nrest_gateway_server_host := \"http://api.drivedeposits.local\"\n```\n\nSend an HTTP request to the REST gateway server, which communicates with other microservices:\n\n`just post-calculate-portfolio-valid`\n\nThis command tests the full microservices pipeline, from the REST gateway to the gRPC server, through to AWS EventBridge\nand Lambda functions.\n\nFollow up with the [Data population](#data-population) section to see how to query data across the microservices.\n\n#### Test AWS Lambda microservice directly\n\n`just aws-invoke-drive-deposits-event-rules-lambda`\n\n#### Data population\n\n##### AWS Populated with Basic Data for Queries Lambda\n\nSee [Deployment of Microservices](#deployment-of-microservices) section for deploying the microservices and AWS\nresources.\n\n###### send curl post requests to populate\n\n`just post-calculate-portfolio-valid`\n\n`just post-calculate-portfolio-valid-lesser-amount`\n\n`just post-calculate-portfolio-valid-greater-amount`\n\n###### Alternatively, Using check command without servers\n\n`just run-drive-deposits-check-cmd-valid-send-events`\n\n[Back to Table of Contents](#table-of-contents)\n\n#### Querying with custom domain\n\nSuccessfully configured custom domain (https://api-queries.drivedeposits.drinnovations.us) for API Gateway so with some\nexisting data queries can be tried\nby running just recipes for querying\n\nThen you can perform the following from [justfile](justfile) for portfolios delta growth sorted in descending order with\ntop 3 items:\n\n`just get-query-by-level-portfolios-delta-growth`\n\nor directly click link to see some data in your browser or Postman like tool - with pretty print:\n\n[Drive Deposits By Level For Portfolios API Query](https://api-queries.drivedeposits.drinnovations.us/by-level-for-portfolios/delta-growth?order=desc\u0026top_k=3)\n\nwhich actually calls\n\n```curl\nhttps://api-queries.drivedeposits.drinnovations.us/by-level-for-portfolios/delta-growth?order=desc\u0026top_k=3\n```\n\nUpdate aws_portfolio_uuid per the portfolios response to populate\nin [justfile](justfile). Adjusted `justfile` with a test portfolio UUID for querying.\n\n`just get-query-by-level-portfolios-delta-growth`\n\n`just get-query-by-level-for-deposits-delta-growth`\n\n`just get-query-by-level-for-deposits-maturity-date`\n\n[Back to Table of Contents](#table-of-contents)\n\n### Running Tests\n\n##### Integration tests\n\n`just test-intg`\n\n##### Unit and Integration tests\n\n`just test`\n\n##### End-to-End tests\n\n`just test-e2e`\n\n##### Hybrid Integration Testing Tool\n\n###### Efficient Command for Synchronous Calculation Flow and Asynchronous AWS Serverless Event Testing\n\nThe command drive-deposits-check-cmd is a powerful hybrid integration testing tool that bridges both synchronous and\nasynchronous aspects of the system. It efficiently mimics the synchronous flow of REST and gRPC servers while\ninteracting with asynchronous EventBridge components, all without the need for full server deployment.\nKey features:\n\n* Performs identical type transformations as REST and gRPC servers\n* Enables rapid calculation validation and event routing verification\n* Sends calculations to EventBridge for comprehensive sanity testing\n* Validates event routing to appropriate destinations (log groups, AWS Lambda functions, DynamoDB)\n* Allows developers to verify end-to-end flow of calculations, event handling, and data persistence\n\nExecute the tool with:\n\n`just run-drive-deposits-check-cmd-valid-send-events`\n\nThis streamlined approach significantly enhances development efficiency and system reliability testing.\n\n###### Alias\n\n.cargo/config.toml has alias for command line [drive-deposits-check-cmd](drive-deposits-check-cmd) so can be run using\n`cargo ddcheck` For help see `cargo ddcheck -- --help`\n\n[Back to Table of Contents](#table-of-contents)\n\n### Development Tool: cargo lambda\n\nFollowing is convenience so that in development can iterate faster when skipping sam resources:\n\nfor only build watching lambda without using sam\n`just cargo-lambda-build-watching-drive-deposits-event-rules-lambda`\n\nthis is different from cargo lambda watch used with invoke -- The watch subcommand emulates the AWS Lambda control plane\nAPI. The function is not compiled until the first time that you try to execute it.\n`just cargo-lambda-watch-drive-deposits-event-rules-lambda`\n\n#### command for cargo lambda invoke check directly\n\n`just cargo-lambda-invoke-drive-deposits-event-rules-lambda`\n\n[Back to Table of Contents](#table-of-contents)\n\n### Development Tool: LocalStack\n\nLocalstack, being an ephemeral service, can be started and stopped as needed. This means that unless the state is\npersisted, LocalStack provides a clean slate every time it's restarted. This feature can expedite the development and\ndeployment process as it eliminates the need to manually delete resources.\n\nFollowing is convenience so that in development can iterate faster:\n\n`just localstack-start`\n\nShould see \"Ready.\" -- There is a Terminal now in Docker Desktop itself so that is a good place to run this command.\n\n#### deploy everything serverless in Localstack - aws deployment related commands for EventBridge, EventBus, Cloudwatch log groups and Lambda target function for writing to DynamoDB and Lambda DynamoDB reader\n\n`just localstack-deploy-drive-deposits-dynamodb-queries`\n\n#### LocalStack populated with data\n\nSanity check with LocalStack/ Working with queries lambda directly with cargo lambda and localstack\n\n`just localstack-start`\n\n`just localstack-deploy-drive-deposits-event-rules`\n\nPopulate with\n\n`just localstack-run-drive-deposits-check-cmd-valid-send-events`\n\nand then\n\n`just localstack-run-drive-deposits-check-cmd-valid-send-events-lesser-amount-investments`\n\nand then\n\n`just localstack-run-drive-deposits-check-cmd-valid-send-events-greater-amount-investments`\n\nand can also repeat these commands for more data\n\nThere is also a watch command that can be used to watch for changes in the code and automatically run the check command\n`just localstack-watch-run-drive-deposits-check-cmd-valid-send-events`\n\nFor queries lambda, replace table name in localstack in\ncargo-lambda-watch-drive-deposits-lambda-dynamodb-reader-localstack recipe\n\nThen can use cargo-lambda with dynamodb in localstack already populated with data:\n`just cargo-lambda-watch-drive-deposits-lambda-dynamodb-reader-localstack`\n\nAnd for actual cargo lambda apigw events\n`just cargo-lambda-invoke-drive-deposits-lambda-dynamodb-reader-apigw-event-query-for-portfolios`\n\nReplace proper portfolio uuid\n\n`just cargo-lambda-invoke-drive-deposits-lambda-dynamodb-reader-apigw-event-query-for-banks`\n\n`just cargo-lambda-invoke-drive-deposits-lambda-dynamodb-reader-apigw-event-query-for-deposits`\n\n##### command for awslocal localstack lambda invoke check directly\n\nFollowing is convenience so that in development can iterate faster:\n\n`just awslocal-invoke-drive-deposits-event-rules-lambda`\n\n[Back to Table of Contents](#table-of-contents)\n\n### Clean And Build\n\ncargo clean is used but since there are lambda we have .aws-sam folders created by sam also that we have a clean\n\nso we have\n`just clean-with-lambdas`\n\nfor quick build that can be used after cleaning everything\n`just build-with-lambdas`\n\n[Back to Table of Contents](#table-of-contents)\n\n### Configurations for DriveDeposits\n\nThe project uses custom configurations defined in `.cargo/config.toml`:\n\n- `SEND_CAL_EVENTS`: This environment variable is set to \"true\" by default in the config file. It can be overridden in\n  specific commands as needed.\n- `USE_LOCALSTACK`: This environment variable is set to \"false\" by default in the config file. It can be overridden for\n  local development with LocalStack.\n- Alias: The project includes an alias for the `drive-deposits-check-cmd`. It can be run using `cargo ddcheck`. For\n  help, use `cargo ddcheck -- --help`.\n\nThese configurations allow for flexible development and testing environments, enabling easy switching between local and\nAWS deployments.\n\n[Back to Table of Contents](#table-of-contents)\n\n[Copyright (c) 2024-2025 Rohit Sachdeva](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frsachdeva%2Fdrive-deposits","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frsachdeva%2Fdrive-deposits","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frsachdeva%2Fdrive-deposits/lists"}