{"id":15321793,"url":"https://github.com/eldimious/real_time_streaming_analytics","last_synced_at":"2025-04-15T02:36:20.460Z","repository":{"id":41289511,"uuid":"508599609","full_name":"eldimious/real_time_streaming_analytics","owner":"eldimious","description":"we look at how we can use managed services such as Amazon Kinesis to handle our incoming data analytics streams from our sources while AWS handles the undifferentiated heavy lifting of managing the infrastructure","archived":false,"fork":false,"pushed_at":"2022-07-06T07:29:29.000Z","size":534,"stargazers_count":6,"open_issues_count":6,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T14:22:03.741Z","etag":null,"topics":["amazon-web-services","aws","aws-apigateway","aws-kinesis","aws-lambda","lambda","real-time","streaming","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eldimious.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-06-29T07:59:38.000Z","updated_at":"2023-03-06T11:40:23.000Z","dependencies_parsed_at":"2022-08-27T01:12:43.288Z","dependency_job_id":null,"html_url":"https://github.com/eldimious/real_time_streaming_analytics","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eldimious%2Freal_time_streaming_analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eldimious%2Freal_time_streaming_analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eldimious%2Freal_time_streaming_analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eldimious%2Freal_time_streaming_analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eldimious","download_url":"https://codeload.github.com/eldimious/real_time_streaming_analytics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248995075,"owners_count":21195491,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-web-services","aws","aws-apigateway","aws-kinesis","aws-lambda","lambda","real-time","streaming","terraform"],"created_at":"2024-10-01T09:13:07.422Z","updated_at":"2025-04-15T02:36:20.436Z","avatar_url":"https://github.com/eldimious.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# real_time_streaming_analytics\n\nIn this repository, we look at how we can use managed services such as Amazon Kinesis to handle our incoming data analytics streams from our sources while AWS handles the undifferentiated heavy lifting of managing the infrastructure. \u003c/br\u003e\n\u003c/br\u003e\nOur purpose is to create a real time streaming service, where we produce payment transactions, which are pushed to AWS Kinesis data stream, are stored into AWS S3 for cold storage via AWS Kinesis Firehose, and a AWS Lambda detects anomalies in payment transactions (in payment amount \u003e 1000) stores these anomalies to a database and notifies user via email.\n\n## Architecture Flow\nThe flow pipeline for our microservice ecosystem is the ![following](./ArchitectureFlow.png)\n\n## Deploy services\n\nIn order to deploy microservices in AWS run the following cmd:\n\n```shell\ndeploy.sh\n```\n\n## Destroy services\n\nIn order to destroy clusters from AWS run the following cmd:\n\n```shell\ndestroy.sh\n```\n\n## Endpoints\n\nYou can take a look at API's endpoints navigated to ${api_gateway_url} output from `deploy.sh` cmd.\n\n### Collect transactions ###\n\n```shell\nPOST ${api_gateway_url}/collector/transactions\n```\n\nHeaders:\n```json\n{\n    \"x-api-key\": \"${api_gateway_api_key}\"\n}\n```\n\nBody Params:\n```json\n{\n    \"records\": [\n        {\n\n                \"trxId\": \"004ed073-91b4-4623-ae33-7bef7cd104b0\",\n                \"amount\": 9000,\n                \"senderId\": \"2f9c0f77-623d-4c1f-af82-1148ee062c03\",\n                \"receiverId\": \"340515af-476d-446b-bbdd-ad2b8585afd4\",\n                \"senderIban\": \"GRxxxxxxxxxxxxxxx\",\n                \"receiverIban\": \"BYxxxxxxxxxxxxxxxx\",\n                \"senderBankId\": \"EUROBANK\",\n                \"receiverBankId\": \"ALPHA\",\n                \"transactionDate\": \"2022-06-30T07:41:46+0000\"\n\n        },\n        {\n\n                \"trxId\": \"280eaebf-165e-42e0-9212-22df1127e84c\",\n                \"amount\": 333,\n                \"senderId\": \"2f9c0f77-623d-4c1f-af82-1148ee062c03\",\n                \"receiverId\": \"340515af-476d-446b-bbdd-ad2b8585afd4\",\n                \"senderIban\": \"GRxxxxxxxxxxxxxxx\",\n                \"receiverIban\": \"BYxxxxxxxxxxxxxxxx\",\n                \"senderBankId\": \"EUROBANK\",\n                \"receiverBankId\": \"ALPHA\",\n                \"transactionDate\": \"2022-06-30T07:41:46+0000\"\n            \n        },\n        {\n\n                \"trxId\": \"19754ab1-b19d-470b-be0e-4e1bf511761a\",\n                \"amount\": 300,\n                \"senderId\": \"2f9c0f77-623d-4c1f-af82-1148ee062c03\",\n                \"receiverId\": \"340515af-476d-446b-bbdd-ad2b8585afd4\",\n                \"senderIban\": \"GRxxxxxxxxxxxxxxx\",\n                \"receiverIban\": \"BYxxxxxxxxxxxxxxxx\",\n                \"senderBankId\": \"EUROBANK\",\n                \"receiverBankId\": \"ALPHA\",\n                \"transactionDate\": \"2022-06-30T07:41:46+0000\"\n        }\n    ]\n}\n```\n\nDescription: adds new transactions records in Kinesis streams to be used and analyzed for anomaly_transactions_detector lambda\n\n## Show your support\n\nGive a ⭐️ if this project helped you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feldimious%2Freal_time_streaming_analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feldimious%2Freal_time_streaming_analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feldimious%2Freal_time_streaming_analytics/lists"}