{"id":18456876,"url":"https://github.com/mrdatapsycho/cloud-native-datascience","last_synced_at":"2026-04-19T03:03:24.757Z","repository":{"id":112336958,"uuid":"577413958","full_name":"MrDataPsycho/cloud-native-datascience","owner":"MrDataPsycho","description":"A cloud native nlp translation application build with Python sdks for AWS","archived":false,"fork":false,"pushed_at":"2022-12-28T02:53:30.000Z","size":900,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-17T02:04:37.803Z","etag":null,"topics":["aws","cloud-native-applications","datascience","huggingface","nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MrDataPsycho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-12T17:23:33.000Z","updated_at":"2022-12-28T02:54:27.000Z","dependencies_parsed_at":"2023-05-13T06:45:11.272Z","dependency_job_id":null,"html_url":"https://github.com/MrDataPsycho/cloud-native-datascience","commit_stats":null,"previous_names":["mrdatapsycho/cloud-native-datascience"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrDataPsycho%2Fcloud-native-datascience","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrDataPsycho%2Fcloud-native-datascience/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrDataPsycho%2Fcloud-native-datascience/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MrDataPsycho%2Fcloud-native-datascience/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MrDataPsycho","download_url":"https://codeload.github.com/MrDataPsycho/cloud-native-datascience/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250308548,"owners_count":21409291,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud-native-applications","datascience","huggingface","nlp"],"created_at":"2024-11-06T08:12:55.239Z","updated_at":"2026-04-19T03:03:24.719Z","avatar_url":"https://github.com/MrDataPsycho.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cloud Native Data Science Product\nThe repository contains a fully featured cloud native NLP product build with AWS stack. Following AWS service is being heavily used in the Product development:\n- ApiGateway, Lambda for backend api\n- Dynamodb as database storage\n- Step Function as a BatchJob scheduler\n- Sagemaker Processing Job as batch job container\n- S3 for large file storage\n\n## Roles and Policies:\nTo be able to make the whole application work following roles and policies needed to be created and added into the environment variables.\nHere is the Environment variable list:\n\n```shell\nexport BUCKET=\"\u003cyour-bucket\u003e\"\nexport DB_NAME=\"\u003cyour-database\u003e\"\nexport BATCH_JOB_BUCKET=\"\u003cbucket-for-sagemaker-processing-job\u003e\"\nexport SAGEMAKER_ARN=\"\u003csagemaker-arn-to-initiate-processing-job\u003e\"\nexport LAMBDA_ARN=\"\u003carn for lambda role\u003e\"\nexport STEP_FUNCTION_ARN=\"\u003cstep function role arn\u003e\"\nexport SFN_WORKFLOW_ARN=\"\u003cstep function workflow arn\u003e\"\nexport API_GATEWAY_ARN=\"\u003capi gateway role arn for cloud watch\u003e\"\nexport API_URL=\"\u003curl for the api\u003e\"\n```\n\n## Policies for the Roles:\nThe policies for the roles are loosely defined, you should be more strict while adding policies into the roles.\n\n- SAGEMAKER_ARN:\n  - s3FullAccess\n  - DynamoDBFullAccess\n  - StepFunctionFullAccess\n  - Lambda_FullAccess\n- LAMBDA_ARN:\n  - s3FullAccess\n  - APIGatewayInvokeFullAccess\n  - LambdaExecute\n  - StepFunctionFullAccess\n- STEP_FUNCTION_ARN:\n  - LambdaRole\n  - Lambda_FullAccess\n  - CloudWatchEventsFullAccess\n  - Custom Pass role from Step Function to Sagemaker\n\n```json\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"VisualEditor0\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"events:PutTargets\",\n                \"events:DescribeRule\",\n                \"events:PutRule\"\n            ],\n            \"Resource\": [\n                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule\",\n                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule\",\n                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTuningJobsRule\",\n                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForECSTaskRule\",\n                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForBatchJobsRule\"\n            ]\n        },\n        {\n            \"Sid\": \"VisualEditor1\",\n            \"Effect\": \"Allow\",\n            \"Action\": \"iam:PassRole\",\n            \"Resource\": \"*\",\n            \"Condition\": {\n                \"StringEquals\": {\n                    \"iam:PassedToService\": \"sagemaker.amazonaws.com\"\n                }\n            }\n        },\n        {\n            \"Sid\": \"VisualEditor2\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"batch:DescribeJobs\",\n                \"batch:SubmitJob\",\n                \"batch:TerminateJob\",\n                \"lambda:InvokeFunction\",\n                \"sagemaker:CreateEndpoint\",\n                \"sagemaker:CreateEndpointConfig\",\n                \"sagemaker:CreateHyperParameterTuningJob\",\n                \"sagemaker:CreateModel\",\n                \"sagemaker:CreateProcessingJob\",\n                \"sagemaker:CreateTrainingJob\",\n                \"sagemaker:CreateTransformJob\",\n                \"sagemaker:DeleteEndpoint\",\n                \"sagemaker:DeleteEndpointConfig\",\n                \"sagemaker:DescribeHyperParameterTuningJob\",\n                \"sagemaker:DescribeProcessingJob\",\n                \"sagemaker:DescribeTrainingJob\",\n                \"sagemaker:DescribeTransformJob\",\n                \"sagemaker:ListProcessingJobs\",\n                \"sagemaker:ListTags\",\n                \"sagemaker:StopHyperParameterTuningJob\",\n                \"sagemaker:StopProcessingJob\",\n                \"sagemaker:StopTrainingJob\",\n                \"sagemaker:StopTransformJob\",\n                \"sagemaker:UpdateEndpoint\",\n                \"sns:Publish\",\n                \"sqs:SendMessage\"\n            ],\n            \"Resource\": \"*\"\n        }\n    ]\n}\n```\n**This is not a good example of adding role where I have full access everytime please use more restricted access, this is just for demo purpose.**\n- API_GATEWAY_ARN:\n  - APIGatewayPushToCloudWatchLogs\n  - Lambda_FullAccess\n\n# ClientSide\nThe client side is written in Streamlit which can be found at the clientside directory.\n\n# Serverside\nThe serverside is written as http api with each endpoint has its own pure python script deployed as lambda.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrdatapsycho%2Fcloud-native-datascience","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrdatapsycho%2Fcloud-native-datascience","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrdatapsycho%2Fcloud-native-datascience/lists"}