{"id":16110984,"url":"https://github.com/nikiforovall/aws-batch-dotnet","last_synced_at":"2026-01-03T18:46:53.315Z","repository":{"id":241391577,"uuid":"805317077","full_name":"NikiforovAll/aws-batch-dotnet","owner":"NikiforovAll","description":"Learn how to leverage AWS Batch to efficiently process files in S3 using .NET","archived":false,"fork":false,"pushed_at":"2024-05-27T19:33:18.000Z","size":180,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-26T04:25:21.938Z","etag":null,"topics":["aws","aws-batch","aws-ec2","aws-ecs","aws-s3","cli","dotnet","terraform"],"latest_commit_sha":null,"homepage":"https://nikiforovall.github.io/dotnet/aws/2024/05/26/aws-batch-dotnet.html","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NikiforovAll.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-24T10:20:20.000Z","updated_at":"2024-09-01T16:49:26.000Z","dependencies_parsed_at":"2024-05-28T05:09:56.876Z","dependency_job_id":null,"html_url":"https://github.com/NikiforovAll/aws-batch-dotnet","commit_stats":null,"previous_names":["nikiforovall/aws-batch-dotnet"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikiforovAll%2Faws-batch-dotnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikiforovAll%2Faws-batch-dotnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikiforovAll%2Faws-batch-dotnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NikiforovAll%2Faws-batch-dotnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NikiforovAll","download_url":"https://codeload.github.com/NikiforovAll/aws-batch-dotnet/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244759814,"owners_count":20505709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-batch","aws-ec2","aws-ecs","aws-s3","cli","dotnet","terraform"],"created_at":"2024-10-09T19:39:59.691Z","updated_at":"2026-01-03T18:46:53.264Z","avatar_url":"https://github.com/NikiforovAll.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Batch Processing with .NET\n\nThe goal of this repository is to demonstrate how to use AWS Batch.\n\nIn this example we are processing files from S3 in parallel.\n\n## Deploy resources\n\n```bash\n# cd ./deploy\nterraform init\nterraform apply\n```\n\n## Run Manually\n\nPlan:\n\n```bash\ndotnet run -- plan \\\n    --source s3://aws-batch-demo-dotnet-source-bucket \\\n    --destination s3://aws-batch-demo-dotnet-destination-bucket/output/ \\\n    --plan s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json\n```\n\nMigrate:\n\n```bash\nnumberOfJobs=2\nfor index in $(seq 0 $((numberOfJobs-1)))\ndo\n    export AWS_BATCH_JOB_ARRAY_INDEX=$index\n    dotnet run --no-build -- migrate \\\n        --plan s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json\ndone\n```\n\nMerge:\n\n```bash\ndotnet run -- merge \\\n    --source s3://aws-batch-demo-dotnet-destination-bucket/output/\n```\n\n## Submit Job in AWS batch\n\n```bash\naws batch submit-job \\\n    --job-name aws-batch-dotnet-plan-01 \\\n    --job-queue MainQueue  \\\n    --job-definition aws-batch-dotnet-plan \\\n    --share-identifier \"demobatch*\" \\\n    --scheduling-priority-override 1 \\\n    --container-overrides '{\n        \"command\": [\n            \"plan\",\n            \"--source\",\n            \"s3://aws-batch-demo-dotnet-source-bucket\",\n            \"--destination\",\n            \"s3://aws-batch-demo-dotnet-destination-bucket/output/\",\n            \"--plan\",\n            \"s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json\"\n        ]\n    }'\n```\n\n```bash\naws batch submit-job \\\n    --job-name aws-batch-dotnet-migrate-01 \\\n    --job-queue MainQueue  \\\n    --job-definition aws-batch-dotnet-migrate \\\n    --share-identifier \"demobatch*\" \\\n    --scheduling-priority-override 1 \\\n    --array-properties size=2 \\\n    --container-overrides '{\n        \"command\": [\n            \"migrate\",\n            \"--plan\",\n            \"s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json\"\n        ]\n    }'\n```\n\n```bash\naws batch submit-job \\\n    --job-name aws-batch-dotnet-merge-01 \\\n    --job-queue MainQueue  \\\n    --job-definition aws-batch-dotnet-merge \\\n    --share-identifier \"demobatch*\" \\\n    --scheduling-priority-override 1 \\\n    --container-overrides '{\n        \"command\": [\n            \"merge\",\n            \"--source\",\n            \"s3://aws-batch-demo-dotnet-destination-bucket/output/\"\n        ]\n    }'\n```\n\n## Reference\n\n* \u003chttps://docs.aws.amazon.com/batch/latest/userguide/example_array_job.html\u003e\n* \u003chttps://github.com/aws/aws-cli/issues/5636\u003e - issue for ECR + Windows + SSO login via AWS CLI v2.\n* \u003chttps://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html\u003e\n* \u003chttps://docs.aws.amazon.com/step-functions/latest/dg/workflow-studio-process.html\u003e\n* \u003chttps://aws.amazon.com/blogs/hpc/encoding-workflow-dependencies-in-aws-batch/\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikiforovall%2Faws-batch-dotnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnikiforovall%2Faws-batch-dotnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnikiforovall%2Faws-batch-dotnet/lists"}