Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/nikiforovall/aws-batch-dotnet
Learn how to leverage AWS Batch to efficiently process files in S3 using .NET
https://github.com/nikiforovall/aws-batch-dotnet
aws aws-batch aws-ec2 aws-ecs aws-s3 cli dotnet terraform
Last synced: 7 days ago
JSON representation
Learn how to leverage AWS Batch to efficiently process files in S3 using .NET
- Host: GitHub
- URL: https://github.com/nikiforovall/aws-batch-dotnet
- Owner: NikiforovAll
- Created: 2024-05-24T10:20:20.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-05-27T19:33:18.000Z (5 months ago)
- Last Synced: 2024-10-10T19:40:02.106Z (28 days ago)
- Topics: aws, aws-batch, aws-ec2, aws-ecs, aws-s3, cli, dotnet, terraform
- Language: C#
- Homepage: https://nikiforovall.github.io/dotnet/aws/2024/05/26/aws-batch-dotnet.html
- Size: 176 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# AWS Batch Processing with .NET
The goal of this repository is to demonstrate how to use AWS Batch.
In this example we are processing files from S3 in parallel.
## Deploy resources
```bash
# cd ./deploy
terraform init
terraform apply
```
## Run Manually
Plan:
```bash
dotnet run -- plan \
--source s3://aws-batch-demo-dotnet-source-bucket \
--destination s3://aws-batch-demo-dotnet-destination-bucket/output/ \
--plan s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json
```
Migrate:
```bash
numberOfJobs=2
for index in $(seq 0 $((numberOfJobs-1)))
do
export AWS_BATCH_JOB_ARRAY_INDEX=$index
dotnet run --no-build -- migrate \
--plan s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json
done
```
Merge:
```bash
dotnet run -- merge \
--source s3://aws-batch-demo-dotnet-destination-bucket/output/
```
## Submit Job in AWS batch
```bash
aws batch submit-job \
--job-name aws-batch-dotnet-plan-01 \
--job-queue MainQueue \
--job-definition aws-batch-dotnet-plan \
--share-identifier "demobatch*" \
--scheduling-priority-override 1 \
--container-overrides '{
"command": [
"plan",
"--source",
"s3://aws-batch-demo-dotnet-source-bucket",
"--destination",
"s3://aws-batch-demo-dotnet-destination-bucket/output/",
"--plan",
"s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json"
]
}'
```
```bash
aws batch submit-job \
--job-name aws-batch-dotnet-migrate-01 \
--job-queue MainQueue \
--job-definition aws-batch-dotnet-migrate \
--share-identifier "demobatch*" \
--scheduling-priority-override 1 \
--array-properties size=2 \
--container-overrides '{
"command": [
"migrate",
"--plan",
"s3://aws-batch-demo-dotnet-destination-bucket/plans/plan-01.json"
]
}'
```
```bash
aws batch submit-job \
--job-name aws-batch-dotnet-merge-01 \
--job-queue MainQueue \
--job-definition aws-batch-dotnet-merge \
--share-identifier "demobatch*" \
--scheduling-priority-override 1 \
--container-overrides '{
"command": [
"merge",
"--source",
"s3://aws-batch-demo-dotnet-destination-bucket/output/"
]
}'
```
## Reference
*
* - issue for ECR + Windows + SSO login via AWS CLI v2.
*
*
*