{"id":18284639,"url":"https://github.com/pplu/cfn-fargate-batch","last_synced_at":"2025-04-05T07:31:52.859Z","repository":{"id":66753253,"uuid":"320295305","full_name":"pplu/cfn-fargate-batch","owner":"pplu","description":"A guide for using AWS Batch jobs with Fargate from CloudFormation","archived":false,"fork":false,"pushed_at":"2020-12-11T11:57:47.000Z","size":6,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T00:32:54.907Z","etag":null,"topics":["aws","batch","cfn-templates","cloudformation","cloudformation-templates","fargate","serverless"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pplu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-10T14:22:39.000Z","updated_at":"2023-09-30T12:00:29.000Z","dependencies_parsed_at":"2023-02-22T19:15:19.930Z","dependency_job_id":null,"html_url":"https://github.com/pplu/cfn-fargate-batch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Fcfn-fargate-batch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Fcfn-fargate-batch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Fcfn-fargate-batch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2Fcfn-fargate-batch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pplu","download_url":"https://codeload.github.com/pplu/cfn-fargate-batch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305876,"owners_count":20917198,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","batch","cfn-templates","cloudformation","cloudformation-templates","fargate","serverless"],"created_at":"2024-11-05T13:14:16.777Z","updated_at":"2025-04-05T07:31:52.850Z","avatar_url":"https://github.com/pplu.png","language":null,"readme":"# Batch Fargate via CloudFormation\n\nThis year there was an announcement in re:Invent that caught my attention: \n[Serverless AWS Batch](https://aws.amazon.com/blogs/aws/new-fully-serverless-batch-computing-with-aws-batch-support-for-aws-fargate/).\n\nI had looked at Batch some time before, and it had always been in my head. My vision was that \nif you had a heavy Batch process, the effort of setting AWS Batch up for small tasks was a tad too much for \nmy taste. You could resort to Lambda for short-lived tasks.\n\nThese days a use case for the new Serverless Fargate came up. Since the service was announced as GA, and\nI saw in the CloudFormation User Guide it was supported, I went ahead to try it (I really frown upon services \nthat aren't CloudFormation friendly up to a point where I avoid them, since that lack of support brings me\ninto a situation where you have to develop manuals for getting reproducibility and auditability, which I \nreally consider to be a must in the Cloud).\n\nThe following is my experience in getting everything to work. Note that this was **not** trivial for reasons\nyou will see below.\n\nReading the article, and looking to see if everything is supported in CloudFormation we know that we have to\ncreate three components: Compute Environment, Job Queue and Job Definition:\n\n## Create a Compute Environment \n\nFrom the [launch article](https://aws.amazon.com/blogs/aws/new-fully-serverless-batch-computing-with-aws-batch-support-for-aws-fargate/).\n\n```!json\n{\n    \"computeEnvironmentName\": \"FargateComputeEnvironment\",\n    \"type\": \"MANAGED\",\n    \"state\": \"ENABLED\",\n    \"computeResources\": {\n        \"type\": \"FARGATE\", # or FARGATE_SPOT\n        \"maxvCpus\": 40,\n        \"subnets\": [\n             \"subnet-xxxxxxxx\",\"subnet-xxxxxxxx\",\"subnet-xxxxxxxx\"\n        ],\n        \"securityGroupIds\": [\"sg-xxxxxxxxxxxxxxxx\"],\n        \"tags\": {\n            \"KeyName\": \"fargate\"\n        }\n    },\n    \"serviceRole\": \"arn:aws:iam::xxxxxxxxxxxx:role/service-role/AWSBatchServiceRole\"\n}\n```\n\nLooking at the [AWS::Batch::ComputeEnvironment](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-computeenvironment.html) object:\n\n - `Type: MANAGED` is [supported](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-computeenvironment.html#cfn-batch-computeenvironment-type)\n - `ComputeResources.Type: FARGATE` is [supported](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-batch-computeenvironment-computeresources.html#cfn-batch-computeenvironment-computeresources-type)\n\nSo my corresponding CloudFormation template fragment is:\n```\n  ComputeEnvironment:\n    Type: AWS::Batch::ComputeEnvironment\n    Properties: \n      Type: MANAGED\n      State: ENABLED\n      ServiceRole: !Ref BatchRole\n      ComputeResources: \n        Type: FARGATE\n        MaxvCpus: 1\n        Subnets:\n        - !Ref Subnet1\n        - !Ref Subnet2\n        SecurityGroupIds:\n        - !Ref BatchSG\n```\n\nThis works correctly :)\n\n## Create a Job Queue\n\nThe [launch article](https://aws.amazon.com/blogs/aws/new-fully-serverless-batch-computing-with-aws-batch-support-for-aws-fargate/) references:\n\n```!json\n{\n  \"jobQueueName\": \"FargateJobQueue\",\n  \"state\": \"ENABLED\",\n  \"priority\": 1,\n  \"computeEnvironmentOrder\": [\n    {\n      \"order\": 1,\n      \"computeEnvironment\": \"FargateComputeEnvironment\"\n    }\n  ]\n}\n```\n\nLooking at the [AWS::Batch::JobQueue](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobqueue.html) object, this gets\ntranslated easily:\n\n```\n  Queue:\n    Type: AWS::Batch::JobQueue\n    Properties:\n      ComputeEnvironmentOrder: \n        - ComputeEnvironment: !Ref ComputeEnvironment\n          Order: 1\n      Priority: 1\n      State: ENABLED\n```\n\nThis also works correctly! :)\n\n## Create a Job Definition\n\nThe [launch article](https://aws.amazon.com/blogs/aws/new-fully-serverless-batch-computing-with-aws-batch-support-for-aws-fargate/) has a sample Job Definition. I\nhave to advise you that this is where the *fun* starts!\n\n```!json\n{\n    \"jobDefinitionName\": \"FargateJobDefinition\",\n    \"type\": \"container\",\n    \"propagateTags\": true,\n     \"containerProperties\": {\n        \"image\": \"xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/test:latest\",\n        \"networkConfiguration\": {\n            \"assignPublicIp\": \"ENABLED\"\n        },\n        \"fargatePlatformConfiguration\": {\n            \"platformVersion\": \"LATEST\"\n        },\n        \"resourceRequirements\": [\n            {\n                \"value\": \"0.25\",\n                \"type\": \"VCPU\"\n            },\n            {\n                \"value\": \"512\",\n                \"type\": \"MEMORY\"\n            }\n        ],\n        \"jobRoleArn\": \"arn:aws:iam::xxxxxxxxxxxx:role/ecsTaskExecutionRole\",\n        \"executionRoleArn\":\"arn:aws:iam::xxxxxxxxxxxx:role/ecsTaskExecutionRole\",\n        \"logConfiguration\": {\n            \"logDriver\": \"awslogs\",\n            \"options\": {\n            \"awslogs-group\": \"/ecs/sleepenv\",\n            \"awslogs-region\": \"us-east-1\",\n            \"awslogs-stream-prefix\": \"ecs\"\n            }\n        }\n     },\n   \"platformCapabilities\": [\n        \"FARGATE\"\n    ],\n    \"tags\": {\n    \"Service\": \"Batch\",\n    \"Name\": \"JobDefinitionTag\",\n    \"Expected\": \"MergeTag\"\n    }\n}\n```\n\nSeeing that [AWS::Batch::JobDefinition](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobdefinition.html) resource has explicit mentions\nof Fargate:\n\n - `Type: container`: [supported](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-batch-jobdefinition.html#cfn-batch-jobdefinition-type)\n - `ContainerProperties`: [supported](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-batch-jobdefinition-containerproperties.html)\n - `ResourceRequirements`: [supported](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-batch-jobdefinition-resourcerequirement.html) with special mentions for Fargate\n - `FargatePlatformConfiguration`: not in the resource, but mayeb CloudFormation will just use the latest\n - `PlatformCapabilities`: not in the resource, but maybe CloudFormation will take an educated guess, since the ResourceRequirements states\n\nNote that the ResourceRequirements in the manual specify 0.5, 1, 2, 3, and in the launch we see '512'. This will be important later.\n\nAnd here I start getting errors:\n\n```\nAn error occurred (ClientException) when calling the RegisterJobDefinition operation: \nError executing request, Exception : networkConfiguration not applicable for EC2.,\nRequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\n```\n\nSo it looks like CloudFormation is creating a JobDefinition for EC2 (and not Fargate!). \n\nTaking out the NetworkConfiguration part of my CloudFormation I start getting new errors.\n\n```\nAn error occurred (ClientException) when calling the RegisterJobDefinition operation: \nError executing request, Exception : Memory must be at least 4 Mib, got 1 Mib.,\nRequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\n```\n\nWhy is this so? I had read the [ResourceRequirements documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-batch-jobdefinition-resourcerequirement.html) \nwhere it was explicity saying I set decimal values (1 for 1GB of RAM) when using Fargate.\n\n\u003e For jobs running on Fargate resources, then value is the hard limit (in GiB), represented in decimal form, and must match one of the supported values (0.5 and whole numbers between 1 and 30, inclusive) and the VCPU values must be one of the values supported for that memory value\n\n```\nAn error occurred (ClientException) when calling the RegisterJobDefinition operation: \nError executing request, Exception : Fargate resource requirements (0.50 vCPU, 1 MiB) not valid., \nRequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\n```\n\nSomething here is fishy, because in the launch article the MEMORY parameter is '512'...\n\nSo start to approch the the problem from another angle. Instead of *CloudFormating* my way forward, I tried to hand-configure a JobDefinition on the console. \nLots of tests on the AWS console (which also has some quircky behaviors when doing Fargate Batch) get me convinced that everything **should** work.\n\nThen a breakthrough. Out of desperation I add two undocumented properties to the JobDescription: `FargatePlatformConfiguration` and `PlatformCapabilites`.\n\nI get an error for `Encountered unsupported properties in {/}: [FargatePlatformConfiguration]`. But not for `PlatformCapabilites`!\n\nAfter that I decide to try with that fishy MEMORY value that was in the article: BINGO!\n\nSo after lots of pain, here we have **a working** Batch Fargate CloudFormation resource:\n\n```\n  JobDefinition:\n    Type: AWS::Batch::JobDefinition\n    Properties:\n      Type: container\n      JobDefinitionName: { Ref: \"AWS::StackName\" }\n      PlatformCapabilities:\n      - FARGATE\n      Timeout:\n        AttemptDurationSeconds: 600\n      RetryStrategy:\n        Attempts: 1\n      ContainerProperties:\n        Command:\n        - 'echo'\n        - 'hello'\n        - 'world'\n        Image: 'debian:latest'\n        NetworkConfiguration:\n          AssignPublicIp: ENABLED\n        ResourceRequirements:\n        - Type: VCPU\n          Value: 0.5\n        - Type: MEMORY\n          Value: 1024\n        ExecutionRoleArn: !GetAtt ExecutionRole.Arn\n        LogConfiguration:\n          LogDriver: awslogs\n          Options:\n            \"awslogs-group\": !Ref LogGroup\n            \"awslogs-stream-prefix\": \"deploy\"\n```\n\nNote that I have also eliminated the `\"awslogs-region\": \"us-east-1\"` option from the LogConfiguration Options, since the \nconsole didn't show that option existed, so I decided to go without it. Later I found all the configuration options \n[https://docs.aws.amazon.com/batch/latest/userguide/using_awslogs.html](here).\n\n\n# Conclusions\n\nYou may have a hard time getting Batch running Fargate with CloudFormation. I hope I've saved you the pain of going through\nall the guessing, time and pain. Here you have a [complete template](batch-fagate.yaml). There are a couple of unresolved \nreferences that you need to adapt to your use case.\n\nAs for AWS:\n - The CloudFormation documentation for Fargate Batch is clearly lacking\n   - The PlatformCapabilites property which seems to be needed to get a Fargate Job Definition is not documented\n   - The ResourceRequirements documentation seems incorrect\n\n# Author, Copyright and License\n\nThis article was authored by Jose Luis Martinez Torres.\n\nThis article is (c) 2020 Jose Luis Martinez Torres, Licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\n\nThe canonical, up-to-date source is [GitHub](https://github.com/pplu/cfn-fargate-batch). Feel free to contribute back.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Fcfn-fargate-batch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpplu%2Fcfn-fargate-batch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Fcfn-fargate-batch/lists"}