{"id":22357009,"url":"https://github.com/gordonmurray/tofu_aws_apache_flink","last_synced_at":"2025-07-08T04:11:32.014Z","repository":{"id":223042969,"uuid":"758688282","full_name":"gordonmurray/tofu_aws_apache_flink","owner":"gordonmurray","description":"Using OpenTofu to create an Apache Flink cluster, with Task Managers in an auto scaling group of Spot instances to help reduce costs","archived":false,"fork":false,"pushed_at":"2024-02-17T17:23:03.000Z","size":576,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-19T21:47:20.026Z","etag":null,"topics":["ansible","apache-flink","opentofu"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gordonmurray.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-02-16T21:10:29.000Z","updated_at":"2024-02-17T19:45:33.000Z","dependencies_parsed_at":"2024-02-17T21:46:37.025Z","dependency_job_id":null,"html_url":"https://github.com/gordonmurray/tofu_aws_apache_flink","commit_stats":null,"previous_names":["gordonmurray/tofu_aws_apache_flink"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Ftofu_aws_apache_flink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Ftofu_aws_apache_flink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Ftofu_aws_apache_flink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gordonmurray%2Ftofu_aws_apache_flink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gordonmurray","download_url":"https://codeload.github.com/gordonmurray/tofu_aws_apache_flink/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245659026,"owners_count":20651526,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ansible","apache-flink","opentofu"],"created_at":"2024-12-04T14:12:45.172Z","updated_at":"2025-03-26T13:21:15.876Z","avatar_url":"https://github.com/gordonmurray.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open Tofu, AWS, Apache Flink\n\nUse OpenTofu to create an Apache Flink cluster, with Task Managers in an auto scaling group of Spot instances to help reduce costs\n\n[Terraforms provider for Ansible](https://registry.terraform.io/providers/ansible/ansible/latest/docs) is used to run an Ansible playbook which submits a Flink SQL workloads to the Flink Job Manager.\n\nA simple Flink SQL workload is added to generate some fake events.\n\n```sql\nCREATE TEMPORARY TABLE Events (\n    id INT,\n    message STRING\n) WITH (\n    'connector' = 'datagen',\n    'rows-per-second'='5',\n    'fields.id.kind'='random',\n    'fields.id.min'='1',\n    'fields.id.max'='100',\n    'fields.message.length'='10'\n);\n\nSELECT * FROM Events;\n```\n\n\n\n### 1. Create a base image AMI using Packer\n\nFirst, add your VPC IP and Subnet in to `packer/variables.json`\n\nValidate the packer file\n\n```bash\npacker validate --var-file=variables.json flink.json\n```\n\nBuild the AMI from the packer file\n\n```\npacker build --var-file=variables.json flink.json\n```\n\n\n### 2. Enable Ansible\n\nInstall Ansible\n\n```\nsudo apt install ansible -y\n```\n\nInstall the Ansible provider for Terraform\n```\nansible-galaxy collection install cloud.terraform\n```\n\n### 3. Update Variables\n\nAdd your AWS Account ID, VPC ID and subnets to `terraform.tfvars`.\n\nNext use Open Tofu to create the infrastructure:\n\n### 4. Create the cluster\n\n```\ntofu init\n\nfofu plan\n\ntofu apply\n```\n\nOnce Tofu has finished, it will show an endpoint you can open in your browser, similar to `http://localhost:8083`. This will open up the Flink UI in your browser.\n\n![Flink UI](/images/flink_ui.png)\n\nIn your AWS account you will see 3 new EC2 instances.\n\n![AWS Console](/images/ec2_instances.png)\n\n### 5. Adding Flink workloads\n\nYou can add additional Flink SQL workloads to Flink either by updating the file at `roles/flink_jobs/templates/queries.sql.j2` or create new Roles in `roles/` using the `flink_jobs` folder as a template.\n\n### Debugging Ansible playbook\n\nTofu runs the Ansible playbook automatically. If you'd like to record its output, you can set an ANSIBLE_LOG_PATH environment variable.\n\n```\nexport ANSIBLE_LOG_PATH=./ansible.log\n```\n\nIn `ansible_playbook.tf`, there are 2 variables that are useful to change if you want the playbook to run every time you run Tofu, or if you'd like Tofu to stop if there are Ansible errors.\n\n```\nreplayable              = true # set to true if you want to run the playbook every time\nignore_playbook_failure = true # set to true if you want Tofu to keep running if there is an Ansible error\n```\n\n\n### Flink cluster diagram\n\nA diagram of the resulting Flink cluster created using this Tofu project:\n\n```mermaid\ngraph TD\n    subgraph On-demand EC2 instance\n        JM[Flink Job Manager]\n    end\n    subgraph Spot group 2\n        TM1A[Task Manager]\n        TM2A[Task Manager]\n    end\n    subgraph Spot group 1\n        TM1B[Task Manager]\n        TM2B[Task Manager]\n    end\n\n    JM --\u003e|manages| TM1A\n    JM --\u003e|manages| TM2A\n    JM --\u003e|manages| TM1B\n    JM --\u003e|manages| TM2B\n```\n\n### Estimated cost\n\n```\nProject: gordonmurray/tofu_aws_apache_flink\n\n Name                                                  Monthly Qty  Unit     Monthly Cost\n\n aws_autoscaling_group.flink_taskmanagers_spot_1\n └─ aws_launch_template.flink_taskmanagers_spot_1\n    ├─ Instance usage (Linux/UNIX, spot, m7g.large)            730  hours          $23.87\n    ├─ EC2 detailed monitoring                                   7  metrics         $2.10\n    └─ block_device_mapping[0]\n       └─ Storage (general purpose SSD, gp2)                    50  GB              $5.50\n\n aws_autoscaling_group.flink_taskmanagers_spot_2\n └─ aws_launch_template.flink_taskmanagers_spot_2\n    ├─ Instance usage (Linux/UNIX, spot, m7g.xlarge)           730  hours          $55.77\n    ├─ EC2 detailed monitoring                                   7  metrics         $2.10\n    └─ block_device_mapping[0]\n       └─ Storage (general purpose SSD, gp2)                    50  GB              $5.50\n\n aws_instance.flink_job_managers[0]\n ├─ Instance usage (Linux/UNIX, on-demand, t4g.large)          730  hours          $53.73\n └─ root_block_device\n    └─ Storage (general purpose SSD, gp3)                       50  GB              $4.40\n\n OVERALL TOTAL                                                                    $152.97\n──────────────────────────────────\n17 cloud resources were detected:\n∙ 3 were estimated, all of which include usage-based costs, see https://infracost.io/usage-file\n∙ 14 were free:\n  ∙ 5 x aws_security_group_rule\n  ∙ 2 x aws_iam_role_policy_attachment\n  ∙ 2 x aws_launch_template\n  ∙ 1 x aws_iam_instance_profile\n  ∙ 1 x aws_iam_policy\n  ∙ 1 x aws_iam_role\n  ∙ 1 x aws_key_pair\n  ∙ 1 x aws_security_group\n\n┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓\n┃ Project                                            ┃ Monthly cost ┃\n┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━┫\n┃ gordonmurray/tofu_aws_apache_flink                 ┃ $153         ┃\n┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━┛\n```\n\n#### Notes\n\n* This project deploys the instances in a public subnet. For a real world deployment it is best to deploy in to a private subnet.\n* If you want to remove the Flink cluster and its resources, use `tofu destroy`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Ftofu_aws_apache_flink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgordonmurray%2Ftofu_aws_apache_flink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgordonmurray%2Ftofu_aws_apache_flink/lists"}