{"id":19771996,"url":"https://github.com/danieldacosta/apigateway-sqs-lambda","last_synced_at":"2025-02-28T04:44:18.020Z","repository":{"id":47693319,"uuid":"268384998","full_name":"DanielDaCosta/apiGateway-sqs-lambda","owner":"DanielDaCosta","description":"Setting up an API Gateway endpoint that takes records, put them into an SQS queue that triggers an Event Source for a Lambda function.","archived":false,"fork":false,"pushed_at":"2022-01-26T17:16:28.000Z","size":101,"stargazers_count":15,"open_issues_count":1,"forks_count":14,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-11T01:10:38.346Z","etag":null,"topics":["apigateway","apigateway-sqs-lambda","lambda","sqs","sqs-queue"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DanielDaCosta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-31T23:55:48.000Z","updated_at":"2024-09-26T20:44:13.000Z","dependencies_parsed_at":"2022-08-22T00:20:34.071Z","dependency_job_id":null,"html_url":"https://github.com/DanielDaCosta/apiGateway-sqs-lambda","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2FapiGateway-sqs-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2FapiGateway-sqs-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2FapiGateway-sqs-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DanielDaCosta%2FapiGateway-sqs-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DanielDaCosta","download_url":"https://codeload.github.com/DanielDaCosta/apiGateway-sqs-lambda/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241101665,"owners_count":19909943,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apigateway","apigateway-sqs-lambda","lambda","sqs","sqs-queue"],"created_at":"2024-11-12T05:05:04.427Z","updated_at":"2025-02-28T04:44:18.002Z","avatar_url":"https://github.com/DanielDaCosta.png","language":"HCL","readme":"# Terraform ApiGateway-SQS-Lambda integration\nExample of a terraform script to setup an API Gateway endpoint that takes records and puts them into an SQS queue that will trigger an Event Source for AWS Lambda.\n\nWhen deployed, you'll have a public endpoint that will write to SQS with a Lambda function that will consume from it.\n\nFor more informations check the Medium post [Building an ApiGateway-SQS-Lambda integration using Terraform](https://medium.com/@danieldacosta_75030/building-an-apigateway-sqs-lambda-integration-using-terraform-5617cc0408ad).\n\n## Getting Started\nThis project follows the following file structure:\n```\n├── LICENSE\n├── README.md\n├── apiGateway.tf\n├── iam.tf\n├── lambda: folder for lambda code\n│   ├── handler.py\n│   └── sqs-integration-dev-lambda.zip\n├── lambda.tf\n├── main.tf\n├── policies: all policies created\n│   ├── api-gateway-permission.json\n│   └── lambda-permission.json\n├── sqs.tf\n├── terraform.tfstate\n├── terraform.tfstate.backup\n├── variables.tf: defining variables that will be used inside terraform templates\n└── variables.tfvars: input variables\n```\n\n## Usage\nRun ```terraform init``` to initialize the working directory containing Terraform configuration files.\n\nFor good practices, you should confirm your changes using ```terraform plan -var-file=\"variables.tfvars\"```\n\nRun ```terraform apply -var-file=\"variables.tfvars\"``` for applying environment variables.\n\n## Details\n\n### SQS\n\nBuilding SQS\n\n```\nresource \"aws_sqs_queue\" \"queue\" {\n  name                      = \"apigateway-queue\"\n  delay_seconds             = 0\n  max_message_size          = 262144\n  message_retention_seconds = 86400\n  receive_wait_time_seconds = 10\n\n  tags = {\n    Product = local.app_name\n  }\n}\n```\n\n### IAM\n\nDefining permissions so that API Gateway has the necessary permissions to SendMessage to SQS queue.\n\n```\nresource \"aws_iam_role\" \"apiSQS\" {\n  name = \"apigateway_sqs\"\n\n  assume_role_policy = \u003c\u003cEOF\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"apigateway.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\nEOF\n}\n\ndata \"template_file\" \"gateway_policy\" {\n  template = file(\"policies/api-gateway-permission.json\")\n\n  vars = {\n    sqs_arn   = aws_sqs_queue.queue.arn\n  }\n}\n\nresource \"aws_iam_policy\" \"api_policy\" {\n  name = \"api-sqs-cloudwatch-policy\"\n\n  policy = data.template_file.gateway_policy.rendered\n}\n\n\nresource \"aws_iam_role_policy_attachment\" \"api_exec_role\" {\n  role       =  aws_iam_role.apiSQS.name\n  policy_arn =  aws_iam_policy.api_policy.arn\n}\n```\n\nAdd a Lambda permission that allows the specific SQS to invoke it\n\n```\ndata \"template_file\" \"lambda_policy\" {\n  template = file(\"policies/lambda-permission.json\")\n\n  vars = {\n    sqs_arn   = aws_sqs_queue.queue.arn\n  }\n}\n\nresource \"aws_iam_policy\" \"lambda_sqs_policy\" {\n  name        = \"lambda_policy_db\"\n  description = \"IAM policy for lambda Being invoked by SQS\"\n\n  policy = data.template_file.lambda_policy.rendered\n}\n\nresource \"aws_iam_role\" \"lambda_exec_role\" {\n  name               = \"${var.name}-lambda-db\"\n  assume_role_policy = \u003c\u003cEOF\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\nEOF\n}\n\nresource \"aws_iam_role_policy_attachment\" \"lambda_role_policy\" {\n  role       = aws_iam_role.lambda_exec_role.name\n  policy_arn = aws_iam_policy.lambda_sqs_policy.arn\n}\n```\n\n### ApiGateway\n\nCreating ApiGateway resource\n\n```\nresource \"aws_api_gateway_rest_api\" \"apiGateway\" {\n  name        = \"api-gateway-SQS\"\n  description = \"POST records to SQS queue\"\n}\n```\nAdding a resource to our root with a path of ```form_score```. Adding and validating a *query_string_parameter*.\n\n```\nresource \"aws_api_gateway_resource\" \"form_score\" {\n    rest_api_id = aws_api_gateway_rest_api.apiGateway.id\n    parent_id   = aws_api_gateway_rest_api.apiGateway.root_resource_id\n    path_part   = \"form-score\"\n}\n\nresource \"aws_api_gateway_request_validator\" \"validator_query\" {\n  name                        = \"queryValidator\"\n  rest_api_id                 = aws_api_gateway_rest_api.apiGateway.id\n  validate_request_body       = false\n  validate_request_parameters = true\n}\n\nresource \"aws_api_gateway_method\" \"method_form_score\" {\n    rest_api_id   = aws_api_gateway_rest_api.apiGateway.id\n    resource_id   = aws_api_gateway_resource.form_score.id\n    http_method   = \"POST\"\n    authorization = \"NONE\"\n\n    request_parameters = {\n      \"method.request.path.proxy\"        = false\n      \"method.request.querystring.unity\" = true\n  }\n\n  request_validator_id = aws_api_gateway_request_validator.validator_query.id\n}\n```\nDefining an ApiGateway integration with SQS. A *request_template* was added in order to pass the Method, Body, QueryParameters and path Parameters in the SQS message.\n\n```\nresource \"aws_api_gateway_integration\" \"api\" {\n  rest_api_id             = aws_api_gateway_rest_api.apiGateway.id\n  resource_id             = aws_api_gateway_resource.form_score.id\n  http_method             = aws_api_gateway_method.method_form_score.http_method\n  type                    = \"AWS\"\n  integration_http_method = \"POST\"\n  credentials             = aws_iam_role.apiSQS.arn\n  uri                     = \"arn:aws:apigateway:${var.region}:sqs:path/${aws_sqs_queue.queue.name}\"\n\n  request_parameters = {\n    \"integration.request.header.Content-Type\" = \"'application/x-www-form-urlencoded'\"\n  }\n\n  # Request Template for passing Method, Body, QueryParameters and PathParams to SQS messages\n  request_templates = {\n    \"application/json\" = \u003c\u003cEOF\nAction=SendMessage\u0026MessageBody={\n  \"method\": \"$context.httpMethod\",\n  \"body-json\" : $input.json('$'),\n  \"queryParams\": {\n    #foreach($param in $input.params().querystring.keySet())\n    \"$param\": \"$util.escapeJavaScript($input.params().querystring.get($param))\" #if($foreach.hasNext),#end\n  #end\n  },\n  \"pathParams\": {\n    #foreach($param in $input.params().path.keySet())\n    \"$param\": \"$util.escapeJavaScript($input.params().path.get($param))\" #if($foreach.hasNext),#end\n    #end\n  }\n}\"\nEOF\n  }\n\n  depends_on = [\n    aws_iam_role_policy_attachment.api_exec_role\n  ]\n}\n```\n Define a basic 200 handler for successful requests.\n ```\n # Mapping SQS Response\nresource \"aws_api_gateway_method_response\" \"http200\" {\n  rest_api_id = aws_api_gateway_rest_api.apiGateway.id\n  resource_id = aws_api_gateway_resource.form_score.id\n  http_method = aws_api_gateway_method.method_form_score.http_method\n  status_code = 200\n}\n\nresource \"aws_api_gateway_integration_response\" \"http200\" {\n  rest_api_id       = aws_api_gateway_rest_api.apiGateway.id\n  resource_id       = aws_api_gateway_resource.form_score.id\n  http_method       = aws_api_gateway_method.method_form_score.http_method\n  status_code       = aws_api_gateway_method_response.http200.status_code\n  selection_pattern = \"^2[0-9][0-9]\"                                       // regex pattern for any 200 message that comes back from SQS\n\n  depends_on = [\n    aws_api_gateway_integration.api\n    ]\n}\n ```\nFinally, we can add the API Gateway REST Deployment in order to deploy our endpoint. A redeployment trigger was added. This configuration calculates a hash of the API's Terraform resources to determine changes that should trigger a new deployment.\n ```\n resource \"aws_api_gateway_deployment\" \"api\" {\n  rest_api_id = aws_api_gateway_rest_api.apiGateway.id\n  stage_name  = var.environment\n\n  depends_on = [\n    aws_api_gateway_integration.api,\n  ]\n\n  # Redeploy when there are new updates\n  triggers = {\n    redeployment = sha1(join(\",\", list(\n      jsonencode(aws_api_gateway_integration.api),\n    )))\n  }\n\n  lifecycle {\n    create_before_destroy = true\n  }\n}\n ```\n\n### Lambda\n```\ndata \"archive_file\" \"lambda_with_dependencies\" {\n  source_dir  = \"lambda/\"\n  output_path = \"lambda/${local.app_name}-${var.lambda_name}.zip\"\n  type        = \"zip\"\n}\n\nresource \"aws_lambda_function\" \"lambda_sqs\" {\n  function_name    = \"${local.app_name}-${var.lambda_name}\"\n  handler          = \"handler.lambda_handler\"\n  role             = aws_iam_role.lambda_exec_role.arn\n  runtime          = \"python3.7\"\n\n  filename         = data.archive_file.lambda_with_dependencies.output_path\n  source_code_hash = data.archive_file.lambda_with_dependencies.output_base64sha256\n\n  timeout          = 30\n  memory_size      = 128\n\n  depends_on = [\n    aws_iam_role_policy_attachment.lambda_role_policy\n  ]\n}\n```\nLastly, adding a permission so that SQS can invoke the lambda and adding an event source so that SQS can trigger the lambda.\n```\nresource \"aws_lambda_permission\" \"allows_sqs_to_trigger_lambda\" {\n  statement_id  = \"AllowExecutionFromSQS\"\n  action        = \"lambda:InvokeFunction\"\n  function_name = aws_lambda_function.lambda_sqs.function_name\n  principal     = \"sqs.amazonaws.com\"\n  source_arn    = aws_sqs_queue.queue.arn\n}\n\n# Trigger lambda on message to SQS\nresource \"aws_lambda_event_source_mapping\" \"event_source_mapping\" {\n  batch_size       = 1\n  event_source_arn =  aws_sqs_queue.queue.arn\n  enabled          = true\n  function_name    =  aws_lambda_function.lambda_sqs.arn\n}\n```\n\n## License\nThis project is licensed under the *MIT License* - see LICENSE.md file for details\n\n## References \u0026 Acknowlegments\n\n- Special thanks to [Andrew Loesch GitHub Gist](https://gist.github.com/afloesch/dc7d8865eeb91100648330a46967be25)\n- https://www.terraform.io\n- https://medium.com/appetite-for-cloud-formation/setup-lambda-to-event-source-from-sqs-in-terraform-6187c5ac2df1\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanieldacosta%2Fapigateway-sqs-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanieldacosta%2Fapigateway-sqs-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanieldacosta%2Fapigateway-sqs-lambda/lists"}