{"id":31646446,"url":"https://github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws","last_synced_at":"2026-04-11T14:02:35.615Z","repository":{"id":313457702,"uuid":"1051470917","full_name":"sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws","owner":"sqlxpert","description":"Containerized Python API → Kafka cluster → AWS Lambda consumer, via Terraform","archived":false,"fork":false,"pushed_at":"2025-11-23T01:47:34.000Z","size":281,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-11-23T03:32:02.676Z","etag":null,"topics":["aws","aws-lambda","cloudformation","connexion","container","docker","ecr","ecs","fargate","kafka","lambda","msk","openapi","python","terraform","vpc"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sqlxpert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE_CODE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-06T04:11:20.000Z","updated_at":"2025-11-23T01:47:32.000Z","dependencies_parsed_at":"2025-09-06T08:14:43.666Z","dependency_job_id":"7aa38e3e-3667-4561-ac87-ca57596a575c","html_url":"https://github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws","commit_stats":null,"previous_names":["sqlxpert/z-container-api-kafka-aws-terraform","sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sqlxpert","download_url":"https://codeload.github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29575213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T08:38:15.585Z","status":"ssl_error","status_checked_at":"2026-02-18T08:38:14.917Z","response_time":162,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","cloudformation","connexion","container","docker","ecr","ecs","fargate","kafka","lambda","msk","openapi","python","terraform","vpc"],"created_at":"2025-10-07T05:50:18.724Z","updated_at":"2026-04-11T14:02:35.602Z","avatar_url":"https://github.com/sqlxpert.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Containerized Python API, Kafka, AWS Lambda Consumer\n\nHello! This is a high-quality **containerized Python API \u0026rarr; managed Kafka\ncluster \u0026rarr; AWS Lambda consumer function** reference architecture,\nprovisioned with Terraform. (CloudFormation is used indirectly, for a modular\nKafka consumer stack.) I hope you will be able to adapt it for your own\nprojects, under the terms of the license.\n\nJump to:\n[Installation](#installation)\n\u0026bull;\n[Recommendations](#recommendations)\n\u0026bull;\n[Licenses](#licenses)\n\n## Innovations and Best Practices\n\n\u003cdetails name=\"innovations\" open=\"true\"\u003e\n  \u003csummary\u003eLow-cost\u003c/summary\u003e\n\n- Expensive AWS resources can be toggled off during development\n- [Spot pricing reduces compute costs up to 70%](https://aws.amazon.com/fargate/pricing#Fargate_Spot_Pricing_for_Amazon_ECS)\n  even without a long-term, always-on Savings Plan commitment\n- ARM CPU architecture offers a\n  [better\u0026nbsp;price/performance\u0026nbsp;ratio](https://aws.amazon.com/ec2/graviton)\n  than Intel\n\n\u003c/details\u003e\n\n\u003cdetails name=\"innovations\"\u003e\n  \u003csummary\u003eSecure Docker container\u003c/summary\u003e\n\n- Amazon Linux starts with fewer vulnerabilities, is updated frequently by AWS\n  staff, and uses\n  [deterministic\u0026nbsp;operating\u0026nbsp;system\u0026nbsp;package\u0026nbsp;versions](https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades.html)\n- [AWS\u0026nbsp;CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html)\n  or EC2 provides a controlled, auditable environment for building container\n  images\n- The API server process runs as a non-root user, reducing the impact if it is\n  compromised\n\n\u003c/details\u003e\n\n\u003cdetails name=\"innovations\"\u003e\n  \u003csummary\u003eSecure private network\u003c/summary\u003e\n\n- Security group rules reference named security groups rather than ranges of\n  numeric addresses; only known pairs of resources can communicate\n- [PrivateLink endpoints](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-aws-services.html)\n  keep AWS API traffic off the public Internet\n- No public Internet access from private subnets\n\n\u003c/details\u003e\n\n\u003cdetails name=\"innovations\"\u003e\n  \u003csummary\u003eCompatible with continuous integration/continuous deployment (CI/CD)\u003c/summary\u003e\n\n- Getting container image build properties from Terraform\u0026nbsp;variables allows\n  separate versions for development, testing and blue/green deployment\n- [AWS\u0026nbsp;IP\u0026nbsp;Address\u0026nbsp;Manager\u0026nbsp;(IPAM)](https://docs.aws.amazon.com/vpc/latest/ipam/what-it-is-ipam.html)\n  takes a single address range input and divides the space flexibly,\n  accommodating multiple environments of different sizes\n- An AWS Lambda function test event in the\n  [shared\u0026nbsp;registry](https://builder.aws.com/content/33YuiyDjF5jHyRUhjoma00QwwbM/cloudformation-and-terraform-for-realistic-shareable-aws-lambda-test-events)\n  allows realistic, centralized testing\n- Amazon Linux on EC2 provides a consistent, central build platform\n\n\u003c/details\u003e\n\n\u003cdetails name=\"innovations\"\u003e\n  \u003csummary\u003eSmall Docker container image\u003c/summary\u003e\n\n- [Docker\u0026nbsp;cache\u0026nbsp;mounts](https://docs.docker.com/build/cache/optimize/#use-cache-mounts)\n  prevent image bloat _and_ avoid slow re-downloading on re-build (others\n  needlessly disable or clear operating system package and Python module\n  caches)\n- Temporary software is installed, used and removed in the same step, avoiding\n  extra layers _and_\n  [multi\u0026#8209;stage\u0026nbsp;build](https://docs.docker.com/build/building/multi-stage#use-multi-stage-builds)\n  complexity\n- Temporary Python modules are removed, just like temporary operating system\n  packages (others retain `pip` even though it will never be used again)\n\n\u003c/details\u003e\n\n\u003cdetails name=\"innovations\"\u003e\n  \u003csummary\u003eLow-code\u003c/summary\u003e\n\n- API methods, parameters and input validation rules are defined declaratively,\n  in a standard\n  [OpenAPI specification](https://learn.openapis.org/introduction.html#api-description-using-the-oas);\n  API code need only process requests\n- A managed container service (ECS) and a serverless computing option (Fargate)\n  reduce infrastructure-as-code lines and eliminate scripts\n- The\n  [AWS\u0026nbsp;event\u0026nbsp;source\u0026nbsp;mapping](https://docs.aws.amazon.com/lambda/latest/dg/with-msk-configure.html#msk-esm-overview)\n  interacts with Kafka, so that the consumer Lambda function need only process\n  JSON input (I re-used a simple _SQS_ consumer CloudFormation template from my\n  other projects!)\n\n\u003c/details\u003e\n\nJump to:\n[Recommendations](#recommendations)\n\u0026bull;\n[Licenses](#licenses)\n\n## Installation\n\n 1. Choose between\n    [AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html)\n    or an EC2 instance for building the Docker image and running Terraform.\n\n    - **CloudShell**\u003cbr/\u003e_Easy_ \u0026check;\n\n      - Authenticate to the AWS Console. Use a non-production AWS account and\n        a privileged role.\n\n      - Open an\n        [AWS CloudShell](https://console.aws.amazon.com/cloudshell/home)\n        terminal.\n\n      - Prepare for a cross-platform container image build. CloudShell seems to\n        provide Intel CPUs. The following instructions are from\n        \"[Multi-platform builds](https://docs.docker.com/build/building/multi-platform/#prerequisites)\"\n        in the Docker Build manual.\n\n        ```shell\n        sudo docker buildx create --name 'container-builder' --driver 'docker-container' --bootstrap --use\n\n        ```\n\n        ```shell\n        sudo docker run --privileged --rm 'tonistiigi/binfmt' --install all\n\n        ```\n\n      - Review the\n        [Terraform S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3)\n        and\n        [create an S3 bucket](https://console.aws.amazon.com/s3/bucket/create?bucketType=general)\n        to store Terraform state.\n\n      - If at any time you find that your previous CloudShell session has\n        expired, repeat any necessary software installation steps. Your home\n        directory is preserved between sessions, subject to\n        [CloudShell persistent storage limitations](https://docs.aws.amazon.com/cloudshell/latest/userguide/limits.html#persistent-storage-limitations).\n\n    - **EC2 instance**\n\n      \u003cdetails\u003e\n        \u003csummary\u003eEC2 instructions...\u003c/summary\u003e\n\n      \u003cbr/\u003e\n\n      - Create and/or connect to an EC2 instance. I recommend:\n\n        - `arm64`\n        - `t4g.micro` \u0026#9888; The ARM-based AWS Graviton `g` architecture\n          avoids multi-platform build complexity.\n        - Amazon Linux 2023\n        - A 30\u0026nbsp;GiB EBS volume, with default encryption (supports\n          hibernation)\n        - No key pair; connect through\n          [Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-sessions-start.html)\n        - A custom security group with no ingress rules (yay for Session\n          Manager!)\n        - A `sched-stop` = `d=_ H:M=07:00` tag for automatic nightly\n          shutdown (this example corresponds to midnight Pacific Daylight Time)\n          with\n          [sqlxpert/lights-off-aws](https://github.com/sqlxpert/lights-off-aws#quick-start)\n\n      - During the instance creation workflow (Advanced details \u0026rarr; IAM\n        instance profile \u0026rarr; Create new IAM profile) or afterward, give\n        your EC2 instance a custom role. Terraform must be able to\n        list/describe, get tags for, create, tag, untag, update, and delete all\n        of the AWS resource types included in this project's `.tf` files.\n\n      - Update operating system packages (thanks to AWS's\n        [deterministic upgrade philosophy](https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades.html),\n        there shouldn't be any updates if you chose the latest Amazon Linux\n        2023 image), install Docker, and start it.\n\n        ```shell\n        sudo dnf check-update\n        ```\n\n        ```shell\n        sudo dnf --releasever=latest update\n        ```\n\n        ```shell\n        sudo dnf install docker\n        ```\n\n        ```shell\n        sudo systemctl start docker\n        ```\n\n      \u003c/details\u003e\n\n 2. Install Terraform. I've standardized on\n    [Terraform v1.10.0 (2024-11-27)](https://github.com/hashicorp/terraform/releases/tag/v1.10.0)\n    as the minimum supported version for my open-source projects.\n\n    ```shell\n    sudo dnf --assumeyes install 'dnf-command(config-manager)'\n    sudo dnf config-manager --add-repo 'https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo'\n    # sudo dnf --assumeyes install terraform-1.10.0-1\n    sudo dnf --assumeyes install terraform\n\n    ```\n\n 3. Clone this repository and create `terraform.tfvars` to customize variables.\n\n    ```shell\n    git clone 'https://github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws.git' ~/docker-python-openapi-kafka\n    cd ~/docker-python-openapi-kafka/terraform\n    touch terraform.tfvars\n\n    ```\n\n    \u003cdetails\u003e\n      \u003csummary\u003eGenerate a terraform.tfvars skeleton...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    ```shell\n    # Requires an up-to-date GNU sed (not the MacOS default!)\n    sed --regexp-extended --silent  \\\n        --expression='s/^variable \"(.+)\" \\{$/\\n\\n# \\1 =/p' \\\n        --expression='s/^  description = \"(.+)\"$/#\\n# \\1/p' \\\n        --expression='s/^  default = (.+)$/#\\n# Default: \\1/p' variables.tf\n\n    ```\n\n    \u003c/details\u003e\n\n    _Optional:_ To save money while building the Docker container image, set\n    `hello_api_aws_ecs_service_desired_count_tasks = 0` and\n    `create_vpc = false`\u0026nbsp;.\n\n 4. In CloudShell (optional if you chose EC2), create an override file to\n    configure your Terraform S3 backend.\n\n    ```shell\n    cat \u003e terraform_override.tf \u003c\u003c 'EOF'\n    terraform {\n      backend \"s3\" {\n        insecure = false\n\n        region = \"RegionCodeForYourS3Bucket\"\n        bucket = \"NameOfYourS3Bucket\"\n        key    = \"DesiredTerraformStateFileName\"\n\n        use_lockfile = true # No more DynamoDB; now S3-native!\n      }\n    }\n    EOF\n\n    ```\n\n 5. Initialize Terraform and create the AWS infrastructure. There's no need for\n    a separate `terraform plan` step. `terraform apply` outputs the plan and\n    gives you a chance to approve before anything is done. If you don't like\n    the plan, don't type `yes`\u0026nbsp;!\n\n    ```shell\n    terraform init\n\n    ```\n\n    ```shell\n    terraform apply -target='aws_vpc_ipam_pool_cidr_allocation.hello_vpc_subnets'\n\n    ```\n\n    ```shell\n    terraform apply\n\n    ```\n\n    \u003cdetails\u003e\n      \u003csummary\u003eWhy two stages?\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    CloudPosse's otherwise excellent\n    [dynamic-subnets](https://registry.terraform.io/modules/cloudposse/dynamic-subnets/aws/latest)\n    module isn't dynamic enough to co-operate with\n    [AWS\u0026nbsp;IP\u0026nbsp;Address\u0026nbsp;Manager\u0026nbsp;(IPAM)](https://docs.aws.amazon.com/vpc/latest/ipam/what-it-is-ipam.html),\n    so you have _finalize_ the IPAM subnet IP address range allocations before\n    calling the module.\n\n    Creating the IPAM resource planning pool for VPC private IP addresses can\n    take as long as 30\u0026nbsp;minutes, as noted in the Terraform documentation\n    for the\n    [aws_vpc_ipam_pool](https://registry.terraform.io/providers/hashicorp/aws/6.33.0/docs/resources/vpc_ipam_pool)\n    resource.\n\n    \u003c/details\u003e\n\n    \u003cdetails\u003e\n      \u003csummary\u003eIn case of \"already exists\" errors...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    - If you receive a \"**RepositoryAlreadyExistsException**: The repository\n      with name 'hello_api' already exists\", set\n      `create_aws_ecr_repository = false`\u0026nbsp;.\n\n    - If you receive a \"**ConflictException**: Registry with name\n      lambda-testevent-schemas already exists\", set\n      `create_lambda_testevent_schema_registry = false`\u0026nbsp;.\n\n    After changing the variable(s), run `terraform apply` again.\n\n    \u003c/details\u003e\n\n 6. Set environment variables needed for building, tagging and pushing up the\n    Docker container image, then build it.\n\n    ```shell\n    BASE_AMAZONLINUX_REGISTRY_DOMAIN='public.ecr.aws'\n    BASE_AMAZONLINUX_TAG=$(terraform output -raw 'base_amazonlinux_tag')\n    BASE_AMAZONLINUX_DIGEST=$(terraform output -raw 'base_amazonlinux_digest')\n    AWS_ECR_REGISTRY_REGION=$(terraform output -raw 'hello_api_aws_ecr_registry_region')\n    AWS_ECR_REGISTRY_URI=$(terraform output -raw 'hello_api_aws_ecr_registry_uri')\n    AWS_ECR_REPOSITORY_URL=$(terraform output -raw 'hello_api_aws_ecr_repository_url')\n    HELLO_API_IMAGE_TAG=$(terraform output -raw 'hello_api_image_tag')\n\n    aws ecr-public get-login-password --region 'us-east-1' | sudo docker login --username 'AWS' --password-stdin \"${BASE_AMAZONLINUX_REGISTRY_DOMAIN}\"\n\n    aws ecr get-login-password --region \"${AWS_ECR_REGISTRY_REGION}\" | sudo docker login --username 'AWS' --password-stdin \"${AWS_ECR_REGISTRY_URI}\"\n\n    cd ../python_docker\n\n    ```\n\n    ```shell\n    sudo docker buildx build --build-arg BASE_AMAZONLINUX_REGISTRY_DOMAIN=\"${BASE_AMAZONLINUX_REGISTRY_DOMAIN}\" --build-arg BASE_AMAZONLINUX_TAG=\"${BASE_AMAZONLINUX_TAG}\" --build-arg BASE_AMAZONLINUX_DIGEST=\"${BASE_AMAZONLINUX_DIGEST}\" --platform='linux/arm64' --tag \"${AWS_ECR_REPOSITORY_URL}:${HELLO_API_IMAGE_TAG}\" --output 'type=docker' .\n\n    ```\n\n    ```shell\n    sudo docker push \"${AWS_ECR_REPOSITORY_URL}:${HELLO_API_IMAGE_TAG}\"\n\n    ```\n\n    \u003cdetails\u003e\n      \u003csummary\u003eScanning and updating the container image...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    In case you have _not_ configured ECR for automatic security scanning on\n    image push, you may be able to initiate a free operating system-level\n    vulnerability scan once per image per day. If you have opted-in to paid,\n    enhanced scanning, you cannot initiate a scan manually. See\n    [Scan images for software vulnerabilities in Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html)\n    for all options.\n\n    ```shell\n    aws ecr start-image-scan --repository-name 'hello_api' --image-id \"imageTag=${HELLO_API_IMAGE_TAG}\"\n\n    ```\n\n    Carefully review findings from a manual or automatic vulnerability scan.\n\n    ```shell\n    aws ecr describe-image-scan-findings --repository-name 'hello_api' --image-id \"imageTag=${HELLO_API_IMAGE_TAG}\"\n\n    ```\n\n    You can resolve most or all operating system-level findings by specifying\n    the version number and digest that correspond to the latest Amazon Linux\n    2023 release. Its tag is `2023`\u0026nbsp;, not the usual \"latest\". Resolving\n    Python-level findings (from a paid, enhanced scan) might be as simple as\n    re-building to pick up newer versions of secondary dependencies, or it\n    might require updating primary module version numbers, in:\n\n    - [`/python_docker/requirements.txt`](https://github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws/blob/main/python_docker/requirements.txt)\n      _or_\n    - [`/python_docker/Dockerfile`](https://github.com/sqlxpert/docker-python-openapi-kafka-terraform-cloudformation-aws/blob/main/python_docker/Dockerfile)\u0026nbsp;.\n\n    Note: For the Kafka consumer function,\n    [AWS Lambda automatically applies security updates](https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html)\n    to the Lambda runtime.\n\n    Set the `amazon_linux_base_version` and `amazon_linux_base_digest`\n    variables in Terraform, run `terraform apply`\u0026nbsp;, and re-set the\n    environment variables.\n\n    Then, to re-build the image, run `HELLO_API_IMAGE_TAG='1.0.1'`\n    (choose an appropriate new version number, taking\n    [semantic\u0026nbsp;versioning](https://semver.org/#semantic-versioning-specification-semver)\n    into account) in the shell and repeat the build and push commands.\n\n    To deploy the new image version, set\n    `hello_api_image_tag = \"1.0.1\"` (for example) in Terraform and run\n    `terraform apply` one more time.\n\n    \u003c/details\u003e\n\n 7. _If_ you changed Terraform variables at the end of Step\u0026nbsp;3, revert the\n    changes and run _both_ `terraform apply` commands from Step\u0026nbsp;5.\n\n 8. In the Amazon Elastic Container Service section of the AWS Console, check\n    the `hello_api` cluster. Eventually, you should see 2\u0026nbsp;tasks running.\n\n    - It will take a few minutes for ECS to notice, and then deploy, the\n      container image. Relax, and let it happen. If you are impatient, or if\n      there is a problem, you can navigate to the `hello_api` service, open the\n      orange \"Update service\" pop-up menu, and select \"Force new deployment\".\n\n 9. Generate the URLs and then test your API.\n\n    ```shell\n    cd ../terraform\n    HELLO_API_DOMAIN_NAME=$(terraform output -raw 'hello_api_load_balander_domain_name')\n    echo -e \"curl --location --insecure 'http://${HELLO_API_DOMAIN_NAME}/\"{'healthcheck','hello','current_time?name=Paul','current_time?name=;echo','error'}\"'\\n\"\n\n    ```\n\n    Try the different URLs using your Web browser or\n    `curl --location --insecure`\n    (these options allow redirection and self-signed TLS certificates).\n\n    |Method, parameters|Result expected|\n    |:---|:---|\n    |`/healthcheck`|Empty response|\n    |`/hello`|Fixed greeting, in a JSON object|\n    |`/current_time?name=Paul`|Reflected greeting and timestamp, in a JSON object|\n    |`/current_time?name=;echo`|HTTP `400` \"bad request\" error;\u003cbr/\u003eDemonstrates protection from command injection|\n    |`/error`|HTTP `404` \"not found\" error|\n\n    \u003cdetails\u003e\n      \u003csummary\u003eAbout redirection to HTTPS, and certificates...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    Your Web browser should redirect you from `http:` to `https:` and (let's\n    hope!) warn you about the untrusted, self-signed TLS certificate used in\n    this system (which of course is not tied to a pre-determined domain name).\n    Proceed to view the responses from your new API...\n\n    If your browser configuration does not allow accessing Web sites with\n    untrusted certificates, change the `enable_https` variable to `false` and\n    run `terraform apply`\u0026nbsp;. Now, `http:` links will work without\n    redirection. After you have used `https:` with a particular domain, your\n    browser might no longer allow `http:`\u0026nbsp;. Try with another browser.\n\n    \u003c/details\u003e\n\n10. Access the\n    [`/hello/hello_api_web_log`](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups/log-group/$252Fhello$252Fhello_api_web_log)\n    CloudWatch log group in the AWS Console.\n\n    Periodic internal health checks, plus your occasional Web requests, should\n    appear.\n\n    \u003cdetails\u003e\n      \u003csummary\u003eAPI access log limitations...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    The Python\n    [connexion](https://connexion.readthedocs.io/en/stable)\n    module, which I chose because it serves an API from a precise\n    [OpenAPI-format specification](https://learn.openapis.org/introduction.html#api-description-using-the-oas),\n    uses\n    [uvicorn](https://uvicorn.dev)\n    workers. Unfortunately,\n    [uvicorn has lousy log format customization support](https://github.com/Kludex/uvicorn/issues/527).\n\n    \u003c/details\u003e\n\n11. If you wish to run commands remotely, or to open an interactive shell\n    inside a `hello_api` container, use\n    [ECS Exec](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html).\n\n    \u003cdetails\u003e\n      \u003csummary\u003eECS Exec instructions...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    Change the `enable_ecs_exec` variable to `true`\u0026nbsp;, run\n    `terraform apply`\u0026nbsp;, and replace the container(s) using \"Force new\n    deployment\", as explained at the end of Step\u0026nbsp;8.\n\n    In the Amazon Elastic Container Service section of the AWS Console, click\n    `hello_api` to open the cluster's page. Open the \"Tasks\" tab and click an\n    identifier in the \"Task\" column. Under \"Containers\", select the container,\n    then click \"Connect\". Confirm the command that will be executed.\n\n    You can also use the AWS command-line interface from your main CloudShell\n    session (or, with sufficient permissions, from an EC2 instance if you chose\n    to deploy from EC2).\n\n    ```shell\n    aws ecs list-tasks --cluster 'hello_api' --query 'taskArns' --output text\n    read -p 'Task ID: ' HELLO_API_ECS_TASK_ID\n    aws ecs execute-command --cluster 'hello_api' --task \"${HELLO_API_ECS_TASK_ID}\" --interactive --command '/bin/bash'\n    ```\n\n    Activities are logged in the\n    [`/hello/hello_api_ecs_exec_log`](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups/log-group/$252Fhello$252Fhello_api_ecs_exec_log)\n    CloudWatch log group.\n\n    \u003c/details\u003e\n\n12. If you don't wish use Kafka, skip to Step\u0026nbsp;14.\n\n    AWS MSK is expensive, so enable Kafka only after confirming that the rest\n    of the system is working for you.\n\n    Before proceeding, consider whether you need the extra features that\n    [provisioned polling](https://docs.aws.amazon.com/lambda/latest/dg/kafka-scaling-modes.html)\n    provides, and whether you accept the extra cost. If yes, change\n    `msk_provisioned_poll` to `true`\u0026nbsp;.\n\n    To add the Kafka-related infrastructure, change `enable_kafka` to `true`\n    and run `terraform apply`\u0026nbsp;.\n\n    \u003cdetails\u003e\n      \u003csummary\u003eIn case HelloApiKafkaConsumer CloudFormation stack creation fails...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    Creation of the Kafka consumer might fail for various reasons. Once the\n    `HelloApiKafkaConsumer` CloudFormation stack is in `ROLLBACK_COMPLETE`\n    status, delete it, then run `terraform apply` again.\n\n    \u003c/details\u003e\n\n13. Access the `/current_time?name=Paul` method several times (adjust the\n    `name` parameter as you wish). The first use of this method prompts\n    creation of the `events` Kafka topic. From now on, use of this method (not\n    the other methods) will send a message to the `events` Kafka topic.\n\n    The [AWS MSK event source mapping](https://docs.aws.amazon.com/lambda/latest/dg/with-msk-configure.html#msk-esm-overview)\n    reads from the Kafka topic and triggers the consumer Lambda function, which\n    logs decoded Kafka messages to the\n    [HelloApiKafkaConsumer CloudWatch log group](https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups$3FlogGroupNameFilter$3DHelloApiKafkaConsumer-LambdaFnLogGrp-).\n\n    If you chose provisioned polling, an additional log stream in the same log\n    group captures event source mapping status, and special CloudWatch metrics\n    are available. To view the special metrics in the AWS Console, open the\n    [HelloApiKafkaConsumer Lambda function](https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions?fo=and\u0026o0=%3A\u0026v0=HelloApiKafkaConsumer-),\n    click on the \"MSK\" trigger, click on the event source mapping UUID at the\n    bottom, and switch to the \"Monitor\" tab.\n\n14. If you wish to work on the code, you can save money by temporarily deleting\n    expensive AWS resources. The order of the following changes matters. Run\n    `terraform apply` after changing each variable.\n\n    - `hello_api_aws_ecs_service_desired_count_tasks` \u0026rarr; `0`\n    - `enable_kafka` \u0026rarr; `false`\n    - `create_vpc` \u0026rarr; `false`\n\n    When you are finished, delete all resources; the minimum configuration\n    carries a cost.\n\n    If you will be using the container image again soon, you can preserve the\n    Elastic Container Registry repository (at a cost) by removing it from\n    Terraform state.\n\n    ```shell\n    cd ../terraform\n    terraform state rm 'aws_schemas_registry.lambda_testevent'\n    # terraform state rm 'aws_ecr_repository.hello' 'aws_ecr_lifecycle_policy.hello' 'data.aws_ecr_lifecycle_policy_document.hello'\n    terraform apply -destroy\n    ```\n\n    \u003cdetails\u003e\n      \u003csummary\u003eDeletion delays and errors...\u003c/summary\u003e\n\n    \u003cbr/\u003e\n\n    - Harmless \"Invalid target address\" errors will occur in some\n      configurations.\n    - A _newly-created_ ECR repository is deleted along with any images (unless\n      you explicitly removed it from Terraform state), but if you _imported_\n      your previously-created ECR repository and it contains images, you will\n      receive a \"**RepositoryNotEmptyException**\". Either delete the images or\n      remove the ECR repository from Terraform state. Run\n      `terraform apply -destroy` again.\n    - Deleting IP Address Manager (IPAM) pools takes a long time; expect\n      30\u0026nbsp;minutes if `create_vpc` was `true`\u0026nbsp;.\n    - Deleting a VPC Lambda function takes a long time because of the network\n      association; expect 30\u0026nbsp;minutes if `enable_kafka` was `true`\u0026nbsp;.\n    - Expect an error message about retiring KMS encryption key grants\n      (harmless, in this case).\n    - If you cancel and re-run `terraform apply -destroy`\u0026nbsp;, a bug in\n      CloudPosse's `dynamic-subnets` module might cause a \"value depends on\n      resource attributes that cannot be determined until apply\" error. For a\n      work-around, edit the cached module file indicated in the error message.\n      Comment out the indicated line and force `count = 0`\u0026nbsp;. Be sure to\n      revert this temporary patch later.\n\n    \u003c/details\u003e\n\n## Comments\n\n### Artificial Intelligence and Large Language Models (LLMs)\n\nThis is my own original work, produced _without_ the use of artificial\nintelligence (AI) and large language model (LLM) code generation. Code from\nother sources is acknowledged.\n\n### Long Option Names\n\nI write long option names in my instructions so that other people don't have to\nlook up unfamiliar single-letter options \u0026mdash; assuming they can _find_ them!\n\nHere's an example that shows why I go to the trouble, even at the expense of\nbeing laughed at by macho Linux users. I started using\n[UNICOS](https://en.wikipedia.org/wiki/UNICOS)\nin 1991, so it's not for lack of experience.\n\n\u003e Search for the literal text `-t` in\n[docs.docker.com/reference/cli/docker/buildx/build](https://docs.docker.com/reference/cli/docker/buildx/build/)\u0026nbsp;,\nusing Command-F, Control-F, `/`\u0026nbsp;, or `grep`\u0026nbsp;. Only\n2\u0026nbsp;of\u0026nbsp;41\u0026nbsp;occurrences of `-t` are relevant!\n\nWhere available, full-text (that is, not strictly literal) search engines\ncan't make sense of a 1-letter search term and are also likely to ignore a\n2-character term as a \"stop-word\" that's too short to search for.\n\n### Recommendations\n\nMy professional and ethical commitment is simple: Only as much technology as a\nbusiness...\n\n- needs,\n- can afford,\n- understands (or can learn), and\n- can maintain.\n\nHaving worked for startups since 2013, I always recommend focusing software\nengineering effort. It is not possible to do everything, let alone to be good\nat everything. Managed services, serverless technology, and low-code\narchitecture free software engineers to _focus on the core product, that is, on\nwhat the company actually sells_. Avoid complex infrastructure and tooling\nunless it offers a unique, tangible, and substantial benefit. Simplicity pays!\n\nSecurity is easier and cheaper to incorporate at the start than to graft on\nafter the architecture has been finalized, the infrastructure has been\ntemplated and created, and the executable code has been written and deployed.\n\nSpecialized knowledge of the chosen cloud provider is indispensable. I call it\n\"idiomatic\" knowledge, a good part of which is awareness of the _range of\noptions_ supported by your cloud provider. Building generically would mean\ngiving up some performance, some security, and some cloud cost savings.\nOptimizing later is difficult. \"Lean to steer the ship you're on.\"\n\n## Licenses\n\n|Scope|Link|Included Copy|\n|:---|:---|:---|\n|Source code, and source code in documentation|[GNU General Public License (GPL) 3.0](http://www.gnu.org/licenses/gpl-3.0.html)|[LICENSE_CODE.md](/LICENSE_CODE.md)|\n|Documentation, including this ReadMe file|[GNU Free Documentation License (FDL) 1.3](http://www.gnu.org/licenses/fdl-1.3.html)|[LICENSE_DOC.md](/LICENSE_DOC.md)|\n\nCopyright Paul Marcelin\n\nContact: `marcelin` at `cmu.edu` (replace \"at\" with `@`)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqlxpert%2Fdocker-python-openapi-kafka-terraform-cloudformation-aws/lists"}