{"id":27100620,"url":"https://github.com/halimdakir/serverless-image-processing-on-aws","last_synced_at":"2025-04-09T20:59:48.508Z","repository":{"id":285935971,"uuid":"959815332","full_name":"halimdakir/Serverless-Image-Processing-on-AWS","owner":"halimdakir","description":"A fully containerized, event-driven image processing pipeline built with Go, AWS Lambda, and Amazon Rekognition. ","archived":false,"fork":false,"pushed_at":"2025-04-03T12:25:11.000Z","size":405,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"Halim","last_synced_at":"2025-04-03T13:21:58.959Z","etag":null,"topics":["amazon-rekognition","amazon-s3","api-gateway","aws","aws-lambda","benchmark","cloudwatch","docker","event-driven","golang","image-processing","serverless"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/halimdakir.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-04-03T11:55:02.000Z","updated_at":"2025-04-03T12:25:15.000Z","dependencies_parsed_at":"2025-04-03T13:22:04.766Z","dependency_job_id":"f6f98aa6-9814-4ae1-aa4f-86d9d83ec31e","html_url":"https://github.com/halimdakir/Serverless-Image-Processing-on-AWS","commit_stats":null,"previous_names":["halimdakir/serverless-image-processing-on-aws"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halimdakir%2FServerless-Image-Processing-on-AWS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halimdakir%2FServerless-Image-Processing-on-AWS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halimdakir%2FServerless-Image-Processing-on-AWS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halimdakir%2FServerless-Image-Processing-on-AWS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/halimdakir","download_url":"https://codeload.github.com/halimdakir/Serverless-Image-Processing-on-AWS/tar.gz/refs/heads/Halim","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247485662,"owners_count":20946451,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-rekognition","amazon-s3","api-gateway","aws","aws-lambda","benchmark","cloudwatch","docker","event-driven","golang","image-processing","serverless"],"created_at":"2025-04-06T13:20:08.922Z","updated_at":"2025-04-06T13:20:09.640Z","avatar_url":"https://github.com/halimdakir.png","language":"Go","readme":"# AWS Lambda Image Uploader \u0026 Processor\n\nThis repository includes two AWS Lambda functions written in Go and deployed using Docker containers:\n\n1. **Image Uploader Lambda**  \n   - Triggered via HTTP (API Gateway)\n   - Accepts base64-encoded image via POST request\n   - Uploads the image to an **input S3 bucket**\n\n2. **Image Processor Lambda**  \n   - Triggered by S3 `ObjectCreated` event\n   - Resizes and grayscales the image\n   - Uses Amazon Rekognition for object detection and OCR\n   - Saves the processed image and JSON metadata to a **result S3 bucket**\n\n\n---\n\n## Architecture Overview\n\nThe following diagram provides a high-level view of how the system components interact from upload to processing:\n![Workflow Diagram](https://github.com/user-attachments/assets/eb217a9a-08c2-41fd-8dcb-c3b918a035c5)\n\n\n\n\n\n---\n\n## Prerequisites\n\n- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)\n- [Docker](https://docs.docker.com/get-docker/)\n- IAM Role (`lambda-execution-role`) with necessary permissions for:\n  - `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`\n  - `logs:*`\n  - `rekognition:*` (for processor only)\n\n\u003e Replace all `{{PLACEHOLDER}}` values with your actual AWS settings.\n\n---\n\n## Folder Structure\n\n``` bash\n/processor-lambda/\n    Dockerfile               # Docker build file for processor Lambda\n    go.mod                   # Go module file\n    main.go                  # Lambda code: resize, grayscale, Rekognition\n    \n/results/                # Local test outputs (before/after images + JSON)\n    bird.png\n    bird (original).png\n    bird (processed).png\n    bird.json\n    formula1.png\n    formula1 (original).png\n    formula1 (processed).png\n    formula1.json\n\n/uploader-lambda/\n    Dockerfile               # Docker build file for uploader Lambda\n    go.mod                   # Go module file\n    main.go                  # Lambda code: decode + upload to S3\n\nbenchmark_test.sh        # ApacheBench benchmarking script\nconvert_image_to_base64.py  # Script to create payload.json from an image\nformula1.png             # Sample test image\npayload.json             # Test input for uploader Lambda\nqueries_cloudwatch.pl    # Saved CloudWatch queries (optional)\n/README.md                 # Project documentation\n```\n\n\n## Deployment Steps\n\n### 1. Build the Docker Image\n\n```bash\n# Uploader\ncd uploader-lambda\ndocker build -t image-uploader .\n\n# Processor\ncd ../processor-lambda\ndocker build -t image-processor .\n```\n\n---\n\n### 2. Authenticate with ECR\n\n```bash\naws ecr get-login-password --region {{REGION}} | docker login --username AWS --password-stdin {{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com\n```\n\n---\n\n### 3. Create ECR Repositories\n\n```bash\naws ecr create-repository --repository-name image-uploader --region {{REGION}}\naws ecr create-repository --repository-name image-processor --region {{REGION}}\n```\n\n---\n\n### 4. Tag and Push Docker Images\n\n```bash\n# Uploader\ndocker tag image-uploader:latest {{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-uploader:latest\ndocker push {{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-uploader:latest\n\n# Processor\ndocker tag image-processor:latest {{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-processor:latest\ndocker push {{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-processor:latest\n```\n\n---\n\n### 5. Create Lambda Functions\n\n```bash\n# Uploader\naws lambda create-function \\\n  --function-name image-uploader-lambda \\\n  --package-type Image \\\n  --code ImageUri={{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-uploader:latest \\\n  --role arn:aws:iam::{{ACCOUNT_ID}}:role/lambda-execution-role \\\n  --region {{REGION}}\n\n# Processor\naws lambda create-function \\\n  --function-name image-processor-lambda \\\n  --package-type Image \\\n  --code ImageUri={{ACCOUNT_ID}}.dkr.ecr.{{REGION}}.amazonaws.com/image-processor:latest \\\n  --role arn:aws:iam::{{ACCOUNT_ID}}:role/lambda-execution-role \\\n  --region {{REGION}}\n```\n\n---\n\n### 6. S3 Trigger Setup (for Processor)\n\n```bash\naws s3api put-bucket-notification-configuration \\\n  --bucket {{INPUT_BUCKET_NAME}} \\\n  --notification-configuration '{\n    \"LambdaFunctionConfigurations\": [\n      {\n        \"LambdaFunctionArn\": \"arn:aws:lambda:{{REGION}}:{{ACCOUNT_ID}}:function:image-processor-lambda\",\n        \"Events\": [\"s3:ObjectCreated:*\"]\n      }\n    ]\n  }'\n```\n\n---\n\n## Sample IAM Role Policy\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": \"s3:PutObject\",\n      \"Resource\": [\n        \"arn:aws:s3:::{{INPUT_BUCKET_NAME}}/*\",\n        \"arn:aws:s3:::{{RESULT_BUCKET_NAME}}/*\"\n      ]\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\"s3:GetObject\", \"s3:ListBucket\"],\n      \"Resource\": [\n        \"arn:aws:s3:::{{INPUT_BUCKET_NAME}}\",\n        \"arn:aws:s3:::{{INPUT_BUCKET_NAME}}/*\"\n      ]\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"rekognition:DetectLabels\",\n        \"rekognition:DetectText\"\n      ],\n      \"Resource\": \"*\"\n    },\n    {\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"logs:CreateLogGroup\",\n        \"logs:CreateLogStream\",\n        \"logs:PutLogEvents\"\n      ],\n      \"Resource\": \"arn:aws:logs:*:*:*\"\n    }\n  ]\n}\n```\n\n---\n\n## Testing\n#### Sending Payload to the Uploader Lambda\n\nTo test the uploader Lambda, you need a valid `payload.json` with a base64-encoded image.  \nUse the provided helper script (`convert_image_to_base64.py`) or the example below to generate it:\n\n```bash\npython generate_payload.py\n```\n\n\u003e This script reads an image file (e.g., `formula1.png`), encodes it in base64, and creates `payload.json`.\n\nThen test using:\n```bash\ncurl -X POST https://{{API_GATEWAY_ENDPOINT}} \\\n     -H \"Content-Type: application/json\" \\\n     -d @payload.json\n```\n\n---\n\n### Performance Benchmarking \u0026 Cloud Monitoring\n\n#### Load Testing with ApacheBench (ab)\n\nYou can test the scalability of the uploader API using the provided benchmarking script (`benchmark.sh`). It simulates increasing and decreasing workloads.\n\nTo run:\n```bash\nchmod +x benchmark.sh\n./benchmark.sh\n```\n\nThis will:\n- Hit the API Gateway with multiple concurrency levels\n- Save results in files like `result_scale_out_100.txt`\n\n#### CloudWatch Log Insights (Lambda Monitoring)\n\nUse these queries in the [CloudWatch Logs Insights](https://console.aws.amazon.com/cloudwatch/home#logs-insights:) console for detailed Lambda metrics:\n\n- **Total Invocations**\n  ```sql\n  fields @timestamp, @message\n  | filter @message like /REPORT RequestId/\n  | stats count(*) as totalInvocations\n  ```\n\n- **Timeouts**\n  ```sql\n  fields @timestamp, @message\n  | filter @message like /Task timed out/\n  | stats count(*) as timeouts\n  ```\n\n- **Cold Starts**\n  ```sql\n  fields @timestamp, @message\n  | filter @message like /INIT_START/\n  | stats count(*) as coldStarts\n  ```\n\n- **Performance Stats (Duration \u0026 Memory)**\n  ```sql\n  fields @timestamp, @message\n  | filter @message like /REPORT RequestId/\n  | parse @message /Duration: (?\u003cduration\u003e[0-9.]+) ms.*Max Memory Used: (?\u003cmaxMem\u003e\\d+) MB/\n  | stats\n      avg(duration) as avgDurationMs,\n      min(duration) as minDurationMs,\n      max(duration) as maxDurationMs,\n      avg(maxMem) as avgMemoryMB,\n      min(maxMem) as minMemoryMB,\n      max(maxMem) as maxMemoryMB\n  ```\n\n---\n\n## Summary\n\nThis solution demonstrates a fully containerized, event-driven image processing pipeline using AWS Lambda, S3, and Rekognition — written in Go, deployed with Docker, and benchmarked for scalability.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalimdakir%2Fserverless-image-processing-on-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhalimdakir%2Fserverless-image-processing-on-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalimdakir%2Fserverless-image-processing-on-aws/lists"}