{"id":50938385,"url":"https://github.com/cebert/aws-lambda-performance-benchmarks","last_synced_at":"2026-06-17T11:31:37.356Z","repository":{"id":324644874,"uuid":"1097935083","full_name":"cebert/aws-lambda-performance-benchmarks","owner":"cebert","description":"Comprehensive performance benchmark comparing AWS Lambda Arm (Graviton) vs x86 across Python 3.14/3.13/3.12/3.11, Node.js 22/20, and Rust","archived":false,"fork":false,"pushed_at":"2025-12-02T15:52:00.000Z","size":54279,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-05T13:53:02.690Z","etag":null,"topics":["arm-vs-x86","aws-cdk","aws-lambda","graviton","performance-benchmarking","rust","serverless"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cebert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":".github/SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-17T03:30:51.000Z","updated_at":"2025-12-03T17:19:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cebert/aws-lambda-performance-benchmarks","commit_stats":null,"previous_names":["cebert/aws-lambda-performance-benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cebert/aws-lambda-performance-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cebert%2Faws-lambda-performance-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cebert%2Faws-lambda-performance-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cebert%2Faws-lambda-performance-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cebert%2Faws-lambda-performance-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cebert","download_url":"https://codeload.github.com/cebert/aws-lambda-performance-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cebert%2Faws-lambda-performance-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34447264,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arm-vs-x86","aws-cdk","aws-lambda","graviton","performance-benchmarking","rust","serverless"],"created_at":"2026-06-17T11:31:36.491Z","updated_at":"2026-06-17T11:31:37.348Z","avatar_url":"https://github.com/cebert.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS Lambda ARM vs x86 Benchmark\n\nThis repository contains a benchmark suite comparing AWS Lambda cost and performance on **x86_64** and **ARM (Graviton2 Neoverse N1)** CPU architectures as of late 2025. It is inspired by the 2023 AWS blog post [Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis](https://aws.amazon.com/blogs/apn/comparing-aws-lambda-arm-vs-x86-performance-cost-and-analysis-2/), which uses older, now-unsupported runtimes.\n\nThis benchmark reimplements similar **single-threaded workloads** in **Node.js, Python, and Rust** using **currently supported AWS runtimes**, with a focus on realistic but controlled workloads and repeatable measurement.\n\n## Supported Runtimes\n\n- **Python:** 3.14, 3.13, 3.12, 3.11\n- **Node.js:** 22, 20\n- **Rust:** Custom runtime on al2023 (Amazon Linux 2023) + `rustc 1.91.0`\n\n## Workloads\n\nSee [docs/benchmark-design.md](./docs/benchmark-design.md) for full details.\n\n- **CPU-intensive**\n  - SHA-256 hashing loop (500k iterations) to exercise raw compute performance.\n  - No dependency on AWS SDKs.\n\n- **Memory-intensive**\n  - Allocates a fixed 100 MB data structure and sorts it as a memory-heavy workload.\n  - No dependency on AWS SDKs.\n\n- **Light (I/O)**\n  - Writes 5 items to DynamoDB in a batch, then reads 5 items in a batch.\n  - Uses the AWS SDK and represents a realistic, light \"business logic + I/O\" workload.\n\n**Test Matrix:**\n\n- 7 runtimes × 2 architectures (ARM64/x86_64) × 3 workloads = 42 Lambda functions\n- 3 workload types: CPU-intensive (SHA-256 hashing), Memory-intensive (array sorting), Light (DynamoDB I/O)\n- Multiple memory configurations (128 MB to 10240 MB)\n- Cold start vs warm start measurements\n\n**Architecture:**\n\n- Zero-overhead metrics collection (CloudWatch REPORT parsing)\n- Dynamic memory configuration (no redeployment needed)\n- DynamoDB storage with pre-computed aggregates\n- Parallel execution across all functions  \n\n📖 Full design documentation in [/docs](./docs)\n\n## Published Results\n\nPre-computed benchmark results are available in the [published-results](./published-results) folder:\n\n- **[December 2025](./published-results/december-2025/)** - Ran a new modified run run with 94,080 invocations (20 cold + 300 warm per config) to benchmark Rust improvements. The focus was to confirm if these changes improved Rust arm performance for the CPU intensive tests. Cold start benchmarks ended up being a less interesting aspect of this benchmark, so I reduced them in this run so that the benchmark could be completed faster.\n  - Includes **Rust ARM64 4-5x performance improvement** via assembly-optimized SHA-256 hashing\n  - Key change: Enabled `asm` feature on `sha2` crate to leverage NEON SIMD instructions on ARM64\n  - Thanks to [Khawaja Shams](https://github.com/khawajashams) for the optimization suggestion\n  - See [Rust SSE Optimization Comparison chart](./published-results/december-2025/charts/rust-sse-optimization-comparison.png)\n\n- **[November 2025](./published-results/november-2025/)** - Production run with 183,750 invocations (125 cold + 500 warm per config)\n  - 7 runtimes (Python 3.11-3.14, Node.js 20/22, Rust)\n  - ARM64 vs x86_64 comparisons\n  - Memory scaling from 128 MB to 10,240 MB\n\nFor analysis and insights, see the accompanying blog post: [Comparing AWS Lambda ARM64 vs x86_64 Performance Across Multiple Runtimes in Late 2025](https://chrisebert.net/comparing-aws-lambda-arm64-vs-x86_64-performance-across-multiple-runtimes-in-late-2025/)\n\n## Quick Start\n\n### Prerequisites\n\n- AWS account with credentials configured (`aws configure`)\n- Node.js 18+\n- Python 3.11+ (via `uv` - recommended)\n- AWS CDK CLI: `npm install -g aws-cdk`\n- [UV package manager](https://docs.astral.sh/uv/getting-started/installation/)\n- For Rust handlers: cargo-lambda or Docker (automatic via CDK)\n  \n\n### Install\n\n```bash\ngit  clone \u003cthis-repo\u003e\ncd  aws-lambda-performance-benchmakrs\nnpm  install \u0026\u0026 npm  run  build\n```  \n\n### Deploy\n \n```bash\nnpm  run  deploy\n```\n\nDeploys 42 Lambda functions, 2 DynamoDB tables, and supporting infrastructure to `us-east-2` (configurable via `AWS_REGION`).\n\n### Cleanup\n\n```bash\ncdk  destroy  LambdaBenchmarkStack\n```\n\n## Run Benchmarks\n\nChoose a test mode based on your needs:\n\n**Test mode** (quick validation - 2 cold + 2 warm per config, ~10 min):\n\n```bash\nnpm  run  benchmark:test\n```\n\n**Balanced mode** (recommended for publication - 50 cold + 200 warm per config, ~$2-4):\n\n```bash\nuv  run  python  scripts/benchmark_orchestrator.py  --balanced\n```\n\n**Production mode** (maximum rigor - 100 cold + 500 warm per config, ~18-24 hours, ~$5-10):\n\n```bash\nuv  run  python  scripts/benchmark_orchestrator.py  --production\n```\n\nEach mode tests all 42 functions across multiple memory configurations. Higher modes provide better statistical confidence.\n\n### Running Long Benchmarks (Balanced/Production Mode)\n\n**IMPORTANT:** For Balanced (~1 hour) and Production (several hours) modes, AWS SSO tokens may expire mid-test, causing benchmark failures.\n\n**Recommended Solution:** Run benchmarks on an EC2 instance with an IAM instance profile (no credential expiration).\n\n**Prerequisites:**\n- LambdaBenchmarkStack must already be deployed (`npm run deploy`)\n- IAM permissions to create EC2 instances, IAM roles, and security groups\n\n**Usage:**\n```bash\n# Launch EC2 instance that auto-runs benchmark and terminates when complete\nuv run python scripts/run_benchmark_on_ec2.py --mode balanced\n# Production mode (several hours)\nuv run python scripts/run_benchmark_on_ec2.py --mode production\n# Keep instance alive for debugging\nuv run python scripts/run_benchmark_on_ec2.py --mode balanced --keep-alive\n# Upload results to S3\nuv run python scripts/run_benchmark_on_ec2.py --mode production --s3-bucket my-results-bucket\n```\n\n**Benefits:**\n- No SSO token expiration issues\n- Runs in background (immune to laptop sleep/network issues)\n- Auto-terminates after completion (unless `--keep-alive` specified)\n- Cost-effective: t4g.micro instance\n\n**Monitor progress:**\n```bash\n# View instance status\naws ec2 describe-instance-status --instance-ids \u003cinstance-id\u003e\n# Stream benchmark logs (once instance is running)\naws logs tail /var/log/cloud-init-output.log --follow\n# SSH into instance (requires AWS Systems Manager)\naws ssm start-session --target \u003cinstance-id\u003e\n```\n\n**Alternative (Local Execution):** If running locally, extend your SSO session duration before starting:\n```bash\naws sso login --profile \u003cyour-profile\u003e  # Refresh token before benchmark\n```\n\nNote: SSO session duration is configurable in your AWS SSO settings (typically 8-12 hours max). For Production mode (18-24 hours), EC2 execution is strongly recommended.\n\n## Analyze Results\n\nAfter running a benchmark, analyze the results:\n\n```bash\n# Analyze all results for a test run\nuv  run  python  scripts/analyze_results.py \u003ctest-run-id\u003e\n# Filter by specific dimensions\nuv  run  python  scripts/analyze_results.py \u003ctest-run-id\u003e --runtime  python3.13\nuv  run  python  scripts/analyze_results.py \u003ctest-run-id\u003e --workload  cpu-intensive\nuv  run  python  scripts/analyze_results.py \u003ctest-run-id\u003e --architecture  arm64\n```\n\n**Output includes:**\n\n- Performance comparisons (ARM vs x86, runtime vs runtime)\n- Cold start vs warm start analysis\n- Memory scaling charts\n- Cost efficiency calculations\n- Statistical summaries (mean, median, p50/p90/p95/p99) \n\n## Metrics Collected\n\nFor each invocation, the benchmark extracts from CloudWatch REPORT logs:\n-  **Execution duration** (ms) - Actual function runtime\n-  **Billed duration** (ms) - Rounded for AWS billing\n-  **Max memory used** (MB) - Peak memory consumption\n-  **Init duration** (ms) - Cold start initialization time (cold starts only)\n-  **Client latency** (ms) - End-to-end time from orchestrator\n\nResults are stored in DynamoDB with pre-computed aggregates (mean, median, p50/p90/p95/p99, std dev) for fast analysis.\n\n**Zero overhead:** Metrics are parsed from CloudWatch logs, NOT measured in-function, ensuring no performance impact on benchmarks.\n\n## Documentation\n\n**Design \u0026 Architecture:**\n- [docs/benchmark-design.md](./docs/benchmark-design.md) - Test matrix, workloads, orchestration\n- [DECISIONS.md](./DECISIONS.md) - Architectural decision records (ADRs)\n\n**Implementation:**\n- [docs/handler-api-spec.md](./docs/handler-api-spec.md) - Lambda handler contract\n- [docs/metrics-collection-implementation.md](./docs/metrics-collection-implementation.md) - CloudWatch REPORT parsing\n- [docs/dynamodb-schema.md](./docs/dynamodb-schema.md) - Results storage schema\n\n## Contributing\n\nThis is a research benchmark project. Feel free to fork it, or if you find issues or have suggestions:\n\n1. Check existing ADRs in [DECISIONS.md](./DECISIONS.md) to understand design rationale\n2. Review [benchmark-design.md](./docs/benchmark-design.md) for methodology\n3. Open an issue with your findings or proposed changes\n\n## Credits \u0026 References\n\n-  **2023 AWS blog benchmark:** [Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis](https://aws.amazon.com/blogs/apn/comparing-aws-lambda-arm-vs-x86-performance-cost-and-analysis-2/)\n-  **Forced cold start technique:** [AJ Stuyvenberg's cold-start-benchmarker](https://github.com/astuyve/cold-start-benchmarker)\n   - This approach helps us test cold start invocations much faster than having to re-deploy Lambdas or wait for all instances to become cold.\n-  **Rust Lambda support:** [Building serverless applications with Rust on AWS Lambda](https://aws.amazon.com/blogs/compute/building-serverless-applications-with-rust-on-aws-lambda/)\n-  **cargo-lambda-cdk:** [cargo-lambda/cargo-lambda-cdk](https://github.com/cargo-lambda/cargo-lambda-cdk)\n\n## Development Tools\n\nThis project was built with assistance from [Claude Code](https://claude.com/claude-code), Anthropic's AI coding assistant. While Claude Code helped accelerate development, all design decisions, code review, and testing were performed by the author.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcebert%2Faws-lambda-performance-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcebert%2Faws-lambda-performance-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcebert%2Faws-lambda-performance-benchmarks/lists"}