https://github.com/discretetom/onnx-on-aws-lambda-arm64

A demo to show how to run ONNX Runtime on AWS Lambda for arm64 runtimes.
https://github.com/discretetom/onnx-on-aws-lambda-arm64

Last synced: 12 days ago
JSON representation

A demo to show how to run ONNX Runtime on AWS Lambda for arm64 runtimes.

• Host: GitHub
• URL: https://github.com/discretetom/onnx-on-aws-lambda-arm64
• Owner: DiscreteTom
• Created: 2024-07-25T07:50:45.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2024-08-22T01:23:42.000Z (6 months ago)
• Last Synced: 2024-12-16T23:46:32.224Z (2 months ago)
• Language: Dockerfile
• Size: 19.1 MB
• Stars: 8
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# ONNX on AWS Lambda ARM64

This is a demo to show how to run [ONNX Runtime](https://onnxruntime.ai/docs/get-started/with-python.html) on AWS Lambda for arm64 runtimes.

## How This Works

ONNX Runtime works fine on x86 lambda, but on arm64 lambda you will get the following error:

```

Error in cpuinfo: failed to parse the list of possible processors in /sys/devices/system/cpu/possible

Error in cpuinfo: failed to parse the list of present processors in /sys/devices/system/cpu/present

Error in cpuinfo: failed to parse both lists of possible and present processors

terminate called after throwing an instance of 'onnxruntime::OnnxRuntimeException'

```

That's because AWS Lambda remove these files for the simplicity of runtimes.

In Zip mode, you can't modify files in `/sys`, but in Container mode you can. This demo will copy [`patch.txt`](./hello_world/patch.txt) to `/sys/devices/system/cpu/possible` and `/sys/devices/system/cpu/present` to fix the issue. See the [Dockerfile](./hello_world/Dockerfile) for more details.

> [!IMPORTANT]

> You should modify the content of [`patch.txt`](./hello_world/patch.txt) corresponding to your Lambda function's memory configuration. The content should be `0-X` where `X` is the cpu count of the Lambda function ***minus 1***. You can use `os.cpu_count()` in python to checkout how many vCPUs are allocated for your memory configuration. E.g. when the memory is 128MB there will be 2 vCPUs and the content should be `0-1`; when the memory is 10240MB there will be 6 vCPUs and the content should be `0-5`.

> Why can't you generate `/sys/devices/system/cpu/possible` and `/sys/devices/system/cpu/present` during the runtime to write the correct vCPU count? That's because in Lambda only `/tmp` is writable. Thus you have to finish the patch process before the Lambda is invoked.

This demo uses [AWS SAM](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) as the development framework.

## Build

Docker is required.

```bash

sam build -u

```

## Deploy

```bash

sam deploy -g

```

## Remote Test

```bash

sam remote invoke HelloWorldFunction --region us-east-1

```

You can also test your function in the AWS management console.

## Clean

```bash

sam delete --stack-name python-onnx

```