{"id":13856957,"url":"https://github.com/bakdata/aws-lambda-r-runtime","last_synced_at":"2025-04-10T23:48:17.377Z","repository":{"id":43393427,"uuid":"160192085","full_name":"bakdata/aws-lambda-r-runtime","owner":"bakdata","description":"Serverless execution of R code on AWS Lambda","archived":false,"fork":false,"pushed_at":"2020-05-29T16:41:39.000Z","size":150,"stargazers_count":143,"open_issues_count":11,"forks_count":53,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-10T23:48:12.935Z","etag":null,"topics":["aws","aws-lambda","r","serverless"],"latest_commit_sha":null,"homepage":"https://medium.com/bakdata/running-r-on-aws-lambda-9d40643551a6","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-03T13:05:10.000Z","updated_at":"2025-03-22T10:57:50.000Z","dependencies_parsed_at":"2022-07-08T07:50:06.419Z","dependency_job_id":null,"html_url":"https://github.com/bakdata/aws-lambda-r-runtime","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Faws-lambda-r-runtime","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Faws-lambda-r-runtime/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Faws-lambda-r-runtime/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Faws-lambda-r-runtime/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/aws-lambda-r-runtime/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-lambda","r","serverless"],"created_at":"2024-08-05T03:01:20.331Z","updated_at":"2025-04-10T23:48:17.356Z","avatar_url":"https://github.com/bakdata.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# aws-lambda-r-runtime\n\n[![Build Status](https://travis-ci.com/bakdata/aws-lambda-r-runtime.svg?branch=master)](https://travis-ci.com/bakdata/aws-lambda-r-runtime)\n\nThis project makes it easy to run AWS Lambda Functions written in R.\n\n## Example\nTo run the example, we need to create a IAM role executing our lambda.\nThis role should have the following properties:\n- Trusted entity – Lambda.\n- Permissions – AWSLambdaBasicExecutionRole.\n\nFurthermore you need a current version of the AWS CLI.\n\nThen create a lambda function which uses the R runtime layer:\n```bash\ncd example/\nchmod 755 script.R\nzip function.zip script.R\n# current region\nregion=$(aws configure get region)\n# latest runtime layer ARN for R 3.6.0 in most regions\n# for an accurate list, please have a look at the deploy section of the travis ci build log\n# https://travis-ci.com/bakdata/aws-lambda-r-runtime\nruntime_layer=arn:aws:lambda:$region:131329294410:layer:r-runtime-3_6_0:13\naws lambda create-function --function-name r-example \\\n    --zip-file fileb://function.zip --handler script.handler \\\n    --runtime provided --timeout 60 \\\n    --layers ${runtime_layer} \\\n    --role \u003crole-arn\u003e\n```\n\nThe function simply increments 'x' by 1.\nInvoke the function:\n```bash\naws lambda invoke --function-name r-example \\\n    --payload '{\"x\":1}' response.txt\ncat response.txt\n```\n\nThe expected result should look similar to this:\n```json\n2\n```\n\n### Using packages\n\nWe also provide a layer which ships with some recommended R packages, such as `Matrix`.\nThis example lambda shows how to use them:\n```bash\ncd example/\nchmod 755 matrix.R\nzip function.zip matrix.R\n# current region\nregion=$(aws configure get region)\n# latest runtime layer ARN for R 3.6.0 in most regions\n# for an accurate list, please have a look at the deploy section of the travis ci build log\n# https://travis-ci.com/bakdata/aws-lambda-r-runtime\nruntime_layer=arn:aws:lambda:$region:131329294410:layer:r-runtime-3_6_0:13\n# latest recommended layer ARN for R 3.6.0 in most regions\n# for an accurate list, please have a look at the deploy section of the travis ci build log\n# https://travis-ci.com/bakdata/aws-lambda-r-runtime\nrecommended_layer=arn:aws:lambda:$region:131329294410:layer:r-recommended-3_6_0:13\naws lambda create-function --function-name r-matrix-example \\\n    --zip-file fileb://function.zip --handler matrix.handler \\\n    --runtime provided --timeout 60 --memory-size 3008 \\\n    --layers ${runtime_layer} ${recommended_layer} \\\n    --role \u003crole-arn\u003e\n```\n\nThe function returns the second column of some static matrix.\nInvoke the function:\n```bash\naws lambda invoke --function-name r-matrix-example response.txt\ncat response.txt\n```\n\nThe expected result should look similar to this:\n```json\n[4,5,6]\n```\n\n## Provided layers\n\nLayers are only accessible in the AWS region they were published.\nWe provide the following layers:\n\n### r-runtime\n\nR,\n[httr](https://cran.r-project.org/package=httr),\n[jsonlite](https://cran.r-project.org/package=jsonlite),\n[aws.s3](https://cran.r-project.org/package=aws.s3),\n[logging](https://cran.r-project.org/package=logging)\n\nAvailable AWS regions:\n- ap-northeast-1\n- ap-northeast-2\n- ap-south-1\n- ap-southeast-1\n- ap-southeast-2\n- ca-central-1\n- eu-central-1\n- eu-north-1\n- eu-west-1\n- eu-west-2\n- eu-west-3\n- sa-east-1\n- us-east-1\n- us-east-2\n- us-west-1\n- us-west-2\n\nAvailable R versions:\n- 3_5_1\n- 3_5_3\n- 3_6_0\n\nLatest ARN can be retrieved from the [Travis CI build log](https://travis-ci.com/bakdata/aws-lambda-r-runtime). In general, it looks this:\n\n`arn:aws:lambda:$region:131329294410:layer:r-runtime-$r_version:$layer_version`\n\nAutomated command for retrieving the ARN does not work currently:\n```bash\naws lambda list-layer-versions --max-items 1 --no-paginate  \\\n    --layer-name arn:aws:lambda:${region}:131329294410:layer:r-runtime-${r_version} \\\n    --query 'LayerVersions[0].LayerVersionArn' --output text\n```\n\n### r-recommended\n\nThe recommended packages that ship with R:\nboot,\nclass,\ncluster,\ncodetools,\nforeign,\nKernSmooth,\nlattice,\nMASS,\nMatrix,\nmgcv,\nnlme,\nnnet,\nrpart,\nspatial,\nsurvival\n\nAvailable AWS regions:\n- ap-northeast-1\n- ap-northeast-2\n- ap-south-1\n- ap-southeast-1\n- ap-southeast-2\n- ca-central-1\n- eu-central-1\n- eu-north-1\n- eu-west-1\n- eu-west-2\n- eu-west-3\n- sa-east-1\n- us-east-1\n- us-east-2\n- us-west-1\n- us-west-2\n\nAvailable R versions:\n- 3_5_1\n- 3_5_3\n- 3_6_0\n\nLatest ARN can be retrieved from the [Travis CI build log](https://travis-ci.com/bakdata/aws-lambda-r-runtime). In general, it looks this:\n\n`arn:aws:lambda:$region:131329294410:layer:r-recommended-$r_version:$layer_version`\n\nAutomated command for retrieving the ARN does not work currently:\n```bash\naws lambda list-layer-versions --max-items 1 --no-paginate  \\\n    --layer-name arn:aws:lambda:${region}:131329294410:layer:r-recommended-${r_version} \\\n    --query 'LayerVersions[0].LayerVersionArn' --output text\n```\n\n### r-awspack\n\nThe [aws.s3](https://cran.r-project.org/package=aws.s3) package.\nIt used to contain the [awspack](https://cran.r-project.org/package=awspack) package but unfortunately this package has been retired.\nYou can still find it in old versions of the layer that have been published before 2020.\n\nAvailable AWS regions:\n- ap-northeast-1\n- ap-northeast-2\n- ap-south-1\n- ap-southeast-1\n- ap-southeast-2\n- ca-central-1\n- eu-central-1\n- eu-north-1\n- eu-west-1\n- eu-west-2\n- eu-west-3\n- sa-east-1\n- us-east-1\n- us-east-2\n- us-west-1\n- us-west-2\n\nAvailable R versions:\n- 3_5_1\n- 3_5_3\n- 3_6_0\n\nLatest ARN can be retrieved from the [Travis CI build log](https://travis-ci.com/bakdata/aws-lambda-r-runtime). In general, it looks this:\n\n`arn:aws:lambda:$region:131329294410:layer:r-awspack-$r_version:$layer_version`\n\nAutomated command for retrieving the ARN does not work currently:\n```bash\naws lambda list-layer-versions --max-items 1 --no-paginate  \\\n    --layer-name arn:aws:lambda:${region}:131329294410:layer:r-awspack-${r_version} \\\n    --query 'LayerVersions[0].LayerVersionArn' --output text\n```\n\n## Documentation\n\nThe lambda handler is used to determine both the file name of the R script and the function to call.\nThe handler must be separated by `.`, e.g., `script.handler`.\n\nThe lambda payload is unwrapped as named arguments to the R function to call, e.g., `{\"x\":1}` is unwrapped to `handler(x=1)`.\n\nThe lambda function returns whatever is returned by the R function as a JSON object.\n\n### Building custom layers\n\nIn order to install additional R packages, you can create a lambda layer containing the libraries, just as in the second example.\nYou must use the the compiled package files.\nThe easiest way is to install the package with `install.packages()` and copy the resulting folder in `$R_LIBS`.\nUsing only the package sources does not suffice.\nThe file structure must be `R/library/\u003cmy-library\u003e`.\nIf your package requires system libraries, place them in `R/lib/`.\n\nYou can use Docker for building your layer.\nYou need to run `./docker_build.sh` first.\nThen you can install your packages inside the container and copy the files to your machine.\nSee `awspack/` for an example.\nThe `build.sh` script is used to run the docker container and copy sources to your machine.\nThe `entrypoint.sh` script is used for installing packages inside the container.\n\n### Debugging\n\nIn order to make the runtime log debugging messages, you can set the environment variable `LOGLEVEL` to `DEBUG`.\n\n## Limitations\n\nAWS Lambda is limited to running with 3GB RAM and must finish within 15 minutes.\nIt is therefore not feasible to execute long running R scripts with this runtime.\nFurthermore, only the `/tmp/` directory is writeable on AWS Lambda.\nThis must be considered when writing to the local disk. \n\n\n## Building\n\nTo build the layer yourself, you need to first build R from source.\nWe provide a Docker image which uses the great [docker-lambda](https://github.com/lambci/docker-lambda) project.\nJust run `./build.sh \u003cversion\u003e` and everything should be build properly.\n\nIf you plan to publish the runtime, you need to have a recent version of aws cli (\u003e=1.16).\nNow run the `\u003clayer\u003e/deploy.sh` script.\nThis creates a lambda layer named `r-\u003clayer\u003e-\u003cversion\u003e` in your AWS account.\nYou can use it as shown in the example.\n\n### Compiling on EC2\n\nIn case the Docker image does not properly represent the lambda environment,\nwe also provide a script which launches an EC2 instance, compiles R, and uploads the zipped distribution to S3.\nYou need to specify the R version, e.g., `3.6.0`, as well as the S3 bucket to upload the distribution to.\nFinally, you need to create an EC2 instance profile which is capable of uploading to the S3 bucket.\nSee the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#create-iam-role) for details.\nWith everything prepared, you can run the script:\n```bash\n./remote_compile_and_deploy.sh \u003cversion\u003e \u003cbucket-name\u003e \u003cinstance-profile\u003e\n```\nThe script will also take care of terminating the launched EC2 instance.\n\nTo manually build R from source, follow these steps:\n\nStart an EC2 instance which uses the [Lambda AMI](https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2):\n```bash\naws ec2 run-instances --image-id ami-657bd20a --count 1 --instance-type t2.medium --key-name \u003cmy-key-pair\u003e\n```\nNow run the `compile.sh` script in `r/`.\nYou must pass the R version as a parameter to the script, e.g., `3.6.0`.\nThe script produces a zip containing a functional R installation in `/opt/R/`.\nThe relevant files can be found in `r/build/bin/`.\nUse this R distribution for building the layers.\n\n### Testing\n\nAfter building all layers, you can test it locally with SAM CLI and Docker.\nInstall it via `pipenv install --dev`.\nThen run `python3 -m unittest`.\nThis will spawn a local lambda server via Docker and invokes the lambdas defined in `template.yaml`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Faws-lambda-r-runtime","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Faws-lambda-r-runtime","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Faws-lambda-r-runtime/lists"}