{"id":13453732,"url":"https://github.com/tobilg/serverless-duckdb","last_synced_at":"2025-04-05T17:05:18.852Z","repository":{"id":81148126,"uuid":"600494605","full_name":"tobilg/serverless-duckdb","owner":"tobilg","description":"An example of how to run DuckDB on AWS Lambda \u0026 API Gateway.","archived":false,"fork":false,"pushed_at":"2025-02-13T08:47:46.000Z","size":631,"stargazers_count":143,"open_issues_count":0,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T16:04:31.185Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobilg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-11T17:08:25.000Z","updated_at":"2025-03-28T00:30:29.000Z","dependencies_parsed_at":"2023-09-26T19:29:04.065Z","dependency_job_id":"7e52e42d-2798-41b6-8256-336589124261","html_url":"https://github.com/tobilg/serverless-duckdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fserverless-duckdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fserverless-duckdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fserverless-duckdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fserverless-duckdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobilg","download_url":"https://codeload.github.com/tobilg/serverless-duckdb/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247369953,"owners_count":20927928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:00:46.203Z","updated_at":"2025-04-05T17:05:18.830Z","avatar_url":"https://github.com/tobilg.png","language":"TypeScript","funding_links":[],"categories":["Resources","TypeScript"],"sub_categories":[],"readme":"# serverless-duckdb\nAn example of how to run DuckDB on AWS Lambda \u0026 API Gateway. This will eventually deploy three Lambda functions:\n\n* An **API Gateway endpoint** to which DuckDB queries can be issued via a POST request, which is authenticated by an API Key\n* An **API Gateway endpoint** to which DuckDB queries can be issued via a POST request, providing access to S3 Express One Zone, and authenticated by an API Key\n* A **Function URL Lambda** that supports streaming the query results as an Apache Arrow IPC stream, which uses **NO** authentication by default (you can add `AWS_IAM` auth manually if you wish)\n\nOnly the first function is deployed by default, to deploy the others, you need to uncomment the specific sections in the [serverless.yml](serverless.yml) file.\n\n## Requirements\nYou'll need a current v3 version installation of the [Serverless Framework](https://serverless.com) on the machine you're planning to deploy the application from.\n\nAlso, you'll have to setup your AWS credentials according to the [Serverless docs](https://www.serverless.com/framework/docs/providers/aws/guide/credentials/).\n\n## Configuration\nDuckDB is automatically configured to use the [HTTPFS extension](https://duckdb.org/docs/extensions/httpfs), and uses the AWS credentials that are given to your Lambda function by its execution role. This means you can potentially query data that is available via HTTP(S) or in AWS S3 buckets.\n\nIf you want to also query data (e.g. Parquet files) that resides in one or more S3 buckets, you'll have to adjust the `iamRoleStatements` part of the function configuration in the [serverless.yml](serverless.yml#L45) file. Just replace the `YOUR-S3-BUCKET-NAME` with your actual S3 bucket name.\n\n## Deployment\nAfter you cloned this repository to your local machine and cd'ed in its directory, the application can be deployed like this (don't forget a `npm i` to install the dependencies!):\n\n```bash\n$ sls deploy\n```\n\nThis will deploy the stack to the default AWS region `us-east-1`. In case you want to deploy the stack to a different region, you can specify a `--region` argument:\n\n```bash\n$ sls deploy --region eu-central-1\n```\n\nThe deployment should take 2-3 minutes. Once the deployment is finished, you should find some output in your console that indicates the API Gateway endpoint URL and the API Key:\n\n```yaml\napi keys:\n  DuckDBKey: REDACTED\nendpoints:\n  POST - https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query\n  streamingQuery: https://REDACTED.lambda-url.us-east-1.on.aws/\n```\n\n## Usage\n\n### API Gateway endpoint\nYou can now query your DuckDB endpoint via HTTP requests (don't forget to exchange `REDACTED` with your real URL and API Key), e.g.\n\n```bash\ncurl -L -XPOST 'https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query' \\\n  --header 'x-api-key: REDACTED' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw '{\n      \"query\": \"SELECT avg(c_acctbal) FROM '\\''https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet'\\'';\"\n  }'\n```\n\n### API Gateway endpoint with S3 Express One Zone\nYou can now query your DuckDB endpoint via HTTP requests (don't forget to exchange `REDACTED` with your real URL and API Key), e.g.\n\n```bash\ncurl -L -XPOST 'https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/queryS3Express' \\\n  --header 'x-api-key: REDACTED' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw '{\n      \"query\": \"SELECT avg(c_acctbal) FROM '\\''https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet'\\'';\"\n  }'\n```\n\n### Function URL Lambda\nYou can query the streaming Lambda by issueing the following command (don't forget to specify an `--output` path, this is where the Apache Arrow file will be stored):\n\n```bash\ncurl -L -XPOST 'https://REDACTED.lambda-url.us-east-1.on.aws/' \\\n  --header 'Content-Type: application/json' \\\n  --data-raw 'SELECT 1' \\\n  --output /tmp/result.arrow\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fserverless-duckdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobilg%2Fserverless-duckdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fserverless-duckdb/lists"}