{"id":19248964,"url":"https://github.com/skar-software/duckdb-go-lambda","last_synced_at":"2025-07-02T20:34:30.005Z","repository":{"id":240156639,"uuid":"795260154","full_name":"skar-software/DuckDB-Go-Lambda","owner":"skar-software","description":"DuckDB Golang runtime running on AWS Lambda","archived":false,"fork":false,"pushed_at":"2024-05-21T04:25:40.000Z","size":8393,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-02T00:45:09.097Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skar-software.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-02T22:46:37.000Z","updated_at":"2024-06-10T14:24:04.000Z","dependencies_parsed_at":"2024-05-21T05:28:47.771Z","dependency_job_id":null,"html_url":"https://github.com/skar-software/DuckDB-Go-Lambda","commit_stats":null,"previous_names":["anonranger/go-duckdb-lambda","anonranger/duckdb-go-lambda","skarcapital/duckdb-go-lambda","skar-software/duckdb-go-lambda"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/skar-software/DuckDB-Go-Lambda","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skar-software%2FDuckDB-Go-Lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skar-software%2FDuckDB-Go-Lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skar-software%2FDuckDB-Go-Lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skar-software%2FDuckDB-Go-Lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skar-software","download_url":"https://codeload.github.com/skar-software/DuckDB-Go-Lambda/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skar-software%2FDuckDB-Go-Lambda/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263211125,"owners_count":23431259,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T18:11:42.129Z","updated_at":"2025-07-02T20:34:29.979Z","avatar_url":"https://github.com/skar-software.png","language":"Go","readme":"# Usage\njust pass your query into the \"query\" key in the input event json of the lambda like below\n```json\n{\n\t\"query\": \"SELECT COUNT(*) AS 'Total Records' FROM 'https://raw.githubusercontent.com/anonranger/Go-DuckDB-Lambda/main/student-data.csv';\"\n}\n```\nif you are going to pass multiple quries make sure to use the termination character (;)\n\nif you require to use double quotes in the query then it must be escaped while its being passed into the event json\n\n# Example queries\nfetch count of records present in example csv\n\n**Input Query**\n```sql\nSELECT COUNT(*) AS 'Total Records' FROM 'https://raw.githubusercontent.com/anonranger/Go-DuckDB-Lambda/main/student-data.csv';\n```\n**Output**\n```\n+---------------+\n| Total Records |\n+---------------+\n| 61953         |\n+---------------+\n```\nfetch a random record for example csv\n\n**Input Query**\n```sql\nSELECT \"Fiscal Year\", Career, \"Program Level\", Campus FROM 'https://raw.githubusercontent.com/anonranger/Go-DuckDB-Lambda/main/student-data.csv' ORDER BY RANDOM() LIMIT 1;\n```\n**Output**\n```\n+-------------+---------------+---------------+------------------------+\n| Fiscal Year | Career        | Program Level | Campus                 |\n+-------------+---------------+---------------+------------------------+\n| 2023        | Undergraduate | Bachelors     | University of Waterloo |\n+-------------+---------------+---------------+------------------------+\n```\n\n\n# Option 1 : Run EXE file (pre-compiled)\n- Go DuckDB in Lambda EXE file is avalaible under the releases tab. You can download the zip and directly upload it to a AWS lambda Golang (AWS Linux 2023) and test it out your self.\n  https://github.com/anonranger/Go-DuckDB-Lambda/releases/tag/v1\n  \n- If you have questions, post it in the issues.\n\n# Option 2: Compile steps\nThis repository contains examples to successfully compile a golang binary to run on a AWS lambda\n\n- main.go has a simple duckdb program where duckdb is run in memory mode and reads data from a csv file thats present in this repository (`student-data.csv`)\n- since duckdb uses C++, the go-duckdb lib requires CGO (via which we can call C Code from Golang)\n- hence we need to have CSGO_ENABLED=true (Envirnoment Variable)\n- but when we build from our host machine (in my case ubuntu 22.04) the binary will reference a perticular version of GLIBC (OS Level dep) where as when we try to run that binary in AWS Lambda where AWS Linux 2023 is used it has a different version of GLIBC hence we will get an error stating that this perticular version of GLIBC is not found\n- hence we will have to compile the binary from that very OS Environment\n- we build it using either a `EC2 Instance` or `Docker File`\n## Build using EC2 Instance\n- spin a ec2 instance with the same OS (AWS Linux 2023 in my case) as your AWS lamba\n- install golang and all required dependencies and compile the program and the resulting binary should work in AWS Lambda functions\n## Build using Docker File\n- this is the easy way, just utilize the Dockerfile given in this repository\n- build the docker file\n- `docker build -t my-golang-builder .`\n- run the docker image built so we build the program\n- `docker run --name my-golang-container my-golang-builder`\n- copy the file from container to host fs \n- `docker cp my-golang-container:/built_file /home/user`\n\n## HOME env not set\n- in few environments such as AWS Lambda the $HOME env var is not set which is required in order to install extensions for duckdb in our example we are using httpfs to fetch a file from this repository (`student-data.csv`), you will get the below error\n- `An error occurred while trying to automatically install the required extension 'httpfs':\n\tCan't find the home directory at ''\n\tSpecify a home directory using the SET home_directory='/path/to/dir' option.`\n- as this https://github.com/duckdb/duckdb/issues/3855 suggests, in AWS Lambda environment variables set variable `HOME` to `/tmp`\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskar-software%2Fduckdb-go-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskar-software%2Fduckdb-go-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskar-software%2Fduckdb-go-lambda/lists"}