{"id":24087986,"url":"https://github.com/tinybirdco/aws-s3-lambda","last_synced_at":"2026-02-12T14:34:50.719Z","repository":{"id":51230290,"uuid":"346817729","full_name":"tinybirdco/aws-s3-lambda","owner":"tinybirdco","description":"Guide and scripts to automate csv import to Tinybird from AWS S3 using Lambda functions","archived":false,"fork":false,"pushed_at":"2022-09-05T11:54:08.000Z","size":4722,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-01T18:52:35.643Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-11T19:42:13.000Z","updated_at":"2023-11-04T13:36:24.000Z","dependencies_parsed_at":"2023-01-17T21:01:50.862Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/aws-s3-lambda","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tinybirdco/aws-s3-lambda","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Faws-s3-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Faws-s3-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Faws-s3-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Faws-s3-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/aws-s3-lambda/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Faws-s3-lambda/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29368703,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T08:51:36.827Z","status":"ssl_error","status_checked_at":"2026-02-12T08:51:26.849Z","response_time":55,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:56:52.732Z","updated_at":"2026-02-12T14:34:50.697Z","avatar_url":"https://github.com/tinybirdco.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS S3 + Lambda -\u003e Automatically ingesting to Tinybird from S3\n\nThis repository contains a shell script that configures an AWS Lambda function that is automatically triggered each time a new CSV file is uploaded to a given AWS S3 bucket.\n\nThe AWS Lambda function appends the contents of the CSV file to a Tinybird Data Source.\n\n## How to start\n\n- Install the `aws` CLI\n- Run `aws configure` and set your AWS client keys and region.\n- Run `cp .env_sample .env` and edit the `.env` file to customize the `run.sh` script. These are the available variables:\n\n```sh\n# name of the aws role (it will be created by the run.sh script)\nTB_ROLE_NAME=\n# name of the aws policy (it will be created by the run.sh script)\nTB_POLICY_NAME=\n# name of the aws lambda function (it will be created by the run.sh script)\nTB_FUNCTION_NAME=\nTB_PERMISSION_ID=1\n# if you upload a file called products__2021-01-01.csv it'll ingest the contents in the Tinybird products Data Source\nFILE_REGEXP='^[^_]+(?=_)'\n# Tinybird token with DATASOURCE:CREATE permission (or admin token). Get it from https://ui.tinybird.co/tokens\nTB_TOKEN=\n# Tinybird API host. It depends on the region or if you have a dedicated host (eu=https://api.tinybird.co) (us-east=https://api.us-east.tinybird.co) default is `eu` host\nTB_HOST=\n# The name of the S3 bucket to listen to. Files need to be in the root of the bucket\nS3_BUCKET=\n```\n\nMost of the variables are provided with default values, so you just have to indicate `TB_TOKEN` to ingest to your Tinybird account and `S3_BUCKET`. TB_HOST is `https://api.tinybird.co` by default, just set it if your account is in a different region than `eu`\n\n## How to use it\n\nRun: \n\n```bash\n./run.sh\n```\n\nOnce finished, go to the AWS console and check the lambda function is there and the `TB_TOKEN` and `FILE_REGEXP` variables required to run the script are available.\n\nOnce deployed, you can just drop CSV files in the `S3_BUCKET` and they'll be automatically ingested to the corresponding Tinybird Data Source defined by the FILE_REGEXP environment variable, by default it uses as the Data Source name the part of the CSV file name before the first underscore character, otherwise it takes the CSV file name.\n\n![](output.gif)\n\n## How to delete the Lambda function\n\nIf you want to clean the function, roles, policies and triggers:\n\n```bash\n./clean_function.sh\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Faws-s3-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Faws-s3-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Faws-s3-lambda/lists"}