{"id":23296816,"url":"https://github.com/undisputed-jay/aws-s3-integration-with-snowflake","last_synced_at":"2026-03-20T01:08:24.297Z","repository":{"id":174059552,"uuid":"651713150","full_name":"Undisputed-jay/AWS-S3-Integration-with-snowflake","owner":"Undisputed-jay","description":"This project sets up an ETL pipeline to load Citibike trip data from an AWS S3 bucket into Snowflake. It establishes a secure integration with S3, defines a CSV file format, stages the data, and loads it into a Snowflake table for analysis.","archived":false,"fork":false,"pushed_at":"2024-11-03T22:27:03.000Z","size":8,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-06T14:41:40.226Z","etag":null,"topics":["aws-s3","snowflake","sql"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Undisputed-jay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-09T22:12:28.000Z","updated_at":"2024-11-03T22:28:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"ffdeadf2-ddbc-4dca-aaa3-b46da86a8264","html_url":"https://github.com/Undisputed-jay/AWS-S3-Integration-with-snowflake","commit_stats":null,"previous_names":["undisputed-jay/aws-s3-integration-with-snowflake"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Undisputed-jay/AWS-S3-Integration-with-snowflake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Undisputed-jay%2FAWS-S3-Integration-with-snowflake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Undisputed-jay%2FAWS-S3-Integration-with-snowflake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Undisputed-jay%2FAWS-S3-Integration-with-snowflake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Undisputed-jay%2FAWS-S3-Integration-with-snowflake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Undisputed-jay","download_url":"https://codeload.github.com/Undisputed-jay/AWS-S3-Integration-with-snowflake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Undisputed-jay%2FAWS-S3-Integration-with-snowflake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29427066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T22:20:51.549Z","status":"ssl_error","status_checked_at":"2026-02-13T22:20:49.838Z","response_time":78,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","snowflake","sql"],"created_at":"2024-12-20T07:15:21.803Z","updated_at":"2026-02-14T00:45:52.543Z","avatar_url":"https://github.com/Undisputed-jay.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003eCitibike Data Integration and ETL Pipeline\u003c/h1\u003e\n    \u003cp\u003eThis repository contains SQL scripts and configurations for a data integration and ETL pipeline that loads Citibike trip data from an Amazon S3 bucket into a Snowflake database. This pipeline uses Snowflake's external storage integration and staging capabilities to access, format, and transform Citibike data for analysis.\u003c/p\u003e\n    \u003ch2\u003eFeatures\u003c/h2\u003e\n    \u003cul\u003e\n        \u003cli\u003e\u003cstrong\u003eAWS S3 and Snowflake Integration\u003c/strong\u003e: Securely connects to a designated S3 bucket using Snowflake's \u003ccode\u003eSTORAGE INTEGRATION\u003c/code\u003e and IAM roles, ensuring a reliable and efficient data connection.\u003c/li\u003e\n        \u003cli\u003e\u003cstrong\u003eData Staging and File Formatting\u003c/strong\u003e: Defines a reusable file format for consistent parsing of CSV files, including error handling for null values and missing data.\u003c/li\u003e\n        \u003cli\u003e\u003cstrong\u003eAutomated Data Loading\u003c/strong\u003e: Copies raw CSV data from S3 into a Snowflake stage, transforming it for storage in a structured Snowflake table.\u003c/li\u003e\n        \u003cli\u003e\u003cstrong\u003eScalable ETL Process\u003c/strong\u003e: Easily scalable for additional Citibike datasets or other data formats, supporting future data ingestion and transformations.\u003c/li\u003e\n    \u003c/ul\u003e\n    \u003ch2\u003eTable Structure\u003c/h2\u003e\n    \u003cp\u003eThe pipeline loads data into a \u003ccode\u003etrips\u003c/code\u003e table, which includes columns for trip duration, start and end times, station details, bike IDs, and user demographic information. This structured format enables quick querying and supports further analytics.\u003c/p\u003e\n    \u003ch2\u003eUsage\u003c/h2\u003e\n    \u003col\u003e\n        \u003cli\u003e\u003cstrong\u003eConfigure Storage Integration\u003c/strong\u003e: Ensure Snowflake has permissions to access the S3 bucket by configuring the \u003ccode\u003eSTORAGE_AWS_ROLE_ARN\u003c/code\u003e with appropriate IAM roles.\u003c/li\u003e\n        \u003cli\u003e\u003cstrong\u003eRun SQL Scripts\u003c/strong\u003e: Execute the SQL scripts in the repository to create the integration, file format, stage, and table, then load the data from S3.\u003c/li\u003e\n        \u003cli\u003e\u003cstrong\u003eQuery the Data\u003c/strong\u003e: Run queries on the \u003ccode\u003etrips\u003c/code\u003e table to analyze Citibike trip data.\u003c/li\u003e\n    \u003c/ol\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fundisputed-jay%2Faws-s3-integration-with-snowflake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fundisputed-jay%2Faws-s3-integration-with-snowflake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fundisputed-jay%2Faws-s3-integration-with-snowflake/lists"}