{"id":19224331,"url":"https://github.com/snowplow/snowplow-rdb-loader","last_synced_at":"2026-03-11T18:04:38.919Z","repository":{"id":38849458,"uuid":"98586542","full_name":"snowplow/snowplow-rdb-loader","owner":"snowplow","description":"Stores Snowplow enriched events in Redshift, Snowflake and Databricks","archived":false,"fork":false,"pushed_at":"2025-04-04T14:43:41.000Z","size":6798,"stargazers_count":31,"open_issues_count":84,"forks_count":16,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-16T01:05:12.816Z","etag":null,"topics":["redshift","scala","snowplow","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snowplow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-07-27T23:05:09.000Z","updated_at":"2025-04-04T14:39:45.000Z","dependencies_parsed_at":"2023-12-04T12:24:51.955Z","dependency_job_id":"fadd693b-0ade-4ae0-8e02-8fe93a1beb7d","html_url":"https://github.com/snowplow/snowplow-rdb-loader","commit_stats":null,"previous_names":[],"tags_count":524,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowplow%2Fsnowplow-rdb-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowplow%2Fsnowplow-rdb-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowplow%2Fsnowplow-rdb-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snowplow%2Fsnowplow-rdb-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snowplow","download_url":"https://codeload.github.com/snowplow/snowplow-rdb-loader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249178211,"owners_count":21225349,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["redshift","scala","snowplow","spark"],"created_at":"2024-11-09T15:11:22.155Z","updated_at":"2026-03-11T18:04:38.874Z","avatar_url":"https://github.com/snowplow.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Relational Database Loader\n\n[![Build Status][build-image]][build]\n[![Release][release-image]][releases]\n[![License][license-image]][license]\n\n## Introduction\n\nThis project contains applications required to load Snowplow data into various data warehouses.\n\nIt consists of two types of applications: Transformers and Loaders\n\n### Transformers\n\nTransformers read Snowplow enriched events, transform them to a format ready to be loaded to a data warehouse, then write them to respective blob storage.\n\nThere are two types of Transformers: Batch and Streaming\n\n#### Stream Transformer\n\nStream Transformers read enriched events from respective stream service, transform them, then write transformed events to specified blob storage path.\nThey write transformed events in periodic windows.\n\nThere are two different Stream Transformer applications: Transformer Kinesis and Transformer Pubsub. As one can predict, they are different variants for GCP, AWS and Azure.\n\n\n#### Batch Transformer\n\nIt is a [Spark][spark] job. It only works with AWS services. It reads enriched events from a given S3 path, transforms them, then writes transformed events to a specified S3 path.\n\n\n### Loaders\n\nTransformers send a message to a message queue after they are finished with transforming some batch and writing it to blob storage.\nThis message contains information about transformed data such as where it is stored and what it looks like.\n\nLoaders subscribe to the message queue. After a message is received, it is parsed, and necessary bits are extracted to load transformed events to the destination.\nLoaders construct necessary SQL statements to load transformed events then they send these SQL statements to the specified destination.\n\nAt the moment, we have loader applications for Redshift, Databricks and Snowflake.\n\n## Find out more\n\n| Technical Docs             | Setup Guide          | Roadmap \u0026 Contributing |\n|----------------------------|----------------------|------------------------|\n| ![i1][techdocs-image]      | ![i2][setup-image]   | ![i3][roadmap-image]   |\n| [Technical Docs][techdocs] | [Setup Guide][setup] | [Roadmap][roadmap]     |\n\n\n## Copyright and license\n\nCopyright (c) 2012-present Snowplow Analytics Ltd. All rights reserved.\n\nLicensed under the [Snowplow Limited Use License Agreement][license]. _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions][faq].)_\n\n\n[techdocs-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/techdocs.png\n[setup-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/setup.png\n[roadmap-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/roadmap.png\n[setup]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/\n[techdocs]: https://docs.snowplow.io/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-rdb-loader/\n[roadmap]: https://github.com/snowplow/snowplow/projects/7\n\n[spark]: http://spark.apache.org/\n\n[build-image]: https://github.com/snowplow/snowplow-rdb-loader/workflows/CI/badge.svg\n[build]: https://github.com/snowplow/snowplow-rdb-loader/actions/workflows/ci.yml\n\n[release-image]: https://img.shields.io/badge/release-6.1.5-blue.svg?style=flat\n[releases]: https://github.com/snowplow/snowplow-rdb-loader/releases\n\n[license]: https://docs.snowplow.io/limited-use-license-1.0\n[license-image]: https://img.shields.io/badge/license-Snowplow--Limited-Use-blue.svg?style=flat\n\n[faq]: https://docs.snowplow.io/docs/contributing/limited-use-license-faq/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowplow%2Fsnowplow-rdb-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnowplow%2Fsnowplow-rdb-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnowplow%2Fsnowplow-rdb-loader/lists"}