{"id":24041943,"url":"https://github.com/sundeck-io/snowflake-connector-framework","last_synced_at":"2025-04-19T19:54:11.753Z","repository":{"id":63218018,"uuid":"549937978","full_name":"sundeck-io/snowflake-connector-framework","owner":"sundeck-io","description":"A Source Connector Framework for Snowflake","archived":false,"fork":false,"pushed_at":"2022-11-09T23:23:22.000Z","size":121,"stargazers_count":12,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T12:34:10.257Z","etag":null,"topics":["aws","elasticsearch","federation","mongodb","mysql","pulumi","snowflake"],"latest_commit_sha":null,"homepage":"https://sundeck.io/blog/creating-a-source-connector-framework-for-snowflake","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sundeck-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-10-12T00:57:18.000Z","updated_at":"2025-03-01T18:08:06.000Z","dependencies_parsed_at":"2022-11-15T06:06:02.298Z","dependency_job_id":null,"html_url":"https://github.com/sundeck-io/snowflake-connector-framework","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sundeck-io%2Fsnowflake-connector-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sundeck-io%2Fsnowflake-connector-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sundeck-io%2Fsnowflake-connector-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sundeck-io%2Fsnowflake-connector-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sundeck-io","download_url":"https://codeload.github.com/sundeck-io/snowflake-connector-framework/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249786684,"owners_count":21325565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","elasticsearch","federation","mongodb","mysql","pulumi","snowflake"],"created_at":"2025-01-08T22:14:02.334Z","updated_at":"2025-04-19T19:54:11.735Z","avatar_url":"https://github.com/sundeck-io.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Source Connector Framework for Snowflake\n\nThis directory holds a pulumi deployment that allows Snowflake users to query alternative sources (specifically MySQL). You can learn more about this by reading our series of [blog posts](https://www.sundeck.io/blog/creating-a-source-connector-framework-for-snowflake) \n\n## Deploy AWS \u0026 Snowflake Assets\n1. Ensure you're logged into AWS and your credentials are setup correctly. \n1. Install Pulumi ([instructions](https://www.pulumi.com/docs/get-started/install/))\n1. Configure pulumi for first use.\n   ```bash\n   # Configure with local settings. You could also configure with Pulumi cloud but that takes more time. \n   pulumi login --local\n   ```\n1. Install node/npm ([instructions](https://nodejs.org/en/download/))\n1. Clone the repository and install the required NPM packages.\n   ```\n   git clone https://github.com/sundeck-io/snowflake-connector-framework.git\n   npm update\n   ```\n   \n1. Configure your account \u0026 username for snowflake: \n   ```\n   cd snowflake-connectors/athena\n   pulumi config set snowflake:account YOURACCOUNTLOCATOR\n   pulumi config set snowflake:username YOURSNOWFLAKEUSERNAME\n   ```\n1. Setup AWS \u0026 Snowflake \n   ```bash\n   # During execution, pulumi will prompt you one or more times to login to Snowflake via browser.\n   # Most of the operations are quick but the following take a while:\n   #\n   # *  Creation of a MySQL RDS instance takes 2-3 minutes.\n   # *  Deployment of MySQL Athena connector Lambda takes 2-3 minutes.\n   # \n   # Unfortunately, since the lambda connector needs info from the RDS instance post \n   # deploy (connection string), they have to run serially.  \n   #\n   # Note: If you see any Snowflake operations taking more than a couple seconds, that typically \n   # means you've missed a login window. Check all your tabs!\n   #\n   pulumi up -y\n   ```\n   \n## Query MySQL (or any Athena source)\n1. Go into Snowflake Snowsight (or your preferred SQL tool).\n1. Execute a query in Snowflake against your new MySQL instance.\n   ```sql\n   use sundeck_connectors.athena;\n   select * from table(query_athena($$ \n     select * from mysql.information_schema.tables\n   $$, 100));\n   ```\n1. By default, all data comes back as a variant column. This is due to the fact that a UDTF needs to have schema declared at creation time. As such, our declared schema is a single variant column called `data`. If you want to make things more typed, you can create a view on top of your table function invocation. For example:\n   ```sql\n   CREATE VIEW mysql_information_schema_tables AS \n   SELECT \n       data:table_catalog::text AS table_catalog,\n       data:table_schema::text AS table_schema,\n       data:table_name::text AS table_name,\n       data:table_type::text AS table_type\n   FROM TABLE(query_athena($$ \n         SELECT * FROM mysql.information_schema.tables\n       $$, 100)); \n   ```\n\n## Frequently Asked Questions\n\u003cdl\u003e\n   \u003cdt\u003eHow does this all work?\u003c/dt\u003e\n   \u003cdd\u003eSee our \u003ca href=\"https://www.sundeck.io/blog/creating-a-source-connector-framework-for-snowflake\"\u003eBlog post\u003c/a\u003e on the topic!\u003c/dd\u003e\n   \u003cdt\u003eWhat is Pulumi?\u003c/dt\u003e\n   \u003cdd\u003ePulumi is a infrastructure automation tool, similar to Terraform or AWS CloudFormation.\u003c/dd\u003e\n   \u003cdt\u003eWhy use Pulumi?\u003c/dt\u003e\n   \u003cdd\u003eIn order to deploy an external function, there is some back and forth between AWS and Snowflake. (You need take information from each and give it to the other). Rather than make people go through a bunch of steps, Pulumi allows us to automatically move the configuration between the two systems to make it easier to setup an external function.\u003c/dd\u003e\n    \u003cdt\u003eWhat if I want to configure things manually?\u003c/dt\u003e\n    \u003cdd\u003ePulumi is largely declarative. Most of the deployment code should readable even if you've never used Pulumi\u003c/dd\u003e\n   \u003cdt\u003eWhat does Sundeck do?\u003c/dt\u003e\n\u003cdd\u003eWe're working on some new ways to enhance Snowflake. More coming soon. Go to our \u003ca href=\"https://sundeck.io\"\u003ewebsite\u003c/a\u003e and sign up for our mailing list to hear more as we progress.\u003c/dd\u003e\n\u003c/dl\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsundeck-io%2Fsnowflake-connector-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsundeck-io%2Fsnowflake-connector-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsundeck-io%2Fsnowflake-connector-framework/lists"}