{"id":19502157,"url":"https://github.com/cyberagent/snowflake-aws-mysql-connector","last_synced_at":"2025-04-25T23:31:24.340Z","repository":{"id":69710189,"uuid":"324287584","full_name":"CyberAgent/snowflake-aws-mysql-connector","owner":"CyberAgent","description":"Integrate with Snowflake and MySQL(Aurora) on AWS","archived":false,"fork":false,"pushed_at":"2023-08-29T03:46:13.000Z","size":169,"stargazers_count":8,"open_issues_count":0,"forks_count":6,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-04-04T05:11:24.650Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyberAgent.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-25T04:32:01.000Z","updated_at":"2024-01-15T09:41:21.000Z","dependencies_parsed_at":"2024-11-10T22:15:30.875Z","dependency_job_id":"57476787-55b0-41d9-9da2-7c6e8010710c","html_url":"https://github.com/CyberAgent/snowflake-aws-mysql-connector","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgent%2Fsnowflake-aws-mysql-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgent%2Fsnowflake-aws-mysql-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgent%2Fsnowflake-aws-mysql-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgent%2Fsnowflake-aws-mysql-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyberAgent","download_url":"https://codeload.github.com/CyberAgent/snowflake-aws-mysql-connector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250912660,"owners_count":21506865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T22:15:26.287Z","updated_at":"2025-04-25T23:31:24.334Z","avatar_url":"https://github.com/CyberAgent.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# snowflake-aws-mysql-connector\n\n## Install\n\n### Copy configuration file\n\n```bash\n$ cp conf/example.json conf/production.json\n```\n\n### Edit configuration file\n\nEdit `conf/production.json`.\n\n|key|value|example|\n|:--|:--|:--|\n|stackName|Stack name of CloudFormation|prod-SnowflakeAWSMySQLConnector-Stack|\n|awsAccountId|AWS account id to deploy (12 digits)|123456789012|\n|awsRegion|AWS region to deploy|ap-northeast-1|\n|lambdaFunctionName|AWS Lambda function name to deploy|prod-snowflake-aws-mysql-connector|\n|vpcId|Amazon VPC id where Lambda functions to run|vpc-abcded12345abcdef|\n|subnets|Subnets(id and availability zone name) where Lambda functions to run|[{\"id\": \"subnet-abcdefghik1234567\", \"availabilityZone\": \"ap-northeast-1a\"}, {\"id\": \"subnet-lmnopqrstu8901234\", \"availabilityZone\": \"ap-northeast-1c\"}]|\n|securityGroupIds|Security group ids to attach Lambda's ENI|[\"sg-abcdef12345678901\", \"sg-ghijkl23456789012\"]|\n|snowflakeApiAwsIamUser|IAM User from Snowflake (This must be empty array first time)|[]|\n|mysqlHost|MySQL hostname or ip to connect. VPC of Lambda function and MySQL must same.|my-rds.cluster-ro-abcde123abcd.ap-northeast-1.rds.amazonaws.com|\n|mysqlUser|MySQL user|user1|\n|mysqlPassword|MySQL password|password|\n\n* ⚠️ snowflakeApiAwsIamUser must be empty array first time.\n\n### Build Lambda function\n\n```bash\n$ cd path/to/snowflake-aws-mysql-connector/\n$ npm install\n$ npm run build:cdk\n```\n\n### Init CDK\n\n```bash\n$ cd path/to/snowflake-aws-mysql-connector/cdk/\n$ npm install\n$ npm run cdk -- bootstrap -c target=production\n```\n\nYou can use `--profile` option if you use profile to switch account or role. (e.g. `npm run cdk -- --profile my-profile ...`)\n\n### Check what to deploy\n\n```bash\n$ npm run cdk -- diff -c target=production\n```\n\n### Deploy Lambda function and API Gateway\n\n```bash\n$ npm run cdk -- deploy -c target=production\n[snip]\nDo you wish to deploy these changes (y/n)? y\n```\n\n### Check output value from CDK\n\nOutput example:\n\n```\nOutputs:\nSnowflakeAWSMySQLConnectorStack.SnowflakeApiAllowedPrefixes = https://abcde12345.execute-api.ap-northeast-1.amazonaws.com/default/\nSnowflakeAWSMySQLConnectorStack.SnowflakeApiAwsRoleArn = arn:aws:iam::123456789012:role/prod-snowflake-aws-mysql-connector-execution-role-for-snowflake\nSnowflakeAWSMySQLConnectorStack.prodsnowflakeawsmysqlconnectorrestapiEndpoint2587587B = https://abcde12345.execute-api.ap-northeast-1.amazonaws.com/default/\n```\n\n`SnowflakeAWSMySQLConnectorStack.SnowflakeApiAllowedPrefixes` and `SnowflakeAWSMySQLConnectorStack.SnowflakeApiAwsRoleArn` are used later.\n\nYou can check these values on CloudFormation outputs of AWS Console.\n\n### Integrate Snowflake and your AWS IAM role\n\nQuery on Snowflake. Maybe you should use role `ACCOUNTADMIN`;\n\n|placeholder|value|example|\n|:--|:--|:--|\n|\u003cintegration\\_name\u003e|integration name|prod\\_api\\_gateway\\_mysql|\n|\u003ciam_\\role_arn\u003e|SnowflakeAWSMySQLConnectorStack.SnowflakeApiAwsRoleArn|arn:aws:iam::123456789012:role/prod-snowflake-aws-mysql-connector-execution-role-for-snowflake|\n|\u003capi\\_prefix\u003e|SnowflakeAWSMySQLConnectorStack.SnowflakeApiAllowedPrefixes|https://abcde12345.execute-api.ap-northeast-1.amazonaws.com/default/|\n\n```sql\ncreate or replace api integration \u003cintegration_name\u003e\n  api_provider = aws_api_gateway\n  api_aws_role_arn = '\u003ciam_role_arn\u003e'\n  enabled = true\n  api_allowed_prefixes = ('\u003capi_prefix\u003e')\n;\n```\n\n### Update configuration\n\nQuery on Snowflake to get `API_AWS_IAM_USER_ARN` and `API_AWS_EXTERNAL_ID`.\n\n|placeholder|value|example|\n|:--|:--|:--|\n|\u003cintegration\\_name\u003e|integration name|prod\\_api\\_gateway\\_mysql|\n\n```sql\ndescribe api integration \u003cintegration_name\u003e;\n```\n\nUpdate `conf/production.json`.\n\n|placeholder|example|\n|:--|:--|\n|\u003cAPI\\_AWS\\_IAM\\_USER\\_ARN\u003e|arn:aws:iam::987654321098:user/abcdef-abcd1234|\n|\u003cAPI\\_AWS\\_EXTERNAL\\_ID\u003e|AbCdEfGh12345=|\n\n|key|value|example|\n|:--|:--|:--|\n|snowflakeApiAwsIamUser|Update with `API_AWS_IAM_USER_ARN` and `API_AWS_EXTERNAL_ID`|[{\"arn\": \"\u003cAPI\\_AWS\\_IAM\\_USER\\_ARN\u003e\", \"externalId\": \"\u003cAPI\\_AWS\\_EXTERNAL\\_ID\u003e\"}]|\n\n### Build Lambda function\n\n```bash\n$ cd path/to/snowflake-aws-mysql-connector/\n$ npm install\n$ npm run build:cdk\n```\n\n### Check what to deploy\n\n```bash\n$ npm run cdk -- diff -c target=production\n```\n\n### Deploy Lambda function and API Gateway\n\n```bash\n$ npm run cdk -- deploy -c target=production\n[snip]\nDo you wish to deploy these changes (y/n)? y\n```\n\n### Create Snowflake External Function\n\n|placeholder|value|example|\n|:--|:--|:--|\n|\u003cexternal\\_function_name\u003e|external function name|prod\\_call\\_api\\_gateway\\_mysql|\n|\u003cintegration\\_name\u003e|integration name|prod\\_api\\_gateway\\_mysql|\n|\u003capi\\_prefix\u003e|SnowflakeAWSMySQLConnectorStack.SnowflakeApiAllowedPrefixes|https://abcde12345.execute-api.ap-northeast-1.amazonaws.com/default/|\n\n```sql\ncreate or replace external function \u003cexternal_function_name\u003e(q string)\n  returns variant\n  api_integration = \u003cintegration_name\u003e\n  max_batch_rows = 1\n  as '\u003capi_prefix\u003e'\n;\n```\n\n### Create Snowflake User Defined Table Function(UDTF)\n\n|placeholder|value|example|\n|:--|:--|:--|\n|\u003cfunction\\_name\u003e|udf name|query\\_mysql|\n|\u003cexternal\\_function\\_name\u003e|external function name|prod\\_call\\_api\\_gateway\\_mysql|\n```sql\ncreate or replace function \u003cfunction_name\u003e(q string)\n    returns table(value variant)\n    as '\nselect value from\n  table(flatten(input =\u003e\n    case \u003cexternal_function_name\u003e(q):compress\n      when \\'zlib\\' then parse_json(decompress_string(base64_decode_binary(\u003cexternal_function_name\u003e(q):encodedRecords), \\'zlib\\'))\n      else parse_json(\u003cexternal_function_name\u003e(q):encodedRecords)\n    end\n  ))\n';\n```\n\n## Usage\nYou can use UDTF(named as \u003cfunction\\_name\u003e);\n\n```sql\n-- select all\nselect * from table(query_mysql('select * from my_table'));\n\n-- select specific columns\nselect value:id, value:name from table(query_mysql('select id, name from users'));\n\n-- join with Snowflake table and MySQL table\nselect log.user_id, users.value:name\nfrom\n  log  left join table(query_mysql('select id, name from users')) as users\n  on\n    log.user_id = users.value:id\nlimit 10;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagent%2Fsnowflake-aws-mysql-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyberagent%2Fsnowflake-aws-mysql-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagent%2Fsnowflake-aws-mysql-connector/lists"}