{"id":16307099,"url":"https://github.com/fujiwara/rin","last_synced_at":"2026-03-08T21:34:28.143Z","repository":{"id":30750881,"uuid":"34307372","full_name":"fujiwara/Rin","owner":"fujiwara","description":"Rin is a Redshift data Importer by SQS messaging.","archived":false,"fork":false,"pushed_at":"2023-01-18T05:59:10.000Z","size":153,"stargazers_count":26,"open_issues_count":5,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-11T21:12:42.363Z","etag":null,"topics":["aws","go","golang","redshift","rin","sqs"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fujiwara.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-21T05:55:05.000Z","updated_at":"2024-01-12T18:05:06.000Z","dependencies_parsed_at":"2023-02-10T13:31:19.779Z","dependency_job_id":null,"html_url":"https://github.com/fujiwara/Rin","commit_stats":null,"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2FRin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2FRin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2FRin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fujiwara%2FRin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fujiwara","download_url":"https://codeload.github.com/fujiwara/Rin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221809032,"owners_count":16883838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","go","golang","redshift","rin","sqs"],"created_at":"2024-10-10T21:12:30.223Z","updated_at":"2026-03-08T21:34:28.098Z","avatar_url":"https://github.com/fujiwara.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rin\n\nRin is a Redshift data Importer by SQS messaging.\n\n## Architecture\n\n1. (Someone) creates a S3 object.\n2. [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) will send to a message to SQS.\n3. Rin will fetch messages from SQS, and publish a \"COPY\" query to Redshift.\n\n## Installation\n\n### Binary packages\n\n[Releases](https://github.com/fujiwara/Rin/releases)\n\n### Homebrew\n\n```console\n$ brew install fujiwara/tap/rin\n```\n\n### Docker\n\n[GitHub Packages](https://github.com/users/fujiwara/packages/container/package/rin)\n\n```console\n$ docker pull ghcr.io/fujiwara/rin:v1.1.3\n```\n\n## Configuration\n\n[Configuring Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html).\n\n1. Create SQS queue.\n2. Attach SQS access policy to the queue. [Example Walkthrough 1:](https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html)\n3. [Enable Event Notifications](http://docs.aws.amazon.com/AmazonS3/latest/UG/SettingBucketNotifications.html) on a S3 bucket.\n4. Run `rin` process with configuration for using the SQS and S3.\n\n### config.yaml\n\n```yaml\nqueue_name: my_queue_name    # SQS queue name\n\ncredentials:\n  aws_region: ap-northeast-1\n\nredshift:\n  host: localhost\n  port: 5439\n  dbname: test\n  user: test_user\n  password: '{{ must_env \"REDSHIFT_PASSWORD\" }}'\n  schema: public\n  reconnect_on_error: true # disconnect Redshift on error occurred\n\ns3:\n  bucket: test.bucket.test\n  region: ap-northeast-1\n\nsql_option: \"JSON 'auto' GZIP\"       # COPY SQL option\n\n# define import target mappings\ntargets:\n  - s3:\n      key_prefix: test/foo/ignore\n    discard: true  # Do not import and do not try following targets. Matches only.\n\n- redshift:\n      table: foo\n    s3:\n      key_prefix: test/foo\n\n  - redshift:\n      schema: xxx\n      table: bar\n    s3:\n      key_prefix: test/bar\n    break: true       # Do not try following targets.\n\n  - redshift:\n      schema: $1      # expand by key_regexp captured value.\n      table: $2\n    s3:\n      key_regexp: test/schema-([a-z]+)/table-([a-z]+)/\n\n  - redshift:\n      host: redshift.example.com       # override default section in this target\n      port: 5439\n      dbname: example\n      user: example_user\n      password: example_pass\n      schema: public\n      table: example\n    s3:\n      bucket: redshift.example.com\n      region: ap-northeast-1\n      key_prefix: logs/example/\n    sql_option: \"CSV DELIMITER ',' ESCAPE\"\n```\n\nA configuration file is parsed by [kayac/go-config](https://github.com/kayac/go-config).\n\ngo-config expands environment variables using syntax `{{ env \"FOO\" }}` or `{{ must_env \"FOO\" }}` in a configuration file.\n\nWhen the password for Redshift is empty, Rin will try call [GetClusterCredentials API](https://docs.aws.amazon.com/redshift/latest/APIReference/API_GetClusterCredentials.html) to get a temporary password for the cluster.\n\n#### Credentials\n\nRin requires credentials for SQS and Redshift.\n\n1. `credentials.aws_access_key_id` and `credentials.aws_secret_access_key`\n  - used for SQS and Redshift(COPY query and Data API access).\n2. `credentials.aws_iam_role`\n  - used for Redshift COPY query only.\n  - for SQS and Redshift Data API, Rin will try to get a instance credentials.\n\n## Run\n\n### daemon mode\n\nRin waits new SQS messages and processing it continually.\n\n```\n$ rin -config config.yaml [-debug]\n```\n\n`-config` also accepts HTTP/S3/File URL to specify the location of configuration file.\nFor example,\n\n```\n$ rin -config s3://rin-config.my-bucket/config.yaml\n```\n\n### batch mode\n\nRin process new SQS messages and exit.\n\n```\n$ rin -config config.yaml -batch [-debug]\n```\n\n## Set max execution time\n\nA CLI option `-max-execution-time` is set max execution time for running SQS worker and batch process.\n\n## SQL Drivers\n\nRin has two ways to connect to Redshift.\n\n### `postgres` driver\n\n`postgres` driver is the default. Rin connects to Redshift with PostgreSQL protocol over TCP in the VPC network.\n\n`host`, `port`, `user` and `password` fields are required in the `redshift` section.\n\n```yaml\nredshift:\n  driver: postgres # default\n  host: localhost\n  port: 5439\n  user: test_user\n  password: '{{ must_env \"REDSHIFT_PASSWORD\" }}'\n```\n\n### `redshift-data` driver\n\n`redshift-data` driver connects to Redshift via [Redshift Data API](https://docs.aws.amazon.com/redshift/latest/mgmt/data-api.html).\n\nRedshift Data API does not require a VPC network.\n\nWith provisoned cluster, `driver`, `cluster` and `user` are required.\n\n```yaml\nredshift:\n  driver: redshift-data\n  cluster: your-cluster-name\n  user: test_user\n```\n\nWith Redshift serverless, `driver`, `workgroup` are required.\n\n```yaml\nredshift:\n  driver: redshift-data\n  workgroup: your-workgroup-name\n```\n\nSee also [github.com/mashiike/redshift-data-sql-driver](https://github.com/mashiike/redshift-data-sql-driver).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffujiwara%2Frin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffujiwara%2Frin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffujiwara%2Frin/lists"}