{"id":16714731,"url":"https://github.com/sakama/embulk-output-dynamodb","last_synced_at":"2025-03-23T14:31:53.017Z","repository":{"id":59153366,"uuid":"51748407","full_name":"sakama/embulk-output-dynamodb","owner":"sakama","description":"AWS DynamoDB output plugin for Embulk","archived":false,"fork":false,"pushed_at":"2022-11-09T03:21:34.000Z","size":114,"stargazers_count":7,"open_issues_count":1,"forks_count":7,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-26T14:03:33.304Z","etag":null,"topics":["aws","aws-dynamodb","embulk","embulk-output-plugin","embulk-plugin"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sakama.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-15T10:33:37.000Z","updated_at":"2022-11-09T03:41:17.000Z","dependencies_parsed_at":"2022-09-13T10:50:30.288Z","dependency_job_id":null,"html_url":"https://github.com/sakama/embulk-output-dynamodb","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sakama%2Fembulk-output-dynamodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sakama%2Fembulk-output-dynamodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sakama%2Fembulk-output-dynamodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sakama%2Fembulk-output-dynamodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sakama","download_url":"https://codeload.github.com/sakama/embulk-output-dynamodb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244310839,"owners_count":20432623,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-dynamodb","embulk","embulk-output-plugin","embulk-plugin"],"created_at":"2024-10-12T21:06:47.874Z","updated_at":"2025-03-23T14:31:52.662Z","avatar_url":"https://github.com/sakama.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AWS DynamoDB output plugin for Embulk\n[![Build Status](https://travis-ci.org/sakama/embulk-output-dynamodb.svg?branch=master)](https://travis-ci.org/sakama/embulk-output-dynamodb)\n\n[Embulk](http://www.embulk.org/) output plugin to load/insert data into [AWS DynamoDB](http://aws.amazon.com/dynamodb/)\n\n## Overview\n\n* **Plugin type**: output\n* **Load all or nothing**: no\n* **Resume supported**: no\n* **Cleanup supported**: no\n\n## Configuration\n\n### General Parameters\n\n- **mode** (string optional, default:upsert)\n    - **upsert**\n    \n        Creates a new item, or replaces an old item with a new item.\n        If an item that has the same primary key as the new item already exists in the specified table, the new item completely replaces the existing item.\n        This mode uses [BatchWriteItem](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) and you can set `max_put_items` parameter.\n    \n    - **upsert_with_expression**\n    \n        Edits an existing item's attributes, or adds a new item to the table if it does not already exist.\n        If you have data with following schema, you can set `update_expression` as follows.\n        ```\n        - col1(Number primaryKey)\n        - col2(Number)\n        - col3(String)\n        - col4(Number)\n        - col5(String)\n        ```\n        \n        ```yaml\n        mode: upsert_with_expression\n        # update where col1 has #col1\n        update_expression: \"add #col2 :col2 set #col3 = :col3 set #col4 = #col4 - :col4 remove #col5\"\n        ```\n\n- **region** (string required) Region of Amazon DynamoDB\n- **auto_create_table** (boolean optional, default: false) [See below]()\n- **table** (string required) Table name of Amazon DynamoDB\n\n    table and option accept [Time#strftime](http://ruby-doc.org/core-2.2.3/Time.html#method-i-strftime) format to construct table name.\n    Table name is formatted at runtime using the local time of the embulk server.\n    \n    For example, with the configuration below, data is inserted into tables table_2015_04, table_2015_05 and so on.\n    ```\n    out:\n      type: dynamodb\n      table: table_%Y_%m\n    ```\n- **primary_key** (string, required when use auto_create_table) primary key name\n- **primary_key_type** (string, required when use auto_create_table) primary key type\n- **write_capacity_units** (int optional) Provisioned write capacity units\n    - **normal** (int optional) value that will be set after execution\n    - **raise** (int optional) value that will be set before execution\n- **read_capacity_units** (int optional) Provisioned read capacity units\n    - **normal** (int optional) value that will be set after execution\n    - **raise** (int optional) value that will be set before execution\n\n        - Up to 30 before execution and down to 5 after execution.\n        ```yaml\n        write_capacity_units:\n        normal: 5\n        raise: 30\n        ```\n\n        - Up to 30 before execution\n        ```yaml\n        write_capacity_units:\n        raise: 30\n        ```\n    \n        - Down to 5 after execution\n        ```yaml\n        write_capacity_units:\n        normal: 5\n        ```\n\n        **NOTICE**: You can decrease the read_capacity_units or write_capacity_units settings for a table, but no more than [4 times per table in a single UTC calendar day](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#d0e53216).\n\n- **max_put_items** (int optional, default:25)\n  Max put items at every batch writing requests when `upsert mode`.\n  A single call to [BatchWriteItem](http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) can write up to 16 MB of data, which can comprise as many as 25 put requests.\n  If you have exceeded data, decrease this value.\n\n### Authentication Parameters\n\n6 authentication methods are supported.\n\n- **basic**\n\n    Use access_key_id and secret_access_key\n    \n    ```yaml\n    auth_method: basic\n    access_key_id: YOUR_ACCESS_KEY_ID\n    secret_access_key: YOUR_SECRET_ACCESS_KEY\n    ```\n\n    - **access_key_id**: AWS access key ID (string, required)\n    \n    - **secret_access_key**: AWS secret access key (string, required)\n\n- **env**\n\n    Read AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY) from environment variables.\n    \n    ```yaml\n    auth_method: env\n    ```\n\n- **instance**\n    \n    Use ECS task role or EC2 instance profile.\n    \n    ```yaml\n    auth_method: instance\n    ```\n\n- **profile**\n\n    Read credentials from file. Format of the file is as following, where `[...]` is a name of profile.\n    \n    ```yaml\n    auth_method: profile\n    profile_file: /path/to/profilefile\n    profile_name: PROFILE_NAME\n    ```\n    \n    - **profile_file**: path to a profiles file. (string, default: given by AWS_CREDENTIAL_PROFILES_FILE environment varialbe, or ~/.aws/credentials).\n    \n    - **profile_name**: name of a profile. (string, default: `\"default\"`)\n    \n    ```\n    [default]\n    aws_access_key_id=YOUR_ACCESS_KEY_ID\n    aws_secret_access_key=YOUR_SECRET_ACCESS_KEY\n    \n    [profile2]\n    ...\n    ```\n\n- **properties**\n\n    Read aws.accessKeyId and aws.secretKey from Java system properties\n    \n    ```yaml\n    auth_method: properties\n    ```\n\n- **session**\n\n    Use temporary-generated access_key_id, secret_access_key and session_token.\n    \n    ```yaml\n    auth_method: session\n    access_key_id: YOUR_ACCESS_KEY_ID\n    secret_access_key: YOUR_SECRET_ACCESS_KEY\n    session_token: SESSION_TOKEN\n    ```\n    \n    - **access_key_id**: AWS access key ID (string, required)\n    \n    - **secret_access_key**: AWS secret access key (string, required)\n    \n    - **session_token**: session token (string, required)\n\n## Example\n\n```yaml\nout:\n  type: dynamodb\n  mode: upsert\n  region: us-west-2\n  auth_method: basic\n  access_key_id: ABCXYZ123ABCXYZ123\n  secret_access_key: ABCXYZ123ABCXYZ123\n  auto_create_table: true\n  table: dynamotable\n  primary_key: id\n  primary_key_type: Number\n  write_capacity_units:\n    normal: 5\n    raise: 20\n  read_capacity_units:\n    normal: 6\n    raise: 30\n```\n\n\n## Build\n\n```\n$ ./gradlew gem  # -t to watch change of files and rebuild continuously\n```\n\n## Test\n\n```\n$ ./gradlew test  # -t to watch change of files and rebuild continuously\n```\n\nTo run unit tests, you need to setup DynamoDB on your local environment.\nPlease reference the documentation [Running DynamoDB on Your Computer](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) on AWS Developer Guide.\n```shell-session\n$ mkdir /path/to/dynamodb\n$ cd /path/to/dynamodb\n$ wget http://dynamodb-local.s3-website-us-west-2.amazonaws.com/dynamodb_local_latest.tar.gz\n$ tar zxvf dynamodb_local_latest.tar.gz\n$ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsakama%2Fembulk-output-dynamodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsakama%2Fembulk-output-dynamodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsakama%2Fembulk-output-dynamodb/lists"}