{"id":13512579,"url":"https://github.com/jitsecurity/dynamo-data-transform","last_synced_at":"2025-03-31T00:30:38.926Z","repository":{"id":37859721,"uuid":"499310496","full_name":"jitsecurity/dynamo-data-transform","owner":"jitsecurity","description":"Dynamo Data Transform is an easy to use data transformation tool for DynamoDB","archived":false,"fork":false,"pushed_at":"2025-02-13T08:28:54.000Z","size":1423,"stargazers_count":51,"open_issues_count":5,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-11T02:39:27.369Z","etag":null,"topics":["aws","dyanmodt","dynamo","dynamodb","migration-tool","transform","transformation-tool"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jitsecurity.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-02T22:36:45.000Z","updated_at":"2025-02-11T09:07:07.000Z","dependencies_parsed_at":"2023-02-18T17:00:32.112Z","dependency_job_id":"520f903a-e6ee-404a-8a6d-732cd17b9b6f","html_url":"https://github.com/jitsecurity/dynamo-data-transform","commit_stats":{"total_commits":68,"total_committers":8,"mean_commits":8.5,"dds":0.4411764705882353,"last_synced_commit":"9da3f82c323e50d3ed71dfc8c2354e027c116518"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jitsecurity%2Fdynamo-data-transform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jitsecurity%2Fdynamo-data-transform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jitsecurity%2Fdynamo-data-transform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jitsecurity%2Fdynamo-data-transform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jitsecurity","download_url":"https://codeload.github.com/jitsecurity/dynamo-data-transform/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246399816,"owners_count":20770907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","dyanmodt","dynamo","dynamodb","migration-tool","transform","transformation-tool"],"created_at":"2024-08-01T04:00:21.260Z","updated_at":"2025-03-31T00:30:38.595Z","avatar_url":"https://github.com/jitsecurity.png","language":"JavaScript","readme":"![ddt_graphic1x_tl](https://user-images.githubusercontent.com/101042972/172161782-f4a3e15c-fdf2-42f1-a14d-434b109e7d2a.png)\n\u003cp\u003e\n  \u003ca href=\"https://www.jit.io/jit-open-source?utm_source=github\u0026utm_medium=badge\u0026utm_campaign=dynamoDataTransform\u0026utm_id=oss\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Secured%20by-Jit-B8287F?style=?style=plastic\" /\u003e \n    \n  \u003ca href=\"https://www.serverless.com\"\u003e\n    \u003cimg src=\"http://public.serverless.com/badges/v3.svg\"\u003e\n  \u003c/a\u003e\n    \n  \u003ca href=\"https://www.npmjs.com/package/dynamo-data-transform\"\u003e\n    \u003cimg src=\"https://img.shields.io/npm/dw/dynamo-data-transform\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n  \n\n\nDynamo Data Transform is an easy to use data transformation tool for DynamoDB.\n\nIt allows performing powerful data transformations using simple Javascript commands, without the risk of breaking your database.\nAvailable as a [Serverless plugin](#-serverless-plugin), [npm package](#standalone-npm-package) and even as an [interactive CLI](#-interactive-cli), Dynamo Data Transform saves you time and keeps you safe with features like dry-running a data transformation and even rolling back your last trasnformation if needed.\n\n**Features**\n\n- Seemless data transformations management.\n- Support for multiple stages.\n- History of executed data transformations.\n- Dry run option for each command (by suppling --dry flag, the data will be printed instead of stored).\n- Safe \u0026 Secure preparation data\n- Store preparation data in a private s3 bucket. [Prepare data for your data transformation](#usage-and-command-line-options)\n\n## Table of contents\n\n- [Quick Start](#quick-start)\n  - [Serverless plugin](#-serverless-plugin)\n  - [Standalone npm package](#standalone-npm-package)\n  - [Interactive CLI](#-interactive-cli)\n- [Creating your first data transformation](#creating-your-first-data-transformation)\n- [Usage and command-line options](#usage-and-command-line-options)\n- [What happens behind the scenes](#what-happens-behind-the-scenes)\n- [Examples](#examples)\n- [The data transformation process](https://github.com/jitsecurity/dynamo-data-transform/blob/main/docs/zero_downtime_data_transformation_process.md)\n\n## Quick Start\n### ⚡ Serverless plugin\n- Install\n```bash\nnpm install dynamo-data-transform --save-dev\n```\n- Add the tool to your serverless.yml\nRun:\n```bash\nnpx serverless plugin install -n dynamo-data-transform\n```\nOr add manually to your serverless.yml:\n```YML\nplugins:\n  - dynamo-data-transform\n```\n- Run\n```bash\nsls dynamodt --help\n```\n\n### Standalone npm package\n- Install the tool\n```bash\nnpm install -g dynamo-data-transform -s\n```\n- Run the tool\n```bash\ndynamodt help\n```\nOr with the shortcut\n```bash\nddt help\n```\n### 💻 Interactive CLI\nAfter installing the npm package, run:\n```bash\ndynamodt -i\n```\n![cli gif](https://user-images.githubusercontent.com/35347793/172045910-d511e735-2d31-4713-bb64-5f55a900941c.gif)\n\n\n\n## Creating your first data transformation\n1. Intialize data-transformations folder\nServerless (the plugin reads the table names from the serverless.yml file):\n```bash\nsls dynamodt init --stage \u003cstage\u003e\n```\nStandalone:\n```bash\nddt init --tableNames \u003ctable_names\u003e\n```\n\nOpen the generated data transformation file 'v1_script-name.js' file and implement the following functions:\n  - transformUp: Executed when running `dynamodt up`\n  - transformDown: Executed when running `dynamodt down -t \u003ctable\u003e`\n  - prepare (optional): Executed when running `dynamodt prepare -t \u003ctable\u003e --tNumber \u003ctransformation_number\u003e`\n\nThe function parameters:\n  - ddb: The DynamoDB Document client object see [DynamoDB Client](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb)\n  - isDryRun: Boolean indicating if --dry run supplied. You can use it to print/log the data instead of storing it.\n  - preparationData: if you stored the preparation data using `dynamodt prepare`, you can use it here.\n\n2. Run the data transformation\n```bash\ndynamodt up\n```\n\n\n## Data Transformation Script Format\n  Make sure your script name contains the transformation number, for example: v1_transformation_script\n```js\nconst { utils } = require('dynamo-data-transform')\n\nconst TABLE_NAME = 'UsersExample'\n\nconst transformUp = async ({ ddb, isDryRun, preparationData }) =\u003e {\n  // your code here... \n  // return { transformed: 50 } // return the number of transformed items\n}\n\nconst transformDown = async ({ ddb, isDryRun, preparationData }) =\u003e {\n  // your code here...\n  // return { transformed: 50 } // return the number of transformed items\n}\n\nconst prepare = async ({ ddb, isDryRun }) =\u003e {\n  // your code here...\n  // return { transformed: 50 } // return the number of transformed items\n}\n\nmodule.exports = {\n  transformUp,\n  transformDown,\n  prepare, // optional\n  transformationNumber: 1,\n}\n```\n\n\n\n## Usage and command-line options\n\nList available commands:\nServerless plugin:\n```bash\nsls dynamodt --help\n```\nStandalone npm package:\n```bash\ndynamodt help\n```\n\n\nTo list all of the options for a specific command run:\nServerless plugin:\n```bash\nsls dynamodt \u003ccommand\u003e --help\n```\n\nStandalone npm package:\n```bash\ndynamodt \u003ccommand\u003e --help\n```\n\n## What happens behind the scenes\n- When a data transformation runs for the first time, a record in your table is created. This record is for tracking the executed transformations on a specific table.\n\n\n\n## Examples\n[Examples of data transformation code](https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations/UsersExample)\n\n\n### Insert records\n\n```js\n// Seed users data transformation\nconst { utils } = require('dynamo-data-transform');\nconst { USERS_DATA } = require('../../usersData');\n\nconst TABLE_NAME = 'UsersExample';\n\n/**\n * @param {DynamoDBDocumentClient} ddb - dynamo db document client https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb\n * @param {boolean} isDryRun - true if this is a dry run\n */\nconst transformUp = async ({ ddb, isDryRun }) =\u003e {\n  return utils.insertItems(ddb, TABLE_NAME, USERS_DATA, isDryRun);\n};\n\nconst transformDown = async ({ ddb, isDryRun }) =\u003e {\n  return utils.deleteItems(ddb, TABLE_NAME, USERS_DATA, isDryRun);\n};\n\nmodule.exports = {\n  transformUp,\n  transformDown,\n  transformationNumber: 1,\n};\n```\n\n### Add a new field to each record\n```js\n// Adding a \"randomNumber\" field to each item\nconst { utils } = require('dynamo-data-transform');\n\nconst TABLE_NAME = 'UsersExample';\n\nconst transformUp = async ({ ddb, isDryRun }) =\u003e {\n  const addRandomNumberField = (item) =\u003e {\n    const updatedItem = { ...item, randomNumber: Math.random() };\n    return updatedItem;\n  };\n  return utils.transformItems(ddb, TABLE_NAME, addRandomNumberField, isDryRun);\n};\n\nconst transformDown = async ({ ddb, isDryRun }) =\u003e {\n  const removeRandomNumberField = (item) =\u003e {\n    const { randomNumber, ...oldItem } = item;\n    return oldItem;\n  };\n  return utils.transformItems(ddb, TABLE_NAME, removeRandomNumberField, isDryRun);\n};\n\nmodule.exports = {\n  transformUp,\n  transformDown,\n  transformationNumber: 2,\n};\n```\n\nFor more examples of data transformation code, see the [examples](https://github.com/jitsecurity/dynamo-data-transform/tree/main/examples/serverless-localstack/data-transformations/UsersExample) folder in the repository.\n\n\nRead how to automate data-transformation process here:\nhttps://www.infoq.com/articles/dynamoDB-data-transformation-safety/\n","funding_links":[],"categories":["Projects by main language","Tools"],"sub_categories":["javascript"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjitsecurity%2Fdynamo-data-transform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjitsecurity%2Fdynamo-data-transform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjitsecurity%2Fdynamo-data-transform/lists"}