{"id":19622183,"url":"https://github.com/commercetools/commercetools-project-sync","last_synced_at":"2025-04-08T03:12:22.855Z","repository":{"id":33960805,"uuid":"161355453","full_name":"commercetools/commercetools-project-sync","owner":"commercetools","description":"Dockerized CLI application which allows to automatically sync different resources between commercetools projects","archived":false,"fork":false,"pushed_at":"2025-03-31T10:01:38.000Z","size":1621,"stargazers_count":46,"open_issues_count":6,"forks_count":26,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-02T23:56:50.069Z","etag":null,"topics":["commercetools","hacktoberfest","import","project","sync"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/commercetools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-11T15:36:02.000Z","updated_at":"2025-03-31T10:01:42.000Z","dependencies_parsed_at":"2024-03-02T04:23:50.498Z","dependency_job_id":"78256b9b-8c5d-4659-afa9-b253787f83f8","html_url":"https://github.com/commercetools/commercetools-project-sync","commit_stats":{"total_commits":716,"total_committers":15,"mean_commits":"47.733333333333334","dds":0.638268156424581,"last_synced_commit":"408d9258c1b27fbc0ab434fcb17983c4db3f6cf7"},"previous_names":[],"tags_count":53,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commercetools%2Fcommercetools-project-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commercetools%2Fcommercetools-project-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commercetools%2Fcommercetools-project-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commercetools%2Fcommercetools-project-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/commercetools","download_url":"https://codeload.github.com/commercetools/commercetools-project-sync/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767236,"owners_count":20992548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commercetools","hacktoberfest","import","project","sync"],"created_at":"2024-11-11T11:26:28.731Z","updated_at":"2025-04-08T03:12:22.848Z","avatar_url":"https://github.com/commercetools.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# commercetools-project-sync\n[![Build Status](https://travis-ci.com/commercetools/commercetools-project-sync.svg?branch=master)](https://travis-ci.com/commercetools/commercetools-project-sync)\n[![codecov](https://codecov.io/gh/commercetools/commercetools-project-sync/branch/master/graph/badge.svg)](https://codecov.io/gh/commercetools/commercetools-project-sync)\n[![Docker Pulls](https://img.shields.io/docker/pulls/commercetools/commercetools-project-sync)](https://hub.docker.com/r/commercetools/commercetools-project-sync)\n\n\n\u003c!-- START doctoc generated TOC please keep comment here to allow auto update --\u003e\n\u003c!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --\u003e\n\n\n- [What is this?](#what-is-this)\n- [Prerequisites](#prerequisites)\n- [Usage](#usage)\n  - [Delta Sync](#delta-sync)\n  - [Running Multiple syncers](#running-multiple-syncers)\n  - [Running the Docker Image](#running-the-docker-image)\n    - [Download](#download)\n    - [Run](#run)\n- [Examples](#examples)\n- [Scopes](#scopes)\n\n\u003c!-- END doctoc generated TOC please keep comment here to allow auto update --\u003e\n\n### What is this?\n\nA Dockerized CLI application which allows you to automatically sync different resources between two commercetools projects.\nAs of now, these are the supported resources:\n\n- CartDiscounts\n- Categories\n- InventoryEntries\n- Products\n- ProductTypes\n- Types\n- States\n- TaxCategories\n- CustomObjects\n- Customers\n- ShoppingLists\n\n\n### Prerequisites\n\n - Make sure you have installed docker to run the docker image.\n - The following fields are **required** to be set on the following resources (and sub-resources), if they will be\n  synced:\n\n     |  Resource/ Sub-resource |  Required Fields |\n     |---|---|\n     | Product | `key` |\n     | Product Variant  | `key`, `sku`  |\n     | Product Variant Asset (if exists) | `key`  |\n     | ProductType  | `key`  |\n     | Type  | `key`  |\n     | Category  | `key`  |\n     | Category Asset (if exists)  | `key`  |\n     | CartDiscount | `key`  |\n     | InventoryEntry  | `sku`  |\n     | State  | `key`  |\n     | TaxCategory  | `key`  |\n     | CustomObject  | `container` AND `key`  |\n     | Customer  | `key`  |\n     | Customer Address | `key` |\n     | ShoppingList | `key` |\n     | ShoppingList LineItem - Product Variant | `sku` |\n     | ShoppingList TextLineItem | `name` |\n\n - Set the following environment variables before running the application\n   ```bash\n   export SOURCE_PROJECT_KEY = \"source-project-key\"\n   export SOURCE_CLIENT_ID = \"sourceClientId\"\n   export SOURCE_CLIENT_SECRET = \"sourceClientSecret\"\n   export SOURCE_AUTH_URL = \"https://auth.eu-central-1.aws.commercetools.com/oauth/token\" #optional parameter\n   export SOURCE_API_URL = \"https://api.eu-central-1.aws.commercetools.com\" #optional parameter\n   export SOURCE_SCOPES = \"manage_project\" #optional parameter\n   \n   export TARGET_PROJECT_KEY = \"target-project-key\"\n   export TARGET_CLIENT_ID = \"targetClientId\"\n   export TARGET_CLIENT_SECRET = \"targetClientSecret\"\n   export TARGET_AUTH_URL = \"https://auth.eu-central-1.aws.commercetools.com/oauth/token\" #optional parameter\n   export TARGET_API_URL = \"https://api.eu-central-1.aws.commercetools.com\" #optional parameter\n   export TARGET_SCOPES = \"manage_project\" #optional parameter\n   ```\n   Note: For *_AUTH_URL and *_API_URL parameter values,\n    you can use different [authentication endpoints](https://docs.commercetools.com/api/authorization#requesting-an-access-token-using-the-composable-commerce-oauth-20-service) and [API endpoints](https://docs.commercetools.com/api/general-concepts#hosts).\n   \n   Note 2: Project-sync uses `manage_project` [scope](https://docs.commercetools.com/api/scopes) by default.\n    if you want to use different scope you might set `SOURCE_SCOPES` and `TARGET_SCOPES` environment variables, for instance:\n    `export SOURCE_SCOPES=\"manage_products\"` or `export TARGET_SCOPES=\"manage_products, manage_customers\"`(separate multiple scope elements with a comma).\n   \n   Note 3: be careful there is no trailing slash in the URLs. Please make sure the URLs do not include `/` as this would result in a wrong URLs like so and fail the process: `https://auth.eu-central-1.gcp.commercetools.com//oauth/token`\n\n### Usage\n\n   ```bash\n   usage: commercetools-project-sync\n    -f,--full                           By default, a delta sync runs using\n                                        last-sync-timestamp logic. Use this\n                                        flag to run a full sync. i.e. sync\n                                        the entire data set. This option must\n                                        be added after `-s` option.\n    -h,--help                           Print help information.\n    -r,--runnerName \u003carg\u003e               Choose a name for the running sync\n                                        instance. Please make sure the name\n                                        is unique, otherwise running more\n                                        than 1 sync instance with the same\n                                        name would lead to an unexpected\n                                        behaviour. This option must\n                                        be added after `-s` option. (optional parameter)\n                                        default: 'runnerName'.\n    -s,--sync \u003cargs\u003e                    Choose one or more of the following modules\n                                        to run: \"types\", \"productTypes\",\n                                        \"cartDiscounts\", \"customObjects\",\n                                        \"categories\", \"products\",\n                                        \"inventoryEntries\", \"states\",\n                                        \"taxCategories\", \"customers\",\n                                        \"shoppingLists\" or \"all\".\n       --syncProjectSyncCustomObjects   Sync custom objects that were created\n                                        with project sync (this application). This option must\n                                        be added after `-s` option.\n       --productQueryParameters         Pass your customized product fetch limit\n                                        and a product projection predicate to filter  \n                                        product resources to sync in the JSON format. \n                                        Example: \"{\\\"limit\\\": 100, \\\"where\\\": \\\"\n                                        published=true\\\"}\" could be used to fetch \n                                        only published products to sync and limit \n                                        max 100 elements in one page. This option must\n                                        be added after `-s` option.                \n    -v,--version                        Print the version of the application.\n   ```\n\n#### Delta Sync\n\nBy default, running the sync without using `-f` or `--full` option would run a delta sync; which means that only resources\nwhich have been modified since the last time the sync has run would be synced. The application achieves that by\npersisting the last sync timestamp on commercetools using `CustomObjects` on every sync run.\n\nThe last sync timestamp `customObject` for a runner name `testRun` running a **Type Sync** from a source commercetools project with the key `java-sync-source-dev1` looks as follows:\n\n```javascript\n{\n  \"id\": \"0ee39da2-21fd-46b4-9f99-44eae7f249a1\",\n  \"version\": 2,\n  \"container\": \"commercetools-project-sync.testRun.typeSync\",\n  \"key\": \"java-sync-source-dev1\",\n  \"value\": {\n    \"lastSyncDurationInMillis\": 972,\n    \"applicationVersion\": \"3.1.0\",\n    \"lastSyncTimestamp\": \"2019-05-24T11:17:00.602Z\",\n    \"lastSyncStatistics\": {\n      \"processed\": 0,\n      \"failed\": 0,\n      \"created\": 0,\n      \"updated\": 0,\n      \"reportMessage\": \"Summary: 0 types were processed in total (0 created, 0 updated and 0 failed to sync).\"\n    }\n  },\n  \"createdAt\": \"2019-05-24T11:18:12.831Z\",\n  \"lastModifiedAt\": \"2019-05-24T11:19:01.822Z\",\n  \"lastModifiedBy\": {\n    \"clientId\": \"8bV3XSW-taCph843-GQTa8lf\",\n    \"isPlatformClient\": false\n  },\n  \"createdBy\": {\n    \"clientId\": \"8bV3XSW-taCph843-GQTa8lf\",\n    \"isPlatformClient\": false\n  }\n}\n```\n\n- The `container` has the convention: `commercetools-project-sync.{runnerName}.{syncModuleName}`.\n- The `key` contains the source project key.\n- The `value` contains the information  `lastSyncDurationInMillis`, `applicationVersion`, `lastSyncTimestamp` and `lastSyncStatistics`.\n- These custom objects will not be synced with the custom object syncer unless the option --syncProjectSyncCustomObjects is added.\n\n_Note:_ Another `customObject` with the `container` convention `commercetools-project-sync.{runnerName}.{syncModuleName}.timestampGenerator` is also created on the target project for capturing a unified timestamp from commercetools.\n\nRunning a **Full sync** using `-f` or `--full` option will not create any `customObjects`.\n\n#### Understanding the summary reportMessage\n\nIn the best case, the reportMessage should be self-explaining like in the example above. However, in case of errors, this kind of message could appear:\n\n```\nSummary: 1 product(s) were processed in total (0 created, 0 updated, 0 failed to sync and 1 product(s) with missing reference(s))\n```\n\n- `failed to sync` means there is an error from the composable commerce API after the sync tried to create/update the product. Therefore, this product could not be created/updated. The root cause is returned in the previous log lines, immediately during the sync process when this problem happens.\n- `product(s) with missing reference(s)` means that the synced product has some [references](https://docs.commercetools.com/api/types#references) to other products in its attributes. These referenced products do not exist in the target project, therefore the synced product cannot be created/updated. The solution to this problem is to make sure all the references are already synced. This is not counted as `failed to sync` because this reference check happens before the sync itself.\n\n#### Running Multiple Syncers\n\nThe application can sync multiple resources. For example, to run `type` and `productType` sync together, \nthe  `-s` option with `types productTypes` as below:\n```bash\n-s types productTypes\n```\n\n#### Running ProductSync with custom product query parameters\n\nYou might pass your customized product fetch limit, and a product projection predicate to filter product resources to sync in the JSON format.\n\nFor instance:\n\n```bash\n-s products -productQueryParameters \"{\\\"limit\\\": 100, \\\"where\\\": \\\"published=true AND masterVariant(attributes(name=\\\\\\\"attribute-name\\\\\\\" and value=\\\\\\\"attribute-value\\\\\\\"))\\\"}\"\n```\n\nPredicates provide a way for complex filter expressions when querying resources. Refer commercetools docs for more details.\n\nNote: The value of the productQueryParameters argument should be in JSON format and as shown in the above example, please use escape character \\ for the nested double quote values.\nExample: \n```bash\n-s products -productQueryParameters \"{\\\"limit\\\": 100, \\\"where\\\": \\\"published=true AND masterVariant(key= \\\\\\\"variantKey\\\\\\\")\\\"}\"\n```\n\n#### Running the Docker Image\n\n##### Download\n\n```bash\ndocker pull commercetools/commercetools-project-sync:5.5.1\n```\n##### Run\n\n```bash\ndocker run \\\n-e SOURCE_PROJECT_KEY=xxxx \\\n-e SOURCE_CLIENT_ID=xxxx \\\n-e SOURCE_CLIENT_SECRET=xxxx \\\n-e TARGET_PROJECT_KEY=xxxx \\\n-e TARGET_CLIENT_ID=xxxx \\\n-e TARGET_CLIENT_SECRET=xxxx \\\ncommercetools/commercetools-project-sync:5.5.1 -s all\n```\n\n\n### Examples\n - To run the all sync modules from a source project to a target project\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s all\n   ```\n   This will run the following sync modules in the given order:\n 1. `Type` Sync and `ProductType` Sync and `States` Sync and `TaxCategory` Sync and `CustomObject` Sync in parallel.\n 2. `Category` Sync and `InventoryEntry` Sync and `CartDiscount` Sync and `Customer` Sync in parallel.\n 3. `Product` Sync.\n 4. `ShoppingList` Sync.\n\n - To run the type sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s types\n   ```\n\n - To run the productType sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s productTypes\n   ```\n\n- To run the states sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s states\n   ```\n- To run the taxCategory sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s taxCategories\n   ```\n\n- To run the category sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s categories\n   ```\n\n- To run the product sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s products\n   ```\n\n- To run the cartDiscount sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s cartDiscounts\n   ```\n\n- To run the inventoryEntry sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s inventoryEntries\n   ```\n\n- To run the customObject sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s customObjects\n   ```\n\n- To run the customer sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s customers\n   ```\n  \n- To run the shoppingList sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s shoppingLists\n   ```\n- To run both products and shoppingList sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s products shoppingLists\n   ```\n  \n- To run type, productType and shoppingList sync\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s types productTypes shoppingLists\n   ```\n\n- To run all sync modules using a runner name\n   ```bash\n   docker run commercetools/commercetools-project-sync:5.5.1 -s all -r myRunnerName\n   ```\n\n## Scopes\nIf you want to narrow down the credential scopes instead of using `manage_project` scope, make sure you grant the correct scopes for every resource you intend to sync.\n\n### Required scope for all resources\nFor project-sync to run successfully, it is **required to grant access to custom objects for the target project**. Therefore, one of the below scope must always be granted for any resources: `manage_products` OR `manage_orders` OR `manage_customers` OR `manage_key_value_documents`\n\n### Required scope per resource\n\nIn addition to the above-mentioned scope, the table below shows the minimal scope for every resource\n\n| Resource          | Scope |\n|-------------------| --- |\n| Cart discounts    | `manage_cart_discounts` |\n| Categories        | `manage_categories`|\n| Inventory entries | `manage_products`|\n| Products          | `manage_products`|\n| Product types     | `manage_products`|\n| Types             | `manage_types`|\n| States            | `manage_states`|\n| Tax categories    | `manage_tax_categories`|\n| Custom objects    | `manage_products` OR `manage_orders` OR `manage_customers` OR `manage_key_value_documents`|\n| Customers         | `manage_customers`|\n| Shopping lists    | `manage_shopping_lists`|\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommercetools%2Fcommercetools-project-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcommercetools%2Fcommercetools-project-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommercetools%2Fcommercetools-project-sync/lists"}