{"id":20610905,"url":"https://github.com/darkroomengineering/migro","last_synced_at":"2025-04-15T04:32:32.788Z","repository":{"id":107025459,"uuid":"471497246","full_name":"darkroomengineering/migro","owner":"darkroomengineering","description":"Script for migrating data into Contentful","archived":false,"fork":false,"pushed_at":"2022-10-06T15:50:42.000Z","size":545,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T16:03:25.309Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/darkroomengineering.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-03-18T19:47:38.000Z","updated_at":"2024-12-21T14:02:28.000Z","dependencies_parsed_at":"2024-02-19T21:14:16.989Z","dependency_job_id":null,"html_url":"https://github.com/darkroomengineering/migro","commit_stats":null,"previous_names":["darkroomengineering/migro"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darkroomengineering%2Fmigro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darkroomengineering%2Fmigro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darkroomengineering%2Fmigro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/darkroomengineering%2Fmigro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/darkroomengineering","download_url":"https://codeload.github.com/darkroomengineering/migro/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249006599,"owners_count":21197307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T10:18:19.284Z","updated_at":"2025-04-15T04:32:32.772Z","avatar_url":"https://github.com/darkroomengineering.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Migro\nScript for migrating data or bulk modifications to existent data into Contentful via the CMA-API.\n\n#### **WHY**\n\nEach migration or changes to existent data is almost unique requiring coding everytime. This Script sets the basic flow and functions that can be reused doing the process easier.\n\nWe may spent lots of hours writing a code to handle migrations and bulk modifications BUT we will NEVER spent tons of hours doing a repetitive task. \n\n#### **PURPOSES**\n\n- Migrate data into Contentful from TSV file:\n\n Migrating data from other CMS can be done by exporting the data into a TSV file and using this to populate the content in Contentful. Check [Blog migration example](https://github.com/studio-freight/into-contentful/tree/blog-example) for more information about this.\n\n- Bulk Modification of existing Data:\n\n Changing existing entries of content type is done by gettin all the entries of the content type and updating the content. Check [Bulk modification example](https://github.com/studio-freight/into-contentful/tree/bulk-modifications) for more information about this.\n \n - Migrate and restructure data from one contentful Org to another one:\n\n Data will be fetch from one Contentful and sent to the other Contentful after restructuring it to the new needs. Check [Migration Between Contentful Organizations](https://github.com/studio-freight/migro/tree/from-contentful-to-contentful) for more information about this.\n\n#### **SETTING UP**\n\n- Install:\n\n```\npnpm i\n```\n\n- Set envs:\n\nUse env.example for guidance. Create a .env file on the root folder with the following variables:\n\n```\nACCESS_TOKEN = This is the CMA access token of contentful, watch out is not the same as the other APIs.\nENVIRONMENT_ID = The Environment ID from the space we are migrating into.\nSPACE_ID = The Space ID of the Contentful organization.\n\n### Only use this if migrating from one Contentful Organization to other\n### These are the environment and space ID from the Organization that has the data to migrate\nEXTERNAL_ENVIRONMENT_ID=The Environment ID from the space we are migrating from.\nEXTERNAL_SPACE_ID=The Space ID of the Contentful organization we are migrating from.\n```\n\n- Posible Configuration:\n\nAll code will be done in index.js in root folder. Here you can set up configs.\n\n```\npathFile: Save your input data file in the input folder or change this path accordingly. Example: const pathFile = \"./input/\" + \"your-file-name.tsv\";\nbatchSize: How many entries of the content type are created in sequence.\noffset: From which value script should begin. Useful when script has stop and want to restart from certain point.\ncontentTypeId: The content type ID from Contentful which we want to create or update.\nexternalContentTypeId: The content type ID from Contentful which we want to migrate data from. (Only use when migrating from contentful to contentful)\n```\n\n#### **METHODS**\n\n```javascript\nintoContentful.getContentTypeStructure();\n```\n\nThe hardest part is structuring our new data in the exact way Contentful expects it. The best and fastest way we find to achieve this is first creating an entry of the content type and gettin the API response. We can use this response as a mock up for our data. \nThis method saves the API response of the content type in a JSON file called content-type-body.json in the input folder.\n\n```javascript\nintoContentful.setDebug();\n```\n\nThis method breaks script before starting the batch creation of the entries of the content type. Is useful for consoling values of the input data or to test how we are manipulating data.\n\n```javascript\nintoContentful.setPublishJustOneBatchForTesting();\n```\n\nThis method breaks script after first batch creation to test if data creation or update is working as expected. Recomend to use ```batchSize = 1``` to just create one entry.\n\n```javascript\nintoContentful.run();\n\n//Options:\n\n// For creating entries from TSV file:\nawait intoContentful.run(\"file\", \"create\");\n\n// For Updating entries from existing data:\nawait intoContentful.run(\"cma\", \"update\");\n\n// For Migrating from one Contentful Organization to another one:\nawait intoContentful.run(\"externalCma\", \"create\");\n```\n\nMandatory method that executes script.\n\n\n#### **USAGE**\n\nFor running the script just:\n\n```\nnode index\n```\n\n#### **HOW IT WORKS**\n\nIf you made it up to here you really need to create/update tons of entries, we hope this helps.\n\nInput data is treated as two dimensional array, where each row of the array will contain the data for each entry. \nFirst row should be use as a guidance using the existing keys of the input data (will always be skipped but **must** be present). Second row should have the content type fields name.\n\nTSV file structrue example:\n```\ntitle url topic\ntitle slug category\nmy-new-title /my-new-title migrations\n```\nThis will create an object with the following structure:\n\n```javascript\nrow = {title: 'my-new-title', slug: '/my-new-title', category: 'migrations'}\n```\n\nTo transform our data to the format expected by Contentful we have to do it inside the ```myDataMunging```function.\n\nThe ```myDataMunging```function will iterate the input data and will return two arrays. \n- ```row```: Has each row of the TSV file as an object (as explained above). \n- ```parsed```: Carries the transformed data. This means that you must apply all the changes and then pushed into parsed.\n\nBeacuse usually content types have one or many referenes to other content types this script runs batches. For each ```N``` size batch script will iterate the TSV file for the N rows. For linked content types we must provide the unique ID, this process of creating or fetching the IDs must happen inside the ```myDataMunging```function. The parsed data for each batch will be saved in a JSON file and then this file will be used for creating or updating the Content Type.   \n\n\n#### AVOID ####\n\nAvoid concurrent loops, it may hit contentful API rate-limits.\n\n```javascript\n\n// Bad\n await Promise.all(data.map(async (i) =\u003e {\n// your code\n}));\n\n// Good\nfor await (const i of data){\n// your code\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarkroomengineering%2Fmigro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdarkroomengineering%2Fmigro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdarkroomengineering%2Fmigro/lists"}