{"id":25176237,"url":"https://github.com/redturtle/redturtle.importer.base","last_synced_at":"2025-10-11T17:13:29.771Z","repository":{"id":49421556,"uuid":"135166204","full_name":"RedTurtle/redturtle.importer.base","owner":"RedTurtle","description":"Imports contents from a json source","archived":false,"fork":false,"pushed_at":"2022-08-10T10:49:45.000Z","size":279,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-10-11T17:13:29.451Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RedTurtle.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":"LICENSE.GPL","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-28T13:41:36.000Z","updated_at":"2021-10-01T07:12:25.000Z","dependencies_parsed_at":"2022-09-24T03:12:46.052Z","dependency_job_id":null,"html_url":"https://github.com/RedTurtle/redturtle.importer.base","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/RedTurtle/redturtle.importer.base","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTurtle%2Fredturtle.importer.base","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTurtle%2Fredturtle.importer.base/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTurtle%2Fredturtle.importer.base/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTurtle%2Fredturtle.importer.base/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RedTurtle","download_url":"https://codeload.github.com/RedTurtle/redturtle.importer.base/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RedTurtle%2Fredturtle.importer.base/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279008111,"owners_count":26084396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2025-02-09T13:16:41.967Z","updated_at":"2025-10-11T17:13:29.747Z","avatar_url":"https://github.com/RedTurtle.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"=======================\nRedTurtle importer base\n=======================\n\nTool to migrate contents between Plone sites based on transmogrifier.\n\nThis tool works in addition with `redturtle.exporter.base`__\n\n__ https://pypi.org/project/redturtle.exporter.base\n\n\nDependencies\n============\n\nThis product is made over other useful tools:\n\n* `collective.jsonmigrator`__\n* `collective.transmogrifier`__\n* `transmigrify.dexterity`__\n\n__ https://github.com/collective/collective.jsonmigrator\n__ https://github.com/collective/collective.transmogrifier\n__ https://github.com/collective/transmogrify.dexterity\n\nThese tools are not yet actively maintained, so we moved useful parts into this\nproject to have a working Python 3 importer based on transmogrifier.\n\n\nFeatures\n========\n\n- Handle migration for basic content-types\n- Discussions migration\n- Customizable import procedure via blueprints\n- Extensible with more specific blueprints\n- Possibility to customize specific step options with custom adapters\n- Review view after migration with process results\n\nInstallation\n============\n\nInstall redturtle.importer.base by adding it to your buildout::\n\n    [buildout]\n\n    ...\n\n    eggs =\n        redturtle.importer.base\n\n\nand then running ``bin/buildout``\n\nYou don't have to install it. In this way, after the data migration, you can\nremove it from the buildout and everything is clean.\n\n\nUsage\n=====\n\nMigration view\n--------------\nTo start a migration, you only need to call `@@data-migration` view on site root.\n\nIn this view you can see the blueprint configuration (base and overrided), and start the process.\n\nPipelines customization\n-----------------------\n\nThis tool is based on transmogrifier and works with blueprints.\nA blueprint is basically a config file that lists all the steps needed for the migration.\n\nThis product has a `default blueprint`__ for basic migrations, that can be used as is.\n\nDefault blueprint can be easily customized using a `.migrationconfig.cfg` file located in buildout root folder.\n\nIn this file you can override already present parts/variables (like `pipelines` into `[transmogrifier]` section) or \nadd new ones (for example a new step).\n\nFor example, catalogsource step can be configured with some queries like this::\n\n    [catalogsource]\n    catalog-query = {'portal_type': ['Document', 'Event', 'News Item']}\n    ...\n\nIn `.migrationconfig.cfg` file, under `[catalogsource]` section, you also need to set some settings about how to retrieve data on source site::\n\n    [catalogsource]\n    ...\n    remote-url = http://localhost:8080\n    remote-root = /Plone\n    catalog-path = /Plone/portal_catalog\n    remote-username = username\n    remote-password = password\n    ...\n\n\nBefore running a migration, you can check the final configuration in `@@data-migration` view.\n\n\n__ https://github.com/RedTurtle/redturtle.importer.base/blob/python3/src/redturtle/importer/base/transmogrifier/redturtleplone5.cfg\n\n\ncatalogsource configuration\n---------------------------\n\nThis is an example of `[catalogsource]` part::\n \n    [catalogsource]\n    ...\n    remote-url = http://localhost:8080\n    remote-root = /Plone\n    catalog-path = /Plone/portal_catalog\n    remote-username = username\n    remote-password = password\n\nRequired options are:\n\n- `remote-url`: The url of source Plone site\n- `remote-root`: The path of Plone site that we want to migrate\n- `remote-username`: Credentials to access to source site\n- `remote-password`: Credentials to access to source site\n\nAdditional options are:\n\n- `default-local-path`: A path where save migrate contents in destination Site. This path will replace item's root path. Destination root path is not needed in this path.\n- `skip-private`: Boolean to migrate or not private items into destination. Default is `False`.\n- `remote_skip_paths`: A list of paths from source site that will be skipped during migration process.\n- `incremental-migration`: Boolean value. If a content already migrate hasn't been modified since last migration, don't override it. Default is `False`.\n- `ignore-cache`: Boolean value. If True, ignore local cache and always get content data from source site.\n- `cache-dir`: Local folder where migration data cache will be stored. Default is `/tmp/migration/migration_cache`.\n- `migration-dir`: Local fodler where migration support files (for final summary for example) will be saved. Default is '/tmp/migration'.\n\n\nCustom types mapping\n--------------------\n\n*contentsmapping* is the section that allows to convert one portal_type to another before object creation.\n\nThere is a plugin system based on subscribers that allows plugins to add custom mappings.\n\nYou need to register a subscriber for `IPortalTypeMapping` like this::\n\n    \u003csubscriber\n        factory=\".types_mapping.MyCustomMapping\"\n        provides=\"redturtle.importer.base.interfaces.IPortalTypeMapping\"/\u003e\n\nAnd then you need to create the class::\n\n    @adapter(IPloneSiteRoot, IBrowserRequest)\n    @implementer(IPortalTypeMapping)\n    class MyCustomMapping(object):\n        order = 100\n\n        def __init__(self, context, request):\n            self.context = context\n            self.request = request\n\n        def __call__(self, item, typekey):\n            \"\"\"\n            \"\"\"\n            portal_type = item[typekey]\n            if portal_type == \"Type-A\":\n                item[typekey] = \"Type-B\"\n                ...\n            return item\n\n\nCustom steps for specific portal types\n--------------------------------------\n\nIf you are migrating a content-type that needs some manual fixes after the creation, you can do it with an adapter.\n\nYou only need to register an adapter for your content-type like this::\n\n    \u003cadapter\n      for=\"my.product.interfaces.IMyInterface\"\n      provides=\"redturtle.importer.base.interfaces.IMigrationContextSteps\"\n      factory=\".steps.MyTypeSteps\"\n    /\u003e\n\n\nAnd then you need to provide a \"doSteps\" method in the class::\n\n    from redturtle.importer.base.interfaces import IMigrationContextSteps\n    from zope.interface import implementer\n\n    @implementer(IMigrationContextSteps)\n    class MyTypeSteps(object):\n\n        def __init__(self, context):\n            self.context = context\n\n        def doSteps(self):\n            \"\"\"\n            do something here\n            \"\"\"\n\nExample specific importers\n==========================\n\nThere are some per-project importers that we used to migrate some projects and you can use them as a starting point\nto develop new ones.\n\nThey are basically packages that you need to include in your buildout and provides some custom steps for specific types:\n\n- `redturtle.importer.rer`__\n- `redturtle.importer.volto`__\n\n__ https://github.com/RedTurtle/redturtle.importer.rer\n__ https://github.com/RedTurtle/redturtle.importer.volto\n\n\nImport Users and groups\n=======================\n\nYou can also import users and groups from source site.\n\nYou only need to add a section to your migration config file like this::\n\n    [users_and_groups]\n    import-users = True\n    import-groups = True\n\nThe tool will call two views from source site and will use the settings \n(remote-url, remote-root and credentials) from *[catalogsource]* section.\n\nThis import is performed after transmogrifier steps.\n\n\nContribute\n==========\n\n- Issue Tracker: https://github.com/RedTurtle/redturtle.importer.base/issues\n- Source Code: https://github.com/RedTurtle/redturtle.importer.base\n\nCredits\n=======\n\nThis product has been developed with some help from\n\n.. image:: https://kitconcept.com/logo.svg\n   :alt: kitconcept\n   :width: 300\n   :height: 80\n   :target: https://kitconcept.com/\n\nLicense\n=======\n\nThe project is licensed under the GPLv2.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredturtle%2Fredturtle.importer.base","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fredturtle%2Fredturtle.importer.base","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fredturtle%2Fredturtle.importer.base/lists"}