{"id":13531742,"url":"https://github.com/collective/collective.exportimport","last_synced_at":"2025-04-01T20:30:31.060Z","repository":{"id":36951608,"uuid":"338090395","full_name":"collective/collective.exportimport","owner":"collective","description":"Export and import content and other data from and to Plone","archived":false,"fork":false,"pushed_at":"2025-03-20T11:04:49.000Z","size":1750,"stargazers_count":16,"open_issues_count":44,"forks_count":19,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-03-24T21:05:46.587Z","etag":null,"topics":["exporting","hacktoberfest","importing","migrations","plone","python","zope"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/collective.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":"LICENSE.GPL","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-11T16:50:56.000Z","updated_at":"2025-03-03T15:07:50.000Z","dependencies_parsed_at":"2023-10-11T11:08:26.926Z","dependency_job_id":"e5f0c0f5-1f3f-4a2f-850e-583f6be256a4","html_url":"https://github.com/collective/collective.exportimport","commit_stats":{"total_commits":516,"total_committers":29,"mean_commits":17.79310344827586,"dds":"0.34302325581395354","last_synced_commit":"bae323d313a668b4444020a825f86e5eb3cbed4d"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collective%2Fcollective.exportimport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collective%2Fcollective.exportimport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collective%2Fcollective.exportimport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/collective%2Fcollective.exportimport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/collective","download_url":"https://codeload.github.com/collective/collective.exportimport/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246709921,"owners_count":20821296,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["exporting","hacktoberfest","importing","migrations","plone","python","zope"],"created_at":"2024-08-01T07:01:05.295Z","updated_at":"2025-04-01T20:30:31.048Z","avatar_url":"https://github.com/collective.png","language":"Python","funding_links":[],"categories":["Export, Import and Migrations"],"sub_categories":[],"readme":".. This README is meant for consumption by humans and PyPI. PyPI can render reStructuredText files, so please do not use Sphinx features.\n If you want to learn more about writing documentation, please check out: https://6.docs.plone.org/contributing/documentation/\n This text does not appear on PyPI or GitHub. It is a comment.\n\n.. image:: https://img.shields.io/pypi/v/collective.exportimport.svg\n :target: https://pypi.org/project/collective.exportimport/\n :alt: Latest Version\n\n.. image:: https://img.shields.io/pypi/status/collective.exportimport.svg\n :target: https://pypi.org/project/collective.exportimport/\n :alt: Egg Status\n\n.. image:: https://img.shields.io/pypi/pyversions/collective.exportimport.svg?style=plastic :alt: Supported - Python Versions\n\n.. image:: https://img.shields.io/pypi/l/collective.exportimport.svg\n :target: https://pypi.org/project/collective.exportimport/\n :alt: License\n\n\n=======================\ncollective.exportimport\n=======================\n\nExport and import content, members, relations, translations, localroles and much more.\n\nExport and import all kinds of data from and to Plone sites using a intermediate json-format.\nThe main use-case is migrations since it enables you to for example migrate from Plone 4 with Archetypes and Python 2 to Plone 6 with Dexterity and Python 3 in one step.\nMost features use `plone.restapi` to serialize and deserialize data.\n\nSee also the training on migrating with ``exportimport``: https://training.plone.org/migrations/exportimport.html\n\n.. contents:: Contents\n :local:\n\nFeatures\n========\n\n* Export \u0026 Import content\n* Export \u0026 Import members and groups with their roles\n* Export \u0026 Import relations\n* Export \u0026 Import translations\n* Export \u0026 Import local roles\n* Export \u0026 Import order (position in parent)\n* Export \u0026 Import discussions/comments\n* Export \u0026 Import versioned content\n* Export \u0026 Import redirects\n\nExport supports:\n\n* Plone 4, 5 and 6\n* Archetypes and Dexterity\n* Python 2 and 3\n* plone.app.multilingual, Products.LinguaPlone, raptus.multilanguagefields\n\nImport supports:\n\n* Plone 5.2+, Dexterity, Python 2 and 3, plone.app.multilingual\n\n\nInstallation\n============\n\nInstall collective.exportimport as you would install any other Python package.\n\nYou don't need to activate the add-on in the Site Setup Add-ons control panel to be able to use the forms ``@@export_content`` and ``@@import_content`` in your site.\n\nIf you need help, see:\n- for Plone 4: https://4.docs.plone.org/adapt-and-extend/install_add_ons.html\n- for Plone 5: https://5.docs.plone.org/manage/installing/installing_addons.html\n- for Plone 6: https://6.docs.plone.org/install/manage-add-ons-packages.html\n\n\nPython 2 compatibility\n----------------------\n\nThis package is compatible with Python 3 and Python 2.\nDepending on the Python version different versions of it's dependencies will be installed.\nIf you run into problems, file an issue at: https://github.com/collective/collective.exportimport/issues\n\n\nUsage\n=====\n\nExport\n------\n\nUse the form with the URL ``/@@export_content``, and select what you want to export:\n\n.. image:: ./docs/export.png\n\nYou can export one or more types and a whole site or only a specific path in a site. Since items are exported ordered by path importing them will create the same structure as you had originally.\n\nThe downloaded json-file will have the name of the path you exported from, e.g. ``Plone.json``.\n\nThe exports for members, relations, localroles and relations are linked to in this form but can also be called individually: ``/@@export_members``, ``/@@export_relations``, ``/@@export_localroles``, ``/@@export_translations``, ``/@@export_ordering``, ``/@@export_discussion``.\n\n\nImport\n------\n\nUse the form with the URL ``/@@import_content``, and upload a json-file that you want to import:\n\n.. image:: ./docs/import.png\n\n\nThe imports for members, relations, localroles and relations are linked to in this form but can also be called individually: ``/@@import_members``, ``/@@import_relations``, ``/@@import_localroles``, ``/@@import_translations``, ``/@@import_ordering``, ``/@@import_discussion``.\n\nAs a last step in a migration there is another view ``@@reset_dates`` that resets the modified date on imported content to the date initially contained in the imported json-file. This is necessary since varous changes during a migration will likely result in a updated modified-date. During import the original is stored as ``obj.modification_date_migrated`` on each new object and this view sets this date.\n\nExport- and import locations\n----------------------------\n\nIf you select 'Save to file on server', the Export view will save json files in the \u003cvar\u003e directory of your Plone instanc in /var/instance.\nThe import view will look for files under /var/instance/import.\nThese directories will normally be different, under different Plone instances and possibly on different servers.\n\nYou can set the environment variable 'COLLECTIVE_EXPORTIMPORT_CENTRAL_DIRECTORY' to add a 'shared' directory on one server or maybe network share.\nWith this variable set, collective.exportimport will both save to and load .json files from the same server directory.\nThis saves time not having to move .json files around from the export- to the import location.\nYou should be aware that the Export views will overwrite any existing previous .json file export that have the same name.\n\n\nUse-cases\n=========\n\nMigrations\n----------\n\nWhen a in-place-migration is not required you can choose this add-on to migrate the most important parts of your site to json and then import it into a new Plone instance of your targeted version:\n\n* Export content from a Plone site (it supports Plone 4 and 5, Archetypes and Dexterity, Python 2 and 3).\n* Import the exported content into a new site (Plone 5.2+, Dexterity, Python 3)\n* Export and import relations, users and groups with their roles, translations, local roles, ordering, default-pages, comments, portlets and redirects.\n\nHow to migrate additional features like Annotations or Marker Interfaces is discussed in the FAQ section.\n\nOther\n-----\n\nYou can use this add-on to\n\n* Archive your content as JSON.\n* Export data to prepare a migration to another system.\n* Combine content from multiple plone-sites into one.\n* Import a plone-site as a subsite into another.\n* Import content from other systems as long as it fits the required format.\n* Update or replace existing data.\n\nDetails\n=======\n\nExport content\n--------------\n\nExporting content is basically a wrapper for the serializers of plone.restapi:\n\n.. code-block:: python\n\n from plone.restapi.interfaces import ISerializeToJson\n from zope.component import getMultiAdapter\n\n serializer = getMultiAdapter((obj, request), ISerializeToJson)\n data = serializer(include_items=False)\n\nImport content\n--------------\n\nImporting content is a elaborate wrapper for the deserializers of plone.restapi:\n\n.. code-block:: python\n\n from plone.restapi.interfaces import IDeserializeFromJson\n from zope.component import getMultiAdapter\n\n container.invokeFactory(item['@type'], item['id'])\n deserializer = getMultiAdapter((new, self.request), IDeserializeFromJson)\n new = deserializer(validate_all=False, data=item)\n\n\nUse for migrations\n------------------\n\nA main use-case of this package is migration from one Plone-Version to another.\n\nExporting Archetypes content and importing that as Dexterity content works fine but due to changes in field-names some settings would get lost.\nFor example the setting to exclude content from the navigation was renamed from ``excludeFromNav`` to ``exclude_from_nav``.\n\nTo fix this you can check the checkbox \"Modify exported data for migrations\".\nThis will modify the data during export:\n\n* Drop unused data (e.g. `next_item` and `components`)\n* Remove all relation fields\n* Change some field names that changed between Archetypes and Dexterity\n\n * ``excludeFromNav`` → ``exclude_from_nav``\n * ``allowDiscussion`` → ``allow_discussion``\n * ``subject`` → ``subjects``\n * ``expirationDate`` → ``expires``\n * ``effectiveDate`` → ``effective``\n * ``creation_date`` → ``created``\n * ``modification_date`` → ``modified``\n * ``startDate`` → ``start``\n * ``endDate`` → ``end``\n * ``openEnd`` → ``open_end``\n * ``wholeDay`` → ``whole_day``\n * ``contactEmail`` → ``contact_email``\n * ``contactName`` → ``contact_name``\n * ``contactPhone`` → ``contact_phone``\n\n* Update view names on Folders and Collection that changed since Plone 4.\n* Export ``ATTopic`` and their criteria to Collections with querystrings.\n* Update Collection-criteria.\n* Links and images in Richtext-Fields of content and portlets have changes since Plone 4.\n the view ``/@@fix_html`` allows you to fix these.\n\n\nControl creating imported content\n---------------------------------\n\nYou can choose between four options how to deal with content that already exists:\n\n * Skip: Don't import at all\n * Replace: Delete item and create new\n * Update: Reuse and only overwrite imported data\n * Ignore: Create with a new id\n\nImported content is initially created with ``invokeFactory`` using portal_type and id of the exported item before deserializing the rest of the data.\nYou can set additional values by specifying a dict ``factory_kwargs`` that will be passed to the factory.\nLike this you can set values on the imported object that are expected to be there by subscribers to IObjectAddedEvent.\n\n\nExport versioned content\n------------------------\n\nExporting versions of Archetypes content will not work because of a bug in plone.restapi (https://github.com/plone/plone.restapi/issues/1335).\nFor export to work you need to use a version between 7.7.0 and 8.0.0 (if released) or a source-checkout of the branch 7.x.x.\n\n\nNotes on speed and large migrations\n===================================\n\nExporting and importing large amounts of content can take a while. Export is pretty fast but import is constrained by some features of Plone, most importantly versioning:\n\n* Importing 5000 Folders takes ~5 minutes\n* Importing 5000 Documents takes \u003e25 minutes because of versioning.\n* Importing 5000 Documents without versioning takes ~7 minutes.\n\nDuring import you can commit every x number of items which will free up memory and disk-space in your TMPDIR (where blobs are added before each commit).\n\nWhen exporting large numbers of blobs (binary files and images) you will get huge json-files and may run out of memory.\nYou have various options to deal with this.\nThe best way depends on how you are going to import the blobs:\n\n- Export as download urls: small download, but ``collective.exportimport`` cannot import the blobs, so you will need an own import script to download them.\n- Export as base-64 encoded strings: large download, but ``collective.exportimport`` can handle the import.\n- Export as blob paths: small download and ``collective.exportimport`` can handle the import, but you need to copy ``var/blobstorage`` to the Plone Site where you do the import or set the environment variable ``COLLECTIVE_EXPORTIMPORT_BLOB_HOME`` to the old blobstorage path: ``export COLLECTIVE_EXPORTIMPORT_BLOB_HOME=/path-to-old-instance/var/blobstorage``.\n To export the blob-path you do not need to have access to the blobs!\n\n\nFormat of export and import of content\n======================================\n\nBy default all content is exported to and imported from one large json-file.\nTo inspect such very large json-files without performance-issues you can use klogg (https://klogg.filimonov.dev).\n\nSince version 1.10 collective.exportimport also supports exporting and importing each content item as a separate json-file.\nTo use that select *Save each item as a separate file on the server* in the form or specify ``download_to_server=2`` when calling the export in python.\nIn the import-form you can manually select a directory on the server or specify ``server_directory=\"/mydir\"`` when calling the import in python.\n\n\nCustomize export and import\n===========================\n\nThis add-on is designed to be adapted to your requirements and has multiple hooks to make that easy.\n\nTo make that easier here are packages you can reuse to override and extend the export and import.\nUse these templates and adapt them to your own projects:\n\n* https://github.com/starzel/contentexport\n* https://github.com/starzel/contentimport\n\nMany examples for customizing the export and import are collected in the chapter \"FAQ, Tips and Tricks\" below.\n\n.. note::\n\n As a rule of thumb you should make changes to the data during import unless you need access to the original object for the required changes.\n One reason is that this way the serialized content in the json-file more closely represents the original data.\n Another reason is that it allows you to fix issues during the process you are currently developing (i.e. without having to redo the export).\n\n\nExport Example\n--------------\n\n.. code-block:: python\n\n from collective.exportimport.export_content import ExportContent\n\n class CustomExportContent(ExportContent):\n\n QUERY = {\n 'Document': {'review_state': ['published', 'pending']},\n }\n\n DROP_PATHS = [\n '/Plone/userportal',\n '/Plone/en/obsolete_content',\n ]\n\n DROP_UIDS = [\n '71e3e0a6f06942fea36536fbed0f6c42',\n ]\n\n def update(self):\n \"\"\"Use this to override stuff before the export starts\n (e.g. force a specific language in the request).\"\"\"\n\n def start(self):\n \"\"\"Hook to do something before export.\"\"\"\n\n def finish(self):\n \"\"\"Hook to do something after export.\"\"\"\n\n def global_obj_hook(self, obj):\n \"\"\"Inspect the content item before serialisation data.\n Bad: Changing the content-item is a horrible idea.\n Good: Return None if you want to skip this particular object.\n \"\"\"\n return obj\n\n def global_dict_hook(self, item, obj):\n \"\"\"Use this to modify or skip the serialized data.\n Return None if you want to skip this particular object.\n \"\"\"\n return item\n\n def dict_hook_document(self, item, obj):\n \"\"\"Use this to modify or skip the serialized data by type.\n Return the modified dict (item) or None if you want to skip this particular object.\n \"\"\"\n return item\n\n\nRegister it with your own browserlayer to override the default.\n\n.. code-block:: text\n\n \u003cbrowser:page\n name=\"export_content\"\n for=\"zope.interface.Interface\"\n class=\".custom_export.CustomExportContent\"\n layer=\"My.Custom.IBrowserlayer\"\n permission=\"cmf.ManagePortal\"\n /\u003e\n\n\nImport Example\n--------------\n\n.. code-block:: python\n\n from collective.exportimport.import_content import ImportContent\n\n class CustomImportContent(ImportContent):\n\n CONTAINER = {'Event': '/imported-events'}\n\n # These fields will be ignored\n DROP_FIELDS = ['relatedItems']\n\n # Items with these uid will be ignored\n DROP_UIDS = ['04d1477583c74552a7fcd81a9085c620']\n\n # These paths will be ignored\n DROP_PATHS = ['/Plone/doormat/', '/Plone/import_files/']\n\n # Default values for some fields\n DEFAULTS = {'which_price': 'normal'}\n\n def start(self):\n \"\"\"Hook to do something before importing one file.\"\"\"\n\n def finish(self):\n \"\"\"Hook to do something after importing one file.\"\"\"\n\n def global_dict_hook(self, item):\n if isinstance(item.get('description', None), dict):\n item['description'] = item['description']['data']\n if isinstance(item.get('rights', None), dict):\n item['rights'] = item['rights']['data']\n return item\n\n def dict_hook_customtype(self, item):\n # change the type\n item['@type'] = 'anothertype'\n # drop a field\n item.pop('experiences', None)\n return item\n\n def handle_file_container(self, item):\n \"\"\"Use this to specify the container in which to create the item in.\n Return the container for this particular object.\n \"\"\"\n return self.portal['imported_files']\n\nRegister it:\n\n.. code-block:: text\n\n \u003cbrowser:page\n name=\"import_content\"\n for=\"zope.interface.Interface\"\n class=\".custom_import.CustomImportContent\"\n layer=\"My.Custom.IBrowserlayer\"\n permission=\"cmf.ManagePortal\"\n /\u003e\n\n\nAutomate export and import\n--------------------------\n\nRun all exports and save all data in ``var/instance/``:\n\n.. code-block:: python\n\n from plone import api\n from Products.Five import BrowserView\n\n class ExportAll(BrowserView):\n\n def __call__(self):\n export_content = api.content.get_view(\"export_content\", self.context, self.request)\n self.request.form[\"form.submitted\"] = True\n export_content(\n portal_type=[\"Folder\", \"Document\", \"News Item\", \"File\", \"Image\"], # only export these\n include_blobs=2, # Export files and images as blob paths\n download_to_server=True,\n migration=True)\n\n other_exports = [\n \"export_relations\",\n \"export_members\",\n \"export_translations\",\n \"export_localroles\",\n \"export_ordering\",\n \"export_defaultpages\",\n \"export_discussion\",\n \"export_portlets\",\n \"export_redirects\",\n ]\n for name in other_exports:\n view = api.content.get_view(name, portal, request)\n # This saves each export in var/instance/export_xxx.json\n view(download_to_server=True)\n\n # Important! Redirect to prevent infinite export loop :)\n return self.request.response.redirect(self.context.absolute_url())\n\nRun all imports using the data exported in the example above:\n\n.. code-block:: python\n\n from collective.exportimport.fix_html import fix_html_in_content_fields\n from collective.exportimport.fix_html import fix_html_in_portlets\n from pathlib import Path\n from plone import api\n from Products.Five import BrowserView\n\n\n class ImportAll(BrowserView):\n\n def __call__(self):\n portal = api.portal.get()\n\n # Import content\n view = api.content.get_view(\"import_content\", portal, request)\n request.form[\"form.submitted\"] = True\n request.form[\"commit\"] = 500\n view(server_file=\"Plone.json\", return_json=True)\n transaction.commit()\n\n # Run all other imports\n other_imports = [\n \"relations\",\n \"members\",\n \"translations\",\n \"localroles\",\n \"ordering\",\n \"defaultpages\",\n \"discussion\",\n \"portlets\",\n \"redirects\",\n ]\n cfg = getConfiguration()\n directory = Path(cfg.clienthome) / \"import\"\n for name in other_imports:\n view = api.content.get_view(f\"import_{name}\", portal, request)\n path = Path(directory) / f\"export_{name}.json\"\n results = view(jsonfile=path.read_text(), return_json=True)\n logger.info(results)\n transaction.commit()\n\n # Run cleanup steps\n results = fix_html_in_content_fields()\n logger.info(\"Fixed html for %s content items\", results)\n transaction.commit()\n\n results = fix_html_in_portlets()\n logger.info(\"Fixed html for %s portlets\", results)\n transaction.commit()\n\n reset_dates = api.content.get_view(\"reset_dates\", portal, request)\n reset_dates()\n transaction.commit()\n\n.. note::\n\n The views ``@@export_all`` and ``@@import_all`` are also contained in the helper-packages https://github.com/starzel/contentexport and https://github.com/starzel/contentimport\n\nFAQ, Tips and Tricks\n====================\n\nThis section covers frequent use-cases and examples for features that are not required for all migrations.\n\nUsing global_obj_hook during export\n-----------------------------------\n\nUsing ``global_obj_hook`` during export to inspect content and decide to skip it.\n\n.. code-block:: python\n\n def global_obj_hook(self, obj):\n # Drop subtopics\n if obj.portal_type == \"Topic\" and obj.__parent__.portal_type == \"Topic\":\n return\n\n # Drop files and images from PFG formfolders\n if obj.__parent__.portal_type == \"FormFolder\":\n return\n return obj\n\n\nUsing dict-hooks during export\n------------------------------\n\nUse ``global_dict_hook`` during export to inspect content and modify the serialized json.\nYou can also use ``dict_hook_\u003csomecontenttype\u003e`` to better structure your code for readability.\n\nSometimes you need to handle data that you add in ``global_dict_hook`` during export in corresponding code in ``global_object_hook`` during import.\n\nThe following example about placeful workflow policy is a perfect example for that pattern:\n\n\nExport/Import placeful workflow policy\n--------------------------------------\n\nExport:\n\n.. code-block:: python\n\n def global_dict_hook(self, item, obj):\n if obj.isPrincipiaFolderish and \".wf_policy_config\" in obj.keys():\n wf_policy = obj[\".wf_policy_config\"]\n item[\"exportimport.workflow_policy\"] = {\n \"workflow_policy_below\": wf_policy.workflow_policy_below,\n \"workflow_policy_in\": wf_policy.workflow_policy_in,\n }\n return item\n\nImport:\n\n.. code-block:: python\n\n def global_obj_hook(self, obj, item):\n wf_policy = item.get(\"exportimport.workflow_policy\")\n if wf_policy:\n obj.manage_addProduct[\"CMFPlacefulWorkflow\"].manage_addWorkflowPolicyConfig()\n wf_policy_config = obj[\".wf_policy_config\"]\n wf_policy_config.setPolicyIn(wf_policy[\"workflow_policy_in\"], update_security=True)\n wf_policy_config.setPolicyBelow(wf_policy[\"workflow_policy_below\"], update_security=True)\n\n\nUsing dict-hooks during import\n------------------------------\n\nA lot of fixes can be done during import using the ``global_dict_hook`` or ``dict_hook_\u003ccontenttype\u003e``.\n\nHere we prevent the expire-date to be before the effective date since that would lead to validation-errors during deserializing:\n\n.. code-block:: python\n\n def global_dict_hook(self, item):\n effective = item.get('effective', None)\n expires = item.get('expires', None)\n if effective and expires and expires \u003c= effective:\n item.pop('expires')\n return item\n\nHere we drop empty lines from the creators:\n\n.. code-block:: python\n\n def global_dict_hook(self, item):\n item[\"creators\"] = [i for i in item.get(\"creators\", []) if i]\n return item\n\nThis example migrates a ``PloneHelpCenter`` to a simple folder/document structure during import.\nThere are a couple more types to handle (as folder or document) but you get the idea, don't you?\n\n.. code-block:: python\n\n def dict_hook_helpcenter(self, item):\n item[\"@type\"] = \"Folder\"\n item[\"layout\"] = \"listing_view\"\n return item\n\n def dict_hook_helpcenterglossary(self, item):\n item[\"@type\"] = \"Folder\"\n item[\"layout\"] = \"listing_view\"\n return item\n\n def dict_hook_helpcenterinstructionalvideo(self, item):\n item[\"@type\"] = \"File\"\n if item.get(\"video_file\"):\n item[\"file\"] = item[\"video_file\"]\n return item\n\n def dict_hook_helpcenterlink(self, item):\n item[\"@type\"] = \"Link\"\n item[\"remoteUrl\"] = item.get(\"url\", None)\n return item\n\n def dict_hook_helpcenterreferencemanualpage(self, item):\n item[\"@type\"] = \"Document\"\n return item\n\nIf you change types during import you need to take care of other cases where types are referenced.\\\nExamples are collection-queries (see \"Fixing invalid collection queries\" below) or constrains (see here):\n\n.. code-block:: python\n\n PORTAL_TYPE_MAPPING = {\n \"Topic\": \"Collection\",\n \"FormFolder\": \"EasyForm\",\n \"HelpCenter\": \"Folder\",\n }\n\n def global_dict_hook(self, item):\n if item.get(\"exportimport.constrains\"):\n types_fixed = []\n for portal_type in item[\"exportimport.constrains\"][\"locally_allowed_types\"]:\n if portal_type in PORTAL_TYPE_MAPPING:\n types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])\n elif portal_type in ALLOWED_TYPES:\n types_fixed.append(portal_type)\n item[\"exportimport.constrains\"][\"locally_allowed_types\"] = list(set(types_fixed))\n\n types_fixed = []\n for portal_type in item[\"exportimport.constrains\"][\"immediately_addable_types\"]:\n if portal_type in PORTAL_TYPE_MAPPING:\n types_fixed.append(PORTAL_TYPE_MAPPING[portal_type])\n elif portal_type in ALLOWED_TYPES:\n types_fixed.append(portal_type)\n item[\"exportimport.constrains\"][\"immediately_addable_types\"] = list(set(types_fixed))\n return item\n\n\nChange workflow\n---------------\n\n.. code-block:: python\n\n REVIEW_STATE_MAPPING = {\n \"internal\": \"published\",\n \"internally_published\": \"published\",\n \"obsolete\": \"private\",\n \"hidden\": \"private\",\n }\n\n def global_dict_hook(self, item):\n if item.get(\"review_state\") in REVIEW_STATE_MAPPING:\n item[\"review_state\"] = REVIEW_STATE_MAPPING[item[\"review_state\"]]\n return item\n\n\nExport/Import Annotations\n-------------------------\n\nSome core-features of Plone (e.g. comments) use annotations to store data.\nThe core features are already covered but your custom code or community add-ons may use annotations as well.\nHere is how you can migrate them.\n\n**Export**: Only export those Annotations that your really need.\n\n.. code-block:: python\n\n from zope.annotation.interfaces import IAnnotations\n ANNOTATIONS_TO_EXPORT = [\n \"syndication_settings\",\n ]\n ANNOTATIONS_KEY = 'exportimport.annotations'\n\n class CustomExportContent(ExportContent):\n\n def global_dict_hook(self, item, obj):\n item = self.export_annotations(item, obj)\n return item\n\n def export_annotations(self, item, obj):\n results = {}\n annotations = IAnnotations(obj)\n for key in ANNOTATIONS_TO_EXPORT:\n data = annotations.get(key)\n if data:\n results[key] = IJsonCompatible(data, None)\n if results:\n item[ANNOTATIONS_KEY] = results\n return item\n\n**Import**:\n\n.. code-block:: python\n\n from zope.annotation.interfaces import IAnnotations\n ANNOTATIONS_KEY = \"exportimport.annotations\"\n\n class CustomImportContent(ImportContent):\n\n def global_obj_hook(self, obj, item):\n item = self.import_annotations(obj, item)\n return item\n\n def import_annotations(self, obj, item):\n annotations = IAnnotations(obj)\n for key in item.get(ANNOTATIONS_KEY, []):\n annotations[key] = item[ANNOTATIONS_KEY][key]\n return item\n\nSome features also store data in annotations on the portal, e.g. `plone.contentrules.localassignments`, `plone.portlets.categoryblackliststatus`, `plone.portlets.contextassignments`, `syndication_settings`.\nDepending on your requirements you may want to export and import those as well.\n\n\nExport/Import Marker Interfaces\n-------------------------------\n\n**Export**: You may only want to export the marker-interfaces you need.\nIt is a good idea to inspect a list of all used marker interfaces in a portal before deciding what to migrate.\n\n.. code-block:: python\n\n from zope.interface import directlyProvidedBy\n\n MARKER_INTERFACES_TO_EXPORT = [\n \"collective.easyslider.interfaces.ISliderPage\",\n \"plone.app.layout.navigation.interfaces.INavigationRoot\",\n ]\n MARKER_INTERFACES_KEY = \"exportimport.marker_interfaces\"\n\n class CustomExportContent(ExportContent):\n\n def global_dict_hook(self, item, obj):\n item = self.export_marker_interfaces(item, obj)\n return item\n\n def export_marker_interfaces(self, item, obj):\n interfaces = [i.__identifier__ for i in directlyProvidedBy(obj)]\n interfaces = [i for i in interfaces if i in MARKER_INTERFACES_TO_EXPORT]\n if interfaces:\n item[MARKER_INTERFACES_KEY] = interfaces\n return item\n\n**Import**:\n\n.. code-block:: python\n\n from plone.dexterity.utils import resolveDottedName\n from zope.interface import alsoProvides\n\n MARKER_INTERFACES_KEY = \"exportimport.marker_interfaces\"\n\n class CustomImportContent(ImportContent):\n\n def global_obj_hook_before_deserializing(self, obj, item):\n \"\"\"Apply marker interfaces before deserializing.\"\"\"\n for iface_name in item.pop(MARKER_INTERFACES_KEY, []):\n try:\n iface = resolveDottedName(iface_name)\n if not iface.providedBy(obj):\n alsoProvides(obj, iface)\n logger.info(\"Applied marker interface %s to %s\", iface_name, obj.absolute_url())\n except ModuleNotFoundError:\n pass\n return obj, item\n\nSkip versioning during import\n-----------------------------\n\nThe event-handlers of versioning can seriously slow down your imports.\nIt is a good idea to skip it before the import:\n\n.. code-block:: python\n\n VERSIONED_TYPES = [\n \"Document\",\n \"News Item\",\n \"Event\",\n \"Link\",\n ]\n\n def start(self):\n self.items_without_parent = []\n portal_types = api.portal.get_tool(\"portal_types\")\n for portal_type in VERSIONED_TYPES:\n fti = portal_types.get(portal_type)\n behaviors = list(fti.behaviors)\n if 'plone.versioning' in behaviors:\n logger.info(f\"Disable versioning for {portal_type}\")\n behaviors.remove('plone.versioning')\n fti.behaviors = behaviors\n\nRe-enable versioning and create initial versions after all imports and fixes are done, e.g in the view ``@@import_all``.\n\n.. code-block:: python\n\n from Products.CMFEditions.interfaces.IModifier import FileTooLargeToVersionError\n\n VERSIONED_TYPES = [\n \"Document\",\n \"News Item\",\n \"Event\",\n \"Link\",\n ]\n\n class ImportAll(BrowserView):\n\n # re-enable versioning\n portal_types = api.portal.get_tool(\"portal_types\")\n for portal_type in VERSIONED_TYPES:\n fti = portal_types.get(portal_type)\n behaviors = list(fti.behaviors)\n if \"plone.versioning\" not in behaviors:\n behaviors.append(\"plone.versioning\")\n logger.info(f\"Enable versioning for {portal_type}\")\n if \"plone.locking\" not in behaviors:\n behaviors.append(\"plone.locking\")\n logger.info(f\"Enable locking for {portal_type}\")\n fti.behaviors = behaviors\n transaction.get().note(\"Re-enabled versioning\")\n transaction.commit()\n\n # create initial version for all versioned types\n logger.info(\"Creating initial versions\")\n portal_repository = api.portal.get_tool(\"portal_repository\")\n brains = api.content.find(portal_type=VERSIONED_TYPES)\n total = len(brains)\n for index, brain in enumerate(brains):\n obj = brain.getObject()\n try:\n if not portal_repository.getHistoryMetadata(obj=obj):\n portal_repository.save(obj=obj, comment=\"Imported Version\")\n except FileTooLargeToVersionError:\n pass\n if not index % 1000:\n msg = f\"Created versions for {index} of {total} items.\"\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n msg = \"Created initial versions\"\n transaction.get().note(msg)\n transaction.commit()\n\n\nDealing with validation errors\n------------------------------\n\nSometimes you get validation-errors during import because the data cannot be validated.\nThat can happen when options in a field are generated from content in the site.\nIn these cases you cannot be sure that all options already exist in the portal while importing the content.\n\nIt may also happen, when you have validators that rely on content or configuration that does not exist on import.\n\n.. note::\n\n For relation fields this is not necessary since relations are imported after content anyway!\n\nThere are two ways to handle these issues:\n\n* Use a simple setter bypassing the validation used by the restapi\n* Defer the import until all other imports were run\n\n\nUse a simple setter\n*******************\n\nYou need to specify which content-types and fields you want to handle that way.\n\nIt is put in a key, that the normal import will ignore and set using ``setattr()`` before deserializing the rest of the data.\n\n.. code-block:: python\n\n SIMPLE_SETTER_FIELDS = {\n \"ALL\": [\"some_shared_field\"],\n \"CollaborationFolder\": [\"allowedPartnerDocTypes\"],\n \"DocType\": [\"automaticTransferTargets\"],\n \"DPDocument\": [\"scenarios\"],\n \"DPEvent\" : [\"Status\"],\n }\n\n class CustomImportContent(ImportContent):\n\n def global_dict_hook(self, item):\n simple = {}\n for fieldname in SIMPLE_SETTER_FIELDS.get(\"ALL\", []):\n if fieldname in item:\n value = item.pop(fieldname)\n if value:\n simple[fieldname] = value\n for fieldname in SIMPLE_SETTER_FIELDS.get(item[\"@type\"], []):\n if fieldname in item:\n value = item.pop(fieldname)\n if value:\n simple[fieldname] = value\n if simple:\n item[\"exportimport.simplesetter\"] = simple\n\n def global_obj_hook_before_deserializing(self, obj, item):\n \"\"\"Hook to modify the created obj before deserializing the data.\n \"\"\"\n # import simplesetter data before the rest\n for fieldname, value in item.get(\"exportimport.simplesetter\", {}).items():\n setattr(obj, fieldname, value)\n\n.. note::\n\n Using ``global_obj_hook_before_deserializing`` makes sure that data is there when the event-handlers are run after import.\n\nDefer import\n************\n\nYou can also wait until all content is imported before setting the values on these fields.\nAgain you need to find out which fields for which types you want to handle that way.\n\nHere the data is stored in an annotation on the imported object from which it is later read.\nThis example also supports setting some data with ``setattr`` without validating it:\n\n.. code-block:: python\n\n from plone.restapi.interfaces import IDeserializeFromJson\n from zope.annotation.interfaces import IAnnotations\n from zope.component import getMultiAdapter\n\n DEFERRED_KEY = \"exportimport.deferred\"\n DEFERRED_FIELD_MAPPING = {\n \"talk\": [\"somefield\"],\n \"speaker\": [\n \"custom_field\",\n \"another_field\",\n ]\n }\n SIMPLE_SETTER_FIELDS = {\"custom_type\": [\"another_field\"]}\n\n class CustomImportContent(ImportContent):\n\n def global_dict_hook(self, item):\n # Move deferred values to a different key to not deserialize.\n # This could also be done during export.\n item[DEFERRED_KEY] = {}\n for fieldname in DEFERRED_FIELD_MAPPING.get(item[\"@type\"], []):\n if item.get(fieldname):\n item[DEFERRED_KEY][fieldname] = item.pop(fieldname)\n return item\n\n def global_obj_hook(self, obj, item):\n # Store deferred data in an annotation.\n deferred = item.get(DEFERRED_KEY, {})\n if deferred:\n annotations = IAnnotations(obj)\n annotations[DEFERRED_KEY] = {}\n for key, value in deferred.items():\n annotations[DEFERRED_KEY][key] = value\n\nYou then need a new step in the migration to move the deferred values from the annotation to the field:\n\n.. code-block:: python\n\n class ImportDeferred(BrowserView):\n\n def __call__(self):\n # This example reuses the form export_other.pt from collective.exportimport\n self.title = \"Import deferred data\"\n if not self.request.form.get(\"form.submitted\", False):\n return self.index()\n portal = api.portal.get()\n self.results = []\n for brain in api.content.find(DEFERRED_FIELD_MAPPING.keys()):\n obj = brain.getObject()\n self.import_deferred(obj)\n api.portal.show_message(f\"Imported deferred data for {len(self.results)} items!\", self.request)\n\n def import_deferred(self, obj):\n annotations = IAnnotations(obj, {})\n deferred = annotations.get(DEFERRED_KEY, None)\n if not deferred:\n return\n # Shortcut for simple fields (e.g. storing strings, uuids etc.)\n for fieldname in SIMPLE_SETTER_FIELDS.get(obj.portal_type, []):\n value = deferred.pop(fieldname, None)\n if value:\n setattr(obj, fieldname, value)\n if not deferred:\n return\n # This approach validates the values and converts more complex data\n deserializer = getMultiAdapter((obj, self.request), IDeserializeFromJson)\n try:\n obj = deserializer(validate_all=False, data=deferred)\n except Exception as e:\n logger.info(\"Error while importing deferred data for %s\", obj.absolute_url(), exc_info=True)\n logger.info(\"Data: %s\", deferred)\n else:\n self.results.append(obj.absolute_url())\n # cleanup\n del annotations[DEFERRED_KEY]\n\nThis additional view obviously needs to be registered:\n\n.. code-block:: text\n\n \u003cbrowser:page\n name=\"import_deferred\"\n for=\"zope.interface.Interface\"\n class=\".import_content.ImportDeferred\"\n template=\"export_other.pt\"\n permission=\"cmf.ManagePortal\"\n /\u003e\n\n\nHandle LinguaPlone content\n--------------------------\n\nExport:\n\n.. code-block:: python\n\n def global_dict_hook(self, item, obj):\n # Find language of the nearest parent with a language\n # Usefull for LinguaPlone sites where some content is languageindependent\n parent = obj.__parent__\n for ancestor in parent.aq_chain:\n if IPloneSiteRoot.providedBy(ancestor):\n # keep language for root content\n nearest_ancestor_lang = item[\"language\"]\n break\n if getattr(ancestor, \"getLanguage\", None) and ancestor.getLanguage():\n nearest_ancestor_lang = ancestor.getLanguage()\n item[\"parent\"][\"language\"] = nearest_ancestor_lang\n break\n\n # This forces \"wrong\" languages to the nearest parents language\n if \"language\" in item and item[\"language\"] != nearest_ancestor_lang:\n logger.info(u\"Forcing %s (was %s) for %s %s \", nearest_ancestor_lang, item[\"language\"], item[\"@type\"], item[\"@id\"])\n item[\"language\"] = nearest_ancestor_lang\n\n # set missing language\n if not item.get(\"language\"):\n item[\"language\"] = nearest_ancestor_lang\n\n # add info on translations to help find the right container\n # usually this idone by export_translations\n # but when migrating from LP to pam you sometimes want to check the\n # tranlation info during import\n if getattr(obj.aq_base, \"getTranslations\", None) is not None:\n translations = obj.getTranslations()\n if translations:\n item[\"translation\"] = {}\n for lang in translations:\n uuid = IUUID(translations[lang][0], None)\n if uuid == item[\"UID\"]:\n continue\n translation = translations[lang][0]\n if not lang:\n lang = \"no_language\"\n item[\"translation\"][lang] = translation.absolute_url()\n\nImport:\n\n.. code-block:: python\n\n def global_dict_hook(self, item):\n\n # Adapt this to your site\n languages = [\"en\", \"fr\", \"de\"]\n default_language = \"en\"\n portal_id = \"Plone\"\n\n # No language =\u003e lang of parent or default\n if item.get(\"language\") not in languages:\n if item[\"parent\"].get(\"language\"):\n item[\"language\"] = item[\"parent\"][\"language\"]\n else:\n item[\"language\"] = default_language\n\n lang = item[\"language\"]\n\n if item[\"parent\"].get(\"language\") != item[\"language\"]:\n logger.debug(f\"Inconsistent lang: item is {lang}, parent is {item['parent'].get('language')} for {item['@id']}\")\n\n # Move item to the correct language-root-folder\n # This is only relevant for items in the site-root.\n # Most items containers are usually looked up by the uuid of the old parent\n url = item[\"@id\"]\n parent_url = item[\"parent\"][\"@id\"]\n\n url = url.replace(f\"/{portal_id}/\", f\"/{portal_id}/{lang}/\", 1)\n parent_url = parent_url.replace(f\"/{portal_id}\", f\"/{portal_id}/{lang}\", 1)\n\n item[\"@id\"] = url\n item[\"parent\"][\"@id\"] = parent_url\n\n return item\n\nAlternative ways to handle items without parent\n-----------------------------------------------\n\nOften it is better to export and log items for which no container could be found instead of re-creating the original structure.\n\n.. code-block:: python\n\n def update(self):\n self.items_without_parent = []\n\n def create_container(self, item):\n # Override create_container to never create parents\n self.items_without_parent.append(item)\n\n def finish(self):\n # export content without parents\n if self.items_without_parent:\n data = json.dumps(self.items_without_parent, sort_keys=True, indent=4)\n number = len(self.items_without_parent)\n cfg = getConfiguration()\n filename = 'content_without_parent.json'\n filepath = os.path.join(cfg.clienthome, filename)\n with open(filepath, 'w') as f:\n f.write(data)\n msg = u\"Saved {} items without parent to {}\".format(number, filepath)\n logger.info(msg)\n api.portal.show_message(msg, self.request)\n\n\nExport/Import Zope Users\n------------------------\n\nBy default only users and groups stores in Plone are exported/imported.\nYou can export/import Zope user like this.\n\n**Export**\n\n.. code-block:: python\n\n from collective.exportimport.export_other import BaseExport\n from plone import api\n\n import six\n\n class ExportZopeUsers(BaseExport):\n\n AUTO_ROLES = [\"Authenticated\"]\n\n def __call__(self, download_to_server=False):\n self.title = \"Export Zope users\"\n self.download_to_server = download_to_server\n portal = api.portal.get()\n app = portal.__parent__\n self.acl = app.acl_users\n self.pms = api.portal.get_tool(\"portal_membership\")\n data = self.all_zope_users()\n self.download(data)\n\n def all_zope_users(self):\n results = []\n for user in self.acl.searchUsers():\n data = self._getUserData(user[\"userid\"])\n data['title'] = user['title']\n results.append(data)\n return results\n\n def _getUserData(self, userId):\n member = self.pms.getMemberById(userId)\n roles = [\n role\n for role in member.getRoles()\n if role not in self.AUTO_ROLES\n ]\n # userid, password, roles\n props = {\n \"username\": userId,\n \"password\": json_compatible(self._getUserPassword(userId)),\n \"roles\": json_compatible(roles),\n }\n return props\n\n def _getUserPassword(self, userId):\n users = self.acl.users\n passwords = users._user_passwords\n password = passwords.get(userId, \"\")\n return password\n\n**Import**:\n\n.. code-block:: python\n\n class ImportZopeUsers(BrowserView):\n\n def __call__(self, jsonfile=None, return_json=False):\n if jsonfile:\n self.portal = api.portal.get()\n status = \"success\"\n try:\n if isinstance(jsonfile, str):\n return_json = True\n data = json.loads(jsonfile)\n elif isinstance(jsonfile, FileUpload):\n data = json.loads(jsonfile.read())\n else:\n raise (\"Data is neither text nor upload.\")\n except Exception as e:\n status = \"error\"\n logger.error(e)\n api.portal.show_message(\n u\"Failure while uploading: {}\".format(e),\n request=self.request,\n )\n else:\n members = self.import_members(data)\n msg = u\"Imported {} members\".format(members)\n api.portal.show_message(msg, self.request)\n if return_json:\n msg = {\"state\": status, \"msg\": msg}\n return json.dumps(msg)\n\n return self.index()\n\n def import_members(self, data):\n app = self.portal.__parent__\n acl = app.acl_users\n counter = 0\n for item in data:\n username = item[\"username\"]\n password = item.pop(\"password\")\n roles = item.pop(\"roles\", [])\n if not username or not password or not roles:\n continue\n title = item.pop(\"title\", None)\n acl.users.addUser(username, title, password)\n for role in roles:\n acl.roles.assignRoleToPrincipal(role, username)\n counter += 1\n return counter\n\n\nExport/Import properties, registry-settings and installed add-ons\n-----------------------------------------------------------------\n\nWhen you migrate multiple similar sites that are configured manually it can be useful to export and import configuration that was set by hand.\n\nExport/Import installed settings and add-ons\n********************************************\n\nThis custom export exports and imports some selected settings and add-ons from a Plone 4.3 site.\n\n**Export:**\n\n.. code-block:: python\n\n from collective.exportimport.export_other import BaseExport\n from logging import getLogger\n from plone import api\n from plone.restapi.serializer.converters import json_compatible\n\n logger = getLogger(__name__)\n\n\n class ExportSettings(BaseExport):\n \"\"\"Export various settings for haiku sites\n \"\"\"\n\n def __call__(self, download_to_server=False):\n self.title = \"Export installed add-ons various settings\"\n self.download_to_server = download_to_server\n if not self.request.form.get(\"form.submitted\", False):\n return self.index()\n\n data = self.export_settings()\n self.download(data)\n\n def export_settings(self):\n results = {}\n addons = []\n qi = api.portal.get_tool(\"portal_quickinstaller\")\n for product in qi.listInstalledProducts():\n if product[\"id\"].startswith(\"myproject.\"):\n addons.append(product[\"id\"])\n results[\"addons\"] = addons\n\n portal = api.portal.get()\n registry = {}\n registry[\"plone.email_from_name\"] = portal.getProperty('email_from_name', '')\n registry[\"plone.email_from_address\"] = portal.getProperty('email_from_address', '')\n registry[\"plone.smtp_host\"] = getattr(portal.MailHost, 'smtp_host', '')\n registry[\"plone.smtp_port\"] = int(getattr(portal.MailHost, 'smtp_port', 25))\n registry[\"plone.smtp_userid\"] = portal.MailHost.get('smtp_user_id')\n registry[\"plone.smtp_pass\"] = portal.MailHost.get('smtp_pass')\n registry[\"plone.site_title\"] = portal.title\n\n portal_properties = api.portal.get_tool(\"portal_properties\")\n iprops = portal_properties.imaging_properties\n registry[\"plone.allowed_sizes\"] = iprops.getProperty('allowed_sizes')\n registry[\"plone.quality\"] = iprops.getProperty('quality')\n site_props = portal_properties.site_properties\n if site_props.hasProperty(\"webstats_js\"):\n registry[\"plone.webstats_js\"] = site_props.webstats_js\n results[\"registry\"] = json_compatible(registry)\n return results\n\n\n**Import:**\n\nThe import installs the add-ons and load the settings in the registry.\nSince Plone 5 ``portal_properties`` is no longer used.\n\n.. code-block:: python\n\n from logging import getLogger\n from plone import api\n from plone.registry.interfaces import IRegistry\n from Products.CMFPlone.utils import get_installer\n from Products.Five import BrowserView\n from zope.component import getUtility\n from ZPublisher.HTTPRequest import FileUpload\n\n import json\n\n logger = getLogger(__name__)\n\n class ImportSettings(BrowserView):\n \"\"\"Import various settings\"\"\"\n\n def __call__(self, jsonfile=None, return_json=False):\n if jsonfile:\n self.portal = api.portal.get()\n status = \"success\"\n try:\n if isinstance(jsonfile, str):\n return_json = True\n data = json.loads(jsonfile)\n elif isinstance(jsonfile, FileUpload):\n data = json.loads(jsonfile.read())\n else:\n raise (\"Data is neither text nor upload.\")\n except Exception as e:\n status = \"error\"\n logger.error(e)\n api.portal.show_message(\n \"Failure while uploading: {}\".format(e),\n request=self.request,\n )\n else:\n self.import_settings(data)\n msg = \"Imported addons and settings\"\n api.portal.show_message(msg, self.request)\n if return_json:\n msg = {\"state\": status, \"msg\": msg}\n return json.dumps(msg)\n\n return self.index()\n\n def import_settings(self, data):\n installer = get_installer(self.context)\n for addon in data[\"addons\"]:\n if not installer.is_product_installed(addon) and installer.is_product_installable(addon):\n installer.install_product(addon)\n logger.info(f\"Installed addon {addon}\")\n registry = getUtility(IRegistry)\n for key, value in data[\"registry\"].items():\n registry[key] = value\n logger.info(f\"Imported record {key}: {value}\")\n\n\nExport/Import registry settings\n*******************************\n\nThe pull-request https://github.com/collective/collective.exportimport/pull/130 has views ``@@export_registry`` and ``@@import_registry``.\nThese views export and import registry records that do not use the default-setting specified in the schema for that registry record.\nThe export alone could also be usefull to figure out which settings were modified for a site.\n\nThat code will probably not be merged but you can use it in your own projects.\n\nMigrate PloneFormGen to Easyform\n--------------------------------\n\nTo be able to export PFG as easyform you should use the branch ``migration_features_1.x`` of ``collective.easyform`` in your old site.\nEasyform does not need to be installed, we only need the methods ``fields_model`` and ``actions_model``.\n\nExport:\n\n.. code-block:: python\n\n def dict_hook_formfolder(self, item, obj):\n item[\"@type\"] = \"EasyForm\"\n item[\"is_folderish\"] = False\n\n from collective.easyform.migration.fields import fields_model\n from collective.easyform.migration.actions import actions_model\n\n # this does most of the heavy lifting...\n item[\"fields_model\"] = fields_model(obj)\n item[\"actions_model\"] = actions_model(obj)\n\n # handle thankspage\n pfg_thankspage = obj.get(obj.getThanksPage(), None)\n if pfg_thankspage:\n item[\"thankstitle\"] = pfg_thankspage.title\n item[\"thanksdescription\"] = pfg_thankspage.Description()\n item[\"showAll\"] = pfg_thankspage.showAll\n item[\"showFields\"] = pfg_thankspage.showFields\n item[\"includeEmpties\"] = pfg_thankspage.includeEmpties\n item[\"thanksPrologue\"] = json_compatible(pfg_thankspage.thanksPrologue.raw)\n item[\"thanksEpilogue\"] = json_compatible(pfg_thankspage.thanksEpilogue.raw)\n\n # optional\n item[\"exportimport._inputStorage\"] = self.export_saved_data(obj)\n\n # Drop some PFG fields no longer needed\n obsolete_fields = [\n \"layout\",\n \"actionAdapter\",\n \"checkAuthenticator\",\n \"constrainTypesMode\",\n \"location\",\n \"thanksPage\",\n ]\n for key in obsolete_fields:\n item.pop(key, None)\n\n # optional: disable tabs for imported forms\n item[\"form_tabbing\"] = False\n\n # fix some custom validators\n replace_mapping = {\n \"request.form['\": \"request.form['form.widgets.\",\n \"request.form.get('\": \"request.form.get('form.widgets.\",\n \"member and member.id or ''\": \"member and member.getProperty('id', '') or ''\",\n }\n\n # fix overrides in actions and fields to use form.widgets.xyz instead of xyz\n for schema in [\"actions_model\", \"fields_model\"]:\n for old, new in replace_mapping.items():\n if old in item[schema]:\n item[schema] = item[schema].replace(old, new)\n\n # add your own fields if you have these issues...\n for fieldname in [\n \"email\",\n \"replyto\",\n ]:\n if \"request/form/{}\".format(fieldname) in item[schema]:\n item[schema] = item[schema].replace(\"request/form/{}\".format(fieldname), \"python: request.form.get('form.widgets.{}')\".format(fieldname))\n\n return item\n\n def export_saved_data(self, obj):\n actions = {}\n for data_adapter in obj.objectValues(\"FormSaveDataAdapter\"):\n data_adapter_name = data_adapter.getId()\n actions[data_adapter_name] = {}\n cols = data_adapter.getColumnNames()\n column_count_mismatch = False\n for idx, row in enumerate(data_adapter.getSavedFormInput()):\n if len(row) != len(cols):\n column_count_mismatch = True\n logger.debug(\"Column count mismatch at row %s\", idx)\n continue\n data = {}\n for key, value in zip(cols, row):\n data[key] = json_compatible(value)\n id_ = int(time() * 1000)\n while id_ in actions[data_adapter_name]: # avoid collisions during export\n id_ += 1\n data[\"id\"] = id_\n actions[data_adapter_name][id_] = data\n if column_count_mismatch:\n logger.info(\n \"Number of columns does not match for all rows. Some data were skipped in \"\n \"data adapter %s/%s\",\n \"/\".join(obj.getPhysicalPath()),\n data_adapter_name,\n )\n return actions\n\nImport exported ``PloneFormGen`` data into ``Easyform``:\n\n.. code-block:: python\n\n def obj_hook_easyform(self, obj, item):\n if not item.get(\"exportimport._inputStorage\"):\n return\n from collective.easyform.actions import SavedDataBTree\n from persistent.mapping import PersistentMapping\n if not hasattr(obj, '_inputStorage'):\n obj._inputStorage = PersistentMapping()\n for name, data in item[\"exportimport._inputStorage\"].items():\n obj._inputStorage[name] = SavedDataBTree()\n for key, row in data.items():\n obj._inputStorage[name][int(key)] = row\n\n\nExport and import collective.cover content\n------------------------------------------\n\nExport:\n\n.. code-block:: python\n\n from collective.exportimport.serializer import get_dx_blob_path\n from plone.app.textfield.value import RichTextValue\n from plone.namedfile.file import NamedBlobImage\n from plone.restapi.interfaces import IJsonCompatible\n from z3c.relationfield import RelationValue\n from zope.annotation.interfaces import IAnnotations\n\n def global_dict_hook(self, item, obj):\n item = self.handle_cover(item, obj)\n return item\n\n def handle_cover(self, item, obj):\n if ICover.providedBy(obj):\n item['tiles'] = {}\n annotations = IAnnotations(obj)\n for tile in obj.get_tiles():\n annotation_key = 'plone.tiles.data.{}'.format(tile['id'])\n annotation = annotations.get(annotation_key, None)\n if annotation is None:\n continue\n tile_data = self.serialize_tile(annotation)\n tile_data['type'] = tile['type']\n item['tiles'][tile['id']] = tile_data\n return item\n\n def serialize_tile(self, annotation):\n data = {}\n for key, value in annotation.items():\n if isinstance(value, RichTextValue):\n value = value.raw\n elif isinstance(value, RelationValue):\n value = value.to_object.UID()\n elif isinstance(value, NamedBlobImage):\n blobfilepath = get_dx_blob_path(value)\n if not blobfilepath:\n continue\n value = {\n \"filename\": value.filename,\n \"content-type\": value.contentType,\n \"size\": value.getSize(),\n \"blob_path\": blobfilepath,\n }\n data[key] = IJsonCompatible(value, None)\n return data\n\nImport:\n\n.. code-block:: python\n\n from collections import defaultdict\n from collective.exportimport.import_content import get_absolute_blob_path\n from plone.app.textfield.interfaces import IRichText\n from plone.app.textfield.interfaces import IRichTextValue\n from plone.namedfile.file import NamedBlobImage\n from plone.namedfile.interfaces import INamedBlobImageField\n from plone.tiles.interfaces import ITileType\n from zope.annotation.interfaces import IAnnotations\n from zope.component import getUtilitiesFor\n from zope.schema import getFieldsInOrder\n\n COVER_CONTENT = [\n \"collective.cover.content\",\n ]\n\n def global_obj_hook(self, obj, item):\n if item[\"@type\"] in COVER_CONTENT and \"tiles\" in item:\n item = self.import_tiles(obj, item)\n\n def import_tiles(self, obj, item):\n RICHTEXT_TILES = defaultdict(list)\n IMAGE_TILES = defaultdict(list)\n for tile_name, tile_type in getUtilitiesFor(ITileType):\n for fieldname, field in getFieldsInOrder(tile_type.schema):\n if IRichText.providedBy(field):\n RICHTEXT_TILES[tile_name].append(fieldname)\n if INamedBlobImageField.providedBy(field):\n IMAGE_TILES[tile_name].append(fieldname)\n\n annotations = IAnnotations(obj)\n prefix = \"plone.tiles.data.\"\n for uid, tile in item[\"tiles\"].items():\n # TODO: Maybe create all tiles that do not need to be defferred?\n key = prefix + uid\n tile_name = tile.pop(\"type\", None)\n # first set raw data\n annotations[key] = item[\"tiles\"][uid]\n for fieldname in RICHTEXT_TILES.get(tile_name, []):\n raw = annotations[key][fieldname]\n if raw is not None and not IRichTextValue.providedBy(raw):\n annotations[key][fieldname] = RichTextValue(raw, \"text/html\", \"text/x-html-safe\")\n for fieldname in IMAGE_TILES.get(tile_name, []):\n data = annotations[key][fieldname]\n if data is not None:\n blob_path = data.get(\"blob_path\")\n if not blob_path:\n continue\n\n abs_blob_path = get_absolute_blob_path(obj, blob_path)\n if not abs_blob_path:\n logger.info(\"Blob path %s for tile %s of %s %s does not exist!\", blob_path, tile, obj.portal_type, obj.absolute_url())\n continue\n # Determine the class to use: file or image.\n filename = data[\"filename\"]\n content_type = data[\"content-type\"]\n\n # Write the field.\n with open(abs_blob_path, \"rb\") as myfile:\n blobdata = myfile.read()\n image = NamedBlobImage(\n data=blobdata,\n contentType=content_type,\n filename=filename,\n )\n annotations[key][fieldname] = image\n return item\n\n\nFixing invalid collection queries\n---------------------------------\n\nSome queries changes between Plone 4 and 5.\nThis fixes the issues.\n\nThe actual migration of topics to collections in ``collective.exportimport.serializer.SerializeTopicToJson`` does not (yet) take care of that.\n\n.. code-block:: python\n\n class CustomImportContent(ImportContent):\n\n def global_dict_hook(self, item):\n if item[\"@type\"] in [\"Collection\", \"Topic\"]:\n item = self.fix_query(item)\n\n def fix_query(self, item):\n item[\"@type\"] = \"Collection\"\n query = item.pop(\"query\", [])\n if not query:\n logger.info(\"Drop item without query: %s\", item[\"@id\"])\n return\n\n fixed_query = []\n indexes_to_fix = [\n \"portal_type\",\n \"review_state\",\n \"Creator\",\n \"Subject\",\n ]\n operator_mapping = {\n # old -\u003e new\n \"plone.app.querystring.operation.selection.is\":\n \"plone.app.querystring.operation.selection.any\",\n \"plone.app.querystring.operation.string.is\":\n \"plone.app.querystring.operation.selection.any\",\n }\n\n for crit in query:\n if crit[\"i\"] == \"portal_type\" and len(crit[\"v\"]) \u003e 30:\n # Criterion is all types\n continue\n\n if crit[\"o\"].endswith(\"relativePath\") and crit[\"v\"] == \"..\":\n # relativePath no longer accepts ..\n crit[\"v\"] = \"..::1\"\n\n if crit[\"i\"] in indexes_to_fix:\n for old_operator, new_operator in operator_mapping.items():\n if crit[\"o\"] == old_operator:\n crit[\"o\"] = new_operator\n\n if crit[\"i\"] == \"portal_type\":\n # Some types may have changed their names\n fixed_types = []\n for portal_type in crit[\"v\"]:\n fixed_type = PORTAL_TYPE_MAPPING.get(portal_type, portal_type)\n fixed_types.append(fixed_type)\n crit[\"v\"] = list(set(fixed_types))\n\n if crit[\"i\"] == \"review_state\":\n # Review states may have changed their names\n fixed_states = []\n for review_state in crit[\"v\"]:\n fixed_state = REVIEW_STATE_MAPPING.get(review_state, review_state)\n fixed_states.append(fixed_state)\n crit[\"v\"] = list(set(fixed_states))\n\n if crit[\"o\"] == \"plone.app.querystring.operation.string.currentUser\":\n crit[\"v\"] = \"\"\n\n fixed_query.append(crit)\n item[\"query\"] = fixed_query\n\n if not item[\"query\"]:\n logger.info(\"Drop collection without query: %s\", item[\"@id\"])\n return\n return item\n\n\nMigrate to Volto\n----------------\n\nYou can reuse the migration-code provided by ``@@migrate_to_volto`` in ``plone.volto`` in a migration.\nThe following example (used for migrating https://plone.org to Volto) can be used to migrate a site from any older version to Plone 6 with Volto.\n\nYou need to have the Blocks Conversion Tool (https://github.com/plone/blocks-conversion-tool) running that takes care of migrating richtext-values to Volto-blocks.\n\nSee https://6.docs.plone.org/backend/upgrading/version-specific-migration/migrate-to-volto.html for more details on the changes the migration to Volto does.\n\nThis code was used in real projects multiple times and is proven to work.\nAfter the migration you need to restart the instance to make all changes work.\n\n.. code-block:: python\n\n from App.config import getConfiguration\n from bs4 import BeautifulSoup\n from collective.exportimport.fix_html import fix_html_in_content_fields\n from collective.exportimport.fix_html import fix_html_in_portlets\n from contentimport.interfaces import IContentimportLayer\n from logging import getLogger\n from pathlib import Path\n from plone import api\n from plone.volto.browser.migrate_to_volto import migrate_richtext_to_blocks\n from plone.volto.setuphandlers import add_behavior\n from plone.volto.setuphandlers import remove_behavior\n from Products.CMFPlone.utils import get_installer\n from Products.Five import BrowserView\n from Products.ZCatalog.ProgressHandler import ZLogHandler\n from zope.interface import alsoProvides\n\n import requests\n import transaction\n\n logger = getLogger(__name__)\n\n # Add you own project-specific add-ons here\n DEFAULT_ADDONS = []\n\n VERSIONED_TYPES = [\n \"Document\",\n \"News Item\",\n \"Event\",\n \"Link\",\n ]\n\n\n class ImportAll(BrowserView):\n\n def __call__(self):\n\n request = self.request\n # Check if Blocks-conversion-tool is running\n headers = {\n \"Accept\": \"application/json\",\n \"Content-Type\": \"application/json\",\n }\n r = requests.post(\n \"http://localhost:5000/html\", headers=headers, json={\"html\": \"\u003cp\u003etext\u003c/p\u003e\"}\n )\n r.raise_for_status()\n\n # Submit a simple form template to trigger the import\n if not request.form.get(\"form.submitted\", False):\n return self.index()\n\n portal = api.portal.get()\n alsoProvides(request, IContentimportLayer)\n\n installer = get_installer(portal)\n if not installer.is_product_installed(\"contentimport\"):\n installer.install_product(\"contentimport\")\n\n # install required add-ons\n for addon in DEFAULT_ADDONS:\n if not installer.is_product_installed(addon):\n installer.install_product(addon)\n\n # Disable versioning before import\n for portal_type in VERSIONED_TYPES:\n remove_behavior(portal_type, \"plone.versioning\")\n remove_behavior(portal_type, \"plone.locking\")\n\n # Fake the target being a classic site even though plone.volto is installed...\n # 1. Allow Folders and Collections (they are disabled in Volto by default)\n portal_types = api.portal.get_tool(\"portal_types\")\n portal_types[\"Collection\"].global_allow = True\n portal_types[\"Folder\"].global_allow = True\n # 2. Enable richtext behavior (otherwise no text will be imported)\n for portal_type in [\"Document\", \"News Item\", \"Event\"]:\n add_behavior(portal_type, \"plone.richtext\")\n\n transaction.commit()\n cfg = getConfiguration()\n directory = Path(cfg.clienthome) / \"import\"\n\n # Import content\n view = api.content.get_view(\"import_content\", portal, request)\n request.form[\"form.submitted\"] = True\n request.form[\"commit\"] = 500\n # Change \"Plone.json\" to the name of your export file\n view(server_file=\"Plone.json\", return_json=True)\n transaction.commit()\n\n # Run all other imports\n other_imports = [\n \"relations\",\n \"members\",\n \"translations\",\n \"localroles\",\n \"ordering\",\n \"defaultpages\",\n \"discussion\",\n \"portlets\", # not really useful in Volto\n \"redirects\",\n ]\n for name in other_imports:\n view = api.content.get_view(f\"import_{name}\", portal, request)\n path = Path(directory) / f\"export_{name}.json\"\n if path.exists():\n results = view(jsonfile=path.read_text(), return_json=True)\n logger.info(results)\n transaction.get().note(f\"Finished import_{name}\")\n transaction.commit()\n else:\n logger.info(f\"Missing file: {path}\")\n\n # Optional: Run html-fixers on richtext\n fixers = [\n table_class_fixer,\n img_variant_fixer,\n scale_unscaled_images,\n fix_image_align,\n ]\n results = fix_html_in_content_fields(fixers=fixers)\n msg = \"Fixed html for {} content items\".format(results)\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n\n results = fix_html_in_portlets()\n msg = \"Fixed html for {} portlets\".format(results)\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n\n # Add blocks behavior to collections to convert richtext to blocks\n for portal_type in [\"Collection\"]:\n add_behavior(portal_type, \"volto.blocks\")\n\n # Update linksintegrity\n view = api.content.get_view(\"updateLinkIntegrityInformation\", portal, request)\n results = view.update()\n msg = f\"Updated linkintegrity for {results} items\"\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n\n # Rebuilding the catalog is necessary to prevent issues later on\n catalog = api.portal.get_tool(\"portal_catalog\")\n logger.info(\"Rebuilding catalog...\")\n catalog.clearFindAndRebuild()\n msg = \"Finished rebuilding catalog!\"\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n\n # This uses the blocks-conversion-tool to migrate to blocks\n logger.info(\"Start migrating richtext to blocks...\")\n migrate_richtext_to_blocks(purge_richtext=True)\n msg = \"Finished migrating richtext to blocks\"\n transaction.get().note(msg)\n transaction.commit()\n\n # Reuse the migration-form from plone.volto to do some more tasks\n view = api.content.get_view(\"migrate_to_volto\", portal, request)\n # Yes, we want to migrate default pages\n view.migrate_default_pages = True\n view.slate = True\n view.purge_richtext = True\n view.service_url = \"http://localhost:5000/html\"\n logger.info(\"Start migrating Folders to Documents...\")\n view.do_migrate_folders()\n msg = \"Finished migrating Folders to Documents!\"\n transaction.get().note(msg)\n transaction.commit()\n\n logger.info(\"Start migrating Collections to Documents...\")\n view.migrate_collections()\n msg = \"Finished migrating Collections to Documents!\"\n transaction.get().note(msg)\n transaction.commit()\n\n reset_dates = api.content.get_view(\"reset_dates\", portal, request)\n reset_dates()\n transaction.commit()\n\n # Reindex created and modified\n catalog = api.portal.get_tool(\"portal_catalog\")\n pghandler = ZLogHandler(5000)\n catalog.reindexIndex([\"created\", \"modified\"], None, pghandler=pghandler)\n\n # re-enable versioning and add initial versions\n for portal_type in VERSIONED_TYPES:\n add_behavior(portal_type, \"plone.versioning\")\n add_behavior(portal_type, \"plone.locking\")\n logger.info(\"Creating initial versions\")\n portal_repository = api.portal.get_tool(\"portal_repository\")\n brains = api.content.find(portal_type=VERSIONED_TYPES, sort_on=\"path\")\n total = len(brains)\n for index, brain in enumerate(brains):\n obj = brain.getObject()\n try:\n if not portal_repository.getHistoryMetadata(obj=obj):\n portal_repository.save(obj=obj, comment=\"Imported Version\")\n except FileTooLargeToVersionError:\n pass\n if not index % 1000:\n msg = f\"Created versions for {index} of {total} items.\"\n logger.info(msg)\n transaction.get().note(msg)\n transaction.commit()\n msg = \"Created initial versions\"\n transaction.get().note(msg)\n transaction.commit()\n\n # Disallow folders and collections again\n portal_types[\"Collection\"].global_allow = False\n portal_types[\"Folder\"].global_allow = False\n\n # Disable richtext behavior again\n for type_ in [\"Document\", \"News Item\", \"Event\"]:\n remove_behavior(type_, \"plone.richtext\")\n\n # Remove contentimport to also drop the BrowserLayer\n if installer.is_product_installed(\"contentimport\"):\n installer.uninstall_product(\"contentimport\")\n\n logger.info(\"Finished import_all\")\n return request.response.redirect(portal.absolute_url())\n\n\n def table_class_fixer(text, obj=None):\n if \"table\" not in text:\n return text\n dropped_classes = [\n \"MsoNormalTable\",\n \"MsoTableGrid\",\n ]\n replaced_classes = {\n \"invisible\": \"invisible-grid\",\n }\n soup = BeautifulSoup(text, \"html.parser\")\n for table in soup.find_all(\"table\"):\n table_classes = table.get(\"class\", [])\n for dropped in dropped_classes:\n if dropped in table_classes:\n table_classes.remove(dropped)\n for old, new in replaced_classes.items():\n if old in table_classes:\n table_classes.remove(old)\n table_classes.append(new)\n # all tables get the default bootstrap table class\n if \"table\" not in table_classes:\n table_classes.insert(0, \"table\")\n\n return soup.decode()\n\n\n def img_variant_fixer(text, obj=None):\n \"\"\"Set image-variants\"\"\"\n if not text:\n return text\n\n picture_variants = api.portal.get_registry_record(\"plone.picture_variants\")\n scale_variant_mapping = {\n k: v[\"sourceset\"][0][\"scale\"] for k, v in picture_variants.items()\n }\n scale_variant_mapping[\"thumb\"] = \"mini\"\n fallback_variant = \"preview\"\n\n soup = BeautifulSoup(text, \"html.parser\")\n for tag in soup.find_all(\"img\"):\n if \"data-val\" not in tag.attrs:\n # maybe external image\n continue\n scale = tag[\"data-scale\"]\n variant = scale_variant_mapping.get(scale, fallback_variant)\n tag[\"data-picturevariant\"] = variant\n\n classes = tag[\"class\"]\n new_class = f\"picture-variant-{variant}\"\n if new_class not in classes:\n classes.append(new_class)\n tag[\"class\"] = classes\n\n return soup.decode()\n\n\n def scale_unscaled_images(text, obj=None):\n \"\"\"Scale unscaled image\"\"\"\n if not text:\n return text\n fallback_scale = \"huge\"\n\n soup = BeautifulSoup(text, \"html.parser\")\n for tag in soup.find_all(\"img\"):\n if \"data-val\" not in tag.attrs:\n # maybe external image\n continue\n\n scale = tag[\"data-scale\"]\n # Prevent unscaled images!\n if not scale:\n scale = fallback_scale\n tag[\"data-scale\"] = fallback_scale\n if not tag[\"src\"].endswith(scale):\n tag[\"src\"] = tag[\"src\"] + \"/\" + scale\n\n return soup.decode()\n\n\n def fix_image_align(text, obj=None):\n \"\"\"Replace align='xx' with css-classes\"\"\"\n if not text:\n return text\n\n soup = BeautifulSoup(text, \"html.parser\")\n for tag in soup.find_all(\"img\"):\n if \"align\" not in tag.attrs:\n continue\n\n classes = tag.get(\"class\", [])\n direction = tag[\"align\"]\n if direction == \"left\":\n classes.append(\"image-left\")\n elif direction == \"right\":\n classes.append(\"image-right\")\n if \"image-inline\" in classes:\n classes.remove(\"image-inline\")\n del tag[\"align\"]\n return soup.decode()\n\n\n\nMigrate very old Plone Versions with data created by collective.jsonify\n-----------------------------------------------------------------------\n\nVersions older than Plone 4 do not support ``plone.restapi`` which is required to serialize the content used by ``collective.exportimport``.\n\nTo migrate Plone 1, 2 and 3 to Plone 6 you can use ``collective.jsonify`` for the export and ``collective.exportimport`` for the import.\n\n\nExport with collective.jsonify\n******************************\n\nUse https://github.com/collective/collective.jsonify to export content.\n\nYou include the methods of ``collective.jsonify`` using `External Methods`.\nSee https://github.com/collective/collective.jsonify/blob/master/docs/install.rst for more info.\n\nTo work better with ``collective.exportimport`` you could extend the exported data using the feature ``additional_wrappers``.\nAdd info on the parent of an item to make it easier for ``collective.exportimport`` to import the data.\n\nHere is a full example for ``json_methods.py`` which should be in ``BUILDOUT_ROOT/parts/instance/Extensions/``\n\n.. code-block:: python\n\n def extend_item(obj, item):\n \"\"\"Extend to work better well with collective.exportimport\"\"\"\n from Acquisition import aq_parent\n parent = aq_parent(obj)\n item[\"parent\"] = {\n \"@id\": parent.absolute_url(),\n \"@type\": getattr(parent, \"portal_type\", None),\n }\n if getattr(parent.aq_base, \"UID\", None) is not None:\n item[\"parent\"][\"UID\"] = parent.UID()\n\n return item\n\n\nHere is a full example for ``json_methods.py`` which should be in ``\u003cBUILDOUT_ROOT\u003e/parts/instance/Extensions/``\n\n.. code-block:: python\n\n from collective.jsonify.export import export_content as export_content_orig\n from collective.jsonify.export import get_item\n\n EXPORTED_TYPES = [\n \"Folder\",\n \"Document\",\n \"News Item\",\n \"Event\",\n \"Link\",\n \"Topic\",\n \"File\",\n \"Image\",\n \"RichTopic\",\n ]\n\n EXTRA_SKIP_PATHS = [\n \"/Plone/archiv/\",\n \"/Plone/do-not-import/\",\n ]\n\n # Path from which to continue the export.\n # The export walks the whole site respecting the order.\n # It will ignore everything untill this path is reached.\n PREVIOUS = \"\"\n\n def export_content(self):\n return export_content_orig(\n self,\n basedir=\"/var/lib/zope/json\",\n skip_callback=skip_item,\n extra_skip_classname=[],\n extra_skip_id=[],\n extra_skip_paths=EXTRA_SKIP_PATHS,\n batch_start=0,\n batch_size=10000,\n batch_previous_path=PREVIOUS or None,\n )\n\n def skip_item(item):\n \"\"\"Return True if the item should be skipped\"\"\"\n portal_type = getattr(item, \"portal_type\", None)\n if portal_type not in EXPORTED_TYPES:\n return True\n\n def extend_item(obj, item):\n \"\"\"Extend to work better well with collective.exportimport\"\"\"\n from Acquisition import aq_parent\n parent = aq_parent(obj)\n item[\"parent\"] = {\n \"@id\": parent.absolute_url(),\n \"@type\": getattr(parent, \"portal_type\", None),\n }\n if getattr(parent.aq_base, \"UID\", None) is not None:\n item[\"parent\"][\"UID\"] = parent.UID()\n\n return item\n\nTo use these create three \"External Method\" in the ZMI root at the Zope root to use that:\n\n* id: \"export_content\", module name: \"json_methods\", function name: \"export_content\"\n* id: \"get_item\", module name: \"json_methods\", function name: \"get_item\"\n* id: \"extend_item\", module name: \"json_methods\", function name: \"extend_item\"\n\nThen you can pass the extender to the export using a query-string: http://localhost:8080/Plone/export_content?additional_wrappers=extend_item\n\n\nImport with collective.jsonify\n******************************\n\nTwo issues need to be dealt with to allow ``collective.exportimport`` to import the data generated by ``collective.jsonify``.\n\n#. The data is in directories instead of in one large json-file.\n#. The json is not in the expected format.\n\nStarting with version 1.8 you can pass an iterator to the import.\n\nYou need to create a directory-walker that sorts the json-files the right way.\nBy default it would import them in the order `1.json`, `10.json`, `100.json`, `101.json` and so on.\n\n.. code-block:: python\n\n from pathlib import Path\n\n def filesystem_walker(path=None):\n root = Path(path)\n assert(root.is_dir())\n folders = sorted([i for i in root.iterdir() if i.is_dir() and i.name.isdecimal()], key=lambda i: int(i.name))\n for folder in folders:\n json_files = sorted([i for i in folder.glob(\"*.json\") if i.stem.isdecimal()], key=lambda i: int(i.stem))\n for json_file in json_files:\n logger.debug(\"Importing %s\", json_file)\n item = json.loads(json_file.read_text())\n item[\"json_file\"] = str(json_file)\n item = prepare_data(item)\n if item:\n yield item\n\nThe walker takes the path to be the root with one or more directories holding the json-files.\nThe sorting of the files is done using the number in the filename.\n\nThe method ``prepare_data`` modifies the data before passing it to the import.\nA very similar task is done by ``collective.exportimport`` during export.\n\n.. code-block:: python\n\n def prepare_data(item):\n \"\"\"modify jsonify data to work with c.exportimport\"\"\"\n\n # Drop relationfields or defer the import\n item.pop(\"relatedItems\", None)\n\n mapping = {\n # jsonify =\u003e exportimport\n \"_uid\": \"UID\",\n \"_type\": \"@type\",\n \"_path\": \"@id\",\n \"_layout\": \"layout\",\n # AT fieldnames =\u003e DX fieldnames\n \"excludeFromNav\": \"exclude_from_nav\",\n \"allowDiscussion\": \"allow_discussion\",\n \"subject\": \"subjects\",\n \"expirationDate\": \"expires\",\n \"effectiveDate\": \"effective\",\n \"creation_date\": \"created\",\n \"modification_date\": \"modified\",\n \"startDate\": \"start\",\n \"endDate\": \"end\",\n \"openEnd\": \"open_end\",\n \"eventUrl\": \"event_url\",\n \"wholeDay\": \"whole_day\",\n \"contactEmail\": \"contact_email\",\n \"contactName\": \"contact_name\",\n \"contactPhone\": \"contact_phone\",\n \"imageCaption\": \"image_caption\",\n }\n for old, new in mapping.items():\n item = migrate_field(item, old, new)\n\n if item.get(\"constrainTypesMode\", None) == 1:\n item = migrate_field(item, \"constrainTypesMode\", \"constrain_types_mode\")\n else:\n item.pop(\"locallyAllowedTypes\", None)\n item.pop(\"immediatelyAddableTypes\", None)\n item.pop(\"constrainTypesMode\", None)\n\n if \"id\" not in item:\n item[\"id\"] = item[\"_id\"]\n return item\n\n\n def migrate_field(item, old, new):\n if item.get(old, _marker) is not _marker:\n item[new] = item.pop(old)\n return item\n\nYou can pass the generator ``filesystem_walker`` to the import:\n\n.. code-block:: python\n\n class ImportAll(BrowserView):\n\n def __call__(self):\n # ...\n cfg = getConfiguration()\n directory = Path(cfg.clienthome) / \"import\"\n\n # import content\n view = api.content.get_view(\"import_content\", portal, request)\n request.form[\"form.submitted\"] = True\n request.form[\"commit\"] = 1000\n view(iterator=filesystem_walker(directory / \"mydata\"))\n\n # import default-pages\n import_deferred = api.content.get_view(\"import_deferred\", portal, request)\n import_deferred()\n\n\n class ImportDeferred(BrowserView):\n\n def __call__(self):\n self.title = \"Import Deferred Settings (default pages)\"\n if not self.request.form.get(\"form.submitted\", False):\n return self.index()\n\n for brain in api.content.find(portal_type=\"Folder\"):\n obj = brain.getObject()\n annotations = IAnnotations(obj)\n if DEFERRED_KEY not in annotations:\n continue\n\n default = annotations[DEFERRED_KEY].pop(\"_defaultpage\", None)\n if default and default in obj:\n logger.info(\"Setting %s as default page for %s\", default, obj.absolute_url())\n obj.setDefaultPage(default)\n if not annotations[DEFERRED_KEY]:\n annotations.pop(DEFERRED_KEY)\n api.portal.show_message(\"Done\", self.request)\n return self.index()\n\n``collective.jsonify`` puts the info on relations, translations and default-pages in the export-file.\nYou can use the approach to defer imports to deal with that data after all items were imported.\nThe example ``ImportDeferred`` above uses that approach to set the default pages.\n\nThis ``global_obj_hook`` below stores that data in a annotation:\n\n.. code-block:: python\n\n def global_obj_hook(self, obj, item):\n # Store deferred data in an annotation.\n keys = [\"_defaultpage\"]\n data = {}\n for key in keys:\n if value := item.get(key, None):\n data[key] = value\n if data:\n annotations = IAnnotations(obj)\n annotations[DEFERRED_KEY] = data\n\n\nTranslations\n============\n\nThis product has been translated into\n\n- Spanish\n\n\nContribute\n==========\n\n- Issue Tracker: https://github.com/collective/collective.exportimport/issues\n- Source Code: https://github.com/collective/collective.exportimport\n\n\nSupport\n-------\n\nIf you are having issues, please let us know.\n\n\nLicense\n-------\n\nThe project is licensed under the GPLv2.\n\n\nWritten by\n==========\n\n.. image:: ./docs/starzel.png\n :target: https://www.starzel.de\n :alt: Starzel.de\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollective%2Fcollective.exportimport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcollective%2Fcollective.exportimport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcollective%2Fcollective.exportimport/lists"}