{"id":20961828,"url":"https://github.com/heimrichhannot/contao-entity-import-bundle","last_synced_at":"2025-05-14T07:31:07.003Z","repository":{"id":51291598,"uuid":"242151273","full_name":"heimrichhannot/contao-entity-import-bundle","owner":"heimrichhannot","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-18T07:29:36.000Z","size":1028,"stargazers_count":2,"open_issues_count":5,"forks_count":2,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-21T12:56:03.895Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/heimrichhannot.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-02-21T13:58:27.000Z","updated_at":"2024-07-18T07:29:32.000Z","dependencies_parsed_at":"2023-09-22T18:22:00.913Z","dependency_job_id":null,"html_url":"https://github.com/heimrichhannot/contao-entity-import-bundle","commit_stats":{"total_commits":136,"total_committers":5,"mean_commits":27.2,"dds":"0.49264705882352944","last_synced_commit":"270bccbe9de3d7ceed89c521b6ae831cf57926a2"},"previous_names":[],"tags_count":73,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heimrichhannot%2Fcontao-entity-import-bundle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heimrichhannot%2Fcontao-entity-import-bundle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heimrichhannot%2Fcontao-entity-import-bundle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/heimrichhannot%2Fcontao-entity-import-bundle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/heimrichhannot","download_url":"https://codeload.github.com/heimrichhannot/contao-entity-import-bundle/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254094858,"owners_count":22013649,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T02:17:45.595Z","updated_at":"2025-05-14T07:31:04.199Z","avatar_url":"https://github.com/heimrichhannot.png","language":"PHP","readme":"# Contao Entity Import Bundle\n\nThis bundle offers a generic importer to migrate data from various sources to contao database entities.\n\n**This Bundle is still in development.**\n\n## Features\n\n- import data from either file content or database into arbitrary contao database entities (`tl_*`)\n- support for various data types (json, csv, ...)\n- support for various source types (contao file system, http, absolute path)\n- executable from contao backend, as cronjob and symfony command\n- automatic field manipulation (sorting, alias, dateAdded, tstamp)\n- merge and insert mode\n- email and contao_log notifications while executing import with exceptions (will be sent once per importer \nconfiguration, will be reset after next successful import)\n- (optional) support for [DC_Multilingual](https://github.com/terminal42/contao-DC_Multilingual) if installed\n\n## Impressions\n\nImporter source backend settings:\n\n![alt import_source_1](./docs/img/importer_source.png)\n\nImporter config backend settings:\n\n![alt privacy config](./docs/img/importer_config.png)\n\n## Installation\n\nInstall via composer: `composer require heimrichhannot/contao-entity-import-bundle` and update your database.\n\n## Configuration\n\n1. Navigate to \"Import\" in the Contao backend in the section \"system\".\n1. Create an importer source to your needs.\n1. Create an importer using the source created in the step before.\n1. Run the importer either using dry-run or the normal mode.\n\n### config.yml\n\n```yaml\nhuh_entity_import:\n  debug:\n    contao_log: true # log errors while importing to contao system log\n    email: false # report errors while importing via email to the admin email defined in the contao settings\n```\n\n## Technical instructions\n\n### Output messages during import process respecting the context\n\nAn importer can be run as command, cronjob or in the web context. The context implies how to output messages (or error messages).\n\nFor example, when running an importer by clicking the \"import\" button in the website (i.e. in the browser), the messages should be\nprinted out using `\\Contao\\Message::addError()` (or `addConfirmation()`, ...). If the import is run as command, the same messages\nneed to be printed out on command line using a `SymfonyStyle` instance. If the command is also run in web context, the messages even\nneed to be stored to the `tl_entity_import_config` entity, because these are retrieved by ajax in the progress bar view.\n\nAs you can see, it's not just \"printing a message\", but taking care of the context, as well.\n\nFor this you can use the method `ImporterInterface::outputResultMessage()` which takes care of the correct context itself.\n\n### Run as symfony command\n\n`huh:entity-import:execute config-ids [--dry-run]`\n\nwhere `config-ids` needs to be a comma separated list of importer config IDs.\n\n##### Arguments\nArgument | Mandatory | Type | Description\n--------|--------|-------|---\nconfig-id | true | integer |The ID of the importer configuration\ndry-run | false | boolean |Run importer without writing data into database\n\n### Run as contao cron\n\nImport is executable with contao poor man's cron. The interval of execution is similar to the contao definition.\nThe import configuration allows to enable cron execution and picking of the cron interval.\nPossible to choose between `minutely`, `hourly`, `daily`, `weekly`, `monthly` interval. It is recommended to setup\nthe debug options in config.yml before importing via cronjob.\n\n### Add custom sources\n\n1. Create the source and extend from `\\HeimrichHannot\\EntityImportBundle\\Source\\AbstractSource` or implement the\n   interface `\\HeimrichHannot\\EntityImportBundle\\Source\\SourceInterface`.\n1. Create an event listener for the event `SourceFactoryCreateSourceEvent`:\n   ```php\n   class SourceFactoryCreateSourceEventListener {\n       public function __invoke(SourceFactoryCreateSourceEvent $event) {\n           $sourceModel = $event-\u003egetSourceModel();\n           $source = $event-\u003egetSource();\n   \n           switch ($sourceModel-\u003etype) {\n               case 'new_stuff':\n                   $source = new NewStuffSource();\n   \n                   break;\n               // ...\n           }\n   \n           $event-\u003esetSource($source);\n       }\n   }\n   ```\n1. Create a `tl_entity_import_source.php` and add your sources in the `type` field's options:\n   ```php\n   $dca = \u0026$GLOBALS['TL_DCA']['tl_entity_import_source'];\n   \n   $dca['fields']['type']['options'] = array_merge($dca['fields']['type']['options'], [\n       'new_stuff'\n   ]);\n   ```\n1. Create your palette in your `tl_entity_import_source.php`, e.g. by copying one from this bundle.\n\n### Add presets for source/config field mapping\n\nAdding a field mapping might be tedious sometimes. Hence we provide an event for adding presets which then are selectable in the backend entity (source and config).\n\nJust register an event listener and add your presets the following way:\n\n```\n$presets = $event-\u003egetPresets();\n\n$presets = array_merge($presets, [\n    'twitter' =\u003e [\n        [\n            'name'        =\u003e 'externalId',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'id',\n        ],\n        [\n            'name'        =\u003e 'text',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'full_text',\n        ],\n        [\n            'name'        =\u003e 'publishedAt',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'created_at',\n        ],\n        [\n            'name'        =\u003e 'imageUrl',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'image_url',\n        ],\n        [\n            'name'        =\u003e 'imageId',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'image_id',\n        ],\n        [\n            'name'        =\u003e 'url',\n            'valueType'   =\u003e 'source_value',\n            'sourceValue' =\u003e 'url',\n        ],\n    ]\n]);\n\n$event-\u003esetPresets($presets);\n```\n\n### Add quick importers for editors\n\nTo be honest: A \"quick importer\" is a nicer term for \"easy importer\", so that editors don't get sad ;-) The use case for quick importers\nis to have a user interface which is *way* easier to use for editors.\n\nSimply create an import source and import config as you would normally. Afterwards editors can create quick importers on their own\nand add custom file sources (currently only files of the type CSV are supported).\n\n![alt quick importer example](./docs/img/quick_importer.png)\n\n### Run the import via cronjob in web context\n\nIf you have large datasets the import might take too much memory of the web server. In this case it's often useful to\n\n1. Run the importer as symfony command. This has to be done by a developer.\n2. Run the importer in the web context by cronjob. This can be done by editors or developers.\n\nIn most cases, running an importer by command on the shell takes way less memory compared to running it in the web context via website.\n\nFor option 2 to work you need to do the following steps:\n\n1. Add the bundle's import command as cronjob using the parameter `--web-cron-mode`:\u003cbr\u003e\n   `* * * * * /path/to/vendor/bin/contao-console huh:entity-import:execute \u003cimport config ids\u003e --web-cron-mode \u003e/dev/null 2\u003e\u00261`\n2. Activate `useCron` in the importer config and fill out the needed fields.\n3. Activate `useCronInWebContext` in the importer config.\n\n## Events\n\nName | Description\n-----|------------\n`AddConfigFieldMappingPresetsEvent` | Add config field mapping presets\n`AddSourceFieldMappingPresetsEvent` | Add source field mapping presets\n`AfterCsvFileSourceGetRowEvent` | Modify csv rows e.g. for fixing file encoding issues\n`AfterFileSourceGetContentEvent` | Configure the data after receiving from source\n`AfterImportEvent` | Get imported data after finished import\n`AfterItemImportEvent` | Get imported item data after finished import\n`BeforeAuthenticationEvent` | Configure authentication data before sending GET request to http source\n`BeforeImportEvent` | Configure the data before importing\n`BeforeItemImportEvent` | Configure the item data before importing; call `setSkipped(true)` in order to skip the import\n`SourceFactoryCreateSourceEvent` | Implement custom logic for new custom file sources\n`BeforeFileImportEvent` | Implement a custom logic for setting the folder and/or file path when importing files\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheimrichhannot%2Fcontao-entity-import-bundle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheimrichhannot%2Fcontao-entity-import-bundle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheimrichhannot%2Fcontao-entity-import-bundle/lists"}