{"id":42652560,"url":"https://github.com/burnbright/silverstripe-importexport","last_synced_at":"2026-01-29T07:41:39.742Z","repository":{"id":28832868,"uuid":"32356499","full_name":"burnbright/silverstripe-importexport","owner":"burnbright","description":"Import and export data from SilverStripe in a variety of forms. Looking for maintainer, see https://github.com/burnbright/silverstripe-importexport/issues/48","archived":false,"fork":false,"pushed_at":"2017-07-11T01:25:11.000Z","size":108,"stargazers_count":46,"open_issues_count":28,"forks_count":33,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-08-31T04:39:36.690Z","etag":null,"topics":["bulk-loader","export","import","maintainer-wanted","php","silverstripe"],"latest_commit_sha":null,"homepage":"","language":"PHP","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"missxiaolin/webpack-many-pages","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/burnbright.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-03-16T22:21:19.000Z","updated_at":"2023-12-25T13:12:07.000Z","dependencies_parsed_at":"2022-09-05T06:50:22.141Z","dependency_job_id":null,"html_url":"https://github.com/burnbright/silverstripe-importexport","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/burnbright/silverstripe-importexport","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burnbright%2Fsilverstripe-importexport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burnbright%2Fsilverstripe-importexport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burnbright%2Fsilverstripe-importexport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burnbright%2Fsilverstripe-importexport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/burnbright","download_url":"https://codeload.github.com/burnbright/silverstripe-importexport/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/burnbright%2Fsilverstripe-importexport/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28870640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T07:35:32.468Z","status":"ssl_error","status_checked_at":"2026-01-29T07:33:31.463Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk-loader","export","import","maintainer-wanted","php","silverstripe"],"created_at":"2026-01-29T07:41:39.167Z","updated_at":"2026-01-29T07:41:39.733Z","avatar_url":"https://github.com/burnbright.png","language":"PHP","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SilverStripe Import/Export Module\n\n[![Build Status](https://travis-ci.org/burnbright/silverstripe-importexport.svg?branch=master)](https://travis-ci.org/burnbright/silverstripe-importexport)\n\nImport and export data from SilverStripe in various forms, including CSV. This module serves as a replacement/overhaul of BulkLoader functionality found in the framework.\n\n## The loading process\n\n1. Raw data is retrieved from a source (`BulkLoaderSource`).\n2. Data is provided as iterable rows (each row is heading-\u003evalue mapped array).\n3. Rows are mapped to a standardised format, based on a user/developer provided mapping.\n4. Data is set/linked/tranformed onto a placeholder DataObject.\n5. Existing record replaces placeholder, or placeholder becomes the brand new DataObject.\n5. DataObject is validated and saved.\n7. All results are stored in `BulkLoader_Result`.\n\n## User-defined column mapping\n\nUsers can choose which columns map to DataObject fields. This removes any need to define headings, or headings according to a given schema.\nUsers can state if the first line of data is in fact a heading row.\nMappings are saved for the next time an import is done on the same GridField.\n\n![gridfieldimporter](https://cloud.githubusercontent.com/assets/1356335/7108697/24db13c4-e1e2-11e4-9ff1-86d42144201b.gif)\n\n## Grid Field Importer\n\nThis is a grid field component for users to selecting a CSV file and map it's columns to data fields.\n\n```php\n$importer = new GridFieldImporter('before');\n$gridConfig-\u003eaddComponent($importer);\n```\n\nThe importer makes use of the `CSVFieldMapper`, which displays the beginning content of a CSV.\n\n## BulkLoaderSource\n\nA `BulkLoaderSource` provides an iterator to get record data from. Data could come from anywhere such as a CSV file, a web API, etc.\n\nIt can be used independently from the BulkLoader to obtain data.\n\n```php\n$source = new CsvBulkLoaderSource();\n$source-\u003esetFilePath(\"files/myfile.csv\")\n    -\u003esetHasHeader(true)\n    -\u003esetFieldDelimiter(\",\")\n    -\u003esetFieldEnclosure(\"'\");\n\nforeach($source-\u003egetIterator() as $record){\n    //do stuff\n}\n```\n\n## (Better)BulkLoader\n\n* Saves data from a particular source and persists it to database via the ORM.\n* Determines which fields can be mapped to, either scaffolded from the model, provided by configuration, or both.\n* Detects existing records, and either skips or updates them, based on criteria.\n* Maps the source data to new/existing dataobjects, based on a given mapping.\n* Finds, creates, and connects relation objects to objects.\n* Can clear all records prior to processing.\n\n```php\n$source = new CsvBulkLoaderSource();\n$source-\u003esetFilePath(\"files/myfile.csv\");\n\n$loader = new BetterBulkLoader(\"Product\");\n$loader-\u003esetSource($source);\n$loader-\u003eaddNewRecords = false;  // an option to skip new records\n\n$result = $loader-\u003eload();\n```\n\n## ListBulkLoader\n\nOften you'll want to confine bulk loading to a specific DataList. The ListBulkLoader is a variation of BulkLoader that adds and removes records from a given DataList. Of course DataList iself doesn't have an add method implemented, so you'll probably find it more useful for a `HasManyList`.\n\n```php\n$category = ProductCategory::get()-\u003efirst();\n\n$source = new CsvBulkLoaderSource();\n$source-\u003esetFilePath(\"productlist.csv\");\n\n$loader = new ListBulkLoader($category-\u003eProducts());\n$loader-\u003esetSource($source);\n\n$result = $loader-\u003eload();\n```\n\n## Mapping record data to a standard format\n\nYou can provide a `columnMap` to map incoming records to a standard format.\n\n```php \n$loader-\u003ecolumnMap = array(\n    'first name' =\u003e 'FirstName',\n    'Name' =\u003e 'FirstName',\n    'bio' =\u003e 'Biography',\n    'bday' =\u003e 'Birthday',\n    'teamtitle' =\u003e 'Team.Title',\n    'teamsize' =\u003e 'Team.TeamSize',\n    'salary' =\u003e 'Contract.Amount'\n);\n```\n\nThis column map is generated by the `CSVFieldMapper` control inside the `GridFieldImporter` component.\n\n## Scaffolding vs Defining Mappable Fields\n\nMappable fields will be scaffolded if you do not define them yourself. This includes fields that are on relations, so that relations can be linked up.\n\nIt is likely there will be fields you don't want mapped, in which case you should specify a `mappableFields` array on your loader:\n```php\n$loader-\u003emappableFields = array(\n    'FirstName' =\u003e 'First Name',\n    'Surname' =\u003e 'Last Name',\n    'Biography' =\u003e 'Biography',\n    'Birthday' =\u003e 'Birthday',\n    'Team.Title' =\u003e 'Team'\n);\n```\n\n## Transform incoming record data\n\nYou may want to perform some transformations to incoming record data. This can be done by specifying a callback against the record field name.\n\n```php\n$loader-\u003etransforms = array(\n    'Code' =\u003e array(\n        'callback' =\u003e function($value, $placeholder) {\n            //capitalize course codes\n            return strtoupper($value);\n        }\n    )\n);\n```\n\n## Require specific data to be present\n\nIncoming records without required data will be skipped.\n\n```php\n$loader-\u003etransforms = array(\n    'Title' =\u003e array(\n        'required' =\u003e true\n    )\n);\n```\n\nNote that empty records are skipped by default.\n\n## Creating and linking related DataObjects\n\nThe bulk loader can handle linking and creating `has_one` relationship objects, by either providing a callback, or using the `Relation.FieldName` style \"dot notation\". Relationship handling is also performed in the `transformations` array.\n\nYou can specify at the BulkLoader level if records will be created and linked, then you can also specify the behaviour for each field. The default behaviour is to both link and create relation objects.\n\nHere are some configuration examples:\n\n```php\n$loader-\u003etransforms = array(\n    //link and create courses\n    'Course.Title' = array(\n        'link' =\u003e true,\n        'create' =\u003e true\n    ),\n    //only link to existing tutors\n    'Tutor.Name' =\u003e array(\n        'link' =\u003e true,\n        'create' =\u003e false\n    ),\n    //custom way to find parent courses\n    'Parent' =\u003e array(\n        'callback' =\u003e function($value, $placeholder) use ($self){\n            return Course::get()\n                -\u003efilter(\"Title\", $value)\n                -\u003efirst();\n        }\n    )\n);\n```\n\nNote that `$placeholder` in the above example refers to a dummy DataObject that is populated in order to then be saved, or checked against for duplicates. You should not call `$placeholder-\u003ewrite()` in your callback.\n\n## Specify a relation list\n\nIn the same way that you may use a `ListBulkLoader` to constrain records to a given DataList, you may also want to constrain the relation records to a List.\n\n```php\n$loader-\u003etransforms = array(\n    //link and create courses\n    'Course.Title' = array(\n        'list' =\u003e $self-\u003eCourses()\n    )\n);\n```\n\n\n## Determining when to overwrite existing (duplicate) DataObjects\n\nDuplicate checks are performed on record data, mapped into the standardised form.\n\nYou can perform duplicate checking on data fields:\n\n```php\n//course is a duplicate when title is the same\n$loader-\u003eduplicateChecks = array(\n    \"Title\"\n);\n```\n\nOr on a relation:\n```php\n//course selection is a duplicate when course is the same\n$loader-\u003eduplicateChecks = array(\n    \"Course.Title\"\n);\n```\n\nDuplicates can also be found using a callback function:\n```php\n$loader-\u003eduplicateChecks = array(\n    \"FooBar\" =\u003e array(\n        \"callback\" =\u003e function($fieldName, $record) {\n            if(!isset($record[\"FirstName\"]) || !isset($record[\"LastName\"])){\n                return null;\n            }\n\n            return Person::get()\n                -\u003efilter(\"FirstName\", $record['FirstName'])\n                -\u003efilter(\"LastName\", $record['LastName'])\n                -\u003efirst();\n        }\n    )\n);\n```\n\n## Publish pages during import\n\nIf you are importing instances of SiteTree, you can have those pages automatically published using this configuration:\n```php\n$loader-\u003esetPublishPages(true);\n```\n\n## Replace all the \"legacy\" ModelAdmin importers\n\nSome simple yaml config options to help with swapping out all the importer functionality.\n\n```yaml\nModelAdmin:\n    removelegacyimporters: true\n    addbetterimporters: true\n```\n\nRemove only the scafolded (non-custom) importers:\n```yaml\nModelAdmin:\n    removelegacyimporters: scaffolded\n```\n\n## Troubleshooting\n\n### Missing relation objects\n\nIf you are writing relation objects during loading, and they fail validation, the loader will simply ignore that relation object.\n\n### Multiple relation data fields not mapping to the same relation\n\nIf you have mapped multiple fields mapping to the same relation, you may get situations where the incorrect existing relation object is joined. The first field that is mapped is the same field used to find the relation.\nFor example, you'll likely want a Title to be used to find/create a relation, and then an Amount will be added to that same relation, rather than finding/creating a relation by an Amount, and setting the Title.\n\nDefine the correct ordering using the mappableFields array to fix this.\n\n## Contributions\n\nPlease do contribute whatever you can to this module. Check out the [issues](https://github.com/burnbright/silverstripe-importexport/issues) and [milestones](https://github.com/burnbright/silverstripe-importexport/milestones) to see what has needs to be done.\n\n## License\n\nMIT\n\n## Author\n\nJeremy Shipman (http://jeremyshipman.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburnbright%2Fsilverstripe-importexport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fburnbright%2Fsilverstripe-importexport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fburnbright%2Fsilverstripe-importexport/lists"}