{"id":15143741,"url":"https://github.com/contentful/contentful-database-importer.rb","last_synced_at":"2025-09-29T12:30:56.484Z","repository":{"id":66184172,"uuid":"71643568","full_name":"contentful/contentful-database-importer.rb","owner":"contentful","description":"Adapter to extract data from SQL Databases https://www.contentful.com","archived":true,"fork":false,"pushed_at":"2023-10-26T13:46:31.000Z","size":53,"stargazers_count":23,"open_issues_count":8,"forks_count":10,"subscribers_count":39,"default_branch":"master","last_synced_at":"2025-03-06T12:54:12.348Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/contentful.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-22T14:16:52.000Z","updated_at":"2024-05-20T09:29:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"99e609b1-b1dd-406f-9e5d-05f8fa5b22a9","html_url":"https://github.com/contentful/contentful-database-importer.rb","commit_stats":{"total_commits":25,"total_committers":2,"mean_commits":12.5,"dds":0.07999999999999996,"last_synced_commit":"7bba58d3c7e2d801ff707267704322387d26da95"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/contentful/contentful-database-importer.rb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contentful%2Fcontentful-database-importer.rb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contentful%2Fcontentful-database-importer.rb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contentful%2Fcontentful-database-importer.rb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contentful%2Fcontentful-database-importer.rb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/contentful","download_url":"https://codeload.github.com/contentful/contentful-database-importer.rb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/contentful%2Fcontentful-database-importer.rb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277516226,"owners_count":25831795,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-29T02:00:09.175Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T10:03:21.546Z","updated_at":"2025-09-29T12:30:56.082Z","avatar_url":"https://github.com/contentful.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Contentful Database Importer\n\n![TravisCI](https://travis-ci.org/contentful/contentful-database-importer.rb.svg?branch=master)\n\nA simple DSL to define your database schemas, their relation to Contentful and import them to Contentful.\n\nThis gem is intended to be a replacement to [`database_exporter`](https://github.com/contentful/database-exporter.rb). _Warning_: Both gems are incompatible.\n\n## Contentful\n\n[Contentful](https://www.contentful.com) provides a content infrastructure for digital teams to power content in websites, apps, and devices. Unlike a CMS, Contentful was built to integrate with the modern software stack. It offers a central hub for structured content, powerful management and delivery APIs, and a customizable web app that enable developers and content creators to ship digital products faster.\n\n## What does `contentful-database-importer` do?\n\n`contentful-database-importer` let's you define mapping classes between your database and Contentful and allows\nyou to generate a JSON file that's a valid [`contentful_bootstrap`](https://github.com/contentful/contentful-bootstrap.rb) JSON Template,\nor directly import to Contentful, creating a new space and using your data to populate the content.\n\n## Requirements\n\n* Ruby\n* A Relational Database\n\n## Installation\n\n```bash\ngem install contentful-database-importer\n```\n\n## Usage\n\n* Create a new directory for your import configuration:\n\n```bash\nmkdir my_importer_dir \u0026\u0026 cd my_importer_dir\n```\n\n* Create a _`Gemfile`_ with the gem:\n\n```ruby\nsource 'https://rubygems.org'\n\ngem 'contentful-database-importer'\n```\n\n* Add to your _`Gemfile`_ the handler specific to your database (e.g.):\n\n```ruby\ngem 'pg' # if using Postgres\ngem 'sqlite3' # if using SQLite\ngem 'mysql' # if using MySQL\n```\n\n* Create your importer file, for example _`import.rb`_:\n\n```ruby\nrequire 'contentful/database_importer'\n\nclass MyTable\n  include Contentful::DatabaseImporter::Resource\n\n  # ... your schema definition ... (explained in next section)\nend\n\n# ... more table definitions ...\n\nContentful::DatabaseImporter.setup do |config|\n  config.space_name = 'My Cool New Space'\n  config.database_connection = 'postgres://user:pass@host:port'\nend\n\nContentful::DatabaseImporter.run!\n```\n\n* Run your file:\n\n```bash\nbundle exec ruby import.rb\n```\n\n### Defining your Schema\n\n```ruby\nclass MyTable\n  include Contentful::DatabaseImporter::Resource\n\n  self.table_name = 'overrides_table_name' # Optional - By default it's the class name in snake case. E.g. 'my_table'\n  self.content_type_id = 'overrides_content_type_id' # Optional - By default it's the class name in snake case. E.g 'my_table'\n  self.content_type_name = 'Overrides Name' # Optional - By default it's the class name\n\n  id Contentful::DatabaseImporter::IdGenerator::Base, template: '{{content_type_id}}_{{foo}}_{{index}}' # Optional - By default it's the IdGenerator::Base(template: '{{content_type_id}}_{{index}}')\n\n  field :foo, type: :string\n  field :bar, maps_to: :not_bar, type: :string\n  field :image, type: :asset\nend\n```\n\nFor specifying namespaces for your tables, use `self.table_name = :namespace__table_name`.\n\nIf planning to upgrade to Sequel `v5`, and require namespaces, at the beggining of your file add: `Sequel.split_symbols = true`. This will allow to properly handle the namespaces.\n\n#### Overriding Table and Content Type ID\n\nThe methods `::table_name=` and `::content_type_id=` allow you to override the IDs for either the table or content type.\nBy default, they are a `snake_cased` version of the class name.\n\n#### Defining the ID generator\n\nYou can define the ID generation strategy, there are 2 classes currently provided:\n\n- `Contentful::DatabaseImporter::IdGenerator::Base`: Provides a very basic template engine for generating IDs, this is the default strategy.\n- `Contentful::DatabaseImporter::IdGenerator::ContentfulLike`: Provides a Base62 encode that produces IDs similar to the Contentful provided ones,\n  uses the `Base` strategy, then pads it to a minimum length and then Base62 encode it.\n\n##### ID Templates\n\nTheres a minimal template engine provided for the ID Generators.\nA single template looks like `{{foo}}_{{bar}}` and works by replacing the values enclosed between `{{}}` with the corresponding value for each entry.\n\nThere are a few variables globally provided for every class (and will be looked up before the object fields):\n\n- `class_name`: The name of the mapping class\n- `table_name`: The defined table name (or the default)\n- `content_type_id`: The defined content type ID (or the default)\n- `index`: The position of the entry (0-based) on the database table\n\nAfter those globally provided values, you can use the value for any field on the mapping class (using the `:maps_to` value if present).\n\nFor example, using the example template above, if the DB record looks like:\n\n```ruby\n{foo: 'something', bar: 'else', image: 'https://example.com/happycat.jpg'}\n```\n\nThen the resulting template will be `something_else`\n\n**Note**: With relationships, it's useful to use a unique identifier value as part of the ID template.\n\n#### Defining Fields\n\nFor defining the field you have the `::field(name, options = {})` method. It defines how to retrieve and later serialize the field.\n\nThe options are:\n\n- `type: type_name`: **Required** for coercions. Types defined below.\n- `maps_to: name`: **Optional**. Defaults to field name, and defines the field name in Contentful\n- `pre_process: lambda_or_symbol`: **Optional**. Described below.\n- `exclude_from_output: boolean`: **Optional**. Defaults to false. Defines if the field will not be uploaded to Contentful. Useful for ID generation.\n\n##### Regular Field Types\n\n- `:symbol`, `:string`: Short text field (255 characters maximum) in Contentful.\n- `:text`: Long text field.\n- `:number`: Floating point precision number.\n- `:integer`: Integer number.\n- `:boolean`: Boolean.\n- `:location`: Geographical Location (can be coerced from a String, Hash or Array.)\n- `:date`: An ISO8601 Date (can be coerced from a Date/DateTime object or String).\n- `:object`: A JSON Object.\n- `:asset`: A File description. A String containing the file URL needs to be provided.\n- `:array`: An Array of elements.\n\nIn the case of using `:array`, an extra parameter `item_type: type` must be provided.\n\n##### Relationship Field Types\n\nIf your data has a relationship field, the `type:` value will be the related class, and will require additional parameters specifying\nthe relationship type and keys for retrieving the appropiate data.\n\nFor example:\n\n```ruby\nclass Foo\n  field :bars, type: Bar, relationship: :many, id_field: :id, key: :foo_id\n  field :baz, type: Baz, relationship: :one, id_field: :id, key: :baz_id\n  field :quxs, type: Qux, relationship: :through, through: :foo_qux, primary_id_field: :id, primary_key: :foo_id, foreign_key: :qux_id, foreign_id_field: :id\nend\n```\n\nIn Contentful, relationships are unidirectional, and if you want bidirectional relationships, you need to declare them in both classes.\n\nRelationship fields have the particularity that they don't require the `:maps_to` property, as Contentful will always use\nthe field name for the property in Contentful. You define the name of the field in the database with relationship specific parameters.\n\nRelationship Types:\n\n- `:many`: One to Many relationship, looks for all related objects of the associated class that match the value of the `:id_field` via the `:key`.\n  In the example above, it will look for all `Bar` entries which have a `:foo_id` that match the value of `:id` for the current `Foo` entry.\n- `:one`: One to One relationship, looks for the related object of the associated class that matches the value of the `:key` field in the current entry, with the value of `:id_field` in the related entry.\n  In the example above, it will look for the `Baz` entry which has an ID that matches the value of `:baz_id` in the current entry.\n- `:through`: Many to Many relationship, looks for the related object through an intermediate lookup table, after this it behaves like `:many`.\n  In the example above, it will look for all `Qux` entries found in the intermediate table that match the current entry `:id` and looks it up via the `Qux`s `:id`.\n\n**Note**: If you're using relationships, use a custom ID Generator template which includes a unique field for each entry,\nthat way, creating the links in Contentful will be successful. This requires including the field in the class definition.\n\nFor example: `'{{content_type_id}}_{{id}}'`\n\n**Note**: Ruby requires a class to be defined before using it as a parameter, therefore, you should declare all classes that\nare contained within others, before the one in which you use them. If you want to have circular relationships, you need to define a Merge Class pointing to the same table and content type as the desired class (defined below).\n\n##### Pre-processing\n\nIf you want to transform your data before uploading to Contentful,\nyou can use the `:pre_process` parameter in the `::field` definition.\n\nThe pre-process value can be a lambda function (E.g. `-\u003e (value) { value + 1 }`) or a symbol (E.g. `:pre_process_foo`).\n\nIf you use a lambda function, it must receive a single parameter and return a single value.\n\nIf you use a Symbol, it must match the name of a method defined within the class you're calling it from.\nThis method must receive a single parameter and return a single value.\n\n### Merging Tables\n\nYou might want to merge the content of multiple tables into a single content type.\n\nThis is supported by default, but ensure that the classes have the same `content_type_id` defined and if you need to\nmerge the entries as well, that the ID generator template is set in a way that can match the values from the different classes.\n\nIn the case you want to create multiple content types from a single table, the same concepts apply.\n\nIn the case of circular references, you will have to create 2 or more classes pointing to the same table and content type, the same concepts apply.\n\n**Note**: Merge classes require at least 1 field declared, even if it's excluded from output.\n\n### Querying\n\nYou might want to reduce your datasets to specific subsets, in that case, you can use Querying to specify your subsets of data.\n\nA query is an `SQL String`. E.g: `foo = 'bar' AND baz \u003e 2`.\n\nThis is optional and can be specified in the Resource like follows:\n\n```ruby\nclass MyResource\n  include Contentful::DatabaseImporter::Resource\n\n  self.query = \"foo = 'bar' AND baz \u003e 2\"\n\n  field :foo, type: :string\n  field :baz, type: :integer\nend\n```\n\nIf planning to upgrade to Sequel `v5`, use `self.query = Sequel.lit(\"foo = 'bar' AND baz \u003e 2\")`. Sequel is deprecating string literals and allowing only this new method.\n\n### Configuration\n\n```ruby\nContentful::DatabaseImporter.setup do |config|\n  config.space_name = 'My Cool New Space' # Required only for `::run!` - the destination space name\n  config.space_id = 'aAbBcC123foo' # Required only for `::update_space!` - the destination space ID\n  config.environment = 'master' # Optional (only for `::update_space!`) - defaults to `master`\n  config.database_connection = 'postgres://user:pass@host:port' # Required - the DB Connection string\n  config.skip_content_types = true # Optional (only for `::update_space!`) - defaults to `true` - Skips Content Type creation upon updating a space\n  config.locale = 'en-US' # Optional (only for `::update_space!` and `::run!`) - defaults to `'en-US'` - Defines the default locale for Space creation, and locale in which the content will be set for both creation and update\nend\n```\n\n`database_connection` allows the following Database URI Strings:\n\n- **[Postgres (Section 31.1.1.2)](https://www.postgresql.org/docs/9.3/static/libpq-connect.html#LIBPQ-CONNSTRING)**: E.g. `'postgres://user:password@host:port/database_name'`.\n- **SQlite**: E.g. `'sqlite://file_path.db'`.\n- **MySQL**: E.g. `'mysql://user:password@host:post/database_name'`.\n\n### Running the Import Tool\n\nYou can do any of the following operations:\n\n* Generate a JSON Template as a Ruby Hash for reuse within the script:\n\n```ruby\nContentful::DatabaseImporter.generate_json\n```\n\n* Generate JSON Template as a prettyfied JSON string:\n\n```ruby\nContentful::DatabaseImporter.generate_json!\n```\n\n* Generate the JSON and Import it to Contentful (creates a Space with all the content):\n\n```ruby\nContentful::DatabaseImporter.run!\n```\n\n* Generate the JSON and Import it to Contentful (updates a Space with all the content):\n\n```ruby\nContentful::DatabaseImporter.update_space!\n```\n\n## Contributing\n\nFeel free to improve this tool by submitting a Pull Request. For more information, please read [CONTRIBUTING.md](./CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontentful%2Fcontentful-database-importer.rb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcontentful%2Fcontentful-database-importer.rb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcontentful%2Fcontentful-database-importer.rb/lists"}