{"id":13425426,"url":"https://github.com/typedb-osi/typedb-loader","last_synced_at":"2026-01-16T04:05:48.186Z","repository":{"id":40403084,"uuid":"299246085","full_name":"typedb-osi/typedb-loader","owner":"typedb-osi","description":"TypeDB Loader - Data Migration Tool for TypeDB","archived":false,"fork":false,"pushed_at":"2024-03-15T17:33:07.000Z","size":1441,"stargazers_count":60,"open_issues_count":13,"forks_count":17,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-28T03:40:17.067Z","etag":null,"topics":["data-migration-tool","data-migrator","grami","knowledge-base","knowledge-graph","loader","migration","typedb","typedbloader"],"latest_commit_sha":null,"homepage":"https://github.com/typedb-osi/typedb-loader","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/typedb-osi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-09-28T08:46:19.000Z","updated_at":"2024-10-12T20:34:27.000Z","dependencies_parsed_at":"2023-01-19T21:07:18.683Z","dependency_job_id":"ad2bd4d9-133d-41b4-bbca-999b8f8a279e","html_url":"https://github.com/typedb-osi/typedb-loader","commit_stats":null,"previous_names":["bayer-science-for-a-better-life/grami"],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typedb-osi%2Ftypedb-loader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typedb-osi%2Ftypedb-loader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typedb-osi%2Ftypedb-loader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/typedb-osi%2Ftypedb-loader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/typedb-osi","download_url":"https://codeload.github.com/typedb-osi/typedb-loader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243782206,"owners_count":20347222,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-migration-tool","data-migrator","grami","knowledge-base","knowledge-graph","loader","migration","typedb","typedbloader"],"created_at":"2024-07-31T00:01:12.188Z","updated_at":"2026-01-16T04:05:48.179Z","avatar_url":"https://github.com/typedb-osi.png","language":"Java","funding_links":[],"categories":["Open source projects using TypeDB"],"sub_categories":[],"readme":"![TypeDBLoader_icon](https://github.com/bayer-science-for-a-better-life/grami/blob/master/typedbloader.png?raw=true)\n---\n---\n\n###      \n\n[![TypeDB Loader Test](https://github.com/bayer-science-for-a-better-life/grami/actions/workflows/testandbuild.yaml/badge.svg)](https://github.com/bayer-science-for-a-better-life/grami/actions/workflows/testandbuild.yaml)\n[![TypeDB Loader Build](https://github.com/bayer-science-for-a-better-life/grami/actions/workflows/release.yaml/badge.svg)](https://github.com/bayer-science-for-a-better-life/grami/actions/workflows/release.yaml)\n\n###\n\n---\n\nIf your [TypeDB](https://github.com/vaticle/typedb) project\n\n- has a lot of data\n- and you want/need to focus on schema design, inference, and querying\n\nUse TypeDB Loader to take care of your data migration for you. TypeDB Loader streams data from files and migrates them\ninto TypeDB **at scale**!\n\n## Features:\n\n- Data Input:\n    - data is streamed to reduce memory requirements\n    - supports any tabular data file with your separator of choice (i.e.: csv, tsv, whatever-sv...)\n    - supports gzipped files\n    - ignores unnecessary columns\n- [Attribute](https://github.com/typedb-osi/typedb-loader/wiki/02-Loading-Attributes), [Entity](https://github.com/typedb-osi/typedb-loader/wiki/03-Loading-Entities), [Relation](https://github.com/typedb-osi/typedb-loader/wiki/04-Loading-Relations)\n  Loading:\n    - load required/optional attributes of any TypeDB type (string, boolean, long, double, datetime)\n    - load required/optional role players (attribute / entity / relation)\n    - load list-like attribute columns as n attributes (recommended procedure until attribute lists are fully supported\n      by TypeDB)\n    - load list-like player columns as n players for a relation\n    - load entity if not present - if present, either do not write or append attributes\n- [Appending Attributes](https://github.com/typedb-osi/typedb-loader/wiki/05-Appending-Attributes) to existing things\n- [Append-Attribute-Or-Insert-Entity](https://github.com/typedb-osi/typedb-loader/wiki/06-Append-Or-Insert) for entities\n- Data Validation:\n    - validate input data rows and log issues for easy diagnosis input data-related issues (i.e. missing\n      attributes/players, invalid characters...)\n- Configuration Validation:\n    - write your configuration with confidence: warnings will display useful information for fine tuning, errors will\n      let you know what you forgot. All BEFORE the database is touched.\n- Performance:\n    - parallelized asynchronous writes to TypeDB to make the most of your hardware configuration, optimized with\n      engineers @vaticle\n- Stop/Restart (in re-implementation, currently NOT available):\n    - tracking of your migration status to stop/restart, or restart after failure\n\n- [Basic Column Preprocessing using RegEx's](https://github.com/typedb-osi/typedb-loader/wiki/08-Preprocessing)\n\nCreate a Loading\nConfiguration ([example](https://github.com/typedb-osi/typedb-loader/blob/master/src/test/resources/phoneCalls/config.json))\nand use TypeDB Loader\n\n- as an [executable CLI](https://github.com/typedb-osi/typedb-loader/wiki/10-TypeDB-Loader-as-Executable-CLI) - no\n  coding\n- in [your own Java project](https://github.com/typedb-osi/typedb-loader/wiki/09-TypeDB-Loader-as-Dependency) - easy API\n\n## How it works:\n\nTo illustrate how to use TypeDB Loader, we will use a slightly extended version of the \"phone-calls\"\nexample [dataset](https://github.com/typedb-osi/typedb-loader/tree/master/src/test/resources/phoneCalls)\nand [schema](https://github.com/typedb-osi/typedb-loader/blob/master/src/test/resources/phoneCalls/schema.gql) from the\nTypeDB developer documentation:\n\n### Configuration\n\nThe configuration file tells TypeDB Loader what things you want to insert for each of your data files and how to do it.\n\nHere are some example:\n\n- [Attribute Examples](https://github.com/typedb-osi/typedb-loader/wiki/02-Loading-Attributes)\n- [Entity Examples](https://github.com/typedb-osi/typedb-loader/wiki/03-Loading-Entities)\n- [Relation Examples](https://github.com/typedb-osi/typedb-loader/wiki/04-Loading-Relations)\n- [Nested Relation - Match by Attribute(s) Example](https://github.com/typedb-osi/typedb-loader/wiki/04-Loading-Relations#loading-relations-with-entityrelation-players-matched-on-attribute-ownerships-incl-nested-relations)\n- [Nested Relation - Match by Player(s) Example](https://github.com/typedb-osi/typedb-loader/wiki/04-Loading-Relations#loading-relations-relation-players-matching-on-players-in-playing-relation-incl-nested-relations)\n- [Attribute-Player Relation Example](https://github.com/typedb-osi/typedb-loader/wiki/04-Loading-Relations#loading-relations-with-attribute-players)\n- [Custom Migration Order Example](https://github.com/typedb-osi/typedb-loader/wiki/07-Custom-Load-Order)\n\nFor detailed documentation, please refer to the [WIKI](https://github.com/bayer-science-for-a-better-life/grami/wiki).\n\nThe [config](https://github.com/typedb-osi/typedb-loader/tree/master/src/test/resources/phoneCalls/config.json) in the\nphone-calls test is a good starting example of a configuration.\n\n### Migrate Data\n\nOnce your configuration files are complete, you can use TypeDB Loader in one of two ways:\n\n1. As an executable command line interface - no coding required:\n\n```Shell\n./bin/typedbloader load \\\n                -tdb localhost:1729 \\\n                -c /path/to/your/config.json \\\n                -db databaseName \\\n                -cm\n```\n\n[See details here](https://github.com/typedb-osi/typedb-loader/wiki/10-TypeDB-Loader-as-Executable-CLI)\n\n2. As a dependency in your own Java code:\n\n```Java\n\nimport com.vaticle.typedb.osi.loader.cli.LoadOptions;\nimport com.vaticle.typedb.osi.loader.loader.TypeDBLoader;\n\npublic class LoadingData {\n\n    public void loadData() {\n        String uri = \"localhost:1729\";\n        String config = \"path/to/your/config.json\";\n        String database = \"databaseName\";\n\n        String[] args = {\n                \"load\",\n                \"-tdb\", uri,\n                \"-c\", config,\n                \"-db\", database,\n                \"-cm\"\n        };\n\n        LoadOptions options = LoadOptions.parse(args);\n        TypeDBLoader loader = new TypeDBLoader(options);\n        loader.load();\n    }\n}\n```\n\n[See details here](https://github.com/typedb-osi/typedb-loader/wiki/09-TypeDB-Loader-as-Dependency)\n\n## Step-by-Step Tutorial\n\nA complete tutorial for TypeDB version \u003e= 2.5.0 is in work and will be published.\n\nAn example of configuration and usage of TypeDB Loader on real data can be\nfound [in the TypeDB Examples](https://github.com/vaticle/typedb-examples/tree/master/biology/catalogue_of_life).\n\nA complete tutorial for TypeDB (Grakn) version \u003c 2.0 can be\nfound [on Medium](https://medium.com/@hkuich/introducing-grami-a-data-migration-tool-for-grakn-d4051582f867).\n\nThere is an [example repository](https://github.com/bayer-science-for-a-better-life/grami-example) for your convenience.\n\n## Connecting to TypeDB Cluster\n\nTo connect to TypeDB Cluster, a set of options is provided:\n\n```\n--typedb-cluster=\u003caddress:port\u003e\n--username=\u003cusername\u003e\n--password // can be asked for interactively\n--tls-enabled\n--tls-root-ca=\u003cpath/to/CA/cert\u003e\n```\n\n## Compatibility Table\n\nRanges are [inclusive, inclusive].\n\n| TypeDB Loader | TypeDB Driver (internal) |     TypeDB      |  TypeDB Cloud   |\n|:-------------:|:------------------------:|:---------------:|:---------------:|\n|     1.9.x     |          2.26.6          |    2.25.x -     |    2.25.x -     |\n|     1.8.0     |          2.25.6          |    2.25.x -     |    2.25.x -     |\n|     1.7.0     |          2.18.1          |  2.18.x 2.23.x  |  2.18.x 2.23.x  |\n|     1.6.0     |          2.14.2          | 2.14.x - 2.17.x | 2.14.x - 2.16.x |\n| 1.2.0 - 1.5.x |      2.8.0 - 2.14.0      | 2.8.0 - 2.14.0  |       N/A       |\n| 1.1.0 - 1.1.x |          2.8.0           |      2.8.x      |       N/A       |\n|     1.0.0     |      2.5.0 - 2.7.1       |  2.5.x - 2.7.x  |       N/A       |\n|     0.1.1     |      2.0.0 - 2.4.x       |  2.0.x - 2.4.x  |       N/A       |\n|     \u003c0.1      |          1.8.0           |      1.8.x      |       N/A       |\n\n* [Type DB](https://github.com/vaticle/typedb)\n\nFind the Readme for GraMi for grakn \u003c\n2.0 [here](https://github.com/bayer-science-for-a-better-life/grami/blob/b3d6d272c409d6c40254354027b49f90b255e1c3/README.md)\n\n### Package hosting\nPackage repository hosting is graciously provided by  [Cloudsmith](https://cloudsmith.com).\nCloudsmith is the only fully hosted, cloud-native, universal package management solution, that\nenables your organization to create, store and share packages in any format, to any place, with total\nconfidence.\n\n## Contributions\n\nTypeDB Loader was built @[Bayer AG](https://www.bayer.com/) in the Semantic and Knowledge Graph Technology Group with\nthe support of the engineers @[Vaticle](https://github.com/vaticle).\n\n## Licensing\n\nThis repository includes software developed at [Bayer AG](https://www.bayer.com/). It is released under\nthe [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n\n## Credits\n\nIcon in banner by [Freepik](https://www.freepik.com\") from [Flaticon](https://www.flaticon.com/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftypedb-osi%2Ftypedb-loader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftypedb-osi%2Ftypedb-loader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftypedb-osi%2Ftypedb-loader/lists"}