{"id":13530057,"url":"https://github.com/jexp/neo4j-shell-tools","last_synced_at":"2025-04-01T17:31:53.877Z","repository":{"id":9344726,"uuid":"11194460","full_name":"jexp/neo4j-shell-tools","owner":"jexp","description":"A bunch of import/export tools for the neo4j-shell","archived":true,"fork":false,"pushed_at":"2019-01-04T22:44:26.000Z","size":274,"stargazers_count":289,"open_issues_count":65,"forks_count":55,"subscribers_count":29,"default_branch":"3.0","last_synced_at":"2024-11-02T16:35:51.552Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jexp.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-05T07:54:49.000Z","updated_at":"2024-10-26T09:37:22.000Z","dependencies_parsed_at":"2022-09-14T21:40:13.628Z","dependency_job_id":null,"html_url":"https://github.com/jexp/neo4j-shell-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-shell-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-shell-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-shell-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-shell-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jexp","download_url":"https://codeload.github.com/jexp/neo4j-shell-tools/tar.gz/refs/heads/3.0","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246680356,"owners_count":20816686,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T07:00:42.778Z","updated_at":"2025-04-01T17:31:53.513Z","avatar_url":"https://github.com/jexp.png","language":"Java","funding_links":[],"categories":["Shell","REST API"],"sub_categories":["REST API","Other"],"readme":"# Import data into your neo4j database from the neo4j-shell command\n\n`neo4j-shell-tools` adds a number of commands to [neo4j-shell](http://neo4j.com/docs/stable/shell.html) which easily allow import and export data into running [Neo4j](http://www.neo4j.com/) database.\n\n### Installation\n\nDownload [neo4j-shell-tools_3.0.1.zip](http://dist.neo4j.org/jexp/shell/neo4j-shell-tools_3.0.1.zip) and extract it in your\nNeo4j Server's lib directory e.g.\n\n```\ncd /path/to/neo4j-community-3.0.1\ncurl http://dist.neo4j.org/jexp/shell/neo4j-shell-tools_3.0.1.zip -o neo4j-shell-tools.zip\nunzip neo4j-shell-tools.zip -d lib\n```\n\n#### Easier installation on Unix\n\nThe following script does the above installation automatically, and sets the `$NEO4J_HOME` path; works on Unix:\n\n```\nNEO4J_HOME=$(neo4j-shell -c 'dbinfo -g Kernel StoreDirectory' | grep -oE '\\/.*lib.*?\\/') \u0026\u0026 curl -Lk -o neo4j-shell-tools.zip http://dist.neo4j.org/jexp/shell/neo4j-shell-tools_3.0.1.zip \u0026\u0026 unzip neo4j-shell-tools.zip -d ${NEO4J_HOME}lib\n```\n\n### Before you start\n\nRestart neo4j and then launch the neo4j-shell:\n\n```\ncd /path/to/neo4j-community-3.0.1\n./bin/neo4j restart\n./bin/neo4j-shell\n```\n\nThat assumes a default neo4j instance running on port 7474. You can call `./bin/neo4j-shell --help` to get a list of other ways to connect to a neo4j instance.\n\n### Importing workflow\n\nBefore importing data, make sure that you create the necessary [indexes or constraints](http://neo4j.com/docs/stable/cypher-schema.html) for your data, so that you can quickly access the starting points for your graph patterns afterwards.\nFor instance:\n\n```\ncreate constraint on (p:Person) assert p.email is unique;\ncreate index on :Person(age);\n```\n\n\n### Import\n\nThen choose a suitable import command, depending on how your data is structured.\n* If your data is formatted as CSV and you want to use [cypher](http://neo4j.com/docs/stable/cypher-query-lang.html) statements for importing it, use the [Cypher Import](#cypher-import) command.\n* If your data is in [GraphML](http://graphml.graphdrawing.org/) format, use the [GraphML Import](#graphml-import) command.\n* If your data is in [Geoff](http://nigelsmall.com/geoff) format, use the [Geoff Import](#geoff-import) command.\n* If your data is in Binary format, use the [Binary Import](#binary-import) command.\n\n#### Cypher Import\n\nPopulate your database with [write clauses](http://neo4j.com/docs/stable/query-write.html) in the [cypher](http://neo4j.com/docs/stable/cypher-query-lang.html) query language.\n\n```\n$ import-cypher [-i in.csv] [-o out.csv] [-d ,] [-q] [-b 10000] create (n:#{label} {name: {name}, age: {age}}) return id(n) as id, n.name as name\n```\n\n- -i file.csv: tab or comma separated input data file (or URL), with header. Header names are used as param-names. The cypher  statement will be executed one per row\n- -o file.csv: tab or comma separated output data file, all cypher result rows will be written to file, column labels become headers\n- -q: input/output file with quotes\n- -d delim: delim used to separate files (e.g. `-d \" \", -d \\t -d ,` )\n- -b size: batch size for intermediate commits\n\nExample input file: [in.csv](examples/in.csv)\n\n```\nname\tage\nMichael\t38\nSelina\t15\nRana\t8\nSelma\t5\n```\n\nUsage:\n\n```\n$ import-cypher -d\"\\t\" -i in.csv -o out.csv create (n {name: {name}, age: {age}}) return id(n) as id, n.name as name\nQuery: create (n {name: {name}, age: {age}}) return id(n) as id, n.name as name; infile in.csv delim '\t' quoted false outfile out.csv batch-size 20000\nImport statement execution created 4 rows of output.\n```\n\nOutput file: out.csv\n\n```\nid\tname\n1\tMichael\n2\tSelina\n3\tRana\n4\tSelma\n```\n\nOptionally support types for column headers: just use `prop:type` as header in your csv, e.g. `name:string,age:int`\n\nSupported Types\n\n* `int`\n* `long`\n* `double`\n* `float`\n* `boolean`\n* `string`\n* `byte`\n* and arrays thereof with `\u003ctype\u003e_array`, e.g. `int_array`\n\n#### Geoff Import\n\nPopulate your database with [geoff](http://nigelsmall.com/geoff) - a declarative notation for representing graph data in a human-readable format.\n\n```\n$ import-geoff [-i in.geoff]\n```\n\n- -i in.geoff: newline separated geoff rule file (or URL)\n\nExample input file: in.geoff\n\n```\n(A) {\"name\": \"Alice\"}\n(B) {\"name\": \"Bob\"}\n(A)-[r:KNOWS]-\u003e(B)\n```\n\nUsage:\n\n```\n$ import-geoff -i in.geoff\nGeoff import of in.geoff created 3 entities.\n```\n\n#### Binary Import\n\nPopulate your database from a binary dump of a neo4j database.  Internally, Kyro is used to serialize the graph.\n\n```\n$ import-binary [-i in.bin] [-r REL_TYPE] [-b 20000] [-c]\n```\n\n- -i in.bin: binary file from previous database export\n- -r REL_TYPE default relationship-type for relationships without a label attribute\n- -b batch-size batch-commit size\n- -c uses a cache that spills to disk for very large imports\n\nUsage:\n\n```\n$ import-binary -b 10000 -i /tmp/export.bin -c\n```\n\n#### GraphML Import\n\nPopulate your database from [GraphML](http://graphml.graphdrawing.org/) files. GraphML is an XML file format used to describe graphs.\n\n```\n$ import-graphml [-i in.xml] [-r REL_TYPE] [-b 20000] [-c] [-t]\n```\n\n- -i in.xml: graphml file (or URL)\n- -r REL_TYPE default relationship-type for relationships without a label attribute\n- -t also import node labels, see the export format\n- -b batch-size batch-commit size\n- -c uses a cache that spills to disk for very large imports\n\nExample input file: in.xml\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cgraphml xmlns=\"http://graphml.graphdrawing.org/xmlns\"\n         xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n         xsi:schemaLocation=\"http://graphml.graphdrawing.org/xmlns\n        http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd\"\u003e\n    \u003ckey id=\"d0\" for=\"node\" attr.name=\"color\" attr.type=\"string\"\u003e\n        \u003cdefault\u003eyellow\u003c/default\u003e\n    \u003c/key\u003e\n    \u003ckey id=\"d1\" for=\"edge\" attr.name=\"weight\" attr.type=\"double\"/\u003e\n    \u003cgraph id=\"G\" edgedefault=\"undirected\"\u003e\n        \u003cnode id=\"n0\"\u003e\n            \u003cdata key=\"d0\"\u003egreen\u003c/data\u003e\n        \u003c/node\u003e\n        \u003cnode id=\"n1\"/\u003e\n        \u003cedge id=\"e0\" source=\"n0\" target=\"n1\"\u003e\n            \u003cdata key=\"d1\"\u003e1.0\u003c/data\u003e\n        \u003c/edge\u003e\n    \u003c/graph\u003e\n\u003c/graphml\u003e\n```\n\nUsage:\n\n```\n$ import-graphml -i in.xml\nGraphML-Import file in.xml rel-type RELATED_TO batch-size 40000 use disk-cache false\n0. 100%: nodes = 1 rels = 0 properties = 0 time 11 ms\nFinished: nodes = 2 rels = 1 properties = 2 total time 16 ms\nGraphML import created 3 entities.\n```\n\n### Export\n\nChoose a suitable export command, depending on your requirement for the exported data\n\n* To export your data as Cypher statements, use, use the [Cypher Export](#cypher-export) command.\n* To export your data as CSV, use the [Cypher Import](#cypher-import) command with the `-o file` option which will output the results of your queries into a CSV file.\n* To export your data as [GraphML](http://graphml.graphdrawing.org/), use the [GraphML Export](#graphml-export) command.\n* To export your data as a binary file, use the [Export Binary](#binary-export) command.\n\n#### Cypher Export\n\nIf you have a populated database, you can dump it completely or partially (by using a Cypher statement) to a script-file which uses Cypher and neo4j-shell commands to control the import.\n\nThe export order is:\n\n1. Nodes batched in transactions\n1. Schema information like indexes and constraints\n1. Relationships by looking up nodes by primary keys and connecting them, also batched in transactions\n\n```\n$ export-cypher [-o export.cypher] [-r] [-b 10000] [MATCH (p:Person)-[r:ACTED_IN]-\u003e(m:Movie) RETURN *]\n```\n\n- -o file.cypher: output file for dump, if left blank the output is sent to the neo4j-shell output and has to be redirected\n- -b size: batch size for commit batches for nodes and relationships\n- -r add all nodes of selected relationships, also add relationships between selected nodes\n\n**Note:**\nFor nodes that don't have a unique constraint, a temporary label (`UNIQUE IMPORT LABEL`) and property (`UNIQUE IMPORT ID`) are added to allow later lookup in the relationship-generation section.\nA constraint for `UNIQUE IMPORT LABEL`(`UNIQUE IMPORT ID`) is added too. All, the constraint, the label and the property are removed at the end of the import script again.\n\nExample dump file:\n\n```\n$ export-cypher -r -o test.cypher match (n)-[r]-\u003e() return n,r\n\n// create nodes\nbegin\nCREATE (:`UNIQUE IMPORT LABEL` {`UNIQUE IMPORT ID`:0});\nCREATE (:`User` {`age`:43, `name`:\"User1\"});\ncommit\n\n// add schema\nbegin\nCREATE INDEX ON :`User`(`age`);\nCREATE CONSTRAINT ON (node:`User`) ASSERT node.`name` IS UNIQUE;\nCREATE CONSTRAINT ON (node:`UNIQUE IMPORT LABEL`) ASSERT node.`UNIQUE IMPORT ID` IS UNIQUE;\ncommit\nschema await\n\n// create relationships\nbegin\nMATCH (n1:`UNIQUE IMPORT LABEL`{`UNIQUE IMPORT ID`:0}), (n2:`User`{`name`:\"User1\"}) CREATE (n1)-[:`KNOWS` {`since`:2011}]-\u003e(n2);\ncommit\n\n// clean up temporary import keys\nbegin\nMATCH (n:`UNIQUE IMPORT LABEL`)  WITH n LIMIT 1000 REMOVE n:`UNIQUE IMPORT LABEL` REMOVE n.`UNIQUE IMPORT ID`;\ncommit\nbegin\nDROP CONSTRAINT ON (node:`UNIQUE IMPORT LABEL`) ASSERT node.`UNIQUE IMPORT ID` IS UNIQUE;\ncommit\n```\n\n#### Binary Export\n\nExport your Neo4j graph database to a binary file.\n\n```\n$ export-binary -o out.bin\n```\n\n- -o out.bin\n\nUsage:\n\n```\n$ export-binary -o out.bin\n```\n\n#### GraphML Export\n\nExport your Neo4j graph database to [GraphML](http://graphml.graphdrawing.org/) files. GraphML is an XML file format used to describe graphs. Can be used to import and visualize your graph in [Gephi](http://gephi.org).\n\n```\n$ export-graphml [-t] [-r] [-o out.graphml] [match (n:Foo)-[r]-\u003e() return n,r]\n```\n\n- -t write types, do a first pass over the data to determine property-types and write them to the graphml header\n- -r add all nodes of selected relationships\n- -o out.graphml: graphml file to write to\n- optional cypher query to select a subgraph to export\n\nExample output file: out.graphml\n\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cgraphml xmlns=\"http://graphml.graphdrawing.org/xmlns\"\n xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n xsi:schemaLocation=\"http://graphml.graphdrawing.org/xmlns\n http://graphml.graphdrawing.org/xmlns/1.0/graphml.xsd\"\u003e\n\u003cgraph id=\"G\" edgedefault=\"directed\"\u003e\n\u003ckey id=\"name\" for=\"node\" attr.name=\"name\" attr.type=\"string\"/\u003e\n\u003ckey id=\"count\" for=\"edge\" attr.name=\"count\" attr.type=\"int\"/\u003e\n\u003cnode id=\"n0\" labels=\":FOO\" \u003e\u003cdata key=\"labels\"\u003e:FOO\u003c/data\u003e\u003cdata key=\"name\"\u003eJohn Doe\u003c/data\u003e\u003c/node\u003e\n\u003cnode id=\"n1\" labels=\":FOO\" \u003e\u003cdata key=\"labels\"\u003e:FOO\u003c/data\u003e\u003cdata key=\"name\"\u003eJane Doe\u003c/data\u003e\u003c/node\u003e\n\u003cedge id=\"e0\" source=\"n0\" target=\"n1\" label=\"KNOWS\"\u003e\u003cdata key=\"label\"\u003eKNOWS\u003c/data\u003e\u003cdata key=\"count\"\u003e0\u003c/data\u003e\u003c/edge\u003e\n\u003cedge id=\"e1\" source=\"n1\" target=\"n0\" label=\"KNOWS\"\u003e\u003cdata key=\"label\"\u003eKNOWS\u003c/data\u003e\u003cdata key=\"count\"\u003e1\u003c/data\u003e\u003c/edge\u003e\n\u003c/graph\u003e\n\u003c/graphml\u003e\n```\n\nUsage:\n\n```\n$ export-graphml -o out.graphml\n```\n\n### Prerequisites\n\nAn up and running neo4j database which you can [download from here](http://www.neo4j.org/download).\n\n### Other Technical Details\n\n#### Libraries used\n* Cypher Import uses [opencsv-2.3.jar](http://opencsv.sourceforge.net/) for parsing CSV files.\n* GraphML Import uses [mapdb-0.9.3.jar](http://www.mapdb.org/) as part of the cache (-c) flag for very large imports\n* Geoff Import uses [neo4j-geoff-1.7-SNAPSHOT.jar](http://nigelsmall.com/geoff)\n\n#### More on GraphML\n\nThe 'import-graphml' command supports attributes, supports only single pass parsing, optimization for `parse.nodeids=\"canonical\"`\n\nAn import of [@chrisdiehl](https://twitter.com/chrisdiehl)'s [Enron Dataset](http://www.infochimps.com/datasets/enron-email-data-with-manager-subordinate-relationship-metadata) took 5 minutes on a MBA:\n\n`Finished: nodes = 343266 rels = 1903201 properties = 8888993 total time 313491 ms`\n\n### Manual Build \u0026 Install\n\n```\ngit clone git@github.com:jexp/neo4j-shell-tools.git\ncd neo4j-shell-tools\nmvn clean package dependency:copy-dependencies\n```\n\nThen copy the jars that get generated into the neo4j lib directory:\n\n```\ncp target/import-tools-*.jar target/dependency/opencsv-*.jar target/dependency/geoff-*.jar target/dependency/mapdb-*.jar target/dependency/kryo-*.jar target/dependency/reflectasm-*.jar target/dependency/minlog-*.jar target/dependency/objenesis-*.jar /path/to/neo4j-community/lib\n```\n\nor make those two files available on your neo4j-shell classpath\n\n#### Setup auto indexing\n\n**NOTE: Using the \"old\" auto-index is discouraged, except if you know what you're doing. Use the label-based schema indexes instead.**\n\nThe auto index command is used to automatically create indexes on certain properties defined on nodes or relationships. This is in addition to the properties defined in 'conf/neo4j.properties'.\n\n```\nauto-index [-t Node|Relationship] [-r] name age title\n```\n\n- -r stops indexing those properties\n\nUsage:\n\n```\n$ auto-index name age title\nEnabling auto-indexing of Node properties: [name, age, title]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-shell-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjexp%2Fneo4j-shell-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-shell-tools/lists"}