{"id":15994093,"url":"https://github.com/jexp/neo4j-dataset-import","last_synced_at":"2025-04-05T00:15:29.122Z","repository":{"id":29880103,"uuid":"33425438","full_name":"jexp/neo4j-dataset-import","owner":"jexp","description":"Dataset Importer (.e.g Friendster)","archived":false,"fork":false,"pushed_at":"2016-02-13T22:16:52.000Z","size":17,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T08:30:39.633Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jexp.png","metadata":{"files":{"readme":"readme.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-05T00:41:38.000Z","updated_at":"2017-10-23T15:53:43.000Z","dependencies_parsed_at":"2022-07-24T16:32:08.848Z","dependency_job_id":null,"html_url":"https://github.com/jexp/neo4j-dataset-import","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-dataset-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-dataset-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-dataset-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-dataset-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jexp","download_url":"https://codeload.github.com/jexp/neo4j-dataset-import/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266567,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-08T07:05:54.966Z","updated_at":"2025-04-05T00:15:29.101Z","avatar_url":"https://github.com/jexp.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"== Parallel Importer for Line-Based Input Files\n\nImagine you have a bunch of line-based input files as csv.\n\nBut they are not the in the CSV format that the new fast http://neo4j.com/docs/stable/import-tool.html[Neo4j-Import tool] can work with.\n\nIt's your lucky day!\n\nI wanted to import the https://archive.org/details/friendster-dataset-201107[Friendster dataset] with 125M nodes and 2.5BN relationships.\n\nWhich should be pretty straigtforward but their format is rather an edge-list than the plain, denormalized CSV we usually work with.\n\n[source,csv]\n----\n1:private\n2:notdefined\n3:6,7,8,...\n4:23,5,5,66\n----\n\nSo I wrote this little tool to help with it. It uses the same APIs for inserting into Neo4j as the `neo4j-import` tool,\nbut offers some more flexibility for providing the data.\n\n* It works on multiple files (optionally compressed with bzip2 or gzip).\n* You provide two functions, to convert a line into a node or into multiple relationships.\n* Each file is read, each line is passed to both functions and the results imported.\n\nThe import code for Friendster looks like this and runs on my linux box in: 45 minutes (only because my file-reading code is so slow).\nIn that time it reads 11GB compressed csv data and creates a Neo4j datastore of 89GB (2GB nodes, 82GB relationships, 5G properties).\n\n[source,java]\n----\npublic class ImportFriendster {\n    public static final String[] LABELS = new String[]{\"Person\"};\n    public static final String REL_TYPE = \"FRIEND_OF\";\n    public static final Object[] NO_PROPS = new Object[0];\n\n\n    public static void main(String[] args) throws IOException {\n        File store = new File(args[0]);\n        FileUtils.deleteRecursively(store);\n\n        File[] files = getDataFiles(args[1]);\n\n        final MultiFileParallelImporter importFriendster = new MultiFileParallelImporter(store);\n\n        IdMapper idMapper = IdMappers.actual();//.longs(NumberArrayFactory.AUTO); or .strings(NumberArrayFactory.AUTO);\n\n        MultiFileParallelImporter.Generator generator = new MultiFileParallelImporter.Generator() {\n\n            public InputNode createNode(String line, long position, String fileName, int lineNo) {\n                int idx = line.indexOf(':');\n                long nodeId = parseLong(line.substring(0, idx));\n                return new InputNode(fileName, lineNo, position, nodeId, new Object[]{\"id\", nodeId}, null, LABELS, null);\n            }\n\n            public Iterable\u003cInputRelationship\u003e createRelationships(String line, long position, String fileName, int lineNo) {\n                int idx = line.indexOf(':');\n                long nodeId = parseLong(line.substring(0, idx));\n                String[] ids = line.substring(idx + 1).split(\",\");\n                List\u003cInputRelationship\u003e rels =new ArrayList\u003c\u003e(ids.length);\n                for (String id : ids) {\n                    if (id.equals(\"notfound\") || id.equals(\"private\") || id.trim().isEmpty()) continue;\n                    rels.add(new InputRelationship(fileName, lineNo, position, NO_PROPS, null, nodeId, parseLong(id), REL_TYPE, null));\n                }\n                return rels;\n            }\n        };\n        importFriendster.run(files, idMapper, 0, generator);\n    }\n\n    // .... getDataFiles left off ...\n}\n----\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-dataset-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjexp%2Fneo4j-dataset-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-dataset-import/lists"}