{"id":15994171,"url":"https://github.com/jexp/neo4j-large-datagen","last_synced_at":"2025-04-05T00:15:36.149Z","repository":{"id":66743973,"uuid":"159799149","full_name":"jexp/neo4j-large-datagen","owner":"jexp","description":"CSV Data Generator for a simple money transfer model","archived":false,"fork":false,"pushed_at":"2018-11-30T11:12:19.000Z","size":6,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-10T08:30:59.493Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jexp.png","metadata":{"files":{"readme":"readme.adoc","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-30T09:22:24.000Z","updated_at":"2018-12-05T09:53:43.000Z","dependencies_parsed_at":"2023-06-29T13:13:12.652Z","dependency_job_id":null,"html_url":"https://github.com/jexp/neo4j-large-datagen","commit_stats":{"total_commits":3,"total_committers":1,"mean_commits":3.0,"dds":0.0,"last_synced_commit":"c40868c69decabe8d1e98595c4939a6830dbe264"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-large-datagen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-large-datagen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-large-datagen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jexp%2Fneo4j-large-datagen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jexp","download_url":"https://codeload.github.com/jexp/neo4j-large-datagen/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266570,"owners_count":20910837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-08T07:06:47.210Z","updated_at":"2025-04-05T00:15:36.130Z","avatar_url":"https://github.com/jexp.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Data Generator for Large Neo4j Stores\n\nThe domain model is a Money transfer graph `+(:Account)-[:TRANSFER]-\u003e(:Account)+` or  `+(:Account)\u003c-[:FROM]-(:TRANSFER)-[:TO]-\u003e(:Account)+`\n\nThere are two generators one for accounts, one for transfers.\nYou provide them with a total number and a number of files it should generate which also determines concurrency.\nE.g. 100 files for 1bn accounts writes 10M accounts per file.\n\n\n== Usage\n\n----\nexport NEO=/path/to/neo4j-enterprise-3.5.0\nexport JAVA_HOME=/usr/lib/jvm/java-8-oracle\nexport DB=/path/to/transfers/graph.db\n----\n\n----\n$JAVA_HOME/bin/javac AccountFileGenerator.java\n# $JAVA_HOME/bin/java AccountFileGenerator \u003cfiles\u003e \u003ctotal\u003e\n# eg.\n$JAVA_HOME/bin/java AccountFileGenerator 100 1000000000\n----\n\n----\n$JAVA_HOME/bin/javac TransactionFileGenerator.java\n# $JAVA_HOME/bin/java TransactionFileGenerator \u003cbatch\u003e \u003cfiles\u003e \n# eg. total = batch * files\n$JAVA_HOME/bin/java TransactionFileGenerator 100000000 300\n----\n\nThe resulting CSV files are all gzipped, which the import tool can handle without any issues.\n\nOn a machine with 144 CPUs I can write a\n\n== Import\n\nTo use the generated CSV files with neo4j-import you have to create appropriate header files.\n\nDepending on which data model you want to use you can either treat transaction as Node or Relationship.\nIf you're only interested in graph structure (e.g. for graph algorithm testing), you can _ignore_ the properties like `id`, `time` or `value`\n\n=== Transaction as Relationship\n\n\n=== Only Transaction Nodes for Indexing Testing\n\n----\necho 'id:ID,:IGNORE,:IGNORE,value:INT,time:LONG' \u003e transaction.hdr\n----\n\n----\n# import of 1bn tx only\n$NEO/bin/neo4j-import --into $DB --stacktrace true --skip-duplicate-nodes true   --skip-bad-relationships true  --ignore-empty-strings true --bad-tolerance true \\\n--id-type string --db-config neo4j.conf --ignore-extra-columns true --skip-bad-entries-logging true \\\n--nodes:Transaction transaction.hdr,'data/1-payment-1.*\\.csv\\.gz'\n----\n\n=== Transaction as Node\n\n---\necho 'id:ID' \u003e account.hdr\necho 'id:ID,:IGNORE,:IGNORE,value:INT,time:LONG' \u003e transaction.hdr\necho ':START_ID,:END_ID,:IGNORE,:IGNORE,:IGNORE' \u003e sender.hdr\necho ':START_ID,:IGNORE,:END_ID,:IGNORE,:IGNORE' \u003e receiver.hdr\n# for stores \u003e 38bn nodes\necho 'dbms.record_format=high_limit' \u003e neo4j.conf\n----\n\n\n----\nrm -rf $DB\n$NEO/bin/neo4j-import --into $DB --stacktrace true --skip-duplicate-nodes true   --skip-bad-relationships true  --ignore-empty-strings true --bad-tolerance true \\\n--id-type string --db-config neo4j.conf --ignore-extra-columns true --skip-bad-entries-logging true \\\n--nodes:Transaction transaction.hdr,'data/payment-.*\\.csv\\.gz' --nodes:Account account.hdr,'data/account-.*\\.csv\\.gz' \\\n--relationships:SENDER sender.hdr,'data/payment-.*\\.csv\\.gz' --relationships:RECEIVER receiver.hdr,'data/payment-.*\\.csv\\.gz' --max-memory 100%\n----\n\n== Modifications \u0026 Implementation\n\nThe code uses one thread per file, and a fixed-length byte-array for the data, which is modified in-place for each line to write.\n\nCurrently the Range of id's and values to generate is hardcoded in the Java files for optimal performance.\n\nIt uses (depending) a random of range of numbers to write per digit. For all non-leading digits the range is `0..9`.\nFor the leading digits, it's depending on the total scale, e.g. 0..1 for `up \u003c 2xxxx` (e.g. 00000 to 19999) or 0..5 for `\u003c 6xxxx`.\n\nThe increment of `k` moves over the byte array. So if you need different id-ranges e.g. 2.3bn you have to adjust the first 2 leading digits to 0..1 and 0..2 respectively.\n\n.Example from TransactionFileGenerator\n[source,java]\n----\n// sender account id \u003c= 219999999999\nbytes[k++]=numbers[random.nextInt(3)];\nbytes[k++]=numbers[random.nextInt(10)];\nwhile (k!=23) bytes[k++]=numbers[random.nextInt(10)];\n----\n\n== Accessing the store\n\nIf you're on a machine where you don't want to start up a Neo4j server you can use this:\n\n----\nJAVA_OPTS=\"-Xmn4G -Xmx450G -Xms450G -Xss1M -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure -XX:+PrintTenuringDistribution -Xloggc:$DB/logs/gc.log\" $NEO/bin/neo4j-shell -config /mnt/ssd/algo/neo4j.conf -path $DB\n----\n\nIn Neo4j 3.4.x you have to symlink the `plugins` (e.g. for APOC) directory into the db directory, in 3.5.x on level up.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-large-datagen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjexp%2Fneo4j-large-datagen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjexp%2Fneo4j-large-datagen/lists"}