https://github.com/jexp/batchperformance
multi threaded batch-importer
https://github.com/jexp/batchperformance
Last synced: over 1 year ago
JSON representation
multi threaded batch-importer
- Host: GitHub
- URL: https://github.com/jexp/batchperformance
- Owner: jexp
- Created: 2012-10-22T05:47:22.000Z (over 13 years ago)
- Default Branch: master
- Last Pushed: 2012-10-31T02:34:46.000Z (over 13 years ago)
- Last Synced: 2025-02-10T08:29:58.461Z (over 1 year ago)
- Language: Java
- Size: 219 KB
- Stars: 3
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
README
Parallel Batch inserter with Neo4j
Uses the [LMAX Disruptor](http://lmax-exchange.github.com/disruptor/) to parallelize operations during batch-insertion.
The 6 operations are:
1. property encoding
2. property-record creation
3. relationship-id creation and forward handling of reverse relationship chains
4. writing node-records
5. writing relationship-records
6. writing property-records
Dependencies:
(1)<--(2)<--(6)
(2)<--(5)-->(3)
(2)<--(4)-->(3)
It uses the above dependency setup of disruptor handlers to execute the different concerns in parallel. A ringbuffer of about 2^18 elements is used and a heap size of 5-20G, MMIO configuration within the heap limits.
Execution:
MAVEN_OPTS="-Xmx5G -Xms5G -server -d64 -XX:NewRatio=5" mvn exec:java -Dexec.mainClass=org.neo4j.batchimport.DisruptorTest
current limitations, constraints:
* only up to 2bn relationships (due to an int based multi-map)
* have to know max # of rels per node, properties per node and relationship
* relationships have to be pre-sorted by min(start,end)
future improvements:
* implement batch-importer CSV "API" on top of this
* stripe writes across store-files (i.e. strip the relationship-record file over 10 handlers, according to CPUs)
* parallelize writing to dynamic string and arraystore too
* change relationship-record updates for backwards pointers to run in a separate handler that is
RandomAccessFile-based (or nio2) and just writes the 2 int values directly at file-pos
* add a csv analyser / sorter that
* add support & parallelize index addition
* good support for index based lookup for relationship construction (kv-store, better in-memory structure, e.g. a collection of long[])
* use id-compression internally to save memory in structs (write a CompressedLongArray)
* reuse PropertyBlock, PropertyRecords, RelationshipRecords, NodeRecords, probably subclass them and override getId() etc. or copy the code
from the Store's to work with interfaces