{"id":13989458,"url":"https://github.com/linkedin/PalDB","last_synced_at":"2025-07-22T10:32:43.068Z","repository":{"id":34334534,"uuid":"38254684","full_name":"linkedin/PalDB","owner":"linkedin","description":"An embeddable write-once key-value store written in Java","archived":false,"fork":false,"pushed_at":"2019-12-02T23:41:27.000Z","size":365,"stargazers_count":935,"open_issues_count":12,"forks_count":187,"subscribers_count":98,"default_branch":"master","last_synced_at":"2024-08-09T13:15:58.158Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linkedin.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-29T15:20:29.000Z","updated_at":"2024-08-09T11:59:44.000Z","dependencies_parsed_at":"2022-07-11T09:17:28.541Z","dependency_job_id":null,"html_url":"https://github.com/linkedin/PalDB","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FPalDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FPalDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FPalDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linkedin%2FPalDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linkedin","download_url":"https://codeload.github.com/linkedin/PalDB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227079415,"owners_count":17728003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T13:01:40.055Z","updated_at":"2024-11-29T08:31:32.026Z","avatar_url":"https://github.com/linkedin.png","language":"Java","funding_links":[],"categories":["Java","数据库"],"sub_categories":[],"readme":"PalDB\n==========\n\n[![Build Status](https://travis-ci.org/linkedin/PalDB.svg?branch=master)](https://travis-ci.org/linkedin/PalDB)\n[![Coverage Status](https://coveralls.io/repos/linkedin/PalDB/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/linkedin/PalDB?branch=master)\n\nPalDB is an embeddable write-once key-value store written in Java.\n\nWhat is PalDB?\n-------------------\n\nPalDB is an embeddable persistent key-value store with very fast read performance and compact store size. PalDB stores are single binary files written once and ready to be used in applications.\n\nPalDB's JAR is only 110K and has a single dependency (snappy, which isn't mandatory). It's also very easy to use with just a few configuration parameters.\n\nPerformance\n-----------\n\nBecause PalDB is read-only and only focuses on data which can be held in memory it is significantly less complex than other embeddable key-value stores and therefore allows a compact storage format and very high throughput. PalDB is specifically optimized for fast read performance and compact store sizes. Performances can be compared to in-memory data structures such as Java collections (e.g. HashMap, HashSet) or other key-values stores (e.g. LevelDB, RocksDB).\n\nCurrent benchmark on a 3.1Ghz Macbook Pro with 10M integer keys index shows an average performance of ~1.6M reads/s for a memory usage 6X less than using a traditional HashSet. That is 5X faster throughput compared to LevelDB (1.8) or RocksDB (4.0).\n\nResults of a throughput benchmark between PalDB, LevelDB and RocksDB (higher is better):\n\n![throughput](http://linkedin.github.io/PalDB/doc/throughput.png)\n\nMemory usage benchmark between PalDB and a Java HashSet (lower is better):\n\n![memory](http://linkedin.github.com/PalDB/doc/memory.png)\n\nWhat is it suitable for?\n------------------------\n\nSide data can be defined as the extra read-only data needed by a process to do its job. For instance, a list of stopwords used by a natural language processing algorithm is side data. Machine learning models used in machine translation, content classification or spam detection are also side data. When this side data becomes large it can rapidly be a bottleneck for applications depending on them. PalDB aims to fill this gap.\n\nPalDB can replace the usage of in-memory data structures to store this side data with comparable query performances and by using an order of magnitude less memory. It also greatly simplifies the code needed to operate this side data as PalDB stores are single binary files, manipulated with a very simple API (see below for examples).\n\nCode samples\n------------\n\nAPI documentation can be found [here](http://linkedin.github.com/PalDB/doc/javadoc/index.html).\n\nHow to write a store\n```java\nStoreWriter writer = PalDB.createWriter(new File(\"store.paldb\"));\nwriter.put(\"foo\", \"bar\");\nwriter.put(1213, new int[] {1, 2, 3});\nwriter.close();\n```\n\nHow to read a store\n```java\nStoreReader reader = PalDB.createReader(new File(\"store.paldb\"));\nString val1 = reader.get(\"foo\");\nint[] val2 = reader.get(1213);\nreader.close();\n```\n\nHow to iterate on a store\n```java\nStoreReader reader = PalDB.createReader(new File(\"store.paldb\"));\nIterable\u003cMap.Entry\u003cString, String\u003e\u003e iterable = reader.iterable();\nfor (Map.Entry\u003cString, String\u003e entry : iterable) {\n  String key = entry.getKey();\n  String value = entry.getValue();\n}\nreader.close();\n```\n\nFor Scala examples, see [here](https://gist.github.com/mbastian/9b9b49a4b96333da33ec) and [here](https://gist.github.com/mbastian/440a706f5e863bb65622).\n\nUse it\n------\n\nPalDB is available on Maven Central, hence just add the following dependency:\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.linkedin.paldb\u003c/groupId\u003e\n    \u003cartifactId\u003epaldb\u003c/artifactId\u003e\n    \u003cversion\u003e1.2.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\nScala SBT\n```\nlibraryDependencies += \"com.linkedin.paldb\" % \"paldb\" % \"1.2.0\"\n```\n\n\nFrequently asked questions\n--------------------------\n\n**Can you open a store for writing subsequent times?**\n\nNo, the final binary file is created when `StoreWriter.close()` is called.\n\n**Are duplicate keys allowed?**\n\nNo, duplicate keys aren't allowed and an exception will be thrown.\n\n**Do keys have an order when iterating?**\n\nNo, like a hashtable PalDB stores have no order.\n\nBuild\n-----\n\nPalDB requires Java 6+ and gradle. The target Java version is 6.\n\n```bash\ngradle build\n```\n\nPerformance tests are run separately from the build\n```bash\ngradle perfTest\n```\n\nTest\n----\n\nWe use the TestNG framework for our unit tests. You can run them via the `gradle clean test` command.\n\nCoverage\n--------\n\nCoverage is run using JaCoCo. You can run a report via `gradle jacocoTestReport`. The report will be generated in `paldb/build/reports/jacoco/test/html/`.\n\nAdvanced configuration\n----------------------\n\nWrite parameters:\n\n+ `load.factor`,  index load factor (double) [default: 0.75]\n+ `compression.enabled`, enable compression (boolean) [default: false]\n\nRead parameters:\n\n+ `mmap.data.enabled`, enable memory mapping for data (boolean) [default: true]\n+ `mmap.segment.size`, memory map segment size (bytes) [default: 1GB]\n+ `cache.enabled`, LRU cache enabled (boolean) [default: false]\n+ `cache.bytes`, cache limit (bytes) [default: Xmx - 100MB]\n+ `cache.initial.capacity`, cache initial capacity (int) [default: 1000]\n+ `cache.load.factor`, cache load factor (double) [default: 0.75]\n\nConfiguration values are passed at init time. Example:\n```java\nConfiguration config = PalDB.newConfiguration();\nconfig.set(Configuration.CACHE_ENABLED, \"true\");\nStoreReader reader = PalDB.createReader(new File(\"store.paldb\"), config);\n```\n\nA few tips on how configuration can affect performance:\n\n+ Disabling memory mapping will significantly reduce performance as disk seeks will be performed instead.\n+ Enabling the cache makes sense when the value size is large and there's a significant cost in deserialization. Otherwise, the cache adds an overhead. The cache is also useful when memory mapping is disabled.\n+ Compression can be enabled when the store size is a concern and the values are large (e.g. a sparse matrix). By default, PalDB already uses a compact serialization. Snappy is used for compression.\n\nCustom serializer\n-----------------\n\nPalDB is primarily optimized for Java primitives and arrays but supports adding custom serializers so arbitrary Java classes can be supported.\n\nSerializers can be defined by implementing the `Serializer` interface and its methods. Here's an example which supports the `java.awt.Point` class:\n\n```java\npublic class PointSerializer implements Serializer\u003cPoint\u003e {\n\n  @Override\n  public Point read(DataInput input) {\n    return new Point(input.readInt(), input.readInt());\n  }\n\n  @Override\n  public void write(DataOutput output, Point point) {\n    output.writeInt(point.x);\n    output.writeInt(point.y);\n  }\n\n  @Override\n  public int getWeight(Point instance) {\n    return 8;\n  }\n}\n```\n\nThe `write` method serializes the instance to the `DataOutput`. The `read` method deserializes from `DataInput` and creates new object instances. The `getWeight` method returns the estimated memory used by an instance in bytes. The latter is used by the cache to evaluate the amount of memory it's currently using.\n\nSerializer implementation should be registered using the `Configuration`:\n\n```java\nConfiguration configuration = PalDB.newConfiguration();\nconfiguration.registerSerializer(new PointSerializer());\n```\n\nUse cases\n---------\n\nAt LinkedIn, PalDB is used in analytics workflows and machine-learning applications.\n\nIts usage is especially popular in Hadoop workflows because memory is rare yet critical to speed things up. In this context, PalDB often enables map-side operations (e.g. join) which wouldn't be possible with classic in-memory data structures (e.g Java collections). For instance, a set of 35M member ids would only use ~290M of memory with PalDB versus ~1.8GB with a traditional Java HashSet. Moreover, as PalDB's store files are single binary files it is easy to package and use with Hadoop's distributed cache mechanism.\n\nMachine-learning applications often have complex binary model files created in the `training` phase and used in the `scoring` phase. These two phases always happen at different times and often in different environments. For instance, the training phase happens on Hadoop or Spark and the scoring phase in a real-time service. PalDB makes this process easier and more efficient by reducing the need of large CSV files loaded in memory.\n\nLimitations\n-----------\n+ PalDB is optimal in replacing the usage of large in-memory data storage but still use memory (off-heap, yet much less) to do its job. Disabling memory mapping and relying on seeks is possible but is not what PalDB has been optimized for.\n+ The size of the index is limited to 2GB. There's no limitation in the data size however.\n+ PalDB is not thread-safe at the moment so synchronization should be done externally if multi-threaded.\n\nContributions\n-----------\n\nAny helpful feedback is more than welcome. This includes feature requests, bug reports, pull requests, constructive feedback, etc.\n\nCopyright \u0026 License\n-------------------\n\nPalDB © 2015 LinkedIn Corp. Licensed under the terms of the Apache License, Version 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2FPalDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinkedin%2FPalDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinkedin%2FPalDB/lists"}