{"id":26121536,"url":"https://github.com/fizzed/bigmap","last_synced_at":"2025-04-13T12:51:14.182Z","repository":{"id":42462480,"uuid":"203805579","full_name":"fizzed/bigmap","owner":"fizzed","description":"Drop-in replacement for JVM maps when you need to optimize memory or work with large datasets","archived":false,"fork":false,"pushed_at":"2025-01-22T17:09:35.000Z","size":2994,"stargazers_count":15,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T03:51:21.393Z","etag":null,"topics":["java"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fizzed.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-22T13:53:47.000Z","updated_at":"2025-01-22T17:09:38.000Z","dependencies_parsed_at":"2025-01-09T15:28:40.885Z","dependency_job_id":"946b2047-c2f7-4fc2-a4c5-f7a4d516ee57","html_url":"https://github.com/fizzed/bigmap","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fizzed%2Fbigmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fizzed%2Fbigmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fizzed%2Fbigmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fizzed%2Fbigmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fizzed","download_url":"https://codeload.github.com/fizzed/bigmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248717252,"owners_count":21150388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java"],"created_at":"2025-03-10T14:23:41.621Z","updated_at":"2025-04-13T12:51:14.153Z","avatar_url":"https://github.com/fizzed.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BigMap by Fizzed\n\n[![Maven Central](https://img.shields.io/maven-central/v/com.fizzed/bigmap?color=blue\u0026style=flat-square)](https://mvnrepository.com/artifact/com.fizzed/bigmap)\n\n[![Java 8](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/java8.yaml?branch=master\u0026label=Java%208\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/java8.yaml)\n[![Java 11](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/java11.yaml?branch=master\u0026label=Java%2011\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/java11.yaml)\n[![Java 17](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/java17.yaml?branch=master\u0026label=Java%2017\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/java17.yaml)\n[![Java 21](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/java21.yaml?branch=master\u0026label=Java%2021\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/java21.yaml)\n\n[![Linux x64](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/java11.yaml?branch=master\u0026label=Linux%20x64\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/java11.yaml)\n[![MacOS arm64](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/macos-arm64.yaml?branch=master\u0026label=MacOS%20arm64\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/macos-arm64.yaml)\n[![Windows x64](https://img.shields.io/github/actions/workflow/status/fizzed/bigmap/windows-x64.yaml?branch=master\u0026label=Windows%20x64\u0026style=flat-square)](https://github.com/fizzed/bigmap/actions/workflows/windows-x64.yaml)\n\nThe following platforms are tested using the [Fizzed, Inc.](http://fizzed.com) build system:\n\n[![Linux arm64](https://img.shields.io/badge/Linux%20arm64-passing-green)](buildx-results.txt)\n[![Linux riscv64](https://img.shields.io/badge/Linux%20riscv64-passing-green)](buildx-results.txt)\n[![Linux MUSL x64](https://img.shields.io/badge/Linux%20MUSL%20x64-passing-green)](buildx-results.txt)\n[![MacOS x64](https://img.shields.io/badge/MacOS%20x64-passing-green)](buildx-results.txt)\n[![Windows arm64](https://img.shields.io/badge/Windows%20arm64-passing-green)](buildx-results.txt)\n[![FreeBSD x64](https://img.shields.io/badge/FreeBSD%20x64-passing-green)](buildx-results.txt)\n[![OpenBSD x64](https://img.shields.io/badge/OpenBSD%20x64-passing-green)](buildx-results.txt)\n\n## Overview\n\nLightweight Map, SortedMap, LinkedMap, Set, and SortedSet implementation(s) with minimal 3rd party dependencies that alleviates memory\npressure by offloading to disk.  Tested on Java 8, 11, and 17.\n\nWhile there are other alternatives out there, they were almost too complicated.  This is a simple\nway to drop-in as a replacement where you use JVM Maps or Sets and don't want to fuss with settings\ntoo much.\n\nThere are several implementations available, depending on your needs and runtime environment. We evaluated\nRocksDB, LevelDB (pure Java and JNI versions), MVStore, LMDB, KyotoCabinet, TokyoCabinet, and Tkrzw.\n\nBased on our primary need for minimal memory usage (including a desire to not rely on memory-mapped files)\nand as small as possible dependencies, we narrowed in on the older, but still impressive TokyoCabinet vs.\nsome of the more recent entrants like RocksDB.\n\n## Sponsorship \u0026 Support\n\n![](https://cdn.fizzed.com/github/fizzed-logo-100.png)\n\nProject by [Fizzed, Inc.](http://fizzed.com) (Follow on Twitter: [@fizzed_inc](http://twitter.com/fizzed_inc))\n\n**Developing and maintaining opensource projects requires significant time.** If you find this project useful or need\ncommercial support, we'd love to chat. Drop us an email at [ping@fizzed.com](mailto:ping@fizzed.com)\n\nProject sponsors may include the following benefits:\n\n- Priority support (outside of Github)\n- Feature development \u0026 roadmap\n- Priority bug fixes\n- Privately hosted continuous integration tests for their unique edge or use cases\n\n## Usage\n\nWith many of our implementations (e.g. rocksdb or leveldb), you can simply use the dependency below and add it to\nyour maven POM file.  However, with tokyocabinet or tkrzw, there are many native libs you will need to include. To simplify\nmanaging these versions, you should consider importing our bill-of-materials BOM.\n\n```xml\n\u003cdependencyManagement\u003e\n    \u003cdependencies\u003e\n        \u003cdependency\u003e\n            \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n            \u003cartifactId\u003ebigmap-bom\u003c/artifactId\u003e\n            \u003cversion\u003e1.1.0\u003c/version\u003e\n            \u003cscope\u003eimport\u003c/scope\u003e\n            \u003ctype\u003epom\u003c/type\u003e\n        \u003c/dependency\u003e\n    \u003c/dependencies\u003e\n\u003c/dependencyManagement\u003e\n```\n\nThen to use tokyocabinet implementation, add the following:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003ebigmap-tokyocabinet\u003c/artifactId\u003e\n    \u003c!-- you can omit the version if you used our BOM above --\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003etokyocabinet-linux-x64\u003c/artifactId\u003e\n    \u003c!-- you can omit the version if you used our BOM above --\u003e\n    \u003cversion\u003eVERSION-HERE\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThen to use tkrzw implementation, add the following:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003ebigmap-tkrzw\u003c/artifactId\u003e\n    \u003c!-- you can omit the version if you used our BOM above --\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003etkrzw-linux-x64\u003c/artifactId\u003e\n    \u003c!-- you can omit the version if you used our BOM above --\u003e\n    \u003cversion\u003eVERSION-HERE\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nOr for rocksdb\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003ebigmap-rocksdb\u003c/artifactId\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nOr for leveldb\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.fizzed\u003c/groupId\u003e\n    \u003cartifactId\u003ebigmap-leveldb\u003c/artifactId\u003e\n    \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThen in your Java code\n\n```java\nimport com.fizzed.bigmap.leveldb.LevelBigMapBuilder;\n...\nMap\u003cLong,String\u003e map = new LevelBigMapBuilder\u003cLong,String\u003e()\n   .setScratchDirectory(Paths.get(\"target\"))\n   .setKeyType(Long.class)\n   .setValueType(String.class)\n   .build();\n```\n\n```java\nimport com.fizzed.bigmap.leveldb.LevelBigLinkedMapBuilder;\n...\nMap\u003cLong,String\u003e map = new LevelBigLinkedMapBuilder\u003cLong,String\u003e()\n   .setScratchDirectory(Paths.get(\"target\"))\n   .setKeyType(Long.class)\n   .setValueType(String.class)\n   .build();\n\n```\n\n```java\nimport com.fizzed.bigmap.leveldb.LevelBigSetBuilder;\n...\nSet\u003cLong\u003e set = new LevelBigSetBuilder\u003cLong\u003e()\n   .setScratchDirectory(Paths.get(\"target\"))\n   .setValueType(Long.class)\n   .build();\n```\n\nThen standard Map (sorted) and Set (sorted) methods all mostly work.  Some methods make no sense\nwhen you are using this instead (e.g. finding a value vs. lookups by key)\n\n## Serialization\n\nAll keys and values must be serialized to/from byte arrays in order to offload your entries to disk.\nStandard primitives like Strings, Longs, Integers, etc. have built-in codecs in the BigMap library\nfor quick and efficient serialization. For more complex objects that implement the Serializable interface,\nthere is support for that in BigMap as well. However, if you desire high performance for your complex\nobjects, you can either provide your own ByteCodec to your maps, or take a look at the `bigmap-kryo` \ndependency, which offers excellent performance and works great on Java 8+ (including Java 17).\n\n## Performance\n\nAll tests performed with Azul JDK 11, Linux x64, with -Xmx128m settings for JVM.\n\n### TokyoCabinet (B-Tree+)\n\nJNI-based library. Small dependency, most efficient use of memory, and highest performance.\n\n```\nPerformance test: type=TokyoCabinetMap, maps=10, entriesPerMap=3000, totalEntries=30000\nMax openFiles=52, heap=75 (MB), rss=222 (MB)\nMap put throughput: 281514 entries per second\nMap get throughput: 20820 entries per second\nTotal disk used: 1 (MB)\n\nPerformance test: type=TokyoCabinetMap, maps=10, entriesPerMap=300000, totalEntries=3000000\nMax openFiles=53, heap=80 (MB), rss=332 (MB)\nMap put throughput: 872365 entries per second\nMap get throughput: 125808 entries per second\nTotal disk used: 635 (MB)\n\nPerformance test: type=TokyoCabinetMap, maps=10, entriesPerMap=3000000, totalEntries=30000000\nMax openFiles=53, heap=81 (MB), rss=349 (MB)\nMap put throughput: 849108 entries per second\nMap get throughput: 162776 entries per second\nTotal disk used: 6539 (MB)\n\nPerformance test: type=TokyoCabinetMap, maps=10, entriesPerMap=9000000, totalEntries=90000000\nMax openFiles=53, heap=81 (MB), rss=345 (MB)\nMap put throughput: 836712 entries per second\nMap get throughput: 157239 entries per second\nTotal disk used: 19749 (MB)\n\nPerformance test: type=TokyoCabinetMap, maps=30, entriesPerMap=3000000, totalEntries=90000000\nMax openFiles=73, heap=81 (MB), rss=639 (MB)\nMap put throughput: 807945 entries per second\nMap get throughput: 164839 entries per second\nTotal disk used: 19618 (MB)\n```\n\n### RocksDB\n\nJNI-based library.\n\n```\nPerformance test: type=RocksBigMap, maps=10, entriesPerMap=3000, totalEntries=30000\nMax openFiles=103, heap=74 (MB), rss=227 (MB)\nMap put throughput: 155633 entries per second\nMap get throughput: 23344 entries per second\nTotal disk used: 6 (MB)\n\nPerformance test: type=RocksBigMap, maps=10, entriesPerMap=300000, totalEntries=3000000\nMax openFiles=114, heap=80 (MB), rss=362 (MB)\nMap put throughput: 254209 entries per second\nMap get throughput: 108640 entries per second\nTotal disk used: 192 (MB)\n\nPerformance test: type=RocksBigMap, maps=10, entriesPerMap=3000000, totalEntries=30000000\nMax openFiles=214, heap=81 (MB), rss=609 (MB)\nMap put throughput: 177122 entries per second\nMap get throughput: 104554 entries per second\nTotal disk used: 1632 (MB)\n\nPerformance test: type=RocksBigMap, maps=10, entriesPerMap=9000000, totalEntries=90000000\nMax openFiles=444, heap=81 (MB), rss=722 (MB)\nMap put throughput: 160107 entries per second\nMap get throughput: 103470 entries per second\nTotal disk used: 4611 (MB)\n```\n\n### LevelDB (Java version)\n\nPure Java library.\n\n```\nPerformance test: type=LevelBigMap, maps=10, entriesPerMap=3000, totalEntries=30000\nMax openFiles=68, heap=76 (MB), rss=229 (MB)\nMap put throughput: 222263 entries per second\nMap get throughput: 9968 entries per second\nTotal disk used: 20 (MB)\n\nPerformance test: type=LevelBigMap, maps=10, entriesPerMap=300000, totalEntries=3000000\nMax openFiles=71, heap=103 (MB), rss=906 (MB)\nMap put throughput: 214845 entries per second\nMap get throughput: 197670 entries per second\nTotal disk used: 637 (MB)\n\nYIKES - ran out of heap almost towards end of test, adjusted to -Xms256m\nPerformance test: type=LevelBigMap, maps=10, entriesPerMap=3000000, totalEntries=3000000\nMax openFiles=71, heap=223 (MB), rss=6766 (MB)\nMap put throughput: 189189 entries per second\nMap get throughput: 250558 entries per second\nTotal disk used: 6307 (MB)\n```\n\n### MVStore (H2 DB engine)\n\nPure Java library.\n\n```\nPerformance test: type=MVStoreMap, maps=10, entriesPerMap=3000, totalEntries=30000\nMax openFiles=39, heap=64 (MB), rss=299 (MB)\nMap put throughput: 77023 entries per second\nMap get throughput: 15670 entries per second\nTotal disk used: 163 (MB)\n\nPerformance test: type=MVStoreMap, maps=10, entriesPerMap=300000, totalEntries=3000000\nMax openFiles=40, heap=101 (MB), rss=339 (MB)\nMap put throughput: 225884 entries per second\nMap get throughput: 11504 entries per second\nTotal disk used: 2580 (MB)\n\nPerformance test: type=MVStoreMap, maps=10, entriesPerMap=3000000, totalEntries=30000000\nMax openFiles=40, heap=104 (MB), rss=354 (MB)\nMap put throughput: 216259 entries per second\nMap get throughput: 12201 entries per second\nTotal disk used: 24135 (MB)\n```\n\n## License\n\nCopyright (C) 2025 Fizzed, Inc.\n\nThis work is licensed under the Apache License, Version 2.0. See LICENSE for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffizzed%2Fbigmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffizzed%2Fbigmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffizzed%2Fbigmap/lists"}