{"id":38055406,"url":"https://github.com/lmdbjava/benchmarks","last_synced_at":"2026-01-16T20:23:47.176Z","repository":{"id":296995013,"uuid":"60845589","full_name":"lmdbjava/benchmarks","owner":"lmdbjava","description":"Benchmark of open source, embedded, memory-mapped, key-value stores available from Java (JMH)","archived":false,"fork":false,"pushed_at":"2025-11-19T02:11:30.000Z","size":1493,"stargazers_count":144,"open_issues_count":1,"forks_count":23,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-11-19T04:10:10.898Z","etag":null,"topics":["benchmark","java","jmh","leveldb","lmdb","rocksdb"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lmdbjava.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2016-06-10T12:18:32.000Z","updated_at":"2025-11-19T02:11:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"d50bcab8-6d63-4ea2-a392-24750c1c1562","html_url":"https://github.com/lmdbjava/benchmarks","commit_stats":null,"previous_names":["lmdbjava/benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lmdbjava/benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmdbjava%2Fbenchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmdbjava%2Fbenchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmdbjava%2Fbenchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmdbjava%2Fbenchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lmdbjava","download_url":"https://codeload.github.com/lmdbjava/benchmarks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lmdbjava%2Fbenchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28482267,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","java","jmh","leveldb","lmdb","rocksdb"],"created_at":"2026-01-16T20:23:47.076Z","updated_at":"2026-01-16T20:23:47.146Z","avatar_url":"https://github.com/lmdbjava.png","language":"Shell","readme":"[![Library Benchmarks](https://github.com/lmdbjava/benchmarks/workflows/Library%20Benchmarks/badge.svg)](https://github.com/lmdbjava/benchmarks/actions)\n[![Version Benchmarks](https://github.com/lmdbjava/benchmarks/workflows/Version%20Benchmarks/badge.svg)](https://github.com/lmdbjava/benchmarks/actions)\n[![LMDB Benchmarks](https://github.com/lmdbjava/benchmarks/workflows/LMDB%20Benchmarks/badge.svg)](https://github.com/lmdbjava/benchmarks/actions)\n[![License](https://img.shields.io/hexpm/l/plug.svg?maxAge=2592000)](http://www.apache.org/licenses/LICENSE-2.0.txt)\n\n# LmdbJava Benchmarks\n\n**Just want the latest results?**\n\n- **Library Comparisons**: [Full Benchmark](https://libraries-benchmark.lmdbjava.org) | [Smoketest](https://libraries-smoketest.lmdbjava.org)\n- **Version Regressions**: [Full Benchmark](https://versions-benchmark.lmdbjava.org) | [Smoketest](https://versions-smoketest.lmdbjava.org)\n- **LMDB Benchmarks**: [Full Benchmark](https://lmdb-benchmark.lmdbjava.org) | [Smoketest](https://lmdb-smoketest.lmdbjava.org)\n- **Historical**: [2016 Results](https://github.com/lmdbjava/benchmarks/blob/master/results/20160710/README.md)\n\nThis is a [JMH](http://openjdk.java.net/projects/code-tools/jmh/) benchmark\nof open source, embedded, memory-mapped, key-value stores available from Java:\n\n* [LmdbJava](https://github.com/lmdbjava/lmdbjava) (with fast `ByteBuffer`, safe\n  `ByteBuffer` and an [Agrona](https://github.com/real-logic/Agrona) buffer)\n* [LMDBJNI](https://github.com/deephacks/lmdbjni)\n* [Lightweight Java Game Library](https://github.com/LWJGL/lwjgl3/) (LMDB API)\n* [LevelDBJNI](https://github.com/fusesource/leveldbjni)\n* [RocksDB](http://rocksdb.org/)\n* [MVStore](http://h2database.com/html/mvstore.html) (pure Java)\n* [MapDB](http://www.mapdb.org/) (pure Java)\n* [Xodus](https://github.com/JetBrains/xodus) (pure Java)\n* [Chroncile Map](https://github.com/OpenHFT/Chronicle-Map) (pure Java) (**)\n\n(**) does not support ordered keys, so iteration benchmarks not performed\n\nThe benchmark itself is adapted from LMDB's\n[db_bench_mdb.cc](http://lmdb.tech/bench/microbench/db_bench_mdb.cc), which in\nturn is adapted from\n[LevelDB's benchmark](https://github.com/google/leveldb/blob/master/db/db_bench.cc).\n\nThe benchmark includes:\n\n* Writing data\n* Reading all data via each key\n* Reading all data via a reverse iterator\n* Reading all data via a forward iterator\n* Reading all data via a forward iterator and computing a CRC32 (via JDK API)\n* Reading all data via a forward iterator and computing a XXH64 hash\n\nByte arrays (`byte[]`) are always used for the keys and values, avoiding any\nserialization library overhead. For those libraries that support compression,\nit is disabled in the benchmark. In general any special library features that\ndecrease latency (eg batch modes, disable auto-commit, disable journals,\nhint at expected data sizes etc) were used. While we have tried to be fair and\nconsistent, some libraries offer non-obvious tuning settings or usage patterns\nthat might further reduce their latency. We do not claim we have exhausted\nevery tuning option every library exposes, but pull requests are most welcome.\n\n## Build\n\nClone this repository and build:\n\n```bash\nmvn clean package\n```\n\n## Usage\n\nThis benchmark uses POSIX calls to accurately determine consumed disk space and\nonly depends on Linux-specific native library wrappers where a range of such\nwrappers exists. Operation on non-Linux operating systems is unsupported.\n\n### Running Benchmarks\n\n#### Library Comparison Benchmarks\n\nUse the `run-libs.sh` script to compare different key-value store libraries:\n\n```bash\n# Quick smoke test (1K entries, fast verification)\n./run-libs.sh smoketest\n\n# Full benchmark using 25% of system RAM (default)\n./run-libs.sh benchmark\n\n# Full benchmark using 50% of system RAM\n./run-libs.sh benchmark 50\n```\n\nThe benchmark auto-scales based on available RAM and caps at 1 million entries.\nResults are written to `target/benchmark-libs/`.\n\n**File handle limit:** Benchmark mode requires at least 1 million file handles for RocksDB and LevelDB LSM operations:\n\n```bash\nulimit -n 1000000\n./run-libs.sh benchmark\n```\n\nThe script will check and abort if the limit is too low.\n\n**Resumption:** The script skips completed runs by checking for existing result files. For a fresh run, remove the output directory:\n\n```bash\nrm -rf target/benchmark-libs\n./run-libs.sh benchmark\n```\n\n**Long-running benchmarks:** For multi-hour benchmarks, redirect output to a log file and monitor with `tail`:\n\n```bash\n# Start the benchmark with output redirected\n./run-libs.sh benchmark \u003e log.txt 2\u003e\u00261\n\n# Monitor progress in another terminal\ntail -f log.txt\n```\n\n#### Version Regression Testing\n\nUse the `run-vers.sh` script to test LmdbJava performance across versions:\n\n```bash\n# Quick smoke test (1K entries, fast verification)\n./run-vers.sh smoketest\n\n# Full benchmark using 25% of system RAM (default)\n./run-vers.sh benchmark\n\n# Full benchmark using 50% of system RAM\n./run-vers.sh benchmark 50\n```\n\nThis tests selected LmdbJava versions from Maven Central plus current development branches to identify performance regressions.\nResults are written to `target/benchmark-vers/`.\n\n**Resumption:** The script skips completed versions by checking for existing result files. For a fresh run, remove the output directory:\n\n```bash\nrm -rf target/benchmark-vers\n./run-vers.sh benchmark\n```\n\n#### LMDB Benchmarks\n\nUse the `run-lmdb.sh` script to test LmdbJava (master) performance across different LMDB library versions:\n\n```bash\n# Quick smoke test (1K entries, fast verification)\n./run-lmdb.sh smoketest\n\n# Full benchmark using 25% of system RAM (default)\n./run-lmdb.sh benchmark\n\n# Full benchmark using 50% of system RAM\n./run-lmdb.sh benchmark 50\n```\n\nThis tests LmdbJava master against 9 different LMDB library versions (0.9.18-0.9.33) to isolate LMDB native library performance characteristics from LmdbJava wrapper code.\nResults are written to `target/benchmark-lmdb/`.\n\n**Resumption:** The script skips completed LMDB versions by checking for existing result files. For a fresh run, remove the output directory:\n\n```bash\nrm -rf target/benchmark-lmdb\n./run-lmdb.sh benchmark\n```\n\n#### Running All Benchmark Suites\n\nUse the `run-all.sh` script to run all three benchmark suites sequentially (designed for overnight runs):\n\n```bash\n# Run all benchmarks using 25% of system RAM (default)\n./run-all.sh\n\n# Run all benchmarks using 50% of system RAM\n./run-all.sh 50\n```\n\nThis will run LMDB library comparison, library comparison and version regression benchmarks, all in full benchmark mode with 120s iterations. Expect several hours of runtime depending on your system.\n\n### Generating Reports\n\nAfter running library comparison benchmarks, generate a comprehensive report:\n\n```bash\n./report-libs.sh\n```\n\nAfter running version regression tests, generate a version comparison report:\n\n```bash\n./report-vers.sh\n```\n\nAfter running LMDB benchmarks, generate an LMDB performance report:\n\n```bash\n./report-lmdb.sh\n```\n\nReports generate:\n- `target/benchmark/index.html` - Pure HTML report with embedded SVG charts\n- Various SVG chart files\n\n### Publishing Reports\n\nAfter generating a report, you can publish it to Cloudflare Pages:\n\n```bash\nexport CLOUDFLARE_API_TOKEN=\"your-token\"\nexport CLOUDFLARE_ACCOUNT_ID=\"your-account-id\"\n./publish-results.sh\n```\n\nThe script automatically detects:\n- **Type**: Library comparison or version regression (from README heading)\n- **Mode**: Smoketest (3s) or benchmark (120s) (from smoketest warning)\n\nReports are published to:\n- [libraries-smoketest.lmdbjava.org](https://libraries-smoketest.lmdbjava.org) - Fast daily library verification\n- [libraries-benchmark.lmdbjava.org](https://libraries-benchmark.lmdbjava.org) - Full library comparison analysis\n- [versions-smoketest.lmdbjava.org](https://versions-smoketest.lmdbjava.org) - Fast daily version monitoring\n- [versions-benchmark.lmdbjava.org](https://versions-benchmark.lmdbjava.org) - Full version regression analysis\n\n**Workflow for curated reports:**\n1. Run full benchmarks: `./run-all.sh` (or individually with `./run-lmdb.sh benchmark`, `./run-libs.sh benchmark` and `./run-vers.sh benchmark`)\n2. Generate reports: `./report-lmdb.sh`, `./report-libs.sh` and `./report-vers.sh`\n3. Review and edit commentary in report scripts if needed, then re-run\n4. Publish: `./publish-results.sh` (run once after each report generation)\n\n### Performance Bisection\n\nThe `run-bisect.sh` script uses bisection to find the commit that introduced a performance regression:\n\n```bash\n./run-bisect.sh\n```\n\nConfigure the bisection by editing variables at the top of the script:\n- `START_COMMIT`: Known good commit (full hash)\n- `END_COMMIT`: Known bad commit (full hash)\n- `BENCHMARK_NAME`: Full benchmark qualifier (eg `LmdbJavaAgrona.write`)\n- `MAX_BISECTIONS`: Maximum bisection iterations (default 10)\n\nThe script:\n- Tests endpoint commits to confirm regression\n- Bisects the commit range at 50% intervals\n- Uses distance-based decision making (closer to good or bad endpoint)\n- Uses the system LMDB library (`/usr/lib/liblmdb.so`) for all tests\n- Caches built JARs and benchmark results to avoid rebuilding\n- Generates a summary report with an asterisk marking the regression commit\n\nResults are saved in `bisect/results/` with a chronologically-sorted summary report showing scores and percentage changes. To re-run with fresh benchmarks, delete `bisect/results/*.json`.\n\n### Development Testing\n\nThe `run-dev.sh` script enables rapid testing of local LmdbJava changes without full bisection overhead:\n\n```bash\n./run-dev.sh\n```\n\nThis script:\n- Builds LmdbJava from `dev/lmdbjava/` (clone or symlink your working copy here)\n- Patches the benchmark pom.xml to use the local SNAPSHOT version\n- Runs a quick benchmark (10K entries, 3s runtime by default)\n- Saves results to `dev/output/` with commit hash and timestamp\n\nTo match full bisection configuration, edit the script and change:\n- `JMH_RUNTIME=30` (30 second iterations)\n- `NUM_ENTRIES=1000000` (1 million entries)\n\nUse this for iterative development and verification of performance fixes before committing.\n\n## Version Management\n\nUpdate all dependency and plugin versions:\n\n```bash\nmvn versions:update-properties\n```\n\n## Support\n\nIssues are disabled for this repository. Please report any issues or questions\non the [LmdbJava issue tracker](https://github.com/lmdbjava/lmdbjava/issues).\n\n## Contributing\n\nContributions are welcome! Please see the LmdbJava project's\n[Contributing Guidelines](https://github.com/lmdbjava/lmdbjava/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis project is licensed under the\n[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmdbjava%2Fbenchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flmdbjava%2Fbenchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flmdbjava%2Fbenchmarks/lists"}