{"id":20038495,"url":"https://github.com/yahoo/halodb","last_synced_at":"2025-04-04T19:13:39.187Z","repository":{"id":33848089,"uuid":"137792810","full_name":"yahoo/HaloDB","owner":"yahoo","description":"A fast, log structured key-value store.","archived":false,"fork":false,"pushed_at":"2023-04-26T15:56:57.000Z","size":576,"stargazers_count":514,"open_issues_count":30,"forks_count":100,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-03-28T18:16:02.632Z","etag":null,"topics":["big-data","embedded-database","java","key-value-store","storage-engine"],"latest_commit_sha":null,"homepage":"https://yahoodevelopers.tumblr.com/post/178250134648/introducing-halodb-a-fast-embedded-key-value","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yahoo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"Code-of-Conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-06-18T18:46:19.000Z","updated_at":"2025-03-10T11:23:26.000Z","dependencies_parsed_at":"2023-01-15T02:56:33.123Z","dependency_job_id":"78034c38-78a4-44ad-b4b6-9439ac4d46d3","html_url":"https://github.com/yahoo/HaloDB","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FHaloDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FHaloDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FHaloDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yahoo%2FHaloDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yahoo","download_url":"https://codeload.github.com/yahoo/HaloDB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234923,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","embedded-database","java","key-value-store","storage-engine"],"created_at":"2024-11-13T10:29:32.224Z","updated_at":"2025-04-04T19:13:39.170Z","avatar_url":"https://github.com/yahoo.png","language":"Java","readme":"# HaloDB\n\n[![Build Status](https://travis-ci.org/yahoo/HaloDB.svg?branch=master)](https://travis-ci.org/yahoo/HaloDB)\n[![Download](https://api.bintray.com/packages/yahoo/maven/halodb/images/download.svg) ](https://bintray.com/yahoo/maven/halodb/_latestVersion)\n\nHaloDB is a fast and simple embedded key-value store written in Java. HaloDB is suitable for IO bound workloads, and is capable of handling high throughput reads and writes at submillisecond latencies. \n\nHaloDB was written for a high-throughput, low latency distributed key-value database that powers multiple ad platforms at Yahoo, therefore all its design choices and optimizations were\nprimarily for this use case.  \n\nBasic design principles employed in HaloDB are not new. Refer to this [document](docs/WhyHaloDB.md) for more details about the motivation for HaloDB and its inspirations.  \n \nHaloDB comprises of two main components: an index in memory which stores all the keys, and append-only log files on\nthe persistent layer which stores all the data. To reduce Java garbage collection pressure the index \nis allocated in native memory, outside the Java heap. \n\n![HaloDB](https://raw.githubusercontent.com/amannaly/HaloDB-images/master/images/halodb.png)\n\n### Basic Operations. \n```java\n            // Open a db with default options.\n            HaloDBOptions options = new HaloDBOptions();\n    \n            // Size of each data file will be 1GB.\n            options.setMaxFileSize(1024 * 1024 * 1024);\n\n            // Size of each tombstone file will be 64MB\n            // Large file size mean less file count but will slow down db open time. But if set\n            // file size too small, it will result large amount of tombstone files under db folder\n            options.setMaxTombstoneFileSize(64 * 1024 * 1024);\n\n            // Set the number of threads used to scan index and tombstone files in parallel\n            // to build in-memory index during db open. It must be a positive number which is\n            // not greater than Runtime.getRuntime().availableProcessors().\n            // It is used to speed up db open time.\n            options.setBuildIndexThreads(8);\n\n            // The threshold at which page cache is synced to disk.\n            // data will be durable only if it is flushed to disk, therefore\n            // more data will be lost if this value is set too high. Setting\n            // this value too low might interfere with read and write performance.\n            options.setFlushDataSizeBytes(10 * 1024 * 1024);\n    \n            // The percentage of stale data in a data file at which the file will be compacted.\n            // This value helps control write and space amplification. Increasing this value will\n            // reduce write amplification but will increase space amplification.\n            // This along with the compactionJobRate below is the most important setting\n            // for tuning HaloDB performance. If this is set to x then write amplification \n            // will be approximately 1/x. \n            options.setCompactionThresholdPerFile(0.7);\n    \n            // Controls how fast the compaction job should run.\n            // This is the amount of data which will be copied by the compaction thread per second.\n            // Optimal value depends on the compactionThresholdPerFile option.\n            options.setCompactionJobRate(50 * 1024 * 1024);\n    \n            // Setting this value is important as it helps to preallocate enough\n            // memory for the off-heap cache. If the value is too low the db might\n            // need to rehash the cache. For a db of size n set this value to 2*n.\n            options.setNumberOfRecords(100_000_000);\n            \n            // Delete operation for a key will write a tombstone record to a tombstone file.\n            // the tombstone record can be removed only when all previous version of that key\n            // has been deleted by the compaction job.\n            // enabling this option will delete during startup all tombstone records whose previous\n            // versions were removed from the data file.\n            options.setCleanUpTombstonesDuringOpen(true);\n    \n            // HaloDB does native memory allocation for the in-memory index.\n            // Enabling this option will release all allocated memory back to the kernel when the db is closed.\n            // This option is not necessary if the JVM is shutdown when the db is closed, as in that case\n            // allocated memory is released automatically by the kernel.\n            // If using in-memory index without memory pool this option,\n            // depending on the number of records in the database,\n            // could be a slow as we need to call _free_ for each record.\n            options.setCleanUpInMemoryIndexOnClose(false);\n            \n            // ** settings for memory pool **\n            options.setUseMemoryPool(true);\n    \n            // Hash table implementation in HaloDB is similar to that of ConcurrentHashMap in Java 7.\n            // Hash table is divided into segments and each segment manages its own native memory.\n            // The number of segments is twice the number of cores in the machine.\n            // A segment's memory is further divided into chunks whose size can be configured here. \n            options.setMemoryPoolChunkSize(2 * 1024 * 1024);\n    \n            // using a memory pool requires us to declare the size of keys in advance.\n            // Any write request with key length greater than the declared value will fail, but it\n            // is still possible to store keys smaller than this declared size. \n            options.setFixedKeySize(8);\n    \n            // Represents a database instance and provides all methods for operating on the database.\n            HaloDB db = null;\n    \n            // The directory will be created if it doesn't exist and all database files will be stored in this directory\n            String directory = \"directory\";\n    \n            // Open the database. Directory will be created if it doesn't exist.\n            // If we are opening an existing database HaloDB needs to scan all the\n            // index files to create the in-memory index, which, depending on the db size, might take a few minutes.\n            db = HaloDB.open(directory, options);\n    \n            // key and values are byte arrays. Key size is restricted to 128 bytes.\n            byte[] key1 = Ints.toByteArray(200);\n            byte[] value1 = \"Value for key 1\".getBytes();\n    \n            byte[] key2 = Ints.toByteArray(300);\n            byte[] value2 = \"Value for key 2\".getBytes();\n    \n            // add the key-value pair to the database.\n            db.put(key1, value1);\n            db.put(key2, value2);\n    \n            // read the value from the database.\n            value1 = db.get(key1);\n            value2 = db.get(key2);\n    \n            // delete a key from the database.\n            db.delete(key1);\n    \n            // Open an iterator and iterate through all the key-value records.\n            HaloDBIterator iterator = db.newIterator();\n            while (iterator.hasNext()) {\n                Record record = iterator.next();\n                System.out.println(Ints.fromByteArray(record.getKey()));\n                System.out.println(new String(record.getValue()));\n            }\n    \n            // get stats and print it.\n            HaloDBStats stats = db.stats();\n            System.out.println(stats.toString());\n    \n            // reset stats\n            db.resetStats();\n            \n            // pause background compaction thread.\n            // if a file is being compacted the thread\n            // will block until the compaction is complete.\n            db.pauseCompaction();\n            \n            // resume background compaction thread.\n            db.resumeCompaction();\n            \n            // repeatedly calling pause/resume compaction methods will have no effect.\n\n            // Close the database.\n            db.close();\n```\nBinaries for HaloDB are hosted on [Bintray](https://bintray.com/yahoo).   \n``` xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.oath.halodb\u003c/groupId\u003e\n  \u003cartifactId\u003ehalodb\u003c/artifactId\u003e\n  \u003cversion\u003ex.y.x\u003c/version\u003e \n\u003c/dependency\u003e\n\n\u003crepository\u003e\n  \u003cid\u003eyahoo-bintray\u003c/id\u003e\n  \u003cname\u003eyahoo-bintray\u003c/name\u003e\n  \u003curl\u003ehttps://yahoo.bintray.com/maven\u003c/url\u003e\n\u003c/repository\u003e\n``` \n   \n\n\n### Read, Write and Space amplification.\nRead amplification in HaloDB is always 1—for a read request it needs to do at most one disk lookup—hence it is well suited for \nread latency critical workloads. HaloDB provides a configuration which can be tuned to control write amplification \nand space amplification, both of which trade-off with each other; HaloDB has a background compaction thread which removes stale data \nfrom the DB. The percentage of stale data at which a file is compacted can be controlled. Increasing this value will increase space amplification \nbut will reduce write amplification. For example if the value is set to 50% then write amplification will be approximately 2 \n\n\n### Durability and Crash recovery.\nWrite Ahead Logs (WAL) are usually used by databases for crash recovery. Since for HaloDB WAL _is the_ database crash recovery\nis easier and faster. \n\nHaloDB does not flush writes to disk immediately, but, for performance reasons, writes only to the OS page cache. The cache is synced to \ndisk once a configurable size is reached. In the event of a power loss, the data not flushed to disk will be lost. This compromise\nbetween performance and durability is a necessary one. \n\nIn the event of a power loss and data corruption, HaloDB will scan and discard corrupted records. Since the write thread and compaction \nthread could be writing to at most two files at a time only those files need to be repaired and hence recovery times are very short.\n\nIn the event of a power loss HaloDB offers the following consistency guarantees:\n* Writes are atomic.\n* Inserts and updates are committed to disk in the same order they are received.\n* When inserts/updates and deletes are interleaved total ordering is not guaranteed, but partial ordering is guaranteed for inserts/updates and deletes.    \n \n  \n### In-memory index.  \nHaloDB stores all keys and their associated metadata in an index in memory. The size of this index, depending on the \nnumber and length of keys, can be quite big. Therefore, storing this in the Java Heap is a non-starter for a \nperformance critical storage engine. HaloDB solves this problem by storing the index in native memory, \noutside the heap. There are two variants of the index; one with a memory pool and the other \nwithout it. Using the memory pool helps to reduce the memory footprint of the index and reduce \nfragmentation, but requires fixed size keys. A billion 8 byte keys \ncurrently takes around 44GB of memory with memory pool and around 64GB without memory pool.   \n\nThe size of the keys when using a memory pool should be declared in advance, and although this imposes an \nupper limit on the size of the keys it is still possible to store keys smaller than this declared size. \n\nWithout the memory pool, HaloDB needs to allocate native memory for every write request. Therefore, \nmemory fragmentation could be an issue. Using [jemalloc](http://jemalloc.net/) is highly recommended as it \nprovides a significant reduction in the cache's memory footprint and fragmentation.\n\n### Delete operations.\nDelete operation for a key will add a tombstone record to a tombstone file, which is distinct from the data files. \nThis design has the advantage that the tombstone record once written need not be copied again during compaction, but \nthe drawback is that in case of a power loss HaloDB cannot guarantee total ordering when put and delete operations are \ninterleaved (although partial ordering for both is guaranteed).\n\n### DB open time\nOpen db could take a few minutes, depends on number of records and tombstones. If the db open time is critical to your\nuse case, please keep tombstone file size relatively small and increase the number of threads used in building index.\nSee the option setting section in example code above. As best practice, set tombstone file size at 64MB and set build\nindex threads to number of available processors divided by number of dbs being opened simultaneously.\n\n### System requirements. \n* HaloDB requires Java 8 to run, but has not yet been tested with newer Java versions.  \n* HaloDB has been tested on Linux running on x86 and on MacOS. It may run on other platforms, but this hasn't been verified yet.\n* For performance disable Transparent Huge Pages and swapping (vm.swappiness=0).\n* If a thread is interrupted JVM will close those file channels the thread was operating on.\nTherefore, don't interrupt threads while they are doing IO operations.\n\n### Restrictions. \n* Size of keys is restricted to 128 bytes.  \n* HaloDB don't support range scans or ordered access.\n\n# Benchmarks.\n[Benchmarks](docs/benchmarks.md).\n  \n# Contributing\nContributions are most welcome. Please refer to the [CONTRIBUTING](https://github.com/yahoo/HaloDB/blob/master/CONTRIBUTING.md) guide \n\n# Credits\nHaloDB was written by [Arjun Mannaly](https://github.com/amannaly).\n\n# License \nHaloDB is released under the Apache License, Version 2.0  \n  \n  ","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fhalodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyahoo%2Fhalodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyahoo%2Fhalodb/lists"}