{"id":13626762,"url":"https://github.com/RoaringBitmap/RoaringBitmap","last_synced_at":"2025-04-16T19:30:51.855Z","repository":{"id":8996398,"uuid":"10746583","full_name":"RoaringBitmap/RoaringBitmap","owner":"RoaringBitmap","description":"A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others","archived":false,"fork":false,"pushed_at":"2024-10-24T00:58:15.000Z","size":89920,"stargazers_count":3526,"open_issues_count":68,"forks_count":552,"subscribers_count":134,"default_branch":"master","last_synced_at":"2024-10-24T11:54:58.970Z","etag":null,"topics":["bitset","druid","java","lucene","roaring-bitmaps","roaringbitmap","spark"],"latest_commit_sha":null,"homepage":"http://roaringbitmap.org/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RoaringBitmap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-17T20:29:23.000Z","updated_at":"2024-10-24T07:44:11.000Z","dependencies_parsed_at":"2023-02-18T08:46:02.686Z","dependency_job_id":"1b195473-a9c6-4c35-966f-2297651f312b","html_url":"https://github.com/RoaringBitmap/RoaringBitmap","commit_stats":{"total_commits":2213,"total_committers":99,"mean_commits":"22.353535353535353","dds":"0.34297333935833707","last_synced_commit":"11f3bef58232e7d614a0bffa1ff3f25ef1e0b18a"},"previous_names":[],"tags_count":262,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RoaringBitmap%2FRoaringBitmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RoaringBitmap%2FRoaringBitmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RoaringBitmap%2FRoaringBitmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RoaringBitmap%2FRoaringBitmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RoaringBitmap","download_url":"https://codeload.github.com/RoaringBitmap/RoaringBitmap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249268547,"owners_count":21240940,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitset","druid","java","lucene","roaring-bitmaps","roaringbitmap","spark"],"created_at":"2024-08-01T22:00:17.563Z","updated_at":"2025-04-16T19:30:46.845Z","avatar_url":"https://github.com/RoaringBitmap.png","language":"Java","readme":"RoaringBitmap\n=============\n[![][license img]][license]\n[![docs-badge][]][docs]\n![Java 11 CI](https://github.com/RoaringBitmap/RoaringBitmap/workflows/Java%2011%20CI/badge.svg)\n\n- [Introduction](#introduction)\n- [When should you use a bitmap?](#when-should-you-use-a-bitmap)\n- [When should you use compressed bitmaps?](#when-should-you-use-compressed-bitmaps)\n- [How does Roaring compare with the alternatives?](#how-does-roaring-compare-with-the-alternatives)\n- [Code sample](#code-sample)\n- [API docs](#api-docs)\n- [Download](#download)\n- [Usage within a Maven project](#usage-within-a-maven-project)\n- [Usage within a gradle project](#usage-within-a-gradle-project)\n- [Scientific Documentation](#scientific-documentation)\n- [Unsigned integers](#unsigned-integers)\n- [Working with memory-mapped bitmaps](#working-with-memory-mapped-bitmaps)\n- [Thread safety](#thread-safety)\n- [Kryo](#kryo)\n- [64-bit integers (long)](#64-bit-integers-long)\n- [Range Bitmaps](#range-bitmaps)\n- [Prerequisites](#prerequisites)\n- [Usage for RoaringBitmap Developers](#usage-for-roaringbitmap-developers)\n- [IntelliJ and Eclipse](#intellij-and-eclipse)\n- [Contributing](#contributing)\n- [FAQ](#faq)\n- [Benchmark](#benchmark)\n- [Mailing list/discussion group](#mailing-listdiscussion-group)\n- [Funding](#funding)\n\nIntroduction\n-------------\n\nBitsets, also called bitmaps, are commonly used as fast data structures.\nUnfortunately, they can use too much memory. To compensate, we often use\ncompressed bitmaps.\n\nRoaring bitmaps are compressed bitmaps which tend to outperform conventional\ncompressed bitmaps such as WAH, EWAH or Concise. In some instances, roaring bitmaps can\nbe hundreds of times faster and they often offer significantly better compression.\nThey can even be faster than uncompressed bitmaps.\n\nRoaring bitmaps are found to work well in many important applications:\n\n\u003e Use Roaring for bitmap compression whenever possible. Do not use other bitmap compression methods ([Wang et al., SIGMOD 2017](http://db.ucsd.edu/wp-content/uploads/2017/03/sidm338-wangA.pdf))\n\n\u003e kudos for making something that makes my software run 5x faster (Charles Parker from BigML)\n\nThis library is used by\n*   [Apache Spark](http://spark.apache.org/),\n*   [Apache Hive](http://hive.apache.org),\n*   [Apache Tez](http://tez.apache.org),\n*   [Apache Kylin](http://kylin.io),\n*   [Apache CarbonData](https://carbondata.apache.org),\n*   [Netflix Atlas](https://github.com/Netflix/atlas),\n*   [OpenSearchServer](http://www.opensearchserver.com),\n*   [zenvisage](http://zenvisage.github.io/),\n*   [Jive Miru](https://github.com/jivesoftware/miru),\n*   [Tablesaw](https://github.com/jtablesaw/tablesaw),\n*   [Apache Hivemall](http://hivemall.incubator.apache.org),\n*   [Gaffer](https://github.com/gchq/Gaffer),\n*   [Apache Pinot](https://pinot.apache.org/),\n*   [Apache Druid](https://druid.apache.org/),\n*   [SirixDB](https://sirix.io)\n*   [EvitaDB](https://evitadb.io/)\n*   [Apache Iceberg](https://iceberg.apache.org/)\n*   [Apache Pulsar](https://pulsar.apache.org/)\n\nThe library is mature and has been used in production for many years.\n\nThe YouTube SQL Engine, [Google Procella](https://research.google/pubs/pub48388/), uses Roaring bitmaps for indexing. [Apache Lucene](http://lucene.apache.org/) uses  Roaring bitmaps, though they have their own [independent implementation](https://svn.apache.org/viewvc/lucene/dev/branches/branch_5x/lucene/core/src/java/org/apache/lucene/util/RoaringDocIdSet.java?view=markup\u0026pathrev=1629606). Derivatives of Lucene such as Solr and Elastic also use Roaring bitmaps.\nOther platforms such as [Whoosh](https://pypi.python.org/pypi/Whoosh/), [Microsoft Visual Studio Team Services (VSTS)](https://www.visualstudio.com/team-services/) and [Pilosa](https://github.com/pilosa/pilosa) also use Roaring bitmaps with their own implementations. You find Roaring bitmaps in [InfluxDB](https://www.influxdata.com), [Bleve](http://www.blevesearch.com), [Cloud Torrent](https://github.com/jpillora/cloud-torrent), [Redpanda](https://github.com/redpanda-data/redpanda), and so forth.\n\n\n[There is a serialized format specification for interoperability between implementations](https://github.com/RoaringBitmap/RoaringFormatSpec/).\nWe have interoperable [C/C++](https://github.com/RoaringBitmap/CRoaring), Java and [Go](https://github.com/RoaringBitmap/roaring) implementations.\n\n(c) 2013-... the RoaringBitmap authors\n\n\nThis code is licensed under Apache License, Version 2.0 (AL2.0).\n\n\n\nWhen should you use a bitmap?\n-------------------------------\n\n\nSets are a fundamental abstraction in\nsoftware. They can be implemented in various\nways, as hash sets, as trees, and so forth.\nIn databases and search engines, sets are often an integral\npart of indexes. For example, we may need to maintain a set\nof all documents or rows  (represented by numerical identifier)\nthat satisfy some property. Besides adding or removing\nelements from the set, we need fast functions\nto compute the intersection, the union, the difference between sets, and so on.\n\n\nTo implement a set\nof integers, a particularly appealing strategy is the\nbitmap (also called bitset or bit vector). Using n bits,\nwe can represent any set made of the integers from the range\n[0,n): the ith bit is set to one if integer i is present in the set.\nCommodity processors use words of W=32 or W=64 bits. By combining many such words, we can\nsupport large values of n. Intersections, unions and differences can then be implemented\n as bitwise AND, OR and ANDNOT operations.\nMore complicated set functions can also be implemented as bitwise operations.\n\nWhen the bitset approach is applicable, it can be orders of\nmagnitude faster than other possible implementation of a set (e.g., as a hash set)\nwhile using several times less memory.\n\nHowever, a bitset, even a compressed one is not always applicable. For example, if\nyou have 1000 random-looking integers, then a simple array might be the best representation.\nWe refer to this case as the \"sparse\" scenario.\n\nWhen should you use compressed bitmaps?\n--------------------------\n\nAn uncompressed BitSet can use a lot of memory. For example, if you take a BitSet\nand set the bit at position 1,000,000 to true and you have just over 100kB. That is over 100kB\nto store the position of one bit. This is wasteful  even if you do not care about memory:\nsuppose that you need to compute the intersection between this BitSet and another one\nthat has a bit at position 1,000,001 to true, then you need to go through all these zeroes,\nwhether you like it or not. That can become very wasteful.\n\nThis being said, there are definitively cases where attempting to use compressed bitmaps is wasteful.\nFor example, if you have a small universe size. E.g., your bitmaps represent sets of integers\nfrom [0,n) where n is small (e.g., n=64 or n=128). If you can use an uncompressed BitSet and\nit does not blow up your memory usage,  then compressed bitmaps are probably not useful\nto you. In fact, if you do not need compression, then a BitSet offers remarkable speed.\n\nThe sparse scenario is another use case where compressed bitmaps should not be used.\nKeep in mind that random-looking data is usually not compressible. E.g., if you have a small set of\n32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer,\nand attempts at compression can be counterproductive.\n\nHow does Roaring compare with the alternatives?\n------------------------------------------------\n\nMost alternatives to Roaring are part of a larger family of compressed bitmaps that are run-length-encoded\nbitmaps. They identify long runs of 1s or 0s and they represent them with a marker word.\nIf you have a local mix of 1s and 0, you use an uncompressed word.\n\nThere are many formats in this family:\n\n* Oracle's BBC (Byte-aligned Bitmap Code) is an obsolete format at this point: though it may provide good compression,\nit is likely much slower than more recent alternatives due to excessive branching.\n* WAH (Word Aligned Hybrid) is a patented variation on BBC that provides better performance.\n* Concise is a variation on the patented WAH. In some specific instances, it can compress\nmuch better than WAH (up to 2x better), but it is generally slower.\n* EWAH (Enhanced Word Aligned Hybrid) is both free of patent, and it is faster than all the above. On the downside, it\ndoes not compress quite as well. It is faster because it allows some form of \"skipping\"\nover uncompressed words. So though none of these formats are great at random access, EWAH\nis better than the alternatives.\n\n\n\nThere is a big problem with these formats however that can hurt you badly in some cases: there is no random access. If you want to check whether a given value is present in the set, you have to start from the beginning and \"uncompress\" the whole thing. This means that if you want to intersect a big set with a large set, you still have to uncompress the whole big set in the worst case...\n\nRoaring solves this problem. It works in the following manner. It divides the data into chunks of 2\u003csup\u003e16\u003c/sup\u003e integers\n(e.g., [0, 2\u003csup\u003e16\u003c/sup\u003e), [2\u003csup\u003e16\u003c/sup\u003e, 2 x 2\u003csup\u003e16\u003c/sup\u003e), ...). Within a chunk, it can use an uncompressed bitmap, a simple list of integers,\nor a list of runs. Whatever format it uses, they all allow you to check for the presence of any one value quickly\n(e.g., with a binary search). The net result is that Roaring can compute many operations much faster than run-length-encoded\nformats like WAH, EWAH, Concise... Maybe surprisingly, Roaring also generally offers better compression ratios.\n\n\n\nCode sample\n-------------\n\n```java\nimport org.roaringbitmap.RoaringBitmap;\n\npublic class Basic {\n\n  public static void main(String[] args) {\n        RoaringBitmap rr = RoaringBitmap.bitmapOf(1,2,3,1000);\n        RoaringBitmap rr2 = new RoaringBitmap();\n        rr2.add(4000L,4255L);\n        rr.select(3); // would return the third value or 1000\n        rr.rank(2); // would return the rank of 2, which is index 1\n        rr.contains(1000); // will return true\n        rr.contains(7); // will return false\n\n        RoaringBitmap rror = RoaringBitmap.or(rr, rr2);// new bitmap\n        rr.or(rr2); //in-place computation\n        boolean equals = rror.equals(rr);// true\n        if(!equals) throw new RuntimeException(\"bug\");\n        // number of values stored?\n        long cardinality = rr.getLongCardinality();\n        System.out.println(cardinality);\n        // a \"forEach\" is faster than this loop, but a loop is possible:\n        for(int i : rr) {\n          System.out.println(i);\n        }\n  }\n}\n```\n\nPlease see the examples folder for more examples, which you can run with `./gradlew :examples:runAll`, or run a specific one with `./gradlew :examples:runExampleBitmap64`, etc.\n\nAPI docs\n---------\n\nhttp://www.javadoc.io/doc/org.roaringbitmap/RoaringBitmap/\n\n\nDownload\n---------\n\nYou can download releases from github:\nhttps://github.com/RoaringBitmap/RoaringBitmap/releases\n\nUsage within a Maven project\n---------\n\n### 1. Using JitPack\n\n\nAdd the following dependency to your pom.xml file...\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.github.RoaringBitmap.RoaringBitmap\u003c/groupId\u003e\n    \u003cartifactId\u003eroaringbitmap\u003c/artifactId\u003e\n    \u003cversion\u003e1.3.16\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nYou may adjust the version number.\n\nThen add the repository to your pom.xml file:\n\n```xml\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003cid\u003ejitpack.io\u003c/id\u003e\n        \u003curl\u003ehttps://jitpack.io\u003c/url\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n```\nSee https://github.com/RoaringBitmap/JitPackRoaringBitmapProject for a complete example.\n\n\n### 2. Using GitHub Packages\n\nAdd the following dependency to your `pom.xml` file inside the `\u003cdependencies\u003e` element...\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003eorg.roaringbitmap\u003c/groupId\u003e\n    \u003cartifactId\u003eroaringbitmap\u003c/artifactId\u003e\n    \u003cversion\u003e1.3.16\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nAdd the GitHub repository inside the `\u003crepositories\u003e` element (`pom.xml` file)...\n\n```xml\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003cid\u003egithub\u003c/id\u003e\n        \u003cname\u003eRoaring Maven Packages\u003c/name\u003e\n        \u003curl\u003ehttps://maven.pkg.github.com/RoaringBitmap/RoaringBitmap\u003c/url\u003e\n        \u003creleases\u003e\u003cenabled\u003etrue\u003c/enabled\u003e\u003c/releases\u003e\n        \u003csnapshots\u003e\u003cenabled\u003etrue\u003c/enabled\u003e\u003c/snapshots\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n```\n\nSee https://github.com/RoaringBitmap/MavenRoaringBitmapProject for a complete example.\n\nThe registry access is is protected by an authorisation. So you have to add your GitHub credentials to your global settings.xml: `$HOME\\.m2\\settings.xml`.\n\nYou will need a token which you can generate on GitHub.\n\n```\nGitHub \u003e Settings \u003e Developer Settings \u003e Personal access tokens \u003e Generate new token\n```\n\nThe token needs the read:packages permission. The token identifier is a long string such as `ghp_ieOkN`.\n\nPut the following in your `settings.xml` file, within the `\u003cservers\u003e` element.\n\n```xml\n\u003cserver\u003e\n  \u003cid\u003egithub\u003c/id\u003e\n  \u003cusername\u003elemire\u003c/username\u003e\n  \u003cpassword\u003eghp_ieOkN\u003c/password\u003e\n\u003c/server\u003e\n```\n\nReplace `lemire` by your GitHub username and `ghp_ieOkN` by the token identifier\nyou just generated.\n\nUsage within a gradle project\n------------------\n\n### 1. Using JitPack\n\nThen all you need is to edit your `build.gradle` file like so:\n\n\n```groovy\nplugins {\n    id 'java'\n}\n\ngroup 'org.roaringbitmap' // name of your project\nversion '1.0-SNAPSHOT' // version of your project\n\nrepositories {\n    mavenCentral()\n    maven {\n        url 'https://jitpack.io'\n    }\n}\n\ndependencies {\n    implementation 'com.github.RoaringBitmap.RoaringBitmap:roaringbitmap:1.3.16'\n    testImplementation 'junit:junit:3.8.1'\n}\n```\n\n\nSee https://github.com/RoaringBitmap/JitPackRoaringBitmapProject for a complete example.\n\n\n### 2. Using GitHub Packages\n\n\nYou first need your GitHub credentials. Go\nto \n\n```\nGitHub \u003e Settings \u003e Developer Settings \u003e Personal access tokens \u003e Generate new token\n```\n\nAnd create a token with read:packages permission.\n\nIf your GitHub user name is `lemire` and your GitHub personal token `ghp_ieOkN`,\nthen you can set them using system variables. Under bash, you can do it like so:\n```\nexport GITHUB_USER=lemire\nexport GITHUB_PASSWORD=ghp_ieOkN\n```\n\n\nIf you prefer you can write your GitHub credentials in your  gradle.properties\nfile\n\n```\n# gradle.properties\ngithubUser=lemire\ngithubPassword=ghp_ieOkN\n```\n\nThen all you need is to edit your `build.gradle` file like so:\n\n```groovy\nplugins {\n    id 'java'\n}\n\ngroup 'org.roaringbitmap' // name of your project\nversion '1.0-SNAPSHOT' // version of your project\n\nrepositories {\n    mavenCentral()\n    maven {\n        url 'https://maven.pkg.github.com/RoaringBitmap/RoaringBitmap'\n        credentials {\n            username = System.properties['githubUser'] ?: System.env.GITHUB_USER\n            password = System.properties['githubPassword'] ?: System.env.GITHUB_PASSWORD\n        }\n    }\n}\n\ndependencies {\n    implementation 'org.roaringbitmap:roaringbitmap:1.3.16'\n    testImplementation 'junit:junit:3.8.1'\n}\n```\n\nSee https://github.com/RoaringBitmap/MavenRoaringBitmapProject for a complete example.\n\n\n\nScientific Documentation\n--------------------------\n\n- Daniel Lemire, Owen Kaser, Nathan Kurz, Luca Deri, Chris O'Hara, François Saint-Jacques, Gregory Ssi-Yan-Kai, Roaring Bitmaps: Implementation of an Optimized Software Library, Software: Practice and Experience 48 (4), 2018 [arXiv:1709.07821](https://arxiv.org/abs/1709.07821)\n-  Samy Chambi, Daniel Lemire, Owen Kaser, Robert Godin,\nBetter bitmap performance with Roaring bitmaps,\nSoftware: Practice and Experience 46 (5), 2016. [arXiv:1402.6407](http://arxiv.org/abs/1402.6407) This paper used data from http://lemire.me/data/realroaring2014.html\n- Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience 46 (11), 2016. [arXiv:1603.06549](http://arxiv.org/abs/1603.06549)\n- Samy Chambi, Daniel Lemire, Robert Godin, Kamel Boukhalfa, Charles Allen, Fangjin Yang, Optimizing Druid with Roaring bitmaps, IDEAS 2016, 2016. http://r-libre.teluq.ca/950/\n\n\nUnsigned integers\n------------------\n\nJava lacks native unsigned integers but integers are still considered to be unsigned within Roaring and ordered according to ``Integer.compareUnsigned``. This means that Java will order the numbers like so 0, 1, ..., 2147483647, -2147483648, -2147483647,..., -1. To interpret correctly, you can use ``Integer.toUnsignedLong`` and ``Integer.toUnsignedString``.\n\n\nWorking with memory-mapped bitmaps\n---------------------------------------\n\nIf you want to have your bitmaps lie in memory-mapped files, you can\nuse the org.roaringbitmap.buffer package instead. It contains two\nimportant classes, ImmutableRoaringBitmap and MutableRoaringBitmap.\nMutableRoaringBitmaps are derived from ImmutableRoaringBitmap, so that\nyou can convert (cast) a MutableRoaringBitmap to an ImmutableRoaringBitmap\nin constant time.\n\nAn ImmutableRoaringBitmap that is not an instance of a MutableRoaringBitmap\nis backed by a ByteBuffer which comes with some performance overhead, but\nwith the added flexibility that the data can reside anywhere (including outside\nof the Java heap).\n\nAt times you may need to work with bitmaps that reside on disk (instances\nof ImmutableRoaringBitmap) and bitmaps that reside in Java memory. If you\nknow that the bitmaps will reside in Java memory, it is best to use\nMutableRoaringBitmap instances, not only can they be modified, but they\nwill also be faster. Moreover, because MutableRoaringBitmap instances are\nalso ImmutableRoaringBitmap instances, you can write much of your code\nexpecting ImmutableRoaringBitmap.\n\nIf you write your code expecting ImmutableRoaringBitmap instances, without\nattempting to cast the instances, then your objects will be truly immutable.\nThe MutableRoaringBitmap has a convenience method (toImmutableRoaringBitmap)\nwhich is a simple cast back to an ImmutableRoaringBitmap instance.\nFrom a language design point of view, instances of the ImmutableRoaringBitmap class are immutable only when used as per\nthe interface of the ImmutableRoaringBitmap class. Given that the class is not final, it is possible\nto modify instances, through other interfaces. Thus we do not take the term \"immutable\" in a purist manner,\nbut rather in a practical one.\n\nOne of our motivations for this design where MutableRoaringBitmap instances can be casted\ndown to ImmutableRoaringBitmap instances is that bitmaps are often large,\nor used in a context where memory allocations are to be avoided, so we avoid forcing copies.\nCopies could be expected if one needs to mix and match ImmutableRoaringBitmap and MutableRoaringBitmap instances.\n\nThe following code sample illustrates how to create an ImmutableRoaringBitmap\nfrom a ByteBuffer. In such instances, the constructor only loads the meta-data\nin RAM while the actual data is accessed from the ByteBuffer on demand.\n\n```java\n        import org.roaringbitmap.buffer.*;\n\n        //...\n\n        MutableRoaringBitmap rr1 = MutableRoaringBitmap.bitmapOf(1, 2, 3, 1000);\n        MutableRoaringBitmap rr2 = MutableRoaringBitmap.bitmapOf( 2, 3, 1010);\n        ByteArrayOutputStream bos = new ByteArrayOutputStream();\n        DataOutputStream dos = new DataOutputStream(bos);\n        // If there were runs of consecutive values, you could\n        // call rr1.runOptimize(); or rr2.runOptimize(); to improve compression\n        rr1.serialize(dos);\n        rr2.serialize(dos);\n        dos.close();\n        ByteBuffer bb = ByteBuffer.wrap(bos.toByteArray());\n        ImmutableRoaringBitmap rrback1 = new ImmutableRoaringBitmap(bb);\n        bb.position(bb.position() + rrback1.serializedSizeInBytes());\n        ImmutableRoaringBitmap rrback2 = new ImmutableRoaringBitmap(bb);\n```\n\nAlternatively, we can serialize directly to a `ByteBuffer` with the `serialize(ByteBuffer)` method.\n\n\nOperations on an ImmutableRoaringBitmap such as and, or, xor, flip, will\ngenerate a RoaringBitmap which lies in RAM. As the name suggest, the\nImmutableRoaringBitmap itself cannot be modified.\n\n\nThis design was inspired by Apache Druid.\n\nOne can find a complete working example in the test file TestMemoryMapping.java.\n\nNote that you should not mix the classes from the org.roaringbitmap package with the classes\nfrom the org.roaringbitmap.buffer package. They are incompatible. They serialize\nto the same output however. The performance of the code in org.roaringbitmap package is\ngenerally superior because there is no overhead due to the use of ByteBuffer instances.\n\nThread safety\n-----\n\nIn general, it is unsafe to access the same bitmaps using different threads--the bitmaps are unsynchronized for performance. Should you want to access a Bitmap from more than one thread, you should provide synchronization. However, you can access an immutable bitmap from multiple threads, as long as you abide by the `ImmutableBitmapDataProvider` interface.\n\nKryo\n-----\n\nMany applications use Kryo for serialization/deserialization. One can\nuse Roaring bitmaps with Kryo efficiently thanks to a custom serializer (Kryo 5):\n\n```java\npublic class RoaringSerializer extends Serializer\u003cRoaringBitmap\u003e {\n    @Override\n    public void write(Kryo kryo, Output output, RoaringBitmap bitmap) {\n        try {\n            bitmap.serialize(new KryoDataOutput(output));\n        } catch (IOException e) {\n            e.printStackTrace();\n            throw new RuntimeException();\n        }\n    }\n    @Override\n    public RoaringBitmap read(Kryo kryo, Input input, Class\u003c? extends RoaringBitmap\u003e type) {\n        RoaringBitmap bitmap = new RoaringBitmap();\n        try {\n            bitmap.deserialize(new KryoDataInput(input));\n        } catch (IOException e) {\n            e.printStackTrace();\n            throw new RuntimeException();\n        }\n        return bitmap;\n    }\n\n}\n```\n\n64-bit integers (long)\n-----------------------\n\nThough Roaring Bitmaps were designed with the 32-bit case in mind, we have extensions to 64-bit integers.\nWe offer two classes for this purpose: `Roaring64NavigableMap` and `Roaring64Bitmap`.\n\nThe `Roaring64NavigableMap` relies on a conventional red-black-tree. The keys are 32-bit integers representing\nthe most significant 32~bits of  elements\nwhereas the values of the tree are 32-bit Roaring bitmaps. The 32-bit Roaring bitmaps represent the least significant\nbits of a set of elements.\n\nThe newer `Roaring64Bitmap` approach relies on the ART data structure to hold the key/value pair. The key\n is made of the most significant 48~bits of elements whereas the values are 16-bit Roaring containers. It is inspired by\n [The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases](https://db.in.tum.de/~leis/papers/ART.pdf) by Leis et al. (ICDE '13).\n\n```java\n    import org.roaringbitmap.longlong.*;\n\n\n    // first Roaring64NavigableMap\n    LongBitmapDataProvider r = Roaring64NavigableMap.bitmapOf(1,2,100,1000);\n    r.addLong(1234);\n    System.out.println(r.contains(1)); // true\n    System.out.println(r.contains(3)); // false\n    LongIterator i = r.getLongIterator();\n    while(i.hasNext()) System.out.println(i.next());\n\n\n    // second Roaring64Bitmap\n    bitmap1 = new Roaring64Bitmap();\n    bitmap2 = new Roaring64Bitmap();\n    int k = 1 \u003c\u003c 16;\n    long i = Long.MAX_VALUE / 2;\n    long base = i;\n    for (; i \u003c base + 10000; ++i) {\n       bitmap1.add(i * k);\n       bitmap2.add(i * k);\n    }\n    b1.and(bitmap2);\n```\n\nThe serialization of 64-bit Roaring bitmaps is specified: see\nhttps://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations\n\nHowever, it is implemented only by `Roaring64NavigableMap`, by switching:\n\n    Roaring64NavigableMap.SERIALIZATION_MODE = Roaring64NavigableMap.SERIALIZATION_MODE_PORTABLE\n\nRange Bitmaps\n-------------\n\n`RangeBitmap` is a succinct data structure supporting range queries.\nEach value added to the bitmap is associated with an incremental identifier,\nand queries produce a `RoaringBitmap` of the identifiers associated with values\nthat satisfy the query. Every value added to the bitmap is stored separately,\nso that if a value is added twice, it will be stored twice, and if that value\nis less than some threshold, there will be at least two integers in the resultant\n`RoaringBitmap`.\n\nIt is more efficient - in terms of both time and space - to\nprovide a maximum value. If you don't know the maximum value,\nprovide a `Long.MAX_VALUE`. Unsigned order is used like elsewhere in\nthe library.\n\n```java\nvar appender = RangeBitmap.appender(1_000_000);\nappender.add(1L);\nappender.add(1L);\nappender.add(100_000L);\nRangeBitmap bitmap = appender.build();\nRoaringBitmap lessThan5 = bitmap.lt(5); // {0,1}\nRoaringBitmap greaterThanOrEqualTo1 = bitmap.gte(1); // {0, 1, 2}\nRoaringBitmap greaterThan1 = bitmap.gt(1); // {2}\nRoaringBitmap equalTo1 = bitmap.eq(1); // {0, 1}\nRoaringBitmap notEqualTo1 = bitmap.neq(1); // {2}\n```\n\n`RangeBitmap` is can be written to disk and memory mapped:\n\n```java\nvar appender = RangeBitmap.appender(1_000_000);\nappender.add(1L);\nappender.add(1L);\nappender.add(100_000L);\nByteBuffer buffer = mapBuffer(appender.serializedSizeInBytes());\nappender.serialize(buffer);\nRangeBitmap bitmap = RangeBitmap.map(buffer);\n```\n\nThe serialization format uses little endian byte order.\n\nPrerequisites\n-------------\n\n\n - Version 0.7.x requires JDK 8 or better\n - Version 0.6.x requires JDK 7 or better\n - Version 0.5.x requires JDK 6 or better\n\n\n\n\nUsage for RoaringBitmap Developers\n------\n\n* Get java\n\n* ``./gradlew assemble`` will compile\n* ``./gradlew build`` will compile and run the unit tests\n* ``./gradlew test `` will run the tests\n* ``./gradlew :roaringbitmap:test --tests TestIterators.testIndexIterator4`` runs just the test `TestIterators.testIndexIterator4`; `./gradlew -i :roaringbitmap:test --tests TestRoaringBitmap.issue623` runs just the test `issue623` in the class ` TestRoaringBitmap` while printing out to the console.\n* ``./gradlew  bsi:test --tests BufferBSITest.testEQ``  run just the test `BufferBSITest.testEQ` in the `bsi` submodule\n\n\nIntelliJ and Eclipse\n--------\n\nIf you plan to contribute to RoaringBitmap, you can load\nit up in your favorite IDE.\n- For IntelliJ, in the IDE create a new project, possibly from existing sources, choose import, gradle.\n- For Eclipse: File, Import, Existing Gradle Projects, Select RoaringBitmap from your disk.\n\nContributing\n------------\n\nContributions are invited. We use the Google Java style (see `roaring_google_checks.xml`). It can be applied\nautomatically to your code with `./gradlew spotlessApply`\n\nPlease do not reformat the code needlessly (especially on comments/javadoc).\n\nFAQ\n----\n\n* I am getting an error about a bad cookie. What is this about?\n\nIn the serialized files, part of the first 4 bytes are dedicated to a \"cookie\"\nwhich serves to indicate the file format.\n\nIf you try to deserialize or map a bitmap from data that has an\nunrecognized \"cookie\", the code will abort the process and report\nan error.\n\nThis problem will occur to all users who serialized Roaring bitmaps\nusing versions prior to 0.4.x as they upgrade to version 0.4.x or better.\nThese users need to refresh their serialized bitmaps.\n\n* How big can a Roaring bitmap get?\n\nGiven N integers in [0,x), then the serialized size in bytes of\na Roaring bitmap should never exceed this bound:\n\n`` 8 + 9 * ((long)x+65535)/65536 + 2 * N ``\n\nThat is, given a fixed overhead for the universe size (x), Roaring\nbitmaps never use more than 2 bytes per integer. You can call\n``RoaringBitmap.maximumSerializedSize`` for a more precise estimate.\n\n* What is the worst case scenario for Roaring bitmaps?\n\nThere is no such thing as a data structure that is always ideal. You should\nmake sure that Roaring bitmaps fit your application profile.\nThere are at least two cases where Roaring bitmaps can be easily replaced\nby superior alternatives compression-wise:\n\n1. You have few random values spanning in a large interval (i.e., you have\na very sparse set). For example, take the set 0, 65536, 131072, 196608, 262144 ...\nIf this is typical of your application, you might consider using a HashSet or\na simple sorted array.\n\n2. You have dense set of random values that never form runs of continuous\nvalues. For example, consider the set 0,2,4,...,10000. If this is typical of your\napplication, you might be better served with a conventional bitset (e.g., Java's BitSet class).\n\n* How do I select an element at random?\n\n         Random random = new Random();\n         bitmap.select(random.nextInt(bitmap.getCardinality()));\n\n\nBenchmark\n-----------\n\nTo run JMH benchmarks, use the following commands:\n\n         $ ./gradlew jmh::shadowJar\n         $ java -jar jmh/build/libs/benchmarks.jar\n\nYou can also run a specific benchmark:\n\n         $ java -jar jmh/build/libs/benchmarks.jar 'org.roaringbitmap.aggregation.and.identical.*'\n\nIf you have a bash shell, you can also run our script which automatically builds and run specific tests...\n\n         $ ./jmh/run.sh 'org.roaringbitmap.aggregation.and.identical.*'\n\n\nMailing list/discussion group\n-----------------------------\n\nhttps://groups.google.com/forum/#!forum/roaringbitmaps\n\nFunding\n----------\n\nThis work was supported by NSERC grant number 26143.\n\n\n\n[maven img]:https://maven-badges.herokuapp.com/maven-central/org.roaringbitmap/RoaringBitmap/badge.svg\n[maven]:http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.roaringbitmap%22%20\n\n[license]:LICENSE-2.0.txt\n[license img]:https://img.shields.io/badge/License-Apache%202-blue.svg\n\n[docs-badge]:https://img.shields.io/badge/API-docs-blue.svg?style=flat-square\n[docs]:http://www.javadoc.io/doc/org.roaringbitmap/RoaringBitmap/\n","funding_links":[],"categories":["Projects","项目","Java","Memory and concurrency"],"sub_categories":["Data Structures","数据结构"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRoaringBitmap%2FRoaringBitmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FRoaringBitmap%2FRoaringBitmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FRoaringBitmap%2FRoaringBitmap/lists"}