{"id":16572705,"url":"https://github.com/lemire/javaewah","last_synced_at":"2025-05-14T13:07:56.271Z","repository":{"id":43023165,"uuid":"961417","full_name":"lemire/javaewah","owner":"lemire","description":"A compressed alternative to the Java BitSet class","archived":false,"fork":false,"pushed_at":"2023-08-28T18:12:47.000Z","size":2377,"stargazers_count":564,"open_issues_count":1,"forks_count":112,"subscribers_count":43,"default_branch":"master","last_synced_at":"2025-04-11T06:11:21.707Z","etag":null,"topics":["bitmap-indexes","bitset","ewah","java"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lemire.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2010-10-04T18:49:09.000Z","updated_at":"2025-03-20T05:37:17.000Z","dependencies_parsed_at":"2024-06-19T01:49:59.585Z","dependency_job_id":"7fb55006-6952-4a5b-9d33-edf65b66301f","html_url":"https://github.com/lemire/javaewah","commit_stats":null,"previous_names":[],"tags_count":81,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fjavaewah","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fjavaewah/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fjavaewah/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lemire%2Fjavaewah/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lemire","download_url":"https://codeload.github.com/lemire/javaewah/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254149959,"owners_count":22022851,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitmap-indexes","bitset","ewah","java"],"created_at":"2024-10-11T21:28:19.975Z","updated_at":"2025-05-14T13:07:51.240Z","avatar_url":"https://github.com/lemire.png","language":"Java","readme":"JavaEWAH\n==========================================================\n[![Java CI](https://github.com/lemire/javaewah/actions/workflows/basic.yml/badge.svg)](https://github.com/lemire/javaewah/actions/workflows/basic.yml)\n[![][maven img]][maven]\n[![][license img]][license]\n[![docs-badge][]][docs]\n[![Coverage Status](https://coveralls.io/repos/lemire/javaewah/badge.svg?branch=master)](https://coveralls.io/r/lemire/javaewah?branch=master)\n\n\n\nThis is a word-aligned compressed variant of\nthe Java Bitset class. We provide both a 64-bit\nand a 32-bit RLE-like compression scheme. It can\nbe used to implement bitmap indexes. The EWAH format\nit relies upon is used in the git implementation \nthat runs GitHub.\n\nThe goal of word-aligned compression is not to\nachieve the best compression, but rather to\nimprove query processing time. Hence, we try\nto save CPU cycles, maybe at the expense of\nstorage. However, the EWAH scheme we implemented\nis always more efficient storage-wise than an\nuncompressed bitmap (as implemented in the java\nBitSet class by Sun).\n\nJavaEWAH offers competitive speed. In an exhaustive\ncomparison, Guzun et al. (ICDE 2014) found that \"EWAH\noffers the best query time for all distributions.\"\n\nJavaEWAH also supports memory-mapped files: we can\nserialize the bitmaps to disk and then map them to\nmemory using the java.nio classes. This may avoid\nwasteful serialization/deserialization routines.\n\nThe library also provides a drop-in replacement for\nthe standard BitSet class. Like the other bitmap classes\nin JavaEWAH, this uncompressed BitSet class supports\nmemory-mapped files as well as many other conveniences.\n\nFor better performance, use a 64-bit JVM over\n64-bit CPUs when using the 64-bit scheme (javaewah.EWAHCompressedBitmap).\nThe 32-bit version (javaewah32.EWAHCompressedBitmap32) should\ncompress better but be comparatively slower. It is recommended however that you run your own benchmark.\n\n\n\nJava 6 or better is required. We found the very latest OpenJDK release\noffered the best performance.\n\nReal-world usage\n----------------\n\nJavaEWAH is part of Apache Hive and its derivatives (e.g.,  Apache Spark) and Eclipse JGit. It has been used in production systems for many years. It is part of major Linux distributions. It is part of [Twitter algebird](https://github.com/twitter/algebird).\n\n\nEWAH is used to accelerate the distributed version control system Git (http://githubengineering.com/counting-objects/). You can find the C port of EWAH written by the Git team at https://github.com/git/git/tree/master/ewah\n\nWhen should you use a bitmap?\n----------------------------------------\n\nSets are a fundamental abstraction in\nsoftware. They can be implemented in various\nways, as hash sets, as trees, and so forth.\nIn databases and search engines, sets are often an integral\npart of indexes. For example, we may need to maintain a set\nof all documents or rows  (represented by numerical identifier)\nthat satisfy some property. Besides adding or removing\nelements from the set, we need fast functions\nto compute the intersection, the union, the difference between sets, and so on.\n\n\nTo implement a set\nof integers, a particularly appealing strategy is the\nbitmap (also called bitset or bit vector). Using n bits,\nwe can represent any set made of the integers from the range\n[0,n): it suffices to set the ith bit is set to one if integer i is present in the set.\nCommodity processors use words of W=32 or W=64 bits. By combining many such words, we can\nsupport large values of n. Intersections, unions and differences can then be implemented\n as bitwise AND, OR and ANDNOT operations.\nMore complicated set functions can also be implemented as bitwise operations.\n\nWhen the bitset approach is applicable, it can be orders of\nmagnitude faster than other possible implementation of a set (e.g., as a hash set)\nwhile using several times less memory.\n\nHowever, a bitset, even a compressed one is not always applicable. For example, if \nyou have 1000 random-looking integers, then a simple array might be the best representation.\nWe refer to this case as the \"sparse\" scenario.\n\nWhen should you use compressed bitmaps?\n----------------------------------------\n\nAn uncompressed BitSet can use a lot of memory. For example, if you take a BitSet\nand set the bit at position 1,000,000 to true and you have just over 100kB. That's over 100kB\nto store the position of one bit. This is wasteful  even if you do not care about memory:\nsuppose that you need to compute the intersection between this BitSet and another one\nthat has a bit at position 1,000,001 to true, then you need to go through all these zeroes,\nwhether you like it or not. That can become very wasteful.\n\nThis being said, there are definitively cases where attempting to use compressed bitmaps is wasteful.\nFor example, if you have a small universe size. E.g., your bitmaps represent sets of integers\nfrom [0,n) where n is small (e.g., n=64 or n=128). If you can use an uncompressed BitSet and\nit does not blow up your memory usage,  then compressed bitmaps are probably not useful\nto you. In fact, if you do not need compression, then a BitSet offers remarkable speed.\nOne of the downsides of a compressed bitmap like those provided by JavaEWAH is slower random access:\nchecking whether a bit is set to true in a compressed bitmap takes longer.\n\nThe sparse scenario is another use case where compressed bitmaps should not be used.\nKeep in mind that random-looking data is usually not compressible. E.g., if you have a small set of\n32-bit random integers, it is not mathematically possible to use far less than 32 bits per integer,\nand attempts at compression can be counterproductive.\n\nHow does EWAH compares with the alternatives?\n-------------------------------------------\n\nEWAH is part of a larger family of compressed bitmaps that are run-length-encoded\nbitmaps. They identify long runs of 1s or 0s and they represent them with a marker word.\nIf you have a local mix of 1s and 0, you use an uncompressed word.\n\nThere are many formats in this family beside EWAH:\n\n* Oracle's BBC is an obsolete format at this point: though it may provide good compression,\nit is likely much slower than more recent alternatives due to excessive branching.\n* WAH is a patented variation on BBC that provides better performance.\n* Concise is a variation on the patented WAH. It some specific instances, it can compress\nmuch better than WAH (up to 2x better), but it is generally slower.\n* EWAH is both free of patent, and it is faster than all the above. On the downside, it\ndoes not compress quite as well. It is faster because it allows some form of \"skipping\"\nover uncompressed words. So though none of these formats are great at random access, EWAH\nis better than the alternatives.\n\nThere are other alternatives however. For example, the Roaring\nformat (https://github.com/lemire/RoaringBitmap) is not a run-length-encoded hybrid. It provides faster random access\nthan even EWAH.\n\n\nData format\n------------\n\nFor more details regarding the compression format, please\nsee Section 3 of the following paper:\n\nDaniel Lemire, Owen Kaser, Kamel Aouiche, [Sorting improves word-aligned bitmap indexes](http://arxiv.org/abs/0901.3751). Data \u0026 Knowledge Engineering 69 (1), pages 3-28, 2010.  \n \n\nBenchmark\n---------\n\nFor a simple comparison between this library and other libraries such as\nWAH, ConciseSet, BitSet and other options, please see\n\nhttps://github.com/lemire/simplebitmapbenchmark\n\nHowever, this is very naive. It is recommended that you run your own benchmarks.\n\nUnit testing\n------------\n\nAs of October 2011, this package relies on Maven. To\ntest it:\n\n```\nmvn test\n```\n\nSee\nhttp://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html\nfor details.\n\nWe support Java 8 and up, but to build the library, you need Java 9 and up\nwith the default Maven setup, since we rely on the `--release` flag functionality\n(unavailable in Java 8) to sidestep the [Deaded-NoSuchMethodError issue](https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/).\n\n\n\nUsage\n-----\n\n```java\n        EWAHCompressedBitmap ewahBitmap1 = EWAHCompressedBitmap.bitmapOf(0, 2, 55, 64, 1 \u003c\u003c 30);\n        EWAHCompressedBitmap ewahBitmap2 = EWAHCompressedBitmap.bitmapOf(1, 3, 64,\n                1 \u003c\u003c 30);\n        System.out.println(\"bitmap 1: \" + ewahBitmap1);\n        System.out.println(\"bitmap 2: \" + ewahBitmap2);\n        // or\n        EWAHCompressedBitmap orbitmap = ewahBitmap1.or(ewahBitmap2);\n        System.out.println(\"bitmap 1 OR bitmap 2: \" + orbitmap);\n        System.out.println(\"memory usage: \" + orbitmap.sizeInBytes() + \" bytes\");\n        // and\n        EWAHCompressedBitmap andbitmap = ewahBitmap1.and(ewahBitmap2);\n        System.out.println(\"bitmap 1 AND bitmap 2: \" + andbitmap);\n        System.out.println(\"memory usage: \" + andbitmap.sizeInBytes() + \" bytes\");\n        // xor\n        EWAHCompressedBitmap xorbitmap = ewahBitmap1.xor(ewahBitmap2);\n        System.out.println(\"bitmap 1 XOR bitmap 2:\" + xorbitmap);\n        System.out.println(\"memory usage: \" + xorbitmap.sizeInBytes() + \" bytes\");\n        // fast aggregation over many bitmaps\n        EWAHCompressedBitmap ewahBitmap3 = EWAHCompressedBitmap.bitmapOf(5, 55,\n                1 \u003c\u003c 30);\n        EWAHCompressedBitmap ewahBitmap4 = EWAHCompressedBitmap.bitmapOf(4, 66,\n                1 \u003c\u003c 30);\n        System.out.println(\"bitmap 3: \" + ewahBitmap3);\n        System.out.println(\"bitmap 4: \" + ewahBitmap4);\n        andbitmap = EWAHCompressedBitmap.and(ewahBitmap1, ewahBitmap2, ewahBitmap3,\n                ewahBitmap4);\n        System.out.println(\"b1 AND b2 AND b3 AND b4: \" + andbitmap);\n        // serialization\n        ByteArrayOutputStream bos = new ByteArrayOutputStream();\n        // Note: you could use a file output steam instead of ByteArrayOutputStream\n        ewahBitmap1.serialize(new DataOutputStream(bos));\n        EWAHCompressedBitmap ewahBitmap1new = new EWAHCompressedBitmap();\n        byte[] bout = bos.toByteArray();\n        ewahBitmap1new.deserialize(new DataInputStream(new ByteArrayInputStream(bout)));\n        System.out.println(\"bitmap 1 (recovered) : \" + ewahBitmap1new);\n        if (!ewahBitmap1.equals(ewahBitmap1new)) throw new RuntimeException(\"Will not happen\");\n        //\n        // we can use a ByteBuffer as backend for a bitmap\n        // which allows memory-mapped bitmaps\n        //\n        ByteBuffer bb = ByteBuffer.wrap(bout);\n        EWAHCompressedBitmap rmap = new EWAHCompressedBitmap(bb);\n        System.out.println(\"bitmap 1 (mapped) : \" + rmap);\n\n        if (!rmap.equals(ewahBitmap1)) throw new RuntimeException(\"Will not happen\");\n        //\n        // support for threshold function (new as of version 0.8.0):\n        // mark as true a bit that occurs at least T times in the source\n        // bitmaps\n        //\n        EWAHCompressedBitmap threshold2 = EWAHCompressedBitmap.threshold(2,\n                ewahBitmap1, ewahBitmap2, ewahBitmap3, ewahBitmap4);\n        System.out.println(\"threshold 2 : \" + threshold2);\n```\nSee example.java.\n\nYou can use our drop-in replacement for the BitSet class in a memory-mapped file\ncontext as follows:\n\n```java\n\t\tfinal FileOutputStream fos = new FileOutputStream(tmpfile);\n\t\tBitSet Bitmap = BitSet.bitmapOf(0, 2, 55, 64, 512);\n\t\tBitmap.serialize(new DataOutputStream(fos));\n\t\tRandomAccessFile memoryMappedFile = new RandomAccessFile(tmpfile, \"r\");\n\t\tByteBuffer bb = memoryMappedFile.getChannel().map(\n\t\t\t\tFileChannel.MapMode.READ_ONLY, 0, totalcount);\n\t\tImmutableBitSet mapped = new ImmutableBitSet(bb.asLongBuffer());\n```\n\n\nThere are more examples in the \"examples\" folder (e.g.,\nfor memory-file mapping).\n\n\nMaven central repository\n------------------------\n\nYou can download JavaEWAH from the Maven central repository:\nhttp://repo1.maven.org/maven2/com/googlecode/javaewah/JavaEWAH/\n\nYou can also specify the dependency in the Maven \"pom.xml\" file:\n\n```xml\n    \u003cdependencies\u003e\n         \u003cdependency\u003e\n\t     \u003cgroupId\u003ecom.googlecode.javaewah\u003c/groupId\u003e\n\t     \u003cartifactId\u003eJavaEWAH\u003c/artifactId\u003e\n\t     \u003cversion\u003e[1.1,)\u003c/version\u003e\n         \u003c/dependency\u003e\n     \u003c/dependencies\u003e\n```\n\nNaturally, you should replace \"version\" by the version\nyou desire.\n\nUbuntu (Linux)\n------------------\n\nTo install javaewah on Ubuntu, type:\n\n          sudo apt-get install libjavaewah-java\n\nClojure\n-------\n\nJoel Boehland wrote Clojure wrappers:\n\nhttps://github.com/jolby/clojure-ewah-bitmap\n\nFrequent questions\n------------------\n\nQuestion: How do I build javaewah without testing or signing?\n\n        mvn clean install -DskipTests -Dgpg.skip=true\n\nQuestion: Will JavaEWAH support long values?\n\nAnswer: It might, but it does not at the moment.\n\nQuestion: How do I check the value of a bit?\n\nAnswer: If you need to routinely check the value of a given bit quickly, then\nEWAH might not be the right format. However, if you must do it, you can proceed as\nfollows:\n```java\n          /**\n           * Suppose you have the following bitmap:\n           */\n          EWAHCompressedBitmap b = EWAHCompressedBitmap.bitmapOf(0,2,64,1\u003c\u003c30);\n          /**\n           * We want to know if bit 64 is set:\n           */\n          boolean is64set = b.get(64);\n```\n\nAPI Documentation\n-----------------\n\nhttp://www.javadoc.io/doc/com.googlecode.javaewah/JavaEWAH/\n\nMailing list and discussion group\n---------------------------------\n\nhttps://groups.google.com/forum/#!forum/javaewah\n\n\nFurther reading\n---------------\n\n- Daniel Lemire, Owen Kaser, Kamel Aouiche, [Sorting improves word-aligned bitmap indexes](http://arxiv.org/abs/0901.3751), Data \u0026 Knowledge Engineering 69 (1), 2010.\n- Owen Kaser and Daniel Lemire, [Compressed bitmap indexes: beyond unions and intersections](http://arxiv.org/abs/1402.4466), Software: Practice and Experience 46 (2), 2016. \n\n\nCredit\n--------\n\n\n(c) 2009-2023\n[Daniel Lemire](http://lemire.me/en/), Cliff Moon, [David McIntosh](https://github.com/mctofu), [Robert Becho](https://github.com/RBecho), [Colby Ranger](https://github.com/crangeratgoogle), [Veronika Zenz](https://github.com/veronikazenz), [Owen Kaser](https://github.com/owenkaser), [Gregory Ssi-Yan-Kai](https://github.com/gssiyankai), and [Rory Graves](https://github.com/rorygraves)\n\n\nThis code is licensed under Apache License, Version 2.0 (ASL2.0).\n(GPL 2.0 derivatives are allowed.)\n\nAcknowledgement\n---------------\n\nSpecial thanks to Shen Liang for optimization advice.\n\nThis work was supported by NSERC grant number 26143.\n\n[maven img]:https://maven-badges.herokuapp.com/maven-central/com.googlecode.javaewah/JavaEWAH/badge.svg\n[maven]:http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.googlecode.javaewah%22%20\n\n[license]:LICENSE-2.0.txt\n[license img]:https://img.shields.io/badge/License-Apache%202-blue.svg\n\n\n[docs-badge]:https://img.shields.io/badge/API-docs-blue.svg?style=flat-square\n[docs]:http://www.javadoc.io/doc/com.googlecode.javaewah/JavaEWAH/\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fjavaewah","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flemire%2Fjavaewah","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flemire%2Fjavaewah/lists"}