{"id":22712059,"url":"https://github.com/dynatrace-oss/index4j","last_synced_at":"2025-04-12T13:51:19.782Z","repository":{"id":257383209,"uuid":"816165068","full_name":"dynatrace-oss/index4j","owner":"dynatrace-oss","description":"Dynatrace FM-Index library","archived":false,"fork":false,"pushed_at":"2025-02-20T11:45:10.000Z","size":3561,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T08:23:49.412Z","etag":null,"topics":["benchmarks","bit-vector","bitvector-library","burrows-wheeler-transform","fm-index","fm-index-java","index","java","log-processing","logs","rank-queries","string-index","string-indexing","string-matching","string-processing","succinct","succinct-data-structure","system-logs","wavelet-tree","wavelet-tree-java"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dynatrace-oss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-17T07:04:06.000Z","updated_at":"2025-03-07T03:31:39.000Z","dependencies_parsed_at":"2024-09-16T11:51:24.052Z","dependency_job_id":"638f4359-2130-44e0-9231-b6e4cb9e4b37","html_url":"https://github.com/dynatrace-oss/index4j","commit_stats":null,"previous_names":["dynatrace-oss/index4j"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynatrace-oss%2Findex4j","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynatrace-oss%2Findex4j/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynatrace-oss%2Findex4j/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dynatrace-oss%2Findex4j/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dynatrace-oss","download_url":"https://codeload.github.com/dynatrace-oss/index4j/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248575693,"owners_count":21127247,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","bit-vector","bitvector-library","burrows-wheeler-transform","fm-index","fm-index-java","index","java","log-processing","logs","rank-queries","string-index","string-indexing","string-matching","string-processing","succinct","succinct-data-structure","system-logs","wavelet-tree","wavelet-tree-java"],"created_at":"2024-12-10T13:09:16.391Z","updated_at":"2025-04-12T13:51:19.775Z","avatar_url":"https://github.com/dynatrace-oss.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"plots/logo.png\" alt=\"logo\" width=\"50\"/\u003e index4j\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n[![Maven Central Version](https://img.shields.io/maven-central/v/com.dynatrace.index4j/indices)](https://central.sonatype.com/artifact/com.dynatrace.index4j/indices)\n[![javadoc](https://javadoc.io/badge2/com.dynatrace.index4j/indices/javadoc.svg)](https://javadoc.io/doc/com.dynatrace.index4j/indices)\n\nThis repository is a Java library developed by Dynatrace that implements the FM-Index succinct data structure. The FM-Index takes \nadvantage of the relationship between the suffix array and the Burrows-Wheeler transform to enable both compression and\nfast queries. \n\nThis index is specifically suited for compressing and querying text files without requiring prior tokenization. \nThis means that you can compress a text file and then query it for arbitrary substrings without the need to fully \ndecompress it, i.e., only the compressed size is required to be in memory. Furthermore, query performance depends on \nthe number of located matches and total decompressed bytes, rather than whether the query is aligned with a particular\ntoken.\n\nIn addition, this repository also contains further data structures for working with bit vectors and integer sequences in Java. \n\n## Content\n\n- [First steps](#first-steps)\n- [Supported data structures](#supported-data-structures)\n- [How-to and example use cases](#how-to)\n- [Benchmarks](#benchmarks)\n- [Details about the data structures](#details)\n- [About this project](#about-this-project)\n\n## First steps\n\nTo add a dependency on `index4j` using Maven, use the following:\n\n```\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.dynatrace.index4j\u003c/groupId\u003e\n    \u003cartifactId\u003eindices\u003c/artifactId\u003e\n    \u003cversion\u003e0.3.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nTo add the dependency using Gradle:\n\n```\nimplementation group: 'com.dynatrace.index4j', name: 'indices', version: '0.3.0'\n```\n\nThe current minimum Java version should be 17.\n\n## Supported data structures\n\n- [Fm-Index](indices/src/main/java/com/dynatrace/fm/FmIndex.java): supports `count`, `locate`, `extract` and `extractUntilBoundary` (along with its left and right variations) with arbitrary patterns while compressing your data. In the case of logs, roughly 20% of the input size is required to store both data and index.\n- [RrrVector](indices/src/main/java/com/dynatrace/bitsequence/RrrVector.java) (Raman, Raman \u0026 Rao): compressed bit vectors which both compress and enables rank and select queries.\n- [SuffixArray](indices/src/main/java/com/dynatrace/suffixarray/SuffixArray.java): supports querying arbitrary patterns on input text requiring `O(n logn)` time and `O(5n)` space.\n- [Wavelet FBB](indices/src/main/java/com/dynatrace/wavelet/WaveletFixedBlockBoosting.java): Fast wavelet trees that compress and support rank and select queries on arbitrary sequences.\n- [Burrows-Wheeler transform](indices/src/main/java/com/dynatrace/encoding/BurrowsWheelerTransform.java): enables calculating the BWT of inputs as well as compute redundancy metrics.\n\n## How-to\n\n### Using the FM-Index\n\nThe following examples show how to make use of the provided FM-Index implementation. Note that both compression and \nquery speed are controlled by the `sample rate` parameter (find more details [here](#details)). In essence, a larger \n`sample rate` results in better compression but slower query performance, and viceversa. Note that \"larger\" here means \n\"larger distance\" between samples, and therefore, actually translates into fewer samples.\n\n#### Creating an FM-Index\n\nThe following snippet creates an FM-Index for a char array `text` using a sample rate of `32` and enabling\nthe extraction/decompression of the index.\n\n```java\nchar[] text = ...\nFmIndex fmi = new FmIndexBuilder()\n        .setSampleRate(32)\n        .setEnableExtraction(true)\n        .build(text);\n```\n\nTo get some statistics, we can do the following:\n```java\nSystem.out.println(fmi.getAlphabetLength());                            // how many symbols\nSystem.out.println(fmi.getInputLength());                               // the length of the input `text`\nint size = Serialization.writeToByteArray(FmIndex::write, fmi).length;  // actual compressed size\nSystem.out.println((size * 100 / fmi.getInputLength()) + \"%)\");         // relative size to the input\n```\n\nNote that the original input string is no longer required to perform any query at all, and in fact the whole input \ncan be retrieved back by decompressing or extracting between `0` and `|text|`.\n\n#### Counting occurrences of a pattern\n\nCount how many times the pattern ` DEBUG ` (with spaces before and after) is in the input data.\n```java\nchar[] pattern = \" DEBUG \".toCharArray();\nSystem.out.println(fmi.count(pattern));\n```\n\n#### Locating the positions of an occurrence\n\nUsing the previous pattern, find at most 10 matches and their position in the original input.\n```java\nint[] locations = new int[10];\nint found = fmi.locate(pattern, 0, pattern.length, locations, 10); // find max. 10 matches\nSystem.out.println(found); // print number of matches\nSystem.out.println(Arrays.toString(locations)); // print the locations of the matches\n```\n\n#### Extracting key-value pairs\n\nUsing the previously found matches and locations, extract the right-most string containing the match and until a comma `','` is found.\n```java\nchar[] destination = new char[100]; // maximum length per extraction\nfor (int i = 0; i \u003c found; i++) {\n    int length = fmi.extractUntilBoundaryRight(locations[i] - 1, destination, 0, ','); // extract previous locations until finding a comma\n    System.out.println(new String(destination, 0, length)); // print the extracted string\n}\n```\n\n#### Extracting whole records\n\nExtracting the full string, left and right from every match until a line separator `'\\n'` is found.\n```java\nchar[] destination = new char[512]; // maximum length per extraction \nfor (int i = 0; i \u003c found; i++) {\n    int length = fmi.extractUntilBoundary(locations[i], destination, 0, '\\n');\n    System.out.println(new String(destination, 0, length));\n}\n```\n\n#### Serializing\n\nAll data structures (except the naive `WaveletTree`) can be serialized and deserialized\nvia the [SerializationReader](indices/src/main/java/com/dynatrace/serialization/SerializationReader.java) and\nthe [SerializationWriter](indices/src/main/java/com/dynatrace/serialization/SerializationWriter.java), respectively.\n\nYou can do so as shown below:\n\n```java\nFmIndex fmi = new FmIndexBuilder().build(...);\nbyte[] serialized    = Serialization.writeToByteArray(FmIndex::write, fmi);\nFmIndex deserialized = Serialization.readFromByteArray(FmIndex::read, serialized);\n```\n\n### Using RRR Bit vectors\n\nSimilarly to the FM-Index, RRR bit vectors also make use of the sample rate. A higher sample rate results in using less\nspace but at the expense of slower rank and access queries. In particular, a sample rate of e.g. `32`\nwill result in saving a prefix sum every 32 bits, as well as saving the position of offsets\nevery 32 bits. Therefore, in the worst case, up to `32/BLOCK_SIZE` blocks need to be traversed before finding\nthe rank or access answer.\n\n#### Creating RRR Bit vectors\n\nWe have two options to create RRR bit vectors: either through an array of integers, which we interpret as a sequence\nof bits (e.g. if the int value is `2` then we have a bit sequence of `010...000000`) or creating it through an already\nexisting `BitVector` from the `sux4j` library. For example, using a sequence of integers and a sample rate of `32`:\n\n```java\nint[] array = new Random().ints(1000).toArray();\nRrrVector rrr = new RrrVector(array, 32);\n```\n\nOr from a `BitVector`:\n\n```java\nint lengthInBits = 1000;\nBitVector bv = LongArrayBitVector.getInstance().length(lengthInBits);\n// Fill in some values\nRandom random = new Random();\nfor (int i = 0; i \u003c lengthInBits; i++) {\n    bv.set(i, random.nextBoolean());\n}\n// Create it\nRrrVector rrr = new RrrVector(bv, 32);\n```\n\n#### Ranking zeros or ones\n\nThe following snippet shows how to count the number of 1's or 0's until a given position (exclusive).\n\n```java\nint[] array = ...\nRrrVector rrr = new RrrVector(array, 32);\nSystem.out.println(rrr.rankOnes(10)); // Number of 1's until position 10\nSystem.out.println(rrr.rankZeroes(50)); // Number of 0's until position 50\n```\n\n#### Access an arbitrary position\n\nGet the value of the bit at position `i`:\n\n```java\nint[] array = ...\nRrrVector rrr = new RrrVector(array, 32);\nSystem.out.println(rrr.access(7)); // Get the bit value as a boolean (true or false) of the bit sequence at position 7\n```\n\n#### Serializing\n\nSimilar to the FM-Index, we can serialize and deserialize our RRR bit vectors as follows:\n\n```java\nint[] array = ...\nRrrVector rrr = new RrrVector(array, 32);\nbyte[] serialized      = Serialization.writeToByteArray(RrrVector::write, rrr);\nRrrVector deserialized = Serialization.readFromByteArray(RrrVector::read, serialized);\n```\n\n### Using Wavelet trees\n\nCurrently, two implementations of the wavelet tree are supported. However, only the \n[Fixed-block boosting wavelet tree](indices/src/main/java/com/dynatrace/wavelet/WaveletFixedBlockBoosting.java) should be\nused since its performance and memory consumption is much better than the vanilla wavelet tree implementation.\n\nThe Fixed-block boosting wavelet tree (FBB-WT in advance) also uses the `sample rate` parameter since it is composed of multiple RRR bit\nvectors. \n\n#### Creating FBB-WT\n\nThe FBB-WT can be built from either a sequence of chars or shorts. In any case, both are interpreted as symbols and should\nbe mapped to a sequence `{0,1,2, ..., n}` such that `v_{i+1} = v_{i} + 1` for all `i`. This allows for a larger alphabet,\nup to `32,768` different symbols, whereas otherwise if there is a single symbol with a value above `32,768` it will fail.\nThe following snippet shows how to create an FBB-WT without such mapping:\n\n```java\nchar[] text = \"This is an example\".toCharArray();\nWaveletFixedBlockBoosting wavelet = new WaveletFixedBlockBoosting(text);\n```\n\nAnd the following includes a mapping:\n\n```java\nchar[] text = \"Los erizos pasean junto a la torre\".toCharArray();\n\nMap\u003cCharacter, Short\u003e map = new HashMap\u003c\u003e();\nshort code = 0;\nfor (char c : text)\n    if (map.putIfAbsent(c, code) == null) ++code;\n\nshort[] mappedSequence = new short[text.length];\nfor (int i = 0; i \u003c text.length; i++)\n    mappedSequence[i] = map.get(text[i]);\n\nWaveletFixedBlockBoosting wavelet = new WaveletFixedBlockBoosting(mappedSequence);\n```\n\n#### Ranking symbols\n\nBy generalizing the bit vector structure, the FBB-WT is able to rank any symbol:\n\n```java\nchar[] text = ...\nWaveletFixedBlockBoosting wavelet = new WaveletFixedBlockBoosting(text);\nlong howManyZs = wavelet.rank(60, map.get('z')); // counts how many times the letter 'z' appears before position 60\n```\n\n#### Accessing symbols\n\nWe can also retrieve the original symbol at position `i`:\n\n```java\nchar[] text = ...\nint i = 33;\nWaveletFixedBlockBoosting wavelet = new WaveletFixedBlockBoosting(text);\nlong ithAndSymbol = wavelet.inverseSelect(i); \n// Note that this long contains two ints: the first one is the number of the occurrence and the second one is the symbol value\nassert text[i] == (char) ithAndSymbol; // by casting it to char we are taking the symbol bytes only\n```\n\n#### Serializing\n\nSame approach as in previous occasions:\n\n```java\nchar[] text = ...\nWaveletFixedBlockBoosting wavelet = new WaveletFixedBlockBoosting(text);\n\nbyte[] serialized = Serialization.writeToByteArray(WaveletFixedBlockBoosting::write, wavelet);\nWaveletFixedBlockBoosting deserialized = Serialization.readFromByteArray(WaveletFixedBlockBoosting::read, serialized);\n```\n\n## Benchmarks\n\nThe [jmh package](indices/src/jmh/java/com/dynatrace) contains ready-to-run benchmarks for the FM-Index, the RRR bit\nvector and the Suffix Array. Most of the classes contain results which were run with the `Android.log` dataset of the\npublic repository [loghub](https://zenodo.org/records/3227177/files/Android.tar.gz?download=1). The benchmarks are divided into three types:\n- `Ingest`: Measures how long it takes to create the data structure given the input.\n- `Throughput`: Measures the query performance of the data structure as function of the sample rate.\n- `SerializedSize`: Measures the size of the data structure once it is serialized.\n\nThe additional `state` type simply holds the state necessary to run the benchmarks.\nThe benchmarks `jar` can be built with the following command:\n\n`./gradlew clean jmhJar`\n\nAnd then run as follows:\n\n`java -jar indices/build/libs/benchmarks.jar \u003cregex matching name of benchmark\u003e -p parameter1=value1 -p parameter2=value2 ...`\n\nFor example:\n\n`java -jar indices/build/libs/benchmarks.jar FmIndexSerializedSizeBenchmark -p data=Android.log -p sampleRate=8,16,32,64`\n\n### Infrastructure\nThe following benchmarks were run with 3 forks, each one running 5 warmup iterations and 10 measurement iterations.\nThey were run on a `Xeon W-10885@2.40GHz` with disabled Turbo Boost and 64 GB of RAM. Note that a more automated \napproach for benchmarking is intended for future work.\n\n### Trade off plots between speed and space\n\nThe following plots summarize the general behavior of the FM-Index data structure for the locate and extract queries.\nThe `Android.log` dataset of the public repository [loghub](https://zenodo.org/records/3227177/files/Android.tar.gz?download=1) was used.\nIt contains exactly `1,555,005` lines, totalling `184` MB of data with an alphabet containing over `1,000` different symbols (letters).\n\n#### Locate queries\n\n\u003cimg src=\"plots/locate.png\" alt=\"locate-query\" width=\"600\"/\u003e\n\nThe plot depicts the number of microseconds required for locating a variable number of matches (either 1, 10, 100 or 1,000)\nas a function of the size, which depends on the value of the `sample rate`. The red dotted line in the zoomed-in region\nshows the performance that can be achieved if using a `sample rate` that results in roughly 25% of the original input size\n(including both index and data).\n\nAs can be seen, locate queries can be done in microseconds time depending on sample rate and output size,\nfor example, at 25% of the original size (achieved approximately with a sample rate of 64) gets us to 100 microseconds per output match.\nThe higher the number of output matches, the better the per-match average location time, e.g., for 100 matches it goes down to ~25 microseconds per match.\n\n#### Extract/decompress queries\n\n\u003cimg src=\"plots/extract.png\" alt=\"extract-query\" width=\"600\"/\u003e\n\nSimilarly to locate queries, the extract/decompress queries are also affected by sample rate.\nAt a 25% size (or sample rate of 64), we can extract at a rate of one symbol per ~2 microseconds.\nThis translates into the fact that if we need to extract a full\nlog record of ~300 characters, we can expect roughly ~300 to ~600 microseconds, or half a millisecond. On the contrary,\nif working with key/value pairs of e.g. size 30 characters, then ~30 to ~60 microseconds per extraction is reasonable.\n\n#### Compression evaluation\n\n\u003cimg src=\"plots/compression.png\" alt=\"compression\" width=\"600\"/\u003e\n\nHow much can we compress the FM-Index while still being queryable depending on the sample rate?\nFirst, when the size becomes asymptotic it is worth noting that the queries become much slower. However, this can be\nstill be useful if for example the use case requires higher compression and only sporadic querying.\nFor reference, the close to theoretical compression limit with Z-standard is included, at two levels of compression.\nNotice that the FM-Index is the index as well (not just data, as opposed to the Z-standard compression).\nIt can be seen that the FM-Index requires around ~4 times the size of the Z-standard compressed file (which\nalready represents only 25% of the original file size) when\nused with reasonable sample sizes (such as 64 or 128), which still offer high speed queries. Nevertheless,\nwhen reaching the compression limits at a sample rate of ~1024 or higher, the FM-Index only requires ~2 times\nthe space.\n\n#### Index construction runtime\n\nThe construction time of the FM-Index depends on building multiple other data structures which, although all linear\nin runtime, still contribute towards a larger constant build time. In this line, you can expect the provided \nimplementation of the FM-Index to process inputs at roughly 2 MB/s \n(see the [benchmark](indices/src/jmh/java/com/dynatrace/fm/FmIndexIngestBenchmark.java) for more details). This\ntranslates into a 200 MB input text requiring approximately a minute and a half to build the index. The \nconstruction time is also affected by the `sample rate` but to a much smaller degree in comparison with\nthe compression or query performance, and can be almost ignored for practical purposes. Luckily,\nbuilding the index is required only once per input.\n\n## Details\n\n### FM-Index\nAn FM-Index is a compressed full-text substring index based on suffix arrays, bit vectors, wavelet trees and the\nBurrows–Wheeler transform. It can be used to find the number of occurrences of a pattern without prior tokenization\nwithin the compressed text, as well as locate the position of each occurrence and extract (decompress) the matched strings \nor their vicinity. The query time and the required storage space have a sublinear complexity with respect to the size of \nthe input data. It is used extensively in the world of bioinformatics.\n\nSupported operations:\n- `count` enables to count the number of occurrences of a given pattern. This is extremely fast.\n- `locate` enables to find all the leftmost starting positions of a given pattern. Its speed will depend on the parameter `sample rate` which trades off space for speed. A `sample rate` of `4` means to store an additional integer every four input symbols but also means that up to `4` additional rank queries will be necessary per query. Therefore, more position integers require more space but reduce the number of searches.\n- `extract` enables extracting original input text from the compressed index between given positions. Its performance is also affected by the `sample rate`.\n- `extractUntilBoundary` enables extracting a window of the original text delimited by a boundary character (e.g., a record delimiter such as `\\n`)\n\nAs of now, the FM-Index implementation supports up to 32,768 different UTF-8 encoded characters. So far it has been tested\nwith data inputs of around `~400` MB, including general text documents, system logs, system traces, etc.\n\nReferences:\n- [Wikipedia](https://en.wikipedia.org/wiki/FM-index)\n- Ferragina, Paolo, and Giovanni Manzini. [\"Opportunistic data structures with applications.\"](https://doi.org/10.1109/SFCS.2000.892127) In Proceedings 41st annual symposium on foundations of computer science, pp. 390-398. IEEE, 2000.\n\n### Suffix arrays\nA suffix array is a sorted array of all suffixes of an input string. It can be used as index to locate all occurrences of\na given substring `p` by performing two binary searches per pattern. The drawback of a suffix array is its usually large size. \nThe FM-Index, as described above, tackles some of its shortcomings.\n\nThe version used here is a modification of the [JsuffixArray Repository](https://github.com/carrotsearch/jsuffixarrays)\nwhich in turn is a port of the original `divsufsort` implementation (find it [here](https://github.com/y-256/libdivsufsort))\n\n- [Wikipedia](https://en.wikipedia.org/wiki/Suffix_array)\n- [JsuffixArray repository](https://github.com/carrotsearch/jsuffixarrays)\n\n### RRR Bit vectors\nA bit vector is a string of bits, as the name indicates. Without supporting data structures, rank, select or access queries\non the bit strings are very slow (e.g. `O(n)`). The RRR bit vectors are a data structure built on top of the bit string\nthat answers e.g. rank queries in nearly constant time while also compressing it with space requirements close to the\nzero-th entropy.\n\nFor further details see:\n- [RRR bit vector tutorial](https://www.alexbowe.com/rrr/)\n- Raman, Rajeev, Venkatesh Raman, and Srinivasa Rao Satti. \"Succinct indexable dictionaries\n  with applications to encoding k-ary trees, prefix sums and multisets.\" ACM Transactions on Algorithms (TALG) 3, no. 4 (2007): 43-es.\n- Claude, Francisco, and Gonzalo Navarro. \"Practical rank/select queries over arbitrary sequences.\" In International Symposium on String \n  Processing and Information Retrieval, pp. 176-187. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008.\n\n### Wavelet trees / matrix / block-boosting\nThe Wavelet Tree is a succinct data structure to store strings in compressed space.\nIt generalizes the rank, select and access operations defined on bit vectors to arbitrary alphabets. This means that you\ncan perform queries on arbitrary symbols on an arbitrarily long string without requiring to keep the whole\nstring in memory. An example rank query is \"how many times does the symbol `c` appear in string `s` between positions `p1` and `p2`\".\nWith the combination of RRR bit vectors, the wavelet tree can answer e.g. rank queries in `O(log2 |A|)`, where `A` is the alphabet used in the input string.\n\nNote that here the implementation used is the fixed-block boosting wavelet tree which is based on the clever idea that \nbecause we represent the Burrows-Wheeler transform (and not the input corpus itself), it is actually better to build multiple\nwavelet trees over the transform. These are called blocks, and because characters in the transform are\nmore likely to be adjacent to each other (creating the so-called \"BW-Runs\"), the alphabet is usually smaller\nas well, therefore requiring less levels in the tree to represent the transform.\n\nSupported operations:\n- `rank` enables to count the number of occurrences of a symbol `c` between positions `p1` and `p2` in logarithmic time (as opposed to naive `O(p2-p1)` which grows linearly with the difference of positions).\n- `access` enables to retrieve the symbol at position `p`. Remember that the wavelet tree does not store the original input string.\n- `select` answers at which position is the `i-th` appearance of symbol `c`.\n\nReferences:\n- Gog, Simon, Juha Kärkkäinen, Dominik Kempa, Matthias Petri, and Simon J. Puglisi. \"Faster, minuter.\" In 2016 Data Compression Conference (DCC), pp. 53-62. IEEE, 2016.\n- Gog, Simon, Juha Kärkkäinen, Dominik Kempa, Matthias Petri, and Simon J. Puglisi. \"Fixed block compression boosting in FM-indexes: Theory and practice.\" Algorithmica 81 (2019): 1370-1391.\n- [Original implementation](https://github.com/dominikkempa/faster-minuter) by Dominik Kempa\n\n### Burrows-Wheeler transform\n\nThe Burrows-Wheeler transform is a rearrangement of an input string into runs of similar characters. For example, the\nword `^BANANA$` would be rearranged into `BNN^AA$A`, where two `N`s and `A`s are adjacent. In the case of longer strings,\nthis usually means compression can be applied, therefore making it a lossless compression algorithm.\nBecause of its relation to the Suffix array via the LF-Mapping, it is very useful for the FM-Index as a way of navigating \na suffix array without requiring to store all suffixes. It can also be used as a measure of the compressibility or redundancy of a string.\n\nReferences\n- [Wikipedia](https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transform)\n- Manzini, Giovanni. \"An analysis of the Burrows—Wheeler transform.\" Journal of the ACM (JACM) 48, no. 3 (2001): 407-430.\n\n## About this project\n\nPlease note that this product is not officially supported by Dynatrace.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdynatrace-oss%2Findex4j","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdynatrace-oss%2Findex4j","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdynatrace-oss%2Findex4j/lists"}