{"id":21963038,"url":"https://github.com/kampersanda/xcdat","last_synced_at":"2025-10-29T06:05:41.542Z","repository":{"id":93201506,"uuid":"75139204","full_name":"kampersanda/xcdat","owner":"kampersanda","description":"Fast compressed trie dictionary library","archived":false,"fork":false,"pushed_at":"2024-09-20T13:24:10.000Z","size":1187,"stargazers_count":68,"open_issues_count":4,"forks_count":14,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-12T09:10:05.764Z","etag":null,"topics":["compression","cpp17","dictionary","double-array-trie","trie"],"latest_commit_sha":null,"homepage":"https://kampersanda.github.io/xcdat/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kampersanda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-30T01:38:24.000Z","updated_at":"2025-03-20T12:39:21.000Z","dependencies_parsed_at":"2024-11-29T10:59:35.971Z","dependency_job_id":"365601fe-bda1-45d6-9dbb-c5e0cdf601f7","html_url":"https://github.com/kampersanda/xcdat","commit_stats":{"total_commits":122,"total_committers":3,"mean_commits":"40.666666666666664","dds":"0.016393442622950838","last_synced_commit":"2451441dcc09e470dcb3ecd2c1bb304a7dc46347"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kampersanda%2Fxcdat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kampersanda%2Fxcdat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kampersanda%2Fxcdat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kampersanda%2Fxcdat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kampersanda","download_url":"https://codeload.github.com/kampersanda/xcdat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248543848,"owners_count":21121838,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","cpp17","dictionary","double-array-trie","trie"],"created_at":"2024-11-29T10:59:29.884Z","updated_at":"2025-10-29T06:05:41.443Z","avatar_url":"https://github.com/kampersanda.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"#  Xcdat: Fast compressed trie dictionary library\n\n![](https://github.com/kampersanda/xcdat/actions/workflows/cmake.yml/badge.svg)\n\n**Xcdat** is a C++17 header-only library of a fast compressed string dictionary based on an improved double-array trie structure described in the paper: [Compressed double-array tries for string dictionaries supporting fast lookup](https://doi.org/10.1007/s10115-016-0999-8), *Knowledge and Information Systems*, 2017, available [here](https://kampersanda.github.io/pdf/KAIS2017.pdf).\n\n## Table of contents\n\n- [Features](#features)\n- [Build instructions](#build-instructions)\n- [Command line tools](#command-line-tools)\n- [Sample usage](#sample-usage)\n- [API](#api)\n- [Performance](#performance)\n- [Licensing](#licensing)\n- [Todo](#todo)\n- [References](#references)\n\n## Features\n\n- **Compressed string dictionary.** Xcdat implements a (static) *compressed string dictioanry* that stores a set of strings (or keywords) in a compressed space while supporting several search operations [1,2]. For example, Xcdat can store an entire set of English Wikipedia titles at half the size of the raw data. (See [Performance](#performance))\n- **Fast and compact data structure.** Xcdat employs the *double-array trie* [3] known as the fastest trie implementation. However, the double-array trie resorts to many pointers and consumes a large amount of memory. To address this, Xcdat applies the *XCDA* method [2] that represents the double-array trie in a compressed format while maintaining the fast searches.\n- **Cache efficiency.** Xcdat employs a *minimal-prefix trie* [4] that replaces redundant trie nodes into strings to reduce random access and to improve locality of references.\n- **Dictionary encoding.** Xcdat maps `N` distinct keywords into unique IDs from `[0,N-1]`, and supports the two symmetric operations: `lookup` returns the ID corresponding to a given keyword; `decode` returns the keyword associated with a given ID. The mapping is so-called *dictionary encoding* (or *domain encoding*) and is fundamental in many DB applications as described by Martínez-Prieto et al [1] or Müller et al. [5].\n- **Prefix search operations.** Xcdat supports prefix search operations realized by trie search algorithms: `prefix_search` returns all the keywords contained as prefixes of a given string; `predictive search` returns all the keywords starting with a given string. These will be useful in many NLP applications such as auto completions [6], stemmed searches [7], or input method editors [8].\n- **64-bit support.** As mentioned before, since the double array is a pointer-based data structure, most double-array libraries use 32-bit pointers to reduce memory consumption, resulting in limiting the scale of the input dataset. On the other hand, the XCDA method allows Xcdat to represent 64-bit pointers without sacrificing memory efficiency.\n- **Binary key support.** In normal mode, Xcdat will use the `\\0` character as an end marker for each keyword. However, if the dataset include `\\0` characters, it will use bit flags instead of end markers, allowing the dataset to consist of binary keywords.\n- **Memory mapping.** Xcdat supports *memory mapping*, allowing data to be deserialized quickly without loading it into memory. Of course, deserialization by the loading is also supported.\n- **Header only.** The library consists only of header files, and you can easily install it.\n- **Python binding.** You can use Xcdat in Python3 via [pybind11](https://github.com/pybind/pybind11). (Visit the directory [pybind](https://github.com/kampersanda/xcdat/tree/master/pybind))\n\n## Build instructions\n\nYou can download, compile, and install Xcdat with the following commands.\n\n```sh\n$ git clone https://github.com/kampersanda/xcdat.git\n$ cd xcdat\n$ mkdir build\n$ cd build\n$ cmake ..\n$ make -j\n$ make install\n```\n\nOr, since this library consists only of header files, you can easily install it by passing the include path to the directory `include`.\n\n### Requirements\n\nYou need to install a modern C++17 ready compiler such as `g++ \u003e= 7.0` or `clang \u003e= 4.0`. For the build system, you need to install `CMake \u003e= 3.0` to compile the library.\n\nThe library requires that `std::uint64_t` exists. (This is true for nearly any target, even 32-bit ones.) The code has been tested only on Mac OS X and Linux. That is, this library considers only UNIX-compatible OS.\n\n### Python binding\n\nXcdat supports the Python binding via [pybind11](https://github.com/pybind/pybind11). The description can be found in the directory [pybind](https://github.com/kampersanda/xcdat/tree/master/pybind).\n\n## Command line tools\n\n Xcdat provides command line tools to build the dictionary and perform searches, which are inspired by [marisa-trie](https://github.com/s-yata/marisa-trie). All the tools will print the command line options by specifying the parameter `-h`.\n\nThe tools employ the external libraries [cmd_line_parser](https://github.com/jermp/cmd_line_parser), [mm_file](https://github.com/jermp/mm_file), and [tinyformat](https://github.com/c42f/tinyformat), which are contained in the repository.\n\n### `xcdat_build`\n\nIt builds the trie dictionary from a given dataset consisting of keywords separated by newlines. The following command builds the trie dictionary from dataset `enwiki-titles.txt` and writes the dictionary into file `dic.bin`.\n\n```\n$ xcdat_build enwiki-titles.txt dic.bin\nNumber of keys: 15955763\nNumber of trie nodes: 36439320\nNumber of DA units: 36515840\nMemory usage in bytes: 1.64104e+08\nMemory usage in MiB: 156.502\n```\n\n### `xcdat_lookup`\n\nIt tests the `lookup` operation for a given dictionary. Given a query string via `stdin`, it prints the associated ID if found, or `-1` otherwise.\n\n```\n$ xcdat_lookup dic.bin\nAlgorithm\n1255938\tAlgorithm\nDouble_Array\n-1\tDouble_Array\n```\n\n### `xcdat_decode`\n\nIt tests the `decode` operation for a given dictionary. Given a query ID via `stdin`, it prints the corresponding keyword if the ID is in the range `[0,N-1]`, where `N` is the number of stored keywords.\n\n```\n$ xcdat_decode dic.bin\n1255938\n1255938\tAlgorithm\n```\n\n### `xcdat_prefix_search`\n\nIt tests the `prefix_search` operation for a given dictionary. Given a query string via `stdin`, it prints all the keywords contained as prefixes of a given string.\n\n```\n$ xcdat_prefix_search dic.bin\nAlgorithmic\n6 found\n57\tA\n798460\tAl\n1138004\tAlg\n1253024\tAlgo\n1255938\tAlgorithm\n1255931\tAlgorithmic\n```\n\n### `xcdat_predictive_search`\n\nIt tests the `predictive_search` operation for a given dictionary. Given a query string via `stdin`, it prints the first `n` keywords starting with a given string, where `n` is one of the parameters.\n\n```\n$ xcdat_predictive_search dic.bin -n 3\nAlgorithm\n263 found\n1255938\tAlgorithm\n1255944\tAlgorithm's_optimality\n1255972\tAlgorithm_(C++)\n```\n\n### `xcdat_enumerate`\n\nIt prints all the keywords stored in a given dictionary.\n\n```\n$ xcdat_enumerate dic.bin | head -3\n0\t!\n107\t!!\n138\t!!!\n```\n\n### `xcdat_benchmark`\n\nXcdat provides the four dictionary types defined in `xcdat.hpp`. The tool measures the performances of them for a given dataset. To perform search operations, it randomly samples `n` queires from the dataset, where `n` is one of the parameters. It will help you determine the dictionary type.\n\n```\n$ xcdat_benchmark enwiki-titles.txt\n** xcdat::trie_7_type **\nNumber of keys: 15955763\nMemory usage in bytes: 1.70618e+08\nMemory usage in MiB: 162.714\nConstruction time in seconds: 13.501\nLookup time in microsec/query: 0.5708\nDecode time in microsec/query: 1.0846\n** xcdat::trie_8_type **\nNumber of keys: 15955763\nMemory usage in bytes: 1.64104e+08\nMemory usage in MiB: 156.502\nConstruction time in seconds: 13.626\nLookup time in microsec/query: 0.6391\nDecode time in microsec/query: 1.0531\n** xcdat::trie_15_type **\nNumber of keys: 15955763\nMemory usage in bytes: 2.05737e+08\nMemory usage in MiB: 196.206\nConstruction time in seconds: 13.425\nLookup time in microsec/query: 0.3613\nDecode time in microsec/query: 0.7044\n** xcdat::trie_16_type **\nNumber of keys: 15955763\nMemory usage in bytes: 2.15935e+08\nMemory usage in MiB: 205.932\nConstruction time in seconds: 13.704\nLookup time in microsec/query: 0.3832\nDecode time in microsec/query: 0.8362\n```\n\n## Sample usage\n\n`sample/sample.cpp` provides a sample usage.\n\n```c++\n#include \u003ciostream\u003e\n#include \u003cstring\u003e\n\n#include \u003cxcdat.hpp\u003e\n\nint main() {\n    // Dataset of keywords\n    std::vector\u003cstd::string\u003e keys = {\n        \"AirPods\",  \"AirTag\",  \"Mac\",  \"MacBook\", \"MacBook_Air\", \"MacBook_Pro\",\n        \"Mac_Mini\", \"Mac_Pro\", \"iMac\", \"iPad\",    \"iPhone\",      \"iPhone_SE\",\n    };\n\n    // The input keys must be sorted and unique (already satisfied in this case).\n    std::sort(keys.begin(), keys.end());\n    keys.erase(std::unique(keys.begin(), keys.end()), keys.end());\n\n    // The trie dictionary type from the four types\n    using trie_type = xcdat::trie_8_type;\n    // using trie_type = xcdat::trie_16_type;\n    // using trie_type = xcdat::trie_7_type;\n    // using trie_type = xcdat::trie_15_type;\n\n    // The dictionary filename\n    const char* tmp_filename = \"dic.bin\";\n\n    // Build and save the trie dictionary.\n    try {\n        const trie_type trie(keys);\n        xcdat::save(trie, tmp_filename);\n    } catch (const xcdat::exception\u0026 ex) {\n        std::cerr \u003c\u003c ex.what() \u003c\u003c std::endl;\n        return 1;\n    }\n\n    // Load the trie dictionary on memory.\n    const auto trie = xcdat::load\u003ctrie_type\u003e(tmp_filename);\n\n    // Or, you can set the continuous memory block via a memory-mapped file.\n    // const auto trie = xcdat::mmap\u003ctrie_type\u003e(mapped_data);\n\n    // Basic statistics\n    std::cout \u003c\u003c \"Number of keys: \" \u003c\u003c trie.num_keys() \u003c\u003c std::endl;\n    std::cout \u003c\u003c \"Number of trie nodes: \" \u003c\u003c trie.num_nodes() \u003c\u003c std::endl;\n    std::cout \u003c\u003c \"Number of DA units: \" \u003c\u003c trie.num_units() \u003c\u003c std::endl;\n    std::cout \u003c\u003c \"Memory usage in bytes: \" \u003c\u003c xcdat::memory_in_bytes(trie) \u003c\u003c std::endl;\n\n    // Lookup the ID for a query key.\n    {\n        const auto id = trie.lookup(\"Mac_Pro\");\n        std::cout \u003c\u003c \"Lookup(Mac_Pro) = \" \u003c\u003c id.value_or(UINT64_MAX) \u003c\u003c std::endl;\n    }\n    {\n        const auto id = trie.lookup(\"Google_Pixel\");\n        std::cout \u003c\u003c \"Lookup(Google_Pixel) = \" \u003c\u003c id.value_or(UINT64_MAX) \u003c\u003c std::endl;\n    }\n\n    // Decode the key for a query ID.\n    {\n        const auto dec = trie.decode(4);\n        std::cout \u003c\u003c \"Decode(4) = \" \u003c\u003c dec \u003c\u003c std::endl;\n    }\n\n    // Common prefix search\n    {\n        std::cout \u003c\u003c \"CommonPrefixSearch(MacBook_Air) = {\" \u003c\u003c std::endl;\n        auto itr = trie.make_prefix_iterator(\"MacBook_Air\");\n        while (itr.next()) {\n            std::cout \u003c\u003c \"   (\" \u003c\u003c itr.decoded_view() \u003c\u003c \", \" \u003c\u003c itr.id() \u003c\u003c \"),\" \u003c\u003c std::endl;\n        }\n        std::cout \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n\n    // Predictive search\n    {\n        std::cout \u003c\u003c \"PredictiveSearch(Mac) = {\" \u003c\u003c std::endl;\n        auto itr = trie.make_predictive_iterator(\"Mac\");\n        while (itr.next()) {\n            std::cout \u003c\u003c \"   (\" \u003c\u003c itr.decoded_view() \u003c\u003c \", \" \u003c\u003c itr.id() \u003c\u003c \"),\" \u003c\u003c std::endl;\n        }\n        std::cout \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n\n    // Enumerate all the keys (in lexicographical order).\n    {\n        std::cout \u003c\u003c \"Enumerate() = {\" \u003c\u003c std::endl;\n        auto itr = trie.make_enumerative_iterator();\n        while (itr.next()) {\n            std::cout \u003c\u003c \"   (\" \u003c\u003c itr.decoded_view() \u003c\u003c \", \" \u003c\u003c itr.id() \u003c\u003c \"),\" \u003c\u003c std::endl;\n        }\n        std::cout \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n\n    std::remove(tmp_filename);\n\n    return 0;\n}\n```\n\nThe output will be\n\n```\nNumber of keys: 12\nNumber of trie nodes: 28\nNumber of DA units: 256\nMemory usage in bytes: 1766\nLookup(Mac_Pro) = 7\nLookup(Google_Pixel) = 18446744073709551615\nDecode(4) = MacBook_Air\nCommonPrefixSearch(MacBook_Air) = {\n   (Mac, 1),\n   (MacBook, 2),\n   (MacBook_Air, 4),\n}\nPredictiveSearch(Mac) = {\n   (Mac, 1),\n   (MacBook, 2),\n   (MacBook_Air, 4),\n   (MacBook_Pro, 11),\n   (Mac_Mini, 5),\n   (Mac_Pro, 7),\n}\nEnumerate() = {\n   (AirPods, 0),\n   (AirTag, 3),\n   (Mac, 1),\n   (MacBook, 2),\n   (MacBook_Air, 4),\n   (MacBook_Pro, 11),\n   (Mac_Mini, 5),\n   (Mac_Pro, 7),\n   (iMac, 10),\n   (iPad, 6),\n   (iPhone, 8),\n   (iPhone_SE, 9),\n}\n```\n\n## API\n\nXcdat consists of only the header files and can be used by including only the header `xcdat.hpp`. Also, it uses `namespace xcdat`.\n\n### Trie dictionary types\n\nThe four specialization types of class `xcdat::trie` are provided in `xcdat.hpp`. The first two types are based on standard DACs by Brisaboa et al. [9]. The last two types are based on pointer-based DACs by Kanda et al. [2].\n\n```c++\n//! The trie type with standard DACs using 8-bit integers\nusing trie_8_type = trie\u003cbc_vector_8\u003e;\n\n//! The trie type with standard DACs using 16-bit integers\nusing trie_16_type = trie\u003cbc_vector_16\u003e;\n\n//! The trie type with pointer-based DACs using 7-bit integers (for the 1st layer)\nusing trie_7_type = trie\u003cbc_vector_7\u003e;\n\n//! The trie type with pointer-based DACs using 15-bit integers (for the 1st layer)\nusing trie_15_type = trie\u003cbc_vector_15\u003e;\n```\n\n### Trie dictionary class\n\nThe trie dictionary class provides the following functions.\n\n```c++\n//! A compressed string dictionary based on an improved double-array trie.\n//! 'BcVector' is the data type of Base and Check vectors.\ntemplate \u003cclass BcVector\u003e\nclass trie {\n  public:\n    //! The type identifier.\n    static constexpr std::uint32_t type_id;\n\n    //! Default constructor\n    trie() = default;\n\n    //! Default destructor\n    virtual ~trie() = default;\n\n    //! Copy constructor (deleted)\n    trie(const trie\u0026) = delete;\n\n    //! Copy constructor (deleted)\n    trie\u0026 operator=(const trie\u0026) = delete;\n\n    //! Move constructor\n    trie(trie\u0026\u0026) noexcept = default;\n\n    //! Move constructor\n    trie\u0026 operator=(trie\u0026\u0026) noexcept = default;\n\n    //! Build the trie from the input keywords, which are lexicographically sorted and unique.\n    //!\n    //! If bin_mode = false, the NULL character is used for the termination of a keyword.\n    //! If bin_mode = true, bit flags are used istead, and the keywords can contain NULL characters.\n    //! If the input keywords contain NULL characters, bin_mode will be forced to be set to true.\n    //!\n    //! The type 'Strings' and 'Strings::value_type' should be a random iterable container such as std::vector.\n    //! Precisely, they should support the following operations:\n    //!  - size() returns the container size.\n    //!  - operator[](i) accesses the i-th element.\n    //!  - begin() returns the iterator to the beginning.\n    //!  - end() returns the iterator to the end.\n    //! The type 'Strings::value_type::value_type' should be one-byte integer type such as 'char'.\n    template \u003cclass Strings\u003e\n    trie(const Strings\u0026 keys, bool bin_mode = false);\n\n    //! Check if the binary mode.\n    bool bin_mode() const;\n\n    //! Get the number of stored keywords.\n    std::uint64_t num_keys() const;\n\n    //! Get the alphabet size.\n    std::uint64_t alphabet_size() const;\n\n    //! Get the maximum length of keywords.\n    std::uint64_t max_length() const;\n\n    //! Get the number of trie nodes.\n    std::uint64_t num_nodes() const;\n\n    //! Get the number of DA units.\n    std::uint64_t num_units() const;\n\n    //! Get the number of unused DA units.\n    std::uint64_t num_free_units() const;\n\n    //! Get the length of TAIL vector.\n    std::uint64_t tail_length() const;\n\n    //! Lookup the ID of the keyword.\n    std::optional\u003cstd::uint64_t\u003e lookup(std::string_view key) const;\n\n    //! Decode the keyword associated with the ID.\n    std::string decode(std::uint64_t id) const;\n\n    //! Decode the keyword associated with the ID and store it in 'decoded'.\n    //! It can avoid reallocation of memory to store the result.\n    void decode(std::uint64_t id, std::string\u0026 decoded) const;\n\n    //! An iterator class for common prefix search.\n    //! It enumerates all the keywords contained as prefixes of a given string.\n    //! It should be instantiated via the function 'make_prefix_iterator'.\n    class prefix_iterator {\n      public:\n        prefix_iterator() = default;\n\n        //! Increment the iterator.\n        //! Return false if the iteration is terminated.\n        bool next();\n\n        //! Get the result ID.\n        std::uint64_t id() const;\n\n        //! Get the result keyword.\n        std::string decoded() const;\n\n        //! Get the reference to the result keyword.\n        //! Note that the referenced data will be changed in the next iteration.\n        std::string_view decoded_view() const;\n    };\n\n    //! Make the common prefix searcher for the given keyword.\n    prefix_iterator make_prefix_iterator(std::string_view key) const;\n\n    //! Preform common prefix search for the keyword.\n    void prefix_search(std::string_view key, const std::function\u003cvoid(std::uint64_t, std::string_view)\u003e\u0026 fn) const;\n\n    //! An iterator class for predictive search.\n    //! It enumerates all the keywords starting with a given string.\n    //! It should be instantiated via the function 'make_predictive_iterator'.\n    class predictive_iterator {\n      public:\n        predictive_iterator() = default;\n\n        //! Increment the iterator.\n        //! Return false if the iteration is terminated.\n        bool next();\n\n        //! Get the result ID.\n        std::uint64_t id() const;\n\n        //! Get the result keyword.\n        std::string decoded() const;\n\n        //! Get the reference to the result keyword.\n        //! Note that the referenced data will be changed in the next iteration.\n        std::string_view decoded_view() const;\n    };\n\n    //! Make the predictive searcher for the keyword.\n    predictive_iterator make_predictive_iterator(std::string_view key) const;\n\n    //! Preform predictive search for the keyword.\n    void predictive_search(std::string_view key, const std::function\u003cvoid(std::uint64_t, std::string_view)\u003e\u0026 fn) const;\n\n    //! An iterator class for enumeration.\n    //! It enumerates all the keywords stored in the trie.\n    //! It should be instantiated via the function 'make_enumerative_iterator'.\n    using enumerative_iterator = predictive_iterator;\n\n    //! An iterator class for enumeration.\n    enumerative_iterator make_enumerative_iterator() const;\n\n    //! Enumerate all the keywords and their IDs stored in the trie.\n    void enumerate(const std::function\u003cvoid(std::uint64_t, std::string_view)\u003e\u0026 fn) const;\n\n    //! Visit the members (commonly used for I/O).\n    template \u003cclass Visitor\u003e\n    void visit(Visitor\u0026 visitor);\n};\n```\n\n### I/O utilities\n\n`xcdat.hpp` provides some functions for handling I/O operations.\n\n```c++\n//! Set the continuous memory block to a new trie instance (for a memory-mapped file).\ntemplate \u003cclass Trie\u003e\nTrie mmap(const char* address);\n\n//! Load the trie dictionary from the file.\ntemplate \u003cclass Trie\u003e\nTrie load(const std::string\u0026 filepath);\n\n//! Save the trie dictionary to the file and returns the file size in bytes.\n//! The identifier of the trie type will be written in the first 4 bytes.\ntemplate \u003cclass Trie\u003e\nstd::uint64_t save(const Trie\u0026 idx, const std::string\u0026 filepath);\n\n//! Get the dictionary size in bytes.\ntemplate \u003cclass Trie\u003e\nstd::uint64_t memory_in_bytes(const Trie\u0026 idx);\n\n//! Get the identifier of the trie type embedded by the function 'save'.\n//! The identifier corresponds to trie::type_id and will be used to detect the trie type.\nstd::uint32_t get_type_id(const std::string\u0026 filepath);\n```\n\n### Exception class\n\nIf an error occurs in a construction or I/O operation, Xcdat will throw an instance of  `xcdat::exception` as a runtime error.\n\n```c++\nclass exception : public std::exception {\n  public:\n    //! Get the error massage.\n    const char* what() const throw() override;\n};\n```\n\n## Performance\n\nWe compared the performance of Xcdat with those of other selected dictionary libraries written in C++.\n\n### Implementations\n\n- Our compressed double-array tries [2]\n  - [xcdat\u003c8\u003e](https://github.com/kampersanda/xcdat/blob/master/include/xcdat.hpp): `xcdat::trie_8_type`\n  - [xcdat\u003c16\u003e](https://github.com/kampersanda/xcdat/blob/master/include/xcdat.hpp): `xcdat::trie_16_type`\n  - [xcdat\u003c7\u003e](https://github.com/kampersanda/xcdat/blob/master/include/xcdat.hpp): `xcdat::trie_7_type`\n  - [xcdat\u003c15\u003e](https://github.com/kampersanda/xcdat/blob/master/include/xcdat.hpp): `xcdat::trie_15_type`\n- Other double-array tries\n  - [darts](http://chasen.org/~taku/software/darts/): Double-array trie [3].\n  - [darts-clone](https://github.com/s-yata/darts-clone): Compact double-array trie [4].\n  - [cedar](http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar/): Dynamic double-array reduced trie [10,11]\n  - [cedarpp](http://www.tkl.iis.u-tokyo.ac.jp/~ynaga/cedar/): Dynamic double-array prefix trie [10,11]\n  - [dastrie](http://www.chokkan.org/software/dastrie/): Compact double-array prefix trie [4]\n- Succinct tries\n  - [tx](https://github.com/hillbig/tx-trie): LOUDS trie [12]\n  - [marisa](https://github.com/s-yata/marisa-trie): LOUDS nested patricia trie [13]\n  - [fst](https://github.com/kampersanda/fast_succinct_trie): Fast succinct prefix trie [14]\n  - [pdt](https://github.com/ot/path_decomposed_tries): Centroid path-decomposed trie with RePair [15]\n- Tessil's string containers\n  - [hat-trie](https://github.com/Tessil/hat-trie/): HAT-trie [16]\n  - [array-hash](https://github.com/Tessil/array-hash): Array hashing [17]\n\n### Environments\n\n- **Machine:** MacBook Pro (13-inch, 2019)\n- **OS:** macOS Catalina (version 10.15)\n- **Processor:** 2.4 GHz Quad-Core Intel Core i5\n- **Memory:** 16 GB 2133 MHz LPDDR3\n- **Compiler:** AppleClang (version 12.0.0)\n- **Optimization flags:** `-O3 -march=native`\n\n### Datasets\n\n- Natural language corpus\n  - **IPA:** 325,871 different Japanese words from [IPAdic](https://taku910.github.io/mecab/) (3.7 MiB, 11.9 bytes/key)\n  - **Wiki:** 14,130,439 different [English Wikipedia titles](https://dumps.wikimedia.org/) on 2018-09-20 (285 MiB, 21.2 bytes/key)\n- [Askitis's datasets](http://web.archive.org/web/20120206015921/http://www.naskitis.com/)\n  - **Distinct:** 28,772,169 different english words (290 MiB, 10.6 bytes/key)\n  - **Url:** 1,289,458 different URL strings (44 MiB, 36.2 bytes/key)\n\n### Approach\n\nWe constructed a dictionary from a dataset and measured the elapsed time. The dynamic dictionaries, Cedar and Tessil's containers, were constructed by inserting sorted keywords. Each keyword is associated with a unique ID of a 4-byte integer. Since all the libraries support serialization of the data structure, we measured the output file size as the memory usage.\n\nThe time to lookup IDs from keywords was measured for 1,000 query keywords randomly sampled from each dataset. Also, for some libraries supporting to decode keywords from IDs, the time was measured for the 1,000 IDs corresponding to the query keywords. We took the best result of 10 runs.\n\nThe code of the benchmark can be found [here](https://github.com/kampersanda/fast_succinct_trie/tree/master/bench).\n\n### Results\n\n#### Plot: memory usage vs. lookup time\n\n![lookup_vs_memory](https://github.com/kampersanda/xcdat/blob/master/image/lookup_vs_memory.png?raw=true)\n\n#### Plot: memory usage vs. build time\n\n![build_vs_memory](https://github.com/kampersanda/xcdat/blob/master/image/build_vs_memory.png?raw=true)\n\n#### Table: IPA\n\n| Library     | Data structure                            | Memory (MiB) | Build (ns/key) | Lookup (ns/key) | Decode (ns/id) |\n| ----------- | ----------------------------------------- | -----------: | -------------: | --------------: | -------------: |\n| xcdat\u003c8\u003e    | XCDA prefix trie                          |          1.9 |            271 |             124 |            198 |\n| xcdat\u003c16\u003e   | XCDA prefix trie                          |          2.5 |            257 |              94 |            144 |\n| xcdat\u003c7\u003e    | XCDA prefix trie                          |          2.1 |            288 |             122 |            192 |\n| xcdat\u003c15\u003e   | XCDA prefix trie                          |          2.3 |            259 |              89 |            139 |\n| darts       | Double-array trie                         |         10.9 |            725 |              94 |            n/a |\n| darts-clone | Compact double-array trie                 |          5.2 |            231 |              52 |            n/a |\n| cedar       | Double-array reduced trie (dyn)           |         10.3 |            150 |             101 |            n/a |\n| cedarpp     | Double-array prefix trie (dyn)            |          6.1 |            171 |              83 |            n/a |\n| dastrie     | Compact double-array prefix trie          |          4.7 |            233 |             117 |            n/a |\n| tx          | LOUDS trie                                |          1.5 |            256 |            1315 |           1289 |\n| marisa      | LOUDS nested patricia trie                |          1.0 |            466 |             498 |            385 |\n| fst         | Fast succinct prefix trie                 |          1.6 |            529 |             460 |            n/a |\n| pdt         | Centroid path-decomposed trie with RePair |          1.1 |           1436 |             614 |            716 |\n| hat-trie    | HAT-trie (dyn)                            |          5.8 |            330 |              87 |            n/a |\n| array-hash  | Array hashing (dyn)                       |          8.8 |            476 |             147 |            n/a |\n\n#### Table: Wiki\n\n| Library     | Data structure                            | Memory (MiB) | Build (ns/key) | Lookup (ns/key) | Decode (ns/id) |\n| ----------- | ----------------------------------------- | -----------: | -------------: | --------------: | -------------: |\n| xcdat\u003c8\u003e    | XCDA prefix trie                          |          139 |            672 |             826 |           1173 |\n| xcdat\u003c16\u003e   | XCDA prefix trie                          |          182 |            672 |             679 |            902 |\n| xcdat\u003c7\u003e    | XCDA prefix trie                          |          144 |            674 |             829 |           1153 |\n| xcdat\u003c15\u003e   | XCDA prefix trie                          |          174 |            668 |             619 |            821 |\n| darts       | Double-array trie                         |         1121 |           1795 |             452 |            n/a |\n| darts-clone | Compact double-array trie                 |          525 |            584 |             450 |            n/a |\n| cedar       | Dynamic double-array reduced trie         |         1049 |            210 |             445 |            n/a |\n| cedarpp     | Dynamic double-array prefix trie          |          425 |            174 |             484 |            n/a |\n| dastrie     | Compact double-array prefix trie          |          317 |            387 |             488 |            n/a |\n| tx          | LOUDS trie                                |          178 |            674 |            8062 |           7086 |\n| marisa      | LOUDS nested patricia trie                |           69 |           1078 |            1818 |           1515 |\n| fst         | Fast succinct prefix trie                 |          114 |           1132 |            2309 |            n/a |\n| pdt         | Centroid path-decomposed trie with RePair |           89 |           4242 |            1456 |           1514 |\n| hat-trie    | HAT-trie                                  |          358 |            361 |             368 |            n/a |\n| array-hash  | Array hashing                             |          484 |            927 |             385 |            n/a |\n\n#### Table: Distinct\n\n| Library     | Data structure                            | Memory (MiB) | Build (ns/key) | Lookup (ns/key) | Decode (ns/id) |\n| ----------- | ----------------------------------------- | -----------: | -------------: | --------------: | -------------: |\n| xcdat\u003c8\u003e    | XCDA prefix trie                          |          172 |            347 |             757 |           1017 |\n| xcdat\u003c16\u003e   | XCDA prefix trie                          |          215 |            357 |             543 |            742 |\n| xcdat\u003c7\u003e    | XCDA prefix trie                          |          186 |            371 |             676 |           1003 |\n| xcdat\u003c15\u003e   | XCDA prefix trie                          |          206 |            339 |             451 |            691 |\n| darts       | Double-array trie                         |          859 |           1990 |             225 |            n/a |\n| darts-clone | Compact double-array trie                 |          409 |            251 |             198 |            n/a |\n| cedar       | Dynamic double-array reduced trie         |          817 |            127 |             213 |            n/a |\n| cedarpp     | Dynamic double-array prefix trie          |          488 |            164 |             261 |            n/a |\n| tx          | LOUDS trie                                |          113 |            233 |            3523 |           3147 |\n| marisa      | LOUDS nested patricia trie                |           87 |            558 |            1311 |           1041 |\n| fst         | Fast succinct prefix trie                 |          161 |            519 |            1201 |            n/a |\n| pdt         | Centroid path-decomposed trie with RePair |           94 |           2393 |            1453 |           1681 |\n| hat-trie    | HAT-trie                                  |          443 |            275 |             322 |            n/a |\n| array-hash  | Array hashing                             |          693 |            891 |             411 |            n/a |\n\n1. I don't know why, but dastrie did not complete the build.\n\n#### Table: Url\n\n| Library     | Data structure                            | Memory (MiB) | Build (ns/key) | Lookup (ns/key) | Decode (ns/id) |\n| ----------- | ----------------------------------------- | -----------: | -------------: | --------------: | -------------: |\n| xcdat\u003c8\u003e    | XCDA prefix trie                          |           19 |            606 |             469 |            685 |\n| xcdat\u003c16\u003e   | XCDA prefix trie                          |           24 |            633 |             461 |            682 |\n| xcdat\u003c7\u003e    | XCDA prefix trie                          |           19 |            605 |             542 |            772 |\n| xcdat\u003c15\u003e   | XCDA prefix trie                          |           23 |            624 |             423 |            625 |\n| darts       | Double-array trie                         |          131 |           2659 |             366 |            n/a |\n| darts-clone | Compact double-array trie                 |           61 |           1045 |             411 |            n/a |\n| cedar       | Dynamic double-array reduced trie         |          122 |            287 |             403 |            n/a |\n| cedarpp     | Dynamic double-array prefix trie          |           49 |            220 |             442 |            n/a |\n| dastrie     | Compact double-array prefix trie          |           35 |           1009 |             381 |            n/a |\n| tx          | LOUDS trie                                |           21 |            887 |            6389 |           6058 |\n| marisa      | LOUDS nested patricia trie                |            9 |           1261 |            1789 |           1747 |\n| fst         | Fast succinct prefix trie                 |           15 |           1692 |            1948 |            n/a |\n| pdt         | Centroid path-decomposed trie with RePair |           10 |           5385 |            1013 |           1287 |\n| hat-trie    | HAT-trie                                  |           48 |            541 |             235 |            n/a |\n| array-hash  | Array hashing                             |           65 |            868 |             268 |            n/a |\n\n## Licensing\n\nThis library is free software provided under the MIT License.\n\nIf you use the library in academic settings, please cite the following paper.\n\n```\n@article{kanda2017compressed,\n    title={Compressed double-array tries for string dictionaries supporting fast lookup},\n    author={Kanda, Shunsuke and Morita, Kazuhiro and Fuketa, Masao},\n    journal={Knowledge and Information Systems (KAIS)},\n    volume={51},\n    number={3},\n    pages={1023--1042},\n    year={2017},\n    publisher={Springer}\n}\n```\n\n## Todo\n\n- Support other language bindings.\n- Add SIMD-ization.\n\n## References\n\n1. M. A. Martínez-Prieto, N. Brisaboa, R. Cánovas, F. Claude, and G. Navarro. **Practical compressed string dictionaries.** *Information Systems*, 56:73–108, 2016.\n2. S. Kanda, K. Morita, and M. Fuketa. **Compressed double-array tries for string dictionaries supporting fast lookup.** *Knowledge and Information Systems*, 51(3): 1023–1042, 2017.\n3. J. Aoe. **An efficient digital search algorithm by using a double-array structure.** *IEEE Transactions on Software Engineering*, 15(9): 1066–1077, 1989.\n4. S. Yata, M. Oono, K. Morita, M. Fuketa, T. Sumitomo, and J. Aoe. **A compact static double-array keeping character codes.** *Information Processing \u0026 Management*, 43(1): 237–247, 2007.\n5. I. Mülle, R. Cornelius, and F. Franz. **Adaptive string dictionary compression in in-memory column-store database systems.** In *Proc. EDBT*, pp. 283–294, 2014.\n6. S. Gog, GE. Pibiri, and R. Venturini. **Efficient and effective query auto-completion.** In *Proc. SIGIR*, pp. 2271–2280, 2020.\n7. R. Baeza-Yates, and B. Ribeiro-Neto. **Modern Information Retrieval.** 2nd ed. Addison Wesley, Boston, MA, USA, 2011.\n8. T. Kudo, T. Hanaoka, J. Mukai, Y. Tabata, and H. Komatsu. **Efficient dictionary and language model compression for input method editors.** In *Proc. WTIM*, pp. 19–25, 2011.\n9. N. R. Brisaboa, S. Ladra, and G. Navarro. **DACs: Bringing direct access to variable-length codes.** *Information Processing \u0026 Management*, 49(1): 392–404, 2013.\n10. S. Yata, M. Tamura, K. Morita, M. Fuketa and J. Aoe. **Sequential insertions and performance evaluations for double-arrays.** In *Proc. IPSJ*, pp. 1263–1264, 2009.\n12. N. Yoshinaga, and M. Kitsuregawa. **A self-adaptive classifier for efficient text-stream processing.** In *Proc. COLING*, pp. 1091–1102, 2014.\n13. G. Jacobson. **Space-efficient static trees and graphs.** In *Proc. FOCS*, pp. 549–554, 1989.\n14. S. Yata. **Dictionary compression by nesting prefix/patricia tries.** In *Proc. JNLP*, pp. 576–578, 2011.\n15. H. Zhang, H. Lim, V. Leis, DG. Andersen, M. Kaminsky, K. Keeton, and A. Pavlo. **Surf: Practical range query filtering with fast succinct tries.** In *Proc. SIGMOD*, pp. 323–336, 2018.\n16. R. Grossi, and G. Ottaviano. **Fast compressed tries through path decompositions.** *ACM Journal of Experimental Algorithmics*, 19, 2015.\n17. N. Askitis, and R. Sinha. **Engineering scalable, cache and space efficient tries for strings.** *The VLDB Journal*, *19*(5): 633-660, 2010.\n18. N. Askitis, and J. Zobel. **Cache-conscious collision resolution in string hash tables.** In *Proc. SPIRE*, pp. 91–102, 2005.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkampersanda%2Fxcdat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkampersanda%2Fxcdat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkampersanda%2Fxcdat/lists"}