{"id":17085268,"url":"https://github.com/mpetri/ans-large-alphabet","last_synced_at":"2026-03-08T14:32:43.235Z","repository":{"id":141712947,"uuid":"258898267","full_name":"mpetri/ans-large-alphabet","owner":"mpetri","description":"Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems","archived":false,"fork":false,"pushed_at":"2020-04-28T03:15:25.000Z","size":310,"stargazers_count":4,"open_issues_count":1,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T17:13:54.542Z","etag":null,"topics":["compression","data-compression","entropy-coding","research-paper"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpetri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-25T23:50:13.000Z","updated_at":"2023-01-23T13:17:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"8687273a-a330-4661-895f-16aa47aea647","html_url":"https://github.com/mpetri/ans-large-alphabet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpetri%2Fans-large-alphabet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpetri%2Fans-large-alphabet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpetri%2Fans-large-alphabet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpetri%2Fans-large-alphabet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpetri","download_url":"https://codeload.github.com/mpetri/ans-large-alphabet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253009529,"owners_count":21839711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","data-compression","entropy-coding","research-paper"],"created_at":"2024-10-14T13:23:55.104Z","updated_at":"2026-03-08T14:32:43.203Z","avatar_url":"https://github.com/mpetri.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems\n======================\n\nThis repository is the code released in conjunction of with the following paper:\n\n`\nAlistair Moffat, Matthias Petri: Large-Alphabet Semi-Static Entropy Coding Via Asymmetric Numeral Systems. ACM Transactions on Information Systems, doi: 10.1145/3397175\n`\n\nwhich implements several integer list coding schemes which utilise ANS entropy coders.\n\nRequirements\n--------\n\n1. C++ compiler which supports C++-17\n2. Boost installed\n\nInstallation\n-------------\n\n1. `git checkout https://github.com/mpetri/ans-large-alphabet.git`\n2. `cd ans-large-alphabet`\n3. `git submodule update --init --recursive`\n2. `mkdir build`\n3. `cd build`\n4. `cmake ..`\n5. `make -j`\n\n\nDownload and Generating the Data\n----------------\n\nrun the following scripts to download and generate the data used in the paper:\n\n\n1. `bash ./scripts/download_data.sh` will download the \"real world\" datasets into ./data/\n2. `bash ./scripts/generate_data.sh` will generate the synthetic datasets into ./data/ and setup the directory structure to run all experiments\n\n\nRunning Experiments\n----------------\n\nEach experiment in the paper can be reproduced using a self-contained binary. To generate Table 9 in the paper\nwe run:\n\n```\n./build/table_effectiveness.x -t -i ./data/space/\n```\n\nwhich produces:\n\n```\nhuff0 \u0026\n17.2864  \u0026\n    4.1148  \u0026\n        9.1648  \u0026\n            11.2015  \u0026\n                16.6952  \u0026\n                    15.8428  \u0026\n                        28.9523  \\\\\nFSE \u0026\n17.2491  \u0026\n    0.7367  \u0026\n        8.5364  \u0026\n            10.5462  \u0026\n                16.6186  \u0026\n                    15.7553  \u0026\n                        28.9405  \\\\\nvbyte \u0026\n15.7499  \u0026\n    8.0000  \u0026\n        8.2141  \u0026\n            10.9859  \u0026\n                15.2995  \u0026\n                    14.3414  \u0026\n                        31.6357  \\\\\nvbyte+huff0 \u0026\n14.0040  \u0026\n    1.1125  \u0026\n        5.1454  \u0026\n            8.7429  \u0026\n                13.8829  \u0026\n                    13.0518  \u0026\n                        29.3547  \\\\\nvbyte+FSE \u0026\n13.9387  \u0026\n    0.5234  \u0026\n        5.1147  \u0026\n            8.7600  \u0026\n                13.8549  \u0026\n                    13.0309  \u0026\n                        29.2314  \\\\\n...\n```\n\nThe latex table from the paper (requiring some additional reformatting to get the exact).\n\n\nThe main efficiency results table (Table 10) can be reproduced similarily:\n\n```\n./build/table_efficiency.x -t -i ./data/speed/\n```\n\nwhich produces:\n\n```\n\\method{\\vbyte}  \u0026\n\t1287581427.7761  \u0026  1275265333.6182  \u0026\n\t\t1124443361.1807  \u0026  1113121210.1284  \u0026\n\t\t\t\t205508460.2778  \u0026   212470992.1057  \u0026\n\t\t\t\t\t301014107.7998  \u0026   300215782.7958  \\\\\n\n\\method{\\vbyte+huff0}  \u0026\n\t510864371.2260  \u0026   667794304.1286  \u0026\n\t\t458460057.4240  \u0026   557119911.1247  \u0026\n\t\t\t138274224.6237  \u0026   144482479.1039  \u0026\n\t\t\t\t178295869.4397  \u0026   184607891.3527  \\\\\n\n\\method{\\vbyte+FSE}  \u0026\n\t359587723.6133  \u0026   444027800.3319  \u0026\n\t\t338439584.5626  \u0026   414147209.0230  \u0026\n\t\t\t114633629.8415  \u0026   132227626.1772  \u0026\n\t\t\t\t142760372.5347  \u0026   166934064.7120  \\\\\n...\n```\n\n\n\nCode Overview\n----------------\n\nThe repository contains all the code necessary to reproduce the results in the paper. The table below provides an overview of the different files and what they implement:\n\n| File | Description |\n| ---  | ---- |\n| `shuff.hpp` | A version of `https://github.com/turpinandrew/shuff` which implements \"On the Implementation of Minimum-Redundancy Prefix Codes\", IEEE Transactions on Communications, 45(10):1200-1207, October 1997, and \"Housekeeping for Prefix Coding\", IEEE Transactions on Communications, 48(4):622-628, April 2000. |\n| `arith.hpp` | Implementation of a 56-bit arithmetic encoder and decoder pair that carries out semi-static compression of an input array of (in the encoder) strictly positive uint32_t values, not including zero. |\n| `ans_fold.hpp` | The \"ans_fold\" technique described in the paper |\n| `ans_msb.hpp` | The \"ans_fold\" technique was generalized from a previous paper which was called `ans_msb` which is equivalent to `ans_fold_1` |\n| `ans_int.hpp` | A large alphabet implementation of regular ANS coding. Called \"ANS\" in the paper |\n| `ans_reorder_fold.hpp` | The \"ANSfold-X-r\" technique which reorders the most frequent symbols to the front of the alphabet and stores the mapping in the prelude |\n| `methods.hpp` | Interfaces to all the different methods including the external library calls to the `streamvbyte`, `FiniteStateEntropy` and `FastPfor` libraries for fast `vbyte`, `huff0`, `FSE` and `OpfPFor` implementations |\n| `generate_*.cpp` | Generate different datasets used in the paper |\n| `interp.hpp` | A version of interpolative coding: `Alistair Moffat, Lang Stuiver: Binary Interpolative Coding for Effective Index Compression. Inf. Retr. 3(1): 25-47 (2000)` used for prelude compression. | \n| `ans_util.hpp` | Various ANS utility function shared across different ANS implementations in this repository | \n| `pseudo_adaptive.cpp` | A block based ANS coder to used to create Figure 13 in the paper |\n| `ans_sint.hpp` | A version of the ANS coder in `ans_int.hpp` which supports different entropy approximation ratios used to create Figure 12 |\n| `ans_smsb.hpp` | A version of the ANS coder in `ans_fold.hpp` which supports different entropy approximation ratios used to create Figure 12 |\n| `fold_effectiveness.cpp` | Used to generate Figure 11 which allows changing the fidelity (f) for `ans_fold` and `ans_rfold` |","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpetri%2Fans-large-alphabet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpetri%2Fans-large-alphabet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpetri%2Fans-large-alphabet/lists"}