{"id":17931505,"url":"https://github.com/jeremy-rifkin/markov-huffman-coding","last_synced_at":"2026-08-11T16:32:08.583Z","repository":{"id":96958984,"uuid":"330833732","full_name":"jeremy-rifkin/Markov-Huffman-Coding","owner":"jeremy-rifkin","description":"This is a proof of concept for an experimental data compression technique that applies Markov chains to Huffman coding.","archived":false,"fork":false,"pushed_at":"2021-01-31T00:50:07.000Z","size":867,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2026-01-31T14:39:55.928Z","etag":null,"topics":["compression","data-compression"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jeremy-rifkin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-19T01:47:42.000Z","updated_at":"2025-08-05T21:52:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"4732c31d-9df3-44ba-a57b-17f79484fa9c","html_url":"https://github.com/jeremy-rifkin/Markov-Huffman-Coding","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jeremy-rifkin/Markov-Huffman-Coding","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremy-rifkin%2FMarkov-Huffman-Coding","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremy-rifkin%2FMarkov-Huffman-Coding/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremy-rifkin%2FMarkov-Huffman-Coding/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremy-rifkin%2FMarkov-Huffman-Coding/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jeremy-rifkin","download_url":"https://codeload.github.com/jeremy-rifkin/Markov-Huffman-Coding/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jeremy-rifkin%2FMarkov-Huffman-Coding/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":36530557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-08-06T04:43:03.162Z","status":"online","status_checked_at":"2026-08-11T02:00:06.871Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","data-compression"],"created_at":"2024-10-28T21:22:08.148Z","updated_at":"2026-08-11T16:32:08.576Z","avatar_url":"https://github.com/jeremy-rifkin.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Markov-Huffman Coding\n\n![build status](https://github.com/jeremy-rifkin/Markov-Huffman-Coding/workflows/build/badge.svg)\n![tests status](https://github.com/jeremy-rifkin/Markov-Huffman-Coding/workflows/tests/badge.svg)\n\nWelcome to a data compression project. This project is a proof of concept that applies conditional\nprobability to Huffman coding, creating a data compression system which takes better advantage of\npatterns in data than traditional Huffman coding.\n\n### TL;DR: Results\n\nMarkov-Huffman coding offers a 20-30% improvement over traditional Huffman coding. There are some\n(niche) cases where Markov-Huffman coding works really well standalone and there may applications of\nthis idea in conjunction with other compression techniques. Many widely used compression utilities\nsuch as gzip, bzip2, and DEFLATE, use traditional Huffman coding under the hood in conjunction with\nother algorithms.\n\n### Table of Contents\n- [Encoding Details](#encoding-details)\n- [Usage](#usage)\n- [Performance](#performance)\n- [Overhead](#overhead)\n- [Order-3 Markov Chains](#order-3-markov-chains)\n- [Results and Applications](#results-and-applications)\n- [Existing work](#existing-work)\n- [Logistics](#logistics)\n\n## Encoding Details\n\nHuffman coding is a data compression technique that uses variable-length codes to losslessly\ncompress data. By assigning short codewords to the most frequently used symbols and longer codewords\nto more uncommon symbols, huffman coding creates a more compact representation for a given data set.\nEven in a codec where symbols have a uniform frequency, Huffman coding can still save on space if\nthe set of used symbols is smaller than the number of representable symbols in the codec's\nfixed-length coding.\n\nHere are a couple small examples with an alphabet `{a, b, c, d}` and a default 2-bits-per-character\nfixed-length encoding.\n\n```\nsource: aaaabbc\n  7\n / \\\na   3\n   / \\\n  b   c\nFixed-length encoding: 00000000010110 (14 bits)\nHuffman coding:        0000101011     (10 bits)\n```\n\nFour bits are saved. There are some situations where huffman coding may not be able do do anything,\nthough. Another example with the same alphabet and encoding:\n\n```\nsource: abcdabcd\n      8\n  +---+---+\n  4       4\n / \\     / \\\na   b   c   d\nFixed-length encoding: 0001101100011011 (16 bits)\nHuffman coding:        0001101100011011 (16 bits)\n```\n\nBecause every character in the alphabet is used with the same frequency and every symbol in the\nalphabet is used, huffman coding is not able to do anything here. The huffman coding and\nfixed-length encodings are identical.\n\nBut there's an obvious pattern here: `abcd` is just repeated twice. Every `a` is followed by `b`,\nevery `b` by `c`, and so on. Applying a Markov model to our input data will allow us to take\nadvantage of more patterns in the input data than traditional Huffman coding does.\n\n```\nsource: abcdabcd\nprev: ' ' prev: a prev: b prev: c prev: d\n     1        2       2       2       1\n    /        /       /       /       /\n   a        b       c       d       a\nFixed-length encoding: 0001101100011011 (16 bits)\nMarkov-Huffman coding: 00000000         (8 bits)\n```\n\nEight bits saved. This is the encoding system developed in this project. The Markov model can be\nhelpful for better encoding data where there is inter-symbol dependence, such as human language.\n\n### Huffman Trees\n\nTraditional Huffman coding tree from 150 paragraphs of Lorem Ipsum placeholder text:\n\n![](pics/huffman.png)\n\nMarkov-Huffman coding trees from 150 paragraphs of Lorem Ipsum placeholder text:\n\n![](pics/markov-huffman.png)\n\n## Usage\n\nThis repo contains a simple compression/decompression utility which utilizes Markov-Huffman coding.\n\n```\nmarkov-huffman \u003cinput\u003e [-o output] [options]\n    -o output_file\n    -h use simple huffman coding\n\n    -e encoding_file\n    -d output_encoding_file\n\n    -g print huffman trees and tables\n    -x extract\n```\n\nIf no output file is provided, the program will compress/decompress to `stdout`. Markov-Huffman\nencoding relies on encoding table files. Use `-d` to generate and create an encoding table, and `-e`\nto load an existing table.\n\n`-g` will print all huffman encoding tables as well as all huffman trees in dot/graphviz format.\n\n### Example:\n\n```bash\n# compress\nmarkov-huffman document.txt -o compressed -d encoding\n# extract\nmarkov-huffman compressed -e encoding 2\u003e/dev/null | tee decompressed.txt\n```\n\n## Performance\n\nThis project is a proof of concept. There is room for performance improvement in the implementation\nof this algorithm, however, this project is pretty good.\n\nLookup tables are often used to speedup the Huffman decoding process. 8-bit or 16-bit lookup tables\nwould be feasible memory wise (though 16-bit lookup tables would be very memory intensive in the\nMarkov-Huffman system), however we need a way of handling codewords longer than the table key size.\nIn this project I've implemented 8-bit lookup tables which should provide a good balance between\nmemory, complexity, and speedup. The vast majority of codewords are \u003c= 8 bits long, and the lookup\ntable enables these codewords to be decoded in one step. For the rare codewords longer than 8 bits,\nthe decoder falls back on tree traversal and the lookup table is able to skip past 8 levels of the\ntree.\n\n## Overhead\n\nOne of the obstacles with this compression technique is that specialized encoding trees must be\nstored in order to decompress the data later. While a single Huffman tree can be stored very\ncompactly, Markov-Huffman coding requires up to 256 huffman trees to be stored. For some data sets,\nthis may be quite substantial.\n\nThis project makes an effort to store Huffman trees are stored compactly, however, there is still\nroom for improvement. One way to improve the storage of Huffman trees would be to use canonical\nHuffman codes, which I have not implemented for this proof of concept.\n\nThe good thing about the overhead associated with Markov-Huffman coding is that it is constant.\nUnless tables are updated throughout the compression of a file, the overhead will become negligible\nas the input is larger.\n\nThis overhead can be circumvented, too, by constructing general-use Huffman tables stored in a\nshared manner instead of generating special Huffman tables tailored for each incoming file.\n\n## Order-3 Markov Chains\n\nCan compression be further improved with higher-order markov chains? Probably. I haven't explored\nthem in this project, though. The main downsides I can think of are overhead, memory requirements\nrun the algorithm, and degraded cache performance.\n\nIt would be interesting to explore using a limited number of order-3 markov chains where the benefit\noutweighs the cost, however, I'll have to put that on the backlog.\n\n## Results and Applications\n\n```\n+---------------------+---------+----------------+\n| Test file           | Huffman | Markov-Huffman |\n+---------------------+---------+----------------+\n| input_ipsum.txt     | 0.53    | 0.42 (-22%)    |\n| input_wiki_cpp.txt  | 0.65    | 0.48 (-26%)    |\n| input_wiki_cpp.html | 0.68    | 0.47 (-31%)    |\n| markovhuffman.exe   | 0.71    | 0.51 (-28%)    |\n+---------------------+---------+----------------+\n```\n\n**Conclusion:** Markov-Huffman offers a 20-30% improvement upon traditional Huffman coding,\n\nI am happy with how well this technique performed. Markov-Huffman coding offers significant\nimprovement over traditional Huffman coding at reasonable constant overhead.\n\nOther widely used compression utilities such as gzip, bzip2, and DEFLATE, use traditional Huffman\ncoding under the hood in conjunction with other algorithms. I am interested to see whether Markov-\nHuffman coding offers any improvement over other compression utilities when used in conjunction with\nother algorithms.\n\nStandalone, Markov-Huffman coding has applications. Traditional Huffman coding is sometimes used\nstandalone because it's fast and offers a reasonable amount of compression. Markov-Huffman coding is\njust as fast\\* and can only improve upon traditional-Huffman coding.\n\n\\* Theoretically just as fast, however, Markov-Huffman coding may introduce more cache misses and\nuses more memory to run the algorithm. The bottleneck is, nonetheless, expected to lie in the hard\ndisk.\n\nI previously applied huffman coding to [compressing digits of pi][sc]. Using a fixed Huffman\nencoding table, I was able to achieve a compression ratio of `0.425`, better than any other\ncompression utility tested, and the program I wrote was also faster than any other compression\nutility tested. Granted, this is a niche case: tiny alphabet compared to symbol capacity of the\ndefault fixed-length encoding (ascii). One pesky problem with compressing the digits of pi is that\nthe decimal character after the leading `3` only appears once ever but takes up space in the\nencoding tree. Including the decimal place symbol in the encoding tree would have lead to a\ncompression ratio of `0.438`, which may not sound like much of a difference, but this difference is\nfrom a single character among billions of digits. Previously, I added a special case to just ignore\nthat decimal place. But, because the single decimal place only occurs after a `'3'`, applying\nMarkov-Huffman coding to the pi problem leads to an compression ratio of `0.426`; nearly identical\nto the compression ratio and without special handling for the decimal place.\n\n## Existing work\n\nUnsurprisingly, this idea isn't entirely original. I've found a paper from 1985, [Markov-Huffman\ncoding of LPC parameters][1], and a stackoverflow post from 2018, [Huffman Coding for Markov Chain\nbased on conditional distribution][2], which describe similar ideas. Both of these discuss taking\nadvantage of conditional probabilities with huffman coding in the context of engineering problems.\n\nAnother [post][3] on cs stackexchange regarding data stream compression briefly mentions that \"...\nif there are correlations between neighboring symbols (which there are in real human produced text)\nyou can do better by choosing codes for pairs of letters.\" I think the system I've developed here\nshould provide more optimal compression, however, I can't find any examples of this 2-char symbol\nhuffman encoding strategy to compare against.\n\nUpdate: After making this I learned that bzip2 uses multiple huffman tables internally. The\ndifference is that bzip2 selects the optimal tree every 50 symbols (source: wiki).\n\n### Encryption\n\nBecause this encoding algorithm creates a binary blob that is only meaningful in conjunction with an\nencoding tree file, there could be application of this idea in the context of encryption (with\nrandomized trees as opposed to frequency-based Huffman trees). In their paper [On the security of\nmultiple Huffman table based encryption][4], Zhou et al. explore the security of a multi-Huffman\nencryption technique and conclude that the technique is not secure.They use a tree mutation\nalgorithm on Huffman trees which \"have good compression ratio\\[s\\]\". Despite using frequency-based\nHuffman trees initially, I think the attacks described in this paper would still work on randomized\ntrees.\n\n## Logistics\n\nThis project is Copyright (c) Jeremy Rifkin 2021.\n\nThis repository contains Wikipedia source-code and article contents, used as test cases for\nalgorithm correctness and efficiency and located in `test/input/`, which are licensed under the\n[`CC BY-SA 3.0 License`][CC].\n\n[1]: https://ieeexplore.ieee.org/document/1164545\n[2]: https://stackoverflow.com/q/49955585/6529874\n[3]: https://cs.stackexchange.com/a/14161/57552\n[4]: https://www.sciencedirect.com/science/article/pii/S1047320310001434\n[sc]: https://github.com/jeremy-rifkin/strawberry-cheesecake\n[CC]: https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeremy-rifkin%2Fmarkov-huffman-coding","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjeremy-rifkin%2Fmarkov-huffman-coding","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjeremy-rifkin%2Fmarkov-huffman-coding/lists"}