{"id":51543930,"url":"https://github.com/shibatch/simplerangecoder","last_synced_at":"2026-07-09T16:01:20.537Z","repository":{"id":365165386,"uuid":"1270854633","full_name":"shibatch/simplerangecoder","owner":"shibatch","description":"A simple C++ range coder library with python binding","archived":false,"fork":false,"pushed_at":"2026-06-16T05:58:10.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-16T07:50:25.339Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shibatch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-16T05:36:50.000Z","updated_at":"2026-06-16T05:58:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/shibatch/simplerangecoder","commit_stats":null,"previous_names":["shibatch/simplerangecoder"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/shibatch/simplerangecoder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Fsimplerangecoder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Fsimplerangecoder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Fsimplerangecoder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Fsimplerangecoder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shibatch","download_url":"https://codeload.github.com/shibatch/simplerangecoder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shibatch%2Fsimplerangecoder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35304875,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-09T16:01:19.411Z","updated_at":"2026-07-09T16:01:20.513Z","avatar_url":"https://github.com/shibatch.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Simple Range Coder\n\nA simple range coding library implemented in C++20 with a Python wrapper. While simple, it provides better performance than a pure Python implementation. This component provides entropy encoding and decoding for signed integer arrays.\n\n## License\n\nThis project is licensed under [CC0 1.0 Universal](LICENSE).\n\n## Features\n\n- **Algorithms**:\n  - Schindler's 32-bit top-down Range Coder.\n  - rANS (Range Asymmetric Numeral Systems) with LIFO concealment (reverse encoding).\n- **Precision**: 32-bit state for both algorithms.\n- **Parallelization**: OpenMP-accelerated batch encoding and decoding for both backends.\n- **Interfaces**: C-ABI (C++20) and Python (via `ctypes`).\n- **Safety**: Buffer overflow protection and stream consistency checks.\n\n## Component Structure\n\n- `range_coder.h` / `range_coder.cpp`: C++ core logic.\n- `fast_rc.py`: Python wrapper (`SimpleRangeCoder` class).\n- `Makefile`: Build automation.\n- `hello_world.cpp` / `hello_world.py`: Usage examples.\n\n## Requirements\n\n- GCC 9+ or Clang (supporting C++20)\n- Python 3.7+\n- NumPy\n\n## Build and Test\n\nTo build the shared library and testers:\n\n```bash\nmake all\n```\n\nTo run both C++ and Python tests:\n\n```bash\nmake test\n```\n\nTo run hello world examples:\n\n```bash\nmake hello\n```\n\n## API Documentation\n\n### C++ ABI (`extern \"C\"`)\n\nThe library provides two entropy coding backends: Range Coder and rANS. Both follow the same C-ABI for easy interchangeability.\n\n#### `range_encode_interface` / `ans_encode_interface`\nEncodes an array of integers into a byte stream.\n**Note**: For `ans_encode_interface`, `tot_freq` must be `65536`.\n\n```cpp\nint32_t range_encode_interface( // or ans_encode_interface\n    const int32_t* q_vals,    // Input integers\n    int32_t num_vals,         // Number of input integers\n    const int32_t* cum_freqs, // Cumulative frequencies (alphabet_size + 1)\n    const int32_t* freqs,     // Individual frequencies (alphabet_size)\n    int32_t alphabet_size,    // Number of possible symbols\n    int32_t tot_freq,         // Sum of frequencies (Must be 65536 for rANS)\n    int32_t sym_shift,        // Offset: sym = q_val + sym_shift\n    uint8_t* out_buf,         // Output buffer\n    int32_t out_buf_len       // Output buffer capacity\n);\n```\n**Returns**: Number of bytes written, or `-1` on failure (e.g., buffer overflow).\n\n#### `range_decode_interface` / `ans_decode_interface`\nDecodes a byte stream back into integers.\n**Note**: For `ans_decode_interface`, `tot_freq` must be `65536`.\n\n```cpp\nint32_t range_decode_interface( // or ans_decode_interface\n    const uint8_t* data,      // Compressed byte stream\n    int32_t data_len,         // Length of input data\n    int32_t block_len,        // Number of elements to decode\n    const int32_t* cum_freqs, // Cumulative frequencies\n    const int32_t* freqs,     // Individual frequencies\n    int32_t alphabet_size,    // Number of symbols\n    int32_t tot_freq,         // Sum of frequencies (Must be 65536 for rANS)\n    int32_t sym_shift,        // Same offset as used in encoding\n    int32_t* out_q_vals       // Output array for integers\n);\n```\n**Returns**: `0` on success, negative error code on failure.\n\n#### `range_encode_batch` / `ans_encode_batch`\nEncodes multiple blocks in parallel using OpenMP. The argument structure is identical for both Range Coder and rANS backends.\n**Note**: For rANS, `all_tot_freqs` must contain `65536` for all blocks.\n\n```cpp\nvoid range_encode_batch( // or ans_encode_batch\n    int32_t num_blocks,               // Total number of blocks to process\n    const int32_t* all_q_vals,        // Flat array of input symbols [num_blocks * block_size]\n    int32_t block_size,               // Number of symbols per block\n    const int32_t* lut_cum_freqs,     // Flat LUT of cumulative frequencies\n    const int32_t* lut_freqs,         // Flat LUT of frequencies\n    int32_t max_alphabet_size,        // Stride for LUT indexing\n    const int32_t* all_decay_indices, // Indices into LUT for each block\n    const int32_t* all_alphabet_sizes,// Alphabet sizes for each block\n    const int32_t* all_tot_freqs,     // Total frequencies for each block (Must be 65536 for rANS)\n    const int32_t* all_sym_shifts,    // Symbol shifts for each block\n    uint8_t* all_output_buffers,      // Flat pre-allocated output buffer [num_blocks * max_output_size_per_block]\n    int32_t max_output_size_per_block,// Reserved size per block in all_output_buffers\n    int32_t* all_output_sizes         // Array to store actual written size for each block (or -1 on error)\n);\n```\n\n#### `range_decode_batch` / `ans_decode_batch`\nDecodes multiple blocks in parallel using OpenMP.\n**Note**: For rANS, `all_tot_freqs` must contain `65536` for all blocks.\n\n```cpp\nvoid range_decode_batch( // or ans_decode_batch\n    int32_t num_blocks,               // Number of blocks to decode\n    const uint8_t* all_compressed_data, // Flat array of compressed data [num_blocks * max_output_size_per_block]\n    int32_t max_output_size_per_block,// Stride for compressed data input\n    const int32_t* all_compressed_lengths, // Actual length of each compressed block\n    int32_t block_size,               // Number of symbols per block\n    const int32_t* lut_cum_freqs,     // Flat LUT of cumulative frequencies\n    const int32_t* lut_freqs,         // Flat LUT of frequencies\n    int32_t max_alphabet_size,        // Stride for LUT indexing\n    const int32_t* all_decay_indices, // Indices into LUT for each block\n    const int32_t* all_alphabet_sizes,// Alphabet sizes for each block\n    const int32_t* all_tot_freqs,     // Total frequencies for each block (Must be 65536 for rANS)\n    const int32_t* all_sym_shifts,    // Symbol shifts for each block\n    int32_t* all_out_q_vals,          // Flat array to store reconstructed integers [num_blocks * block_size]\n    int32_t* all_ret_codes            // Array to store return codes (0 on success, negative on error)\n);\n```\n\n\n### Python API (`SimpleRangeCoder`)\n\nThe Python API provides methods for both Range Coding and rANS. The argument structures are consistent across both backends.\n\n#### Range Coder Methods\n- `encode(q_vals, cum_freqs, freqs, tot_freq, sym_shift)`: Encodes a single array of integers.\n- `decode(data_bytes, block_len, cum_freqs, freqs, tot_freq, sym_shift)`: Decodes a byte stream.\n- `batch_encode(all_q_vals, lut_cum_freqs, lut_freqs, max_alphabet_size, all_decay_indices, all_alphabet_sizes, all_tot_freqs, all_sym_shifts)`: Encodes multiple blocks in parallel.\n- `batch_decode(all_compressed_data, all_compressed_lengths, block_size, lut_cum_freqs, lut_freqs, max_alphabet_size, all_decay_indices, all_alphabet_sizes, all_tot_freqs, all_sym_shifts)`: Decodes multiple blocks in parallel.\n\n#### rANS Methods\n**Note**: For all rANS methods, `tot_freq` (or values in `all_tot_freqs`) must be `65536`.\n\n- `ans_encode(q_vals, cum_freqs, freqs, tot_freq, sym_shift)`: rANS version of `encode`.\n- `ans_decode(data_bytes, block_len, cum_freqs, freqs, tot_freq, sym_shift)`: rANS version of `decode`.\n- `batch_ans_encode(all_q_vals, lut_cum_freqs, lut_freqs, max_alphabet_size, all_decay_indices, all_alphabet_sizes, all_tot_freqs, all_sym_shifts)`: rANS version of `batch_encode`.\n- `batch_ans_decode(all_compressed_data, all_compressed_lengths, block_size, lut_cum_freqs, lut_freqs, max_alphabet_size, all_decay_indices, all_alphabet_sizes, all_tot_freqs, all_sym_shifts)`: rANS version of `batch_decode`.\n\n### Detailed rANS Constraints and Optimization\n\nThe rANS backend is highly optimized for performance by using a fixed total frequency ($M = 2^{16} = 65536$). This allows the implementation to replace expensive division and multiplication operations with bitwise shifts (`\u003e\u003e 16`) and masks (`\u0026 0xFFFF`).\n\n#### Frequency Normalization for rANS\nWhen using rANS, you must scale your frequencies so they sum exactly to 65536.\n\n**Python Example:**\n```python\ndef normalize_to_65536(freqs):\n    target = 65536\n    s = np.sum(freqs)\n    normalized = (freqs.astype(np.float64) * target / s).astype(np.int32)\n    normalized[normalized == 0] = 1\n    normalized[-1] += target - np.sum(normalized) # Fix rounding\n    return normalized\n```\n\n**C++ Example:**\nSee `example_ans.cpp` for a complete example of how to set up frequency tables and metadata for batch rANS processing.\n\n## Python Integration\n\nTo use this library in your Python project:\n\n1.  **Build the shared library**: Run `make librangecoder.so` to generate the `.so` file.\n2.  **Library Placement**:\n    - By default, `SimpleRangeCoder` looks for `./librangecoder.so` in the current working directory.\n    - You can place the `.so` file anywhere and provide the path to the constructor:\n      ```python\n      rc = SimpleRangeCoder(\"/path/to/librangecoder.so\")\n      ```\n3.  **Dependencies**: Ensure `numpy` is installed in your Python environment.\n\n## Usage Example (Python)\n\n```python\nimport numpy as np\nfrom fast_rc import SimpleRangeCoder\n\n# Initialize with the path to the shared library\nrc = SimpleRangeCoder(\"./librangecoder.so\")\n\n# Prepare probability model (Example: Uniform distribution)\nalphabet_size = 257\nfreqs = np.full(alphabet_size, 4, dtype=np.int32)\ncum_freqs = np.zeros(alphabet_size + 1, dtype=np.int32)\ncum_freqs[1:] = np.cumsum(freqs)\ntot_freq = 1028\nsym_shift = 0\n\n# Input data\ninput_data = np.array([10, 20, 30, 40, 50], dtype=np.int32)\n\n# Encode\nencoded = rc.encode(input_data, cum_freqs, freqs, tot_freq, sym_shift)\n\n# Decode\ndecoded = rc.decode(encoded, len(input_data), cum_freqs, freqs, tot_freq, sym_shift)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibatch%2Fsimplerangecoder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshibatch%2Fsimplerangecoder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshibatch%2Fsimplerangecoder/lists"}