{"id":13936262,"url":"https://github.com/amit1rrr/numcompress","last_synced_at":"2025-12-24T17:17:15.316Z","repository":{"id":54536231,"uuid":"119820234","full_name":"amit1rrr/numcompress","owner":"amit1rrr","description":"Python package to compress numerical series \u0026 numpy arrays into strings","archived":false,"fork":false,"pushed_at":"2022-11-07T07:02:06.000Z","size":46,"stargazers_count":96,"open_issues_count":0,"forks_count":6,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-27T04:30:51.565Z","etag":null,"topics":["compression","compression-library","decompression","numpy-arrays","series-data"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amit1rrr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-01T10:26:49.000Z","updated_at":"2024-11-12T20:24:52.000Z","dependencies_parsed_at":"2022-08-13T19:00:52.271Z","dependency_job_id":null,"html_url":"https://github.com/amit1rrr/numcompress","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/amit1rrr/numcompress","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amit1rrr%2Fnumcompress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amit1rrr%2Fnumcompress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amit1rrr%2Fnumcompress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amit1rrr%2Fnumcompress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amit1rrr","download_url":"https://codeload.github.com/amit1rrr/numcompress/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amit1rrr%2Fnumcompress/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266019657,"owners_count":23864916,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compression","compression-library","decompression","numpy-arrays","series-data"],"created_at":"2024-08-07T23:02:31.756Z","updated_at":"2025-12-24T17:17:15.307Z","avatar_url":"https://github.com/amit1rrr.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![PyPI version](https://badge.fury.io/py/numcompress.svg)](https://badge.fury.io/py/numcompress) [![Build Status](https://travis-ci.org/amit1rrr/numcompress.svg?branch=master)](https://travis-ci.org/amit1rrr/numcompress)  [![Coverage Status](https://coveralls.io/repos/github/amit1rrr/numcompress/badge.svg)](https://coveralls.io/github/amit1rrr/numcompress)\n\n# numcompress\nSimple way to compress and decompress numerical series \u0026 numpy arrays.\n- Easily gets you above 80% compression ratio\n- You can specify the precision you need for floating points (up to 10 decimal points)\n- Useful to store or transmit stock prices, monitoring data \u0026 other time series data in compressed string format\n\nCompression algorithm is based on [google encoded polyline format](https://developers.google.com/maps/documentation/utilities/polylinealgorithm). I modified it to preserve arbitrary precision and apply it to any numerical series. The work is motivated by usefulness of [time aware polyline](https://www.hypertrack.com/blog/2016/09/01/the-missing-dimension-in-geospatial-data-formats/) built by [Arjun Attam](https://github.com/arjun27) at [HyperTrack](https://github.com/hypertrack/time-aware-polyline-py).\nAfter building this I came across [arrays](https://docs.python.org/3/library/array.html) that are much efficient than lists in terms memory footprint. You might consider using that over numcompress if you don't care about conversion to string for transmitting or storing purpose.\n\n# Installation\n```\npip install numcompress\n```\n\n# Usage\n```python\nfrom numcompress import compress, decompress\n\n# Integers\n\u003e\u003e\u003e compress([14578, 12759, 13525])\n'B_twxZnv_nB_bwm@'\n\n\u003e\u003e\u003e decompress('B_twxZnv_nB_bwm@')\n[14578.0, 12759.0, 13525.0]\n```\n\n```python\n# Floats - lossless compression\n# precision argument specifies how many decimal points to preserve, defaults to 3\n\u003e\u003e\u003e compress([145.7834, 127.5989, 135.2569], precision=4)\n'Csi~wAhdbJgqtC'\n\n\u003e\u003e\u003e decompress('Csi~wAhdbJgqtC')\n[145.7834, 127.5989, 135.2569]\n```\n```python\n# Floats - lossy compression\n\u003e\u003e\u003e compress([145.7834, 127.5989, 135.2569], precision=2)\n'Acn[rpB{n@'\n\n\u003e\u003e\u003e decompress('Acn[rpB{n@')\n[145.78, 127.6, 135.26]\n```\n```python\n# compressing and decompressing numpy arrays\n\u003e\u003e\u003e from numcompress import compress_ndarray, decompress_ndarray\n\u003e\u003e\u003e import numpy as np\n\n\u003e\u003e\u003e series = np.random.randint(1, 100, 25).reshape(5, 5)\n\u003e\u003e\u003e compressed_series = compress_ndarray(series)\n\u003e\u003e\u003e decompressed_series = decompress_ndarray(compressed_series)\n\n\u003e\u003e\u003e series\narray([[29, 95, 10, 48, 20],\n       [60, 98, 73, 96, 71],\n       [95, 59,  8,  6, 17],\n       [ 5, 12, 69, 65, 52],\n       [84,  6, 83, 20, 50]])\n\n\u003e\u003e\u003e compressed_series\n'5*5,Bosw@_|_Cn_eD_fiA~tu@_cmA_fiAnyo@o|k@nyo@_{m@~heAnrbB~{BonT~lVotLoinB~xFnkX_o}@~iwCokuCn`zB_ry@'\n\n\u003e\u003e\u003e decompressed_series\narray([[29., 95., 10., 48., 20.],\n       [60., 98., 73., 96., 71.],\n       [95., 59.,  8.,  6., 17.],\n       [ 5., 12., 69., 65., 52.],\n       [84.,  6., 83., 20., 50.]])\n\n\u003e\u003e\u003e (series == decompressed_series).all()\nTrue\n```\n\n\n# Compression Ratio\n\n| Test          | # of Numbers          | Compression ratio |\n| ------------- |-------------- |---------------------------|\n| [Integers](https://github.com/amit1rrr/numcompress/blob/master/test/test_numcompress.py#L29)    | 10k | **91.14%** |\n| [Floats](https://github.com/amit1rrr/numcompress/blob/master/test/test_numcompress.py#L49)      | 10k | **81.35%** |\n\nYou can run the test suite with -s switch to see the compression ratio. You can even modify the tests to see what kind of compression ratio you will get for your own input.\n```\npytest -s\n```\n\nHere's a quick example showing compression ratio:\n\n```python\n\u003e\u003e\u003e series = random.sample(range(1, 100000), 50000)  # generate 50k random numbers between 1 and 100k\n\u003e\u003e\u003e text = compress(series)  # apply compression\n\n\u003e\u003e\u003e original_size = sum(sys.getsizeof(i) for i in series)\n\u003e\u003e\u003e original_size\n1200000\n\n\u003e\u003e\u003e compressed_size = sys.getsizeof(text)\n\u003e\u003e\u003e compressed_size\n284092\n\n\u003e\u003e\u003e compression_ratio = ((original_size - compressed_size) * 100.0) / original_size\n\u003e\u003e\u003e compression_ratio\n76.32566666666666\n```\n\nWe get ~76% compression for 50k random numbers between 1 \u0026 100k. This ratio increases for real world numerical series as the difference between consecutive numbers tends to be lower. Think of stock prices, monitoring \u0026 other time series data.\n\n\n# Contribute\nIf you see any problem, open an issue or send a pull request. You can write to [me](https://blog.amirathi.com/about/) at [hello@amirathi.com](mailto:hello@amirathi.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famit1rrr%2Fnumcompress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famit1rrr%2Fnumcompress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famit1rrr%2Fnumcompress/lists"}