{"id":22194792,"url":"https://github.com/creatingnull/squash-pickle","last_synced_at":"2025-07-26T23:32:11.104Z","repository":{"id":257789897,"uuid":"861065803","full_name":"CreatingNull/Squash-Pickle","owner":"CreatingNull","description":null,"archived":false,"fork":false,"pushed_at":"2024-11-25T20:16:47.000Z","size":17,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-29T03:08:21.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CreatingNull.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-21T22:54:07.000Z","updated_at":"2024-09-22T00:12:41.000Z","dependencies_parsed_at":null,"dependency_job_id":"95606df2-70df-4514-b8b7-d80c938ac7ca","html_url":"https://github.com/CreatingNull/Squash-Pickle","commit_stats":null,"previous_names":["creatingnull/squash-pickle"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CreatingNull%2FSquash-Pickle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CreatingNull%2FSquash-Pickle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CreatingNull%2FSquash-Pickle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CreatingNull%2FSquash-Pickle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CreatingNull","download_url":"https://codeload.github.com/CreatingNull/Squash-Pickle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227732267,"owners_count":17811370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T13:14:26.656Z","updated_at":"2025-07-26T23:32:11.098Z","avatar_url":"https://github.com/CreatingNull.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ![Logo](resources/Squash-Pickle.png) Squash Pickle\n\n[![License](https://img.shields.io/:license-mit-blue.svg)](https://github.com/CreatingNull/Squash-Pickle/blob/master/LICENSE.md)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/squashpickle?logo=python\u0026logoColor=white)](https://pypi.org/project/squashpickle/)\n[![PyPI](https://img.shields.io/pypi/v/squashpickle?logo=pypi\u0026logoColor=white)](https://pypi.org/project/squashpickle/#history)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/CreatingNull/Squash-Pickle/main.svg)](https://results.pre-commit.ci/latest/github/CreatingNull/Squash-Pickle/main)\n[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/CreatingNull/squash-pickle/run-tests.yaml?branch=main\u0026label=tests\u0026logo=github)](https://github.com/CreatingNull/squash-pickle/actions/workflows/run-tests.yaml)\n\nLike a pickle, only smaller\\*.\n\nTiny python package that compresses your pickles using gzip.\nQuacks like a pickle.\n\n\\* For small objects (\u003c 100 bytes) gzip overhead can end up increasing size.\nOnly squash your pickles when you are working with big objects.\n\n______________________________________________________________________\n\n## Getting Started\n\nFirst install the package, this has no additional dependencies:\n\n```shell\npip install squashpickle\n```\n\nThen simply replace your `pickle` calls with `sqaushpickle` ones.\n`squashpickle` implements, `dump`, `dumps`, `load`, and `loads` functions.\n\n______________________________________________________________________\n\n## Performance\n\nThe GZIP compression can have a **HUGE** impact on large objects.\nSay you are pickling something like a polars / pandas dataframe, these pickles may end up being hundreds of MBs.\nWith squashpickle can get compression ratios exceeding 10x.\n\nFor example if we load a large dataframe of australian weather data.\nUsing pickle this object serialises to `37794198` bytes (~37.8MB).\nDumping the same dataframe with `squashpickle` results in `3370363` bytes (~3.4MB), around 9% of the overall file.\n\n```python\nimport polars as pl\nimport pickle\nimport squashpickle\n\ndf = pl.read_csv(r\"C:\\temp\\weatherAUS.csv\", null_values=[\"NA\"])\nprint(len(pickle.dumps(df)), len(squashpickle.dumps(df)))\n```\n\nAs with any compression, there is a performance cost to achieving the smaller files.\nFor objects \\\u003c1MB this is hardly noticeable, but for objects hundreds of MBs the delay can be significant.\nIt'll depend on your use case if this is a worthwhile tradeoff.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcreatingnull%2Fsquash-pickle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcreatingnull%2Fsquash-pickle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcreatingnull%2Fsquash-pickle/lists"}