{"id":18913593,"url":"https://github.com/eerimoq/detools","last_synced_at":"2025-04-06T12:08:11.950Z","repository":{"id":49161353,"uuid":"171528674","full_name":"eerimoq/detools","owner":"eerimoq","description":"Binary delta encoding tools.","archived":false,"fork":false,"pushed_at":"2023-07-20T08:04:09.000Z","size":15760,"stargazers_count":169,"open_issues_count":2,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-30T11:11:11.521Z","etag":null,"topics":["bsdiff","delta-compression","delta-encoding","delta-update","hdiffpatch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eerimoq.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null},"funding":{"github":"eerimoq"}},"created_at":"2019-02-19T18:38:43.000Z","updated_at":"2025-03-28T00:21:30.000Z","dependencies_parsed_at":"2023-01-31T07:30:54.131Z","dependency_job_id":"19afcf5a-b007-4733-b8fe-51aa3448ce0c","html_url":"https://github.com/eerimoq/detools","commit_stats":{"total_commits":503,"total_committers":4,"mean_commits":125.75,"dds":0.007952286282306154,"last_synced_commit":"d607a58cd3f14e5bf90e04f2aa5820070b9da175"},"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eerimoq%2Fdetools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eerimoq%2Fdetools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eerimoq%2Fdetools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eerimoq%2Fdetools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eerimoq","download_url":"https://codeload.github.com/eerimoq/detools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247478321,"owners_count":20945266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bsdiff","delta-compression","delta-encoding","delta-update","hdiffpatch"],"created_at":"2024-11-08T10:08:19.697Z","updated_at":"2025-04-06T12:08:11.917Z","avatar_url":"https://github.com/eerimoq.png","language":"Python","readme":"|nala|_\n\nAbout\n=====\n\nBinary `delta encoding`_ in Python 3.6+ and C.\n\nBased on http://www.daemonology.net/bsdiff/ and `HDiffPatch`_, with\nthe following features:\n\n- bsdiff, hdiffpatch and match-blocks algorithms.\n\n- `sequential`_, hdiffpatch or `in-place`_ (resumable) patch types.\n\n- BZ2, LZ4, LZMA, `Zstandard`_, `heatshrink`_ or CRLE compression.\n\n- Sequential patches allow streaming.\n\n- Maximum file size is 2 GB for the bsdiff algorithm. There is\n  practically no limit for the hdiffpatch and match-blocks algorithms.\n\n- `Incremental apply patch`_ implemented in C, suitable for memory\n  constrained embedded devices. Only the sequential patch type is\n  supported.\n\n- `SA-IS`_ or divsufsort instead of qsufsort for bsdiff.\n\n- Optional experimental data format aware algorithm for potentially\n  smaller patches. I don't recommend anyone to use this functionality\n  as the gain is small in relation to memory usage and code\n  complexity!\n\n  There is a risk this functionality uses patent\n  https://patents.google.com/patent/EP1988455B1/en. Anyway, this\n  patent expires in August 2019 as I understand it.\n\n  Supported data formats:\n\n  - ARM Cortex-M4\n\n  - AArch64\n\nProject homepage: https://github.com/eerimoq/detools\n\nDocumentation: http://detools.readthedocs.org/en/latest\n\nInstallation\n============\n\n.. code-block:: python\n\n    pip install detools\n\nStatistics\n==========\n\nPatch sizes, memory usage (RSS) and elapsed times when creating a\npatch from Python-3.7.3.tar (79M) to Python-3.8.1.tar (84M) for\nvarious algorithm, patch type and compression combinations.\n\nSee `tests/benchmark.sh`_ for details on how the data was collected.\n\n+--------------+------------+--------+------------+------+---------+\n| Algorithm    | Patch type | Compr. | Patch size |  RSS |    Time |\n+==============+============+========+============+======+=========+\n| bsdiff       | sequential | lzma   |       3,5M | 662M | 0:24.29 |\n+--------------+------------+--------+------------+------+---------+\n| bsdiff       | sequential | none   |        86M | 646M | 0:15.20 |\n+--------------+------------+--------+------------+------+---------+\n| hdiffpatch   | hdiffpatch | lzma   |       2,4M | 523M | 0:13.74 |\n+--------------+------------+--------+------------+------+---------+\n| hdiffpatch   | hdiffpatch | none   |       7,2M | 523M | 0:10.24 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | sequential | lzma   |       2,9M | 273M | 0:08.57 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | sequential | none   |        84M | 273M | 0:01.72 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | hdiffpatch | lzma   |       2,6M | 212M | 0:06.07 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | hdiffpatch | none   |       9,7M | 212M | 0:01.30 |\n+--------------+------------+--------+------------+------+---------+\n\nSame as above, but for MicroPython ESP8266 binary releases (from 604k\nto 615k).\n\n+--------------+------------+--------+------------+------+---------+\n| Algorithm    | Patch type | Compr. | Patch size |  RSS |    Time |\n+==============+============+========+============+======+=========+\n| bsdiff       | sequential | lzma   |        71K |  46M | 0:00.64 |\n+--------------+------------+--------+------------+------+---------+\n| bsdiff       | sequential | none   |       609K |  27M | 0:00.33 |\n+--------------+------------+--------+------------+------+---------+\n| hdiffpatch   | hdiffpatch | lzma   |        65K |  42M | 0:00.37 |\n+--------------+------------+--------+------------+------+---------+\n| hdiffpatch   | hdiffpatch | none   |       123K |  25M | 0:00.32 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | sequential | lzma   |       194K |  46M | 0:00.44 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | sequential | none   |       606K |  25M | 0:00.22 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | hdiffpatch | lzma   |       189K |  43M | 0:00.38 |\n+--------------+------------+--------+------------+------+---------+\n| match-blocks | hdiffpatch | none   |       313K |  24M | 0:00.19 |\n+--------------+------------+--------+------------+------+---------+\n\nExample usage\n=============\n\nExamples in C are found in `c`_.\n\nCommand line tool\n-----------------\n\nThe create patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCreate a patch ``foo.patch`` from ``tests/files/foo/old`` to\n``tests/files/foo/new``.\n\n.. code-block:: text\n\n   $ detools create_patch tests/files/foo/old tests/files/foo/new foo.patch\n   Successfully created 'foo.patch' in 0.01 seconds!\n   $ ls -l foo.patch\n   -rw-rw-r-- 1 erik erik 127 feb  2 10:35 foo.patch\n\nCreate the same patch as above, but without compression.\n\n.. code-block:: text\n\n   $ detools create_patch --compression none \\\n         tests/files/foo/old tests/files/foo/new foo-no-compression.patch\n   Successfully created 'foo-no-compression.patch' in 0 seconds!\n   $ ls -l foo-no-compression.patch\n   -rw-rw-r-- 1 erik erik 2792 feb  2 10:35 foo-no-compression.patch\n\nCreate a hdiffpatch patch ``foo-hdiffpatch.patch``.\n\n.. code-block:: text\n\n   $ detools create_patch --algorithm hdiffpatch --patch-type hdiffpatch \\\n         tests/files/foo/old tests/files/foo/new foo-hdiffpatch.patch\n   Successfully created patch 'foo-hdiffpatch.patch' in 0.01 seconds!\n   $ ls -l foo-hdiffpatch.patch\n   -rw-rw-r-- 1 erik erik 146 feb  2 10:37 foo-hdiffpatch.patch\n\nLower memory usage with ``--algorithm match-blocks`` algorithm. Mainly\nuseful for big files. Creates slightly bigger patches than ``bsdiff``\nand ``hdiffpatch``.\n\n.. code-block:: text\n\n   $ detools create_patch --algorithm match-blocks \\\n         tests/files/foo/old tests/files/foo/new foo-hdiffpatch-64.patch\n   Successfully created patch 'foo-hdiffpatch-64.patch' in 0.01 seconds!\n   $ ls -l foo-hdiffpatch-64.patch\n   -rw-rw-r-- 1 erik erik 404 feb  8 11:03 foo-hdiffpatch-64.patch\n\nNon-sequential but smaller patch with ``--patch-type hdiffpatch``.\n\n.. code-block:: text\n\n   $ detools create_patch \\\n         --algorithm match-blocks --patch-type hdiffpatch \\\n         tests/files/foo/old tests/files/foo/new foo-hdiffpatch-sequential.patch\n   Successfully created 'foo-hdiffpatch-sequential.patch' in 0.01 seconds!\n   $ ls -l foo-hdiffpatch-sequential.patch\n   -rw-rw-r-- 1 erik erik 389 feb  8 11:05 foo-hdiffpatch-sequential.patch\n\nThe create in-place patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCreate an in-place patch ``foo-in-place.patch``.\n\n.. code-block:: text\n\n   $ detools create_patch_in_place --memory-size 3000 --segment-size 500 \\\n         tests/files/foo/old tests/files/foo/new foo-in-place.patch\n   Successfully created 'foo-in-place.patch' in 0.01 seconds!\n   $ ls -l foo-in-place.patch\n   -rw-rw-r-- 1 erik erik 672 feb  2 10:36 foo-in-place.patch\n\nThe create bsdiff patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nCreate a bsdiff patch ``foo-bsdiff.patch``, compatible with the\noriginal bsdiff program.\n\n.. code-block:: text\n\n   $ detools create_patch_bsdiff \\\n         tests/files/foo/old tests/files/foo/new foo-bsdiff.patch\n   Successfully created 'foo-bsdiff.patch' in 0 seconds!\n   $ ls -l foo-bsdiff.patch\n   -rw-rw-r-- 1 erik erik 261 feb  2 10:36 foo-bsdiff.patch\n\nThe apply patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nApply the patch ``foo.patch`` to ``tests/files/foo/old`` to create\n``foo.new``.\n\n.. code-block:: text\n\n   $ detools apply_patch tests/files/foo/old foo.patch foo.new\n   Successfully created 'foo.new' in 0 seconds!\n   $ ls -l foo.new\n   -rw-rw-r-- 1 erik erik 2780 feb  2 10:38 foo.new\n\nThe in-place apply patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nApply the in-place patch ``foo-in-place.patch`` to ``foo.mem``.\n\n.. code-block:: text\n\n   $ cp tests/files/foo/in-place-3000-500.mem foo.mem\n   $ detools apply_patch_in_place foo.mem foo-in-place.patch\n   Successfully created 'foo.mem' in 0 seconds!\n   $ ls -l foo.mem\n   -rw-rw-r-- 1 erik erik 3000 feb  2 10:40 foo.mem\n\nThe bsdiff apply patch subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nApply the patch ``foo-bsdiff.patch`` to ``tests/files/foo/old`` to\ncreate ``foo.new``.\n\n.. code-block:: text\n\n   $ detools apply_patch_bsdiff tests/files/foo/old foo-bsdiff.patch foo.new\n   Successfully created 'foo.new' in 0 seconds!\n   $ ls -l foo.new\n   -rw-rw-r-- 1 erik erik 2780 feb  2 10:41 foo.new\n\nThe patch info subcommand\n^^^^^^^^^^^^^^^^^^^^^^^^^\n\nPrint information about the patch ``foo.patch``.\n\n.. code-block:: text\n\n   $ detools patch_info foo.patch\n   Type:               sequential\n   Patch size:         127 bytes\n   To size:            2.71 KiB\n   Patch/to ratio:     4.6 % (lower is better)\n   Diff/extra ratio:   9828.6 % (higher is better)\n   Size/data ratio:    0.3 % (lower is better)\n   Compression:        lzma\n\n   Number of diffs:    2\n   Total diff size:    2.69 KiB\n   Average diff size:  1.34 KiB\n   Median diff size:   1.34 KiB\n\n   Number of extras:   2\n   Total extra size:   28 bytes\n   Average extra size: 14 bytes\n   Median extra size:  14 bytes\n\nContributing\n============\n\n#. Fork the repository.\n\n#. Install prerequisites.\n\n   .. code-block:: text\n\n      pip install -r requirements.txt\n\n#. Implement the new feature or bug fix.\n\n#. Implement test case(s) to ensure that future changes do not break\n   legacy.\n\n#. Run the tests.\n\n   .. code-block:: text\n\n      make test\n\n#. Create a pull request.\n\n.. |nala| image:: https://img.shields.io/badge/nala-test-blue.svg\n.. _nala: https://github.com/eerimoq/nala\n\n.. _SA-IS: https://sites.google.com/site/yuta256/sais\n\n.. _HDiffPatch: https://github.com/sisong/HDiffPatch\n\n.. _Incremental apply patch: https://github.com/eerimoq/detools/tree/master/c\n\n.. _delta encoding: https://en.wikipedia.org/wiki/Delta_encoding\n\n.. _heatshrink: https://github.com/atomicobject/heatshrink\n\n.. _Zstandard: https://facebook.github.io/zstd\n\n.. _sequential: https://detools.readthedocs.io/en/latest/#id1\n\n.. _in-place: https://detools.readthedocs.io/en/latest/#id3\n\n.. _c: https://github.com/eerimoq/detools/tree/master/c\n\n.. _tests/benchmark.sh: https://github.com/eerimoq/detools/tree/master/tests/benchmark.sh\n","funding_links":["https://github.com/sponsors/eerimoq"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feerimoq%2Fdetools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feerimoq%2Fdetools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feerimoq%2Fdetools/lists"}