{"id":28503041,"url":"https://github.com/neurojson/zmat","last_synced_at":"2025-10-26T23:31:59.999Z","repository":{"id":45237878,"uuid":"184329767","full_name":"NeuroJSON/zmat","owner":"NeuroJSON","description":"ZMat: A portable C-library and MATLAB/Octave toolbox for zlib/gzip/lzma/lz4/zstd/blosc2 data compression","archived":false,"fork":false,"pushed_at":"2025-01-18T18:21:56.000Z","size":21708,"stargazers_count":12,"open_issues_count":0,"forks_count":6,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-07T13:37:57.637Z","etag":null,"topics":["blosc","compression","json","library","lz4","lzma","matlab","mex","neurojson","octave","toolbox","zlib"],"latest_commit_sha":null,"homepage":"https://neurojson.org","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NeuroJSON.png","metadata":{"files":{"readme":"README.rst","changelog":"ChangeLog.txt","contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.txt","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-04-30T20:50:13.000Z","updated_at":"2025-06-02T14:40:20.000Z","dependencies_parsed_at":"2023-01-20T15:46:37.502Z","dependency_job_id":"ef69b87a-4af3-4317-8e53-4e1311c70027","html_url":"https://github.com/NeuroJSON/zmat","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/NeuroJSON/zmat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeuroJSON%2Fzmat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeuroJSON%2Fzmat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeuroJSON%2Fzmat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeuroJSON%2Fzmat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NeuroJSON","download_url":"https://codeload.github.com/NeuroJSON/zmat/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NeuroJSON%2Fzmat/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265189130,"owners_count":23725213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blosc","compression","json","library","lz4","lzma","matlab","mex","neurojson","octave","toolbox","zlib"],"created_at":"2025-06-08T16:38:52.196Z","updated_at":"2025-10-26T23:31:54.956Z","avatar_url":"https://github.com/NeuroJSON.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: https://neurojson.org/wiki/upload/neurojson_banner_long.png\n\n##############################################################################                                                      \nZMAT: A portable C-library and MATLAB/Octave toolbox for zlib/gzip/lzma/lz4/zstd/blosc2 data compression\n##############################################################################\n\n* Copyright (C) 2019-2023  Qianqian Fang \u003cq.fang at neu.edu\u003e\n* License: GNU General Public License version 3 (GPL v3), see License*.txt\n* Version: 0.9.9 (Foxy the Fantastic Mr. Fox - RC1)\n* URL: https://github.com/NeuroJSON/zmat\n* Acknowledgement: This project is part of the `NeuroJSON project \u003chttps://neurojson.org\u003e`_\n  supported by US National Institute of Health (NIH)\n  grant `U24-NS124027 \u003chttps://reporter.nih.gov/project-details/10308329\u003e`_\n\n.. image:: https://github.com/NeuroJSON/zmat/actions/workflows/run_test.yml/badge.svg\n    :target: https://github.com/NeuroJSON/zmat/actions/workflows/run_test.yml\n\n#################\nTable of Contents\n#################\n.. contents::\n  :local:\n  :depth: 3\n\n============\nIntroduction\n============\n\nZMat provides both an easy-to-use C-based data compression library - \n``libzmat`` as well a portable mex function to enable ``zlib/gzip/lzma/lz4/zstd/blosc2``\nbased data compression/decompression and ``base64`` encoding/decoding support \nin MATLAB and GNU Octave. It is fast and compact, can process a \nlarge array within a fraction of a second. \n\nAmong all the supported compression methods, or codecs, ``lz4`` is among the fastest for\nboth compression/decompression; ``lzma`` is the slowest but offers the highest \ncompression ratio; ``zlib/gzip`` have excellent balance between speed \nand compression time; ``zstd`` can be fast, although not as fast as ``lz4``,\nat the low-compression levels, and can offer higher compression ratios\nthan ``zlib/gzip``, although not as high as ``lzma``, at high compression\nlevels. We understand there are many other existing general purpose data\ncompression algorithms. We prioritize the support of these compression\nalgorithms because they have widespread use.\n\nStarting in v0.9.9, we added support to a high-performance meta-compressor,\ncalled ``blosc2`` (https://blosc.org). The ``blosc2`` compressor is not single\ncompression method, but a container format that supports a diverse set of \nstrategies to losslessly compress data, especially optimized for storing and\nretrieving N-D numerical data of uniform binary types. Users can choose\nbetween ``blosc2blosclz``, ``blosc2lz4``, ``blosc2lz4hc``, ``blosc2zlib`` and\n``blosc2zstd``, to access these blosc2 codecs.\n\nThe ``libzmat`` library, including the static library (``libzmat.a``) and the\ndynamic library ``libzmat.so`` or ``libzmat.dll``, provides a simple interface to \nconveniently compress or decompress a memory buffer:\n\n.. code:: c\n\n  int zmat_run(\n        const size_t inputsize,     /* input buffer data length */\n        unsigned char *inputstr,    /* input buffer */\n        size_t *outputsize,         /* output buffer data length */\n        unsigned char **outputbuf,  /* output buffer */\n        const int zipid,            /* 0: zlib, 1: gzip, 2: base64, 3: lzma, 4: lzip, 5: lz4, 6: lz4hc \n                                       7: zstd, 8: blosc2blosclz, 9: blosc2lz4, 10: blosc2lz4hc,\n                                       11: blosc2zlib, 12: blosc2zstd */\n        int *status,                /* return status for error handling */\n        const int clevel            /* 1 to compress (default level); 0 to decompress, -1 to -9 (-22 for zstd): setting compression level */\n      );\n\nWhen ``blosc2`` codecs are used, users can set additional compression flags, including\nnumber of threads, byte-shuffling length, can be set via the ``flags.param`` interface\nin the following data structure, and pass on ``flags.iscompress`` as the last \ninput to ``zmat_run``.\n\n.. code:: c\n\n    union cflag {\n        int iscompress;      /* combined flag used to pass on to zmat_run */\n        struct settings {    /* unpacked flags */\n            char clevel;     /* compression level */\n            char nthread;    /* number of compression/decompression threads */\n            char shuffle;    /* byte shuffle length */\n            char typesize;   /* for ND-array, the byte-size for each array element */\n        } param;\n    } flags = {0};\n\nThe zmat library is highly portable and can be directly embedded in the source code \nto provide maximal portability. In the ``test`` folder, we provided sample codes\nto call ``zmat_run/zmat_encode/zmat_decode`` for stream-level compression and \ndecompression in C and Fortran90. The Fortran90 C-binding module can be found \nin the ``fortran90`` folder.\n\nThe ZMat MATLAB function accepts 3 types of inputs: char-based strings, numerical arrays\nor vectors, or logical arrays/vectors. Any other input format will \nresult in an error unless you typecast the input into ``int8/uint8``\nformat. A multi-dimensional numerical array is accepted, and the\noriginal input's type/dimension info is stored in the 2nd output\n``\"info\"``. If one calls ``zmat`` with both the encoded data (in byte vector)\nand the ``\"info\"`` structure, zmat will first decode the binary data \nand then restore the original input's type and size.\n\nThe pre-compiled mex binaries for MATLAB are stored inside the \nsubfolder named ``private``. Those precompiled for GNU Octave are\nstored in the subfolder named ``octave``, with one operating system\nper subfolder. The ``PKG_ADD`` script should automatically select\nthe correct mex file when one types ``addpath`` in Octave.\nThese precompiled mex files are expected to run out-of-box\nacross a wide-range of MATLAB (tested as old as R2008) and Octave (tested\nas old as v3.8).\n\nIf you do not want to compile zmat yourself, you can download the\nprecompiled package by either clicking on the \"Download ZIP\" button\non the above URL, or use the below git command:\n\n.. code:: shell\n\n    git clone https://github.com/NeuroJSON/zmat.git\n\n================\nInstallation\n================\n\nThe installation of ZMat is no different from any other simple\nMATLAB toolboxes. You only need to download/unzip the  package\nto a folder, and add the folder's path (that contains ``zmat.m`` and \nthe ``\"private\"`` folder) to MATLAB's path list by using the \nfollowing command:\n\n.. code:: matlab\n\n    addpath('/path/to/zmat');\n\nFor Octave, one needs to copy the ``zipmat.mat`` file inside the \"``octave``\",\nfrom the subfolder matching the OS into the \"``private``\" subfolder.\n\nIf you want to add this path permanently, you need to type \"``pathtool``\", \nbrowse to the zmat root folder and add to the list, then click \"Save\".\nThen, run \"``rehash``\" in MATLAB, and type \"``which zmat``\", if you see an \noutput, that means ZMat is installed for MATLAB/Octave.\n\nIf you use MATLAB in a shared environment such as a Linux server, the\nbest way to add path is to type \n\n.. code:: shell\n\n   mkdir ~/matlab/\n   nano ~/matlab/startup.m\n\nand type ``addpath('/path/to/zmat')`` in this file, save and quit the editor.\nMATLAB will execute this file every time it starts. For Octave, the file\nyou need to edit is ``~/.octaverc`` , where \"``~``\" is your home directory.\n\n================\nInstalling ZMat on Linux Distributions\n================\n\nOne can directly install zmat on Fedora Linux 29 or later via the \nbelow shell command\n\n.. code:: shell\n\n   sudo dnf install octave-zmat\n\nSimilarly, the below command installs the ``libzmat`` library for developing\nsoftware using this library:\n\n.. code:: shell\n\n   sudo dnf install zmat zmat-devel zmat-static\n\nThe above command installs the dynamic library, C/Fortran90 header files and\nstatic library, respectively\n\nSimilarly, if one uses Debian (11) or Ubuntu 21.04 or newer, the command to\ninstall zmat toolbox for Octave (and optionally for MATLAB) is\n\n.. code:: shell\n\n   sudo apt-get install octave-zmat matlab-zmat\n\nand that for installing the development environment is\n\n.. code:: shell\n\n   sudo apt-get install libzmat1 libzmat1-dev\n\nA Ubuntu (16.04/18.04) user can use the same commands as Debian to install these \npackages but one must first run \n\n.. code:: shell\n\n   sudo add-apt-repository ppa:fangq/ppa\n   sudo apt-get update\n\nto enable the `relevant PPA \u003chttp://https://launchpad.net/~fangq/+archive/ubuntu/ppa\u003e`_\n(personal package achieve) first.\n\n================\nUsing ZMat in MATLAB\n================\n\nZMat provides a single mex function, ``zipmat.mex*`` -- for both compressing/encoding\nor decompresing/decoding data streams. The help info of the function is shown\nbelow\n\n----------\nzmat.m\n----------\n\n.. code-block:: matlab\n\n  output=zmat(input)\n     or\n  [output, info]=zmat(input, iscompress, method)\n  [output, info]=zmat(input, iscompress, method, options ...)\n  output=zmat(input, info)\n \n  A portable data compression/decompression toolbox for MATLAB/GNU Octave\n \n  author: Qianqian Fang \u003cq.fang at neu.edu\u003e\n  initial version created on 04/30/2019\n \n  input:\n       input: a char, non-complex numeric or logical vector or array\n       iscompress: (optional) if iscompress is 1, zmat compresses/encodes the input,\n              if 0, it decompresses/decodes the input. Default value is 1.\n \n              if iscompress is set to a negative integer, (-iscompress) specifies\n              the compression level. For zlib/gzip, default level is 6 (1-9); for\n              lzma/lzip, default level is 5 (1-9); for lz4hc, default level is 8 (1-16).\n              the default compression level is used if iscompress is set to 1.\n \n              zmat removes the trailing newline when iscompress=2 and method='base64'\n              all newlines are kept when iscompress=3 and method='base64'\n \n              if one defines iscompress as the info struct (2nd output of zmat), zmat\n              will perform a decoding/decompression operation and recover the original\n              input using the info stored in the info structure.\n       method: (optional) compression method, currently, zmat supports the below methods\n              'zlib': zlib/zip based data compression (default)\n              'gzip': gzip formatted data compression\n              'lzip': lzip formatted data compression\n              'lzma': lzma formatted data compression\n              'lz4':  lz4 formatted data compression\n              'lz4hc':lz4hc (LZ4 with high-compression ratio) formatted data compression\n              'zstd':  zstd formatted data compression\n              'blosc2blosclz':  blosc2 meta-compressor with blosclz compression\n              'blosc2lz4':  blosc2 meta-compressor with lz4 compression\n              'blosc2lz4hc':  blosc2 meta-compressor with lz4hc compression\n              'blosc2zlib':  blosc2 meta-compressor with zlib/zip compression\n              'blosc2zstd':  blosc2 meta-compressor with zstd compression\n              'base64': encode or decode use base64 format\n      options: a series of ('name', value) pairs, supported options include\n              'nthread': followed by an integer specifying number of threads for blosc2 meta-compressors\n              'typesize': followed by an integer specifying the number of bytes per data element (used for shuffle)\n              'shuffle': shuffle methods in blosc2 meta-compressor, 0 disable, 1, byte-shuffle\n \n  output:\n       output: a uint8 row vector, storing the compressed or decompressed data;\n              empty when an error is encountered\n       info: (optional) a struct storing additional info regarding the input data, may have\n             'type': the class of the input array\n             'size': the dimensions of the input array\n             'byte': the number of bytes per element in the input array\n             'method': a copy of the 3rd input indicating the encoding method\n             'status': the zlib/lzma/lz4 compression/decompression function return value,\n                     including potential error codes; see documentation of the respective\n                     libraries for details\n             'level': a copy of the iscompress flag; if non-zero, specifying compression\n                     level, see above\n \n  example:\n \n    [ss, info]=zmat(eye(5))\n    orig=zmat(ss,0)\n    orig=zmat(ss,info)\n    ss=char(zmat('zmat test',1,'base64'))\n    orig=char(zmat(ss,0,'base64'))\n \n  -- this function is part of the zmat toolbox (https://github.com/NeuroJSON/zmat)\n\n---------\nexamples\n---------\n\nUnder the ``\"example\"`` folder, you can find a demo script showing the \nbasic utilities of ZMat. Running the ``\"demo_zmat_basic.m\"`` script, \nyou can see how to compress/decompress a simple array, as well as apply\nbase64 encoding/decoding to strings.\n\nPlease run these examples and understand how ZMat works before you use\nit to process your data.\n\nUnder the ``\"c\"`` and ``\"f90\"`` folders, sample C/Fortran90 units calling\nthe compression/decompression APIs provided by zmat are also provided.\nYou may run ``\"make\"`` in each of the folders to build the binary and\nexecute the output program.\n\n---------\nunit tests\n---------\n\nUnder the ``\"test\"`` folder, you can run ``\"run_zmat_test.m\"`` script to\nrun unit tests on the key features provided by zmat.\n\n==========================\nCompile ZMat\n==========================\n\nTo recompile ZMat, you first need to check out ZMat source code, along\nwith the needed submodules from the Github repository using the below \ncommand\n\n.. code:: shell\n\n      git clone https://github.com/NeuroJSON/zmat.git zmat\n\nNext, you need to make sure your system has ``gcc``, ``g++``,\n``mex`` and ``mkoctfile`` (if compiling for Octave is needed). If not, \nplease install gcc, MATLAB and GNU Octave and add the paths to \nthese utilities to the system PATH environment variable.\n\nTo compile zmat, you may choose one of the three methods:\n\n1. Method 1: please open MATLAB or Octave, and run the below commands\n\n.. code-block:: matlab\n\n      cd zmat/src\n      compilezmat\n\nThe above script utilizes the MinGW-w64 MATLAB Compiler plugin.\n\nTo install the MinGW-w64 compiler plugin for MATLAB, please follow\nthe below steps\n\n- If you have MATLAB R2017b or later, you may skip this step.\n  To compile mcxlabcl in MATLAB R2017a or earlier on Windows, you must \n  pre-install the MATLAB support for MinGW-w64 compiler \n  https://www.mathworks.com/matlabcentral/fileexchange/52848-matlab-support-for-mingw-w64-c-c-compiler\n\n  Note: it appears that installing the above Add On is no longer working\n  and may give an error at the download stage. In this case, you should\n  install MSYS2 from https://www.msys2.org/. Once you install MSYS2,\n  run MSYS2.0 MinGW 64bit from Start menu, in the popup terminal window,\n  type\n\n.. code-block:: shell\n\n     pacman -Syu\n     pacman -S base-devel gcc git zlib-devel\n\nThen, start MATLAB, and in the command window, run\n\n.. code-block:: matlab\n\n     setenv('MW_MINGW64_LOC','C:\\msys64\\usr');\n\n- After installation of MATLAB MinGW support, you must type \n  ``mex -setup C`` in MATLAB and select \"MinGW64 Compiler (C)\". \n- Once you select the MingW C compiler, you should run ``mex -setup C++``\n  again in MATLAB and select \"MinGW64 Compiler (C++)\" to compile C++.\n\n2. Method 2: Compile with cmake (3.3 or later) \n\nPlease open a terminal, and run the below shall commands\n\n.. code-block:: shell\n\n      cd zmat/src\n      rm -rf build\n      mkdir build \u0026\u0026 cd build\n      cmake ../\n      make clean\n      make\n\nif MATLAB was not installed in a standard path, you may change ``cmake ../`` to\n\n.. code-block:: shell\n\n      cmake Matlab_ROOT_DIR=/path/to/matlab/root ../\n\nby default, this will first compile ``libzmat.a`` and then create the ``.mex`` file \nthat is statically linked with ``libzmat.a``. If one prefers to create a dynamic\nlibrary ``libzmat.so`` and then a dynamically linked ``.mex`` file, this can\nbe done by\n\n.. code-block:: shell\n\n      cmake Matlab_ROOT_DIR=/path/to/matlab/root -DSTATIC_LIB=off ../\n\n\n3. Method 3: please open a terminal, and run the below shall commands\n\n.. code-block:: shell\n\n      cd zmat/src\n      make clean\n      make mex\n\nto create the mex file for MATLAB, and run ``make clean oct`` to compile\nthe mex file for Octave. \n\nThe compilex mex files are named as ``zipmat.mex*`` under the zmat root folder.\nOne may move those into the ``private`` folder to overwrite the existing files,\nor leave them in the root folder. MATLAB/Octave will use these files when \n``zmat`` is called.\n\n==========================\nContribution and feedback\n==========================\n\nZMat is an open-source project. This means you can not only use it and modify\nit as you wish, but also you can contribute your changes back to ZMat so\nthat everyone else can enjoy the improvement. For anyone who want to contribute,\nplease download ZMat source code from its source code repositories by using the\nfollowing command:\n\n\n.. code:: shell\n\n      git clone https://github.com/NeuroJSON/zmat.git zmat\n\nor browsing the github site at\n\n.. code:: shell\n\n      https://github.com/NeuroJSON/zmat\n \n\nYou can make changes to the files as needed. Once you are satisfied with your\nchanges, and ready to share it with others, please submit your changes as a\n\"pull request\" on github.  The project maintainer, Dr. Qianqian Fang will\nreview the changes and choose to accept the patch.\n\nWe appreciate any suggestions and feedbacks from you. Please use the NeuroJSON\nforum to report any questions you may have regarding ZMat:\n\n`NeuroJSON forum \u003chttps://github.com/orgs/NeuroJSON/discussions\u003e`_\n\n(Subscription to the mailing list is needed in order to post messages).\n\n\n==========================\nAcknowledgement\n==========================\n\nZMat is linked against the below open-source data compression libraries\n\n1. ZLib library: https://www.zlib.net/\n  *  Copyright (C) 1995-2017 Jean-loup Gailly and Mark Adler\n  *  License: Zlib license\n2. Eazylzma: https://github.com/lloyd/easylzma\n  *  Author: Lloyd Hilaiel (lloyd)\n  *  License: public domain\n3. Original LZMA library:\n  *  Author: Igor Pavlov\n  *  License: public domain\n4. LZ4 library: https://lz4.github.io/lz4/\n  *  Copyright (C) 2011-2019, Yann Collet.\n  *  License: BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)\n5. C-blosc2: https://blosc.org/\n  *  Copyright (c) 2009-2018 Francesc Alted \u003cfrancesc@blosc.org\u003e\n  *  Copyright (c) 2019-present The Blosc Development Team \u003cblosc@blosc.org\u003e\n  *  License: BSD 3-Clause License (http://www.opensource.org/licenses/bsd-license.php)\n6. ZStd: https://facebook.github.io/zstd/\n  *  Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.\n  *  License: BSD 3-Clause License (http://www.opensource.org/licenses/bsd-license.php)\n7. miniz: https://github.com/richgel999/miniz\n  *  Copyright (c) 2013-2014 RAD Game Tools and Valve Software\n  *  Copyright (c) 2010-2014 Rich Geldreich and Tenacious Software LLC\n  *  License: MIT License (https://github.com/richgel999/miniz/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurojson%2Fzmat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneurojson%2Fzmat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurojson%2Fzmat/lists"}