{"id":15576400,"url":"https://github.com/njsmith/zs","last_synced_at":"2025-09-05T20:31:54.028Z","repository":{"id":9757521,"uuid":"11723913","full_name":"njsmith/zs","owner":"njsmith","description":"ZS is a compressed, read-only file format for efficiently distributing, querying, and archiving arbitrarily large record-oriented datasets.","archived":false,"fork":false,"pushed_at":"2015-10-27T05:42:37.000Z","size":1188,"stargazers_count":45,"open_issues_count":3,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-30T01:42:02.345Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/njsmith.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-28T18:35:57.000Z","updated_at":"2024-11-28T16:27:37.000Z","dependencies_parsed_at":"2022-07-08T10:21:53.565Z","dependency_job_id":null,"html_url":"https://github.com/njsmith/zs","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/njsmith%2Fzs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/njsmith","download_url":"https://codeload.github.com/njsmith/zs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232059243,"owners_count":18466730,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T18:48:38.051Z","updated_at":"2025-01-01T07:33:59.768Z","avatar_url":"https://github.com/njsmith.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"ZS is a simple, read-only, binary file format designed for\ndistributing, querying, and archiving arbitrarily large\nrecord-oriented datasets (up to tens of terabytes and beyond). It\nallows the data to be stored in compressed form, while still\nsupporting very fast queries for either specific entries, or for all\nentries in a specified range of values (e.g., prefix searches), and\nallows highly-CPU-parallel decompression. It also places an emphasis\non data integrity -- all data is protected by 64-bit CRC checksums --\nand on discoverability -- every ZS file includes arbitrarily detailed\nstructured metadata stored directly inside it.\n\nBasically you can think of ZS as a turbo-charged replacement for\nstoring data in line-based text file formats. It was originally\ndeveloped to provide a better way to work with the massive `Google N-grams\n\u003chttp://storage.googleapis.com/books/ngrams/books/datasetsv2.html\u003e`_,\nbut is potentially useful for data sets of any size.\n\n.. image:: https://travis-ci.org/njsmith/zs.png?branch=master\n   :target: https://travis-ci.org/njsmith/zs\n.. image:: https://coveralls.io/repos/njsmith/zs/badge.png?branch=master\n   :target: https://coveralls.io/r/njsmith/zs?branch=master\n\nDocumentation:\n  http://zs.readthedocs.org/\n\nInstallation:\n  You need either Python **2.7**, or else Python **3.3 or greater**.\n\n  Because ``zs`` includes a C extension, you'll also need a C compiler\n  and Python headers. On Ubuntu or Debian, for example, you get these\n  with::\n\n    sudo apt-get install build-essential python-dev\n\n  Once you have the ability to build C extensions, then on Python\n  3 you should be able to just run::\n\n    pip install zs\n\n  On Python 2.7, things are slightly more complicated: here, ``zs``\n  requires the ``backports.lzma`` package, which in turn requires the\n  liblzma library. On Ubuntu or Debian, for example, something like\n  this should work::\n\n    sudo apt-get install liblzma-dev\n    pip install backports.lzma\n    pip install zs\n\n  ``zs`` also requires the following packages: ``six``, ``docopt``,\n  ``requests``. However, these are all pure-Python packages which pip\n  will install for you automatically when you run ``pip install zs``.\n\nDownloads:\n  http://pypi.python.org/pypi/zs/\n\nCode and bug tracker:\n  https://github.com/njsmith/zs\n\nContact:\n  Nathaniel J. Smith \u003cnathaniel.smith@ed.ac.uk\u003e\n\nCitation:\n  If you use this software (or the ZS format in general) in\n  work that leads to a scientific publication, and feel that a\n  citation would be appropriate, then here is a possible citation:\n\n  Smith, N. J. (submitted). ZS: A file format for efficiently\n  distributing, using, and archiving record-oriented data sets of\n  any size. Retrieved from http://vorpus.org/papers/draft/zs-paper.pdf\n\n  In addition, if you wish to document exactly which version of this\n  software you used, then each official release has its own DOI which\n  you can find on the `change history page\n  \u003chttp://zs.readthedocs.org/en/latest/changes.html\u003e`_.\n\n  It may make sense to cite either or both of these at the same time\n  -- the paper gives a general introduction to the ZS format and why\n  it is useful, the version-specific DOIs link directly to archived\n  snapshots of a specific version of this specific implementation of\n  the ZS format. FWIW, I personally benefit more from citations to the\n  paper.\n\nDeveloper dependencies (only needed for hacking on source):\n  * Cython: needed to build from checkout\n  * nose: needed to run tests\n  * nose-cov: because we use multiprocessing, we need this package to\n    get useful test coverage information\n  * nginx: needed to run HTTP tests\n\nLicense:\n  2-clause BSD, see LICENSE.txt for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fzs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnjsmith%2Fzs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnjsmith%2Fzs/lists"}