https://github.com/zylo117/hyckle

A memory-friendly warpper of pickle with better compression.
https://github.com/zylo117/hyckle

data-serialization local-io pickle python

Last synced: about 2 months ago
JSON representation

A memory-friendly warpper of pickle with better compression.

• Host: GitHub
• URL: https://github.com/zylo117/hyckle
• Owner: zylo117
• License: apache-2.0
• Created: 2019-06-26T07:27:45.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-07-02T02:33:00.000Z (over 4 years ago)
• Last Synced: 2024-10-12T00:55:07.093Z (3 months ago)
• Topics: data-serialization, local-io, pickle, python
• Language: Python
• Size: 45.9 KB
• Stars: 1
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Hyckle

## Introduction

Hyckle is a high-level warpper of pickle that provides data serialization for 

all python-pickleable object with the support of local read & write (like H5), 

better compression (85% smaller than original pickle) 

and more memory-efficient. Also it's a drop-in replacement for python dict and list.

Hyckle是pickle的一个高层封装，可以为所有可pickle的python变量提供数据序列化，

同时支持像H5一样的局部读写，拥有更好的压缩率（比原生pickle小了85%）,并更节省内存。

同时，它可以用于直接替换python的字典和列表。

---

## Installation

    pip install hyckle

---

## Demo

### Basic Usage

    # create or reload a hyckle file. 

    # This will create a new hyckle file if not exists or reload a existed hyckle file.

    hk = Hyckle('helloworld.hyckle')

    

    # use hyckle as a dict

    hk['test_0'] = any_pickleable_object_0

    hk['test_1'] = any_pickleable_object_1

    hk['test_2'] = any_pickleable_object_2

    hk['test_3'] = any_pickleable_object_3

    # or

    hk.add('test_0', any_pickleable_object_0)

    hk.add('test_1',any_pickleable_object_1)

    hk.add('test_2',any_pickleable_object_2)

    hk.add('test_3',any_pickleable_object_3)

    

    # use hyckle as a list

    hk.append(any_pickleable_object_4)

    hk.append(any_pickleable_object_5)

    

    # check the length of hyckle

    length = len(hk)

    

    # use hyckle as a iterable object

    for obj in hk:

        do_something(obj)

    

    # get certain object by key: str

    obj_0 = hk.get(key)

    # or using getitem method

    obj_1 = hk[key]

    

    # get certain object by index: int

    obj_2 = hk[index]

    

    # get certain items by slice

    objs = hk[233:666]

    

    # remove key in hyckle

    hk.remove('test_0')

    

    # flush to disk. 

    # Normally hyckle will auto-flush to disk every K item adding (K = buffersize)

    hk.flush()

    

    # close a hyckle file. 

    # After final reading/writing hyckle, you should close it to ensure data integrity.

    hk.close()

### Advanced Usage

    # create a hyckle with custom compression,

    # lzma is much slower but smaller

    hk = Hyckle('helloworld.hyckle', compression='lzma')

    

    # or use gzip (default) with different compression level,

    # bigger number means smaller filesize.

    hk = Hyckle('helloworld.hyckle', compression=9)

    

    # sppedup I/O using bigger buffer.

    # Noted, bigger buffersize means allocating more memory.

    hk = Hyckle('helloworld.hyckle', buffer_size=128)