https://github.com/opencomputeproject/Project-Zipline

Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm.
https://github.com/opencomputeproject/Project-Zipline

huffman-compression-algorithm

Last synced: 2 months ago
JSON representation

Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm.

• Host: GitHub
• URL: https://github.com/opencomputeproject/Project-Zipline
• Owner: opencomputeproject
• License: other
• Created: 2019-03-08T19:05:02.000Z (almost 6 years ago)
• Default Branch: master
• Last Pushed: 2023-04-11T15:22:03.000Z (almost 2 years ago)
• Last Synced: 2024-11-08T06:06:43.841Z (3 months ago)
• Topics: huffman-compression-algorithm
• Language: Verilog
• Homepage:
• Size: 34.3 MB
• Stars: 281
• Watchers: 35
• Forks: 47
• Open Issues: 9
• Metadata Files:
  • Readme: README.md
  • Changelog: HISTORY.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-opensource-hardware - project-zipline

README

        
## Project Zipline

## Introduction

Project Zipline is a program to accelerate innovation in lossless compression.  With this release Microsoft is making available:

* A new compression format called XP10 which is tailored for modern cloud datasets.

* All specifications for the entire pipeline.

* All RTL to support implementation for the entire pipeline.

* A testbench environment specifically developed for the VCS simulator.

Project Zipline is open-sourced under the MIT License, see the LICENSE file.

## Specifications

The following Project Zipline specifications have been included under the “specs” directory in this repository:

* Project_Zipline_Compression_Specification

* Project_Zipline_Huffman_Encoder_Micro_Architecture_Specification

* Project_Zipline_Comp_lz77_Micro_Architecture_Specification

* Project_Zipline_Crypto_Engine_Micro_Architecture_Specification

* Project_Zipline_Decompression_Top_Micro_Architecture_Specification

* Project_Zipline_Keyblob_Micro_Architecture_Specification

* Project_Zipline_KME_Micro_Architecture_Specification

* Project_Zipline_Prefix_Attach_Micro_Architecture_Specification

* Project_Zipline_Prefix_Micro_Architecture_Specification

* Project_Zipline_SSB_Micro_Architecture_Specification

* Project_Zipline_Top_Micro_Architecture_Specification

## Simulation Notes

Edit the file **`./zipline.setup`** for VCS, SYNTH, and VERDI environment settings.

Then source the file as follows:

**`source zipline.setup`**

Go to either the CCE_64, CDD_64, or KME run directory:

**`cd dv/CCE_64/run`**

**`cd dv/CDD_64/run`**

**`cd dv/KME/run`**

To build the simv executable:

**`make build_simv`**

Example for simulating a test:

**`make run_simv TESTNAME=xp10`**

Example for simulating a test with waves (.vpd):

**`make run_simv TESTNAME=xp10 WAVES=1`**

Example for simulating a test with Verdi waves (.fsdb):

**`make run_simv TESTNAME=xp10 VERDI_WAVES=1`**

All of the tests for the CCE_64, CDD_64, and KME can be run via a regress

script in the associated "run" directory:

**`dv/CCE_64/run/regress`**

**`dv/CDD_64/run/regress`**

**`dv/KME/run/regress`**

All test files are located in the associated engine directory "tests".

Each directory also contains a README file with a description of each test.

**`dv/CCE_64/tests`**

**`dv/CDD_64/tests`**

**`dv/KME/tests`**

A description of the programmable registers in the CCE/CDD/KME can be found

in the register_doc directory. Please read register_doc/README for more details.

## KME Notes

The KME RTL (rtl/cr_kme) has been modified to remove the following modules:

* AES engine in the random GUID generator

* SHA engines within the KDF function

* AES engine within the Key Decryption logic

With these reductions, the RTL only supports Key Types 1-6 without KDF and 

Key Type 0 without encryption/authentication

 

Additionally, the KME output has been reduced to support a single engine.

 

All modifications are identified with "KME_MODIFICATION_NOTE" comments

in the RTL code.

## Synthesis Notes

Note: For synthesis, please edit syn/Makefile to choose an LSF or dedicated

machine with at least 16G of memory.  Edit the contents of this line,

before "dc_shell":

**`bsub -Is -q irv-cpx-M16 -R "rusage [mem=16000]" dc_shell -f syn_example.tcl | tee ./LOGS/syn_eample.tcl.`date '+%m%d.%H:%M'`.log`**

To run synthesis for CDD

**`cd syn/cr_cddip`**

**`make syn`**

To run synthesis for CCE

**`cd syn/cr_cceip_64`**

**`make syn`**

To run synthesis for KME

**`cd syn/cr_kme`**

**`make syn`**