{"id":18470435,"url":"https://github.com/ds2-lab/elf","last_synced_at":"2025-04-16T01:28:36.157Z","repository":{"id":234105478,"uuid":"788284941","full_name":"ds2-lab/ELF","owner":"ds2-lab","description":"Everything You Always Wanted to Know About Storage Compressibility of Pre-Trained ML Models but Were Afraid to Ask (VLDB'24)","archived":false,"fork":false,"pushed_at":"2024-04-19T21:27:49.000Z","size":60,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-29T03:32:02.385Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ds2-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-18T05:58:35.000Z","updated_at":"2024-11-05T13:20:44.000Z","dependencies_parsed_at":null,"dependency_job_id":"c8e5d06d-7954-4485-b110-95c083fc2add","html_url":"https://github.com/ds2-lab/ELF","commit_stats":null,"previous_names":["ds2-lab/elf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2FELF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2FELF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2FELF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ds2-lab%2FELF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ds2-lab","download_url":"https://codeload.github.com/ds2-lab/ELF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249181981,"owners_count":21225986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T10:13:58.938Z","updated_at":"2025-04-16T01:28:36.139Z","avatar_url":"https://github.com/ds2-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELF\n\nELF (Exponent-Less Float-point encoding) is a simple yet effective, near-lossless floating-point compression method, that transforms floating-point parameters within pre-trained models (PTMs) in such a way that the common exponent field of the transformed parameters can be completely eliminated to save storage space. ELF is embarrassingly parallel via data parallelism, achieving an extreme compression \u0026 decompression speed. \n\u003cdiv align=\"center\"\u003e\n  \u003cimg width=\"350\" alt=\"elf_com_decom\" src=\"https://github.com/ds2-lab/ELF/assets/21178173/9c673335-a588-48f7-a8a1-88e6e916781a\"\u003e\n\u003c/div\u003e\n\n\nWe also developed ELVES, a compression framework integrating ELF and several other data reduction methods. ELVES uses the most effective method to compress PTMs that exhibit different patterns. \n\n## Publication\nEverything You Always Wanted to Know About Storage Compressibility of Pre-Trained ML Models but Were Afraid to Ask (VLDB'24 to appear).\n\nThe preprint of our VLDB'24 paper can be viewed at: https://arxiv.org/abs/2402.13429.\n\nThis branch contains the source code of ELF \u0026 ELVES.\n\n## ELF Build\nPackage Installation\n```bash\nsudo apt-get update\nsudo apt-get upgrade -y\nsudo apt-get install -y gcc g++ libeigen3-dev make python3-pip\n```\nBuild ELF and DE\n```bash\nmkdir build\nmake\n```\n\n## ELVES Compression Examples\nPackage Installation\n```\npip3 install -r requirements.txt\n```\nRun the Example\n```\ncd ELVES\npython3 main.py\n```\n\n## ELF Usage\n```\n./build/elf_pthread -c -i ELVES/model_compressed/elves_compression/distilbert-base-cased/fl_weights/f32_65783040.bin -p f32 -o ELVES/model_compressed/elves_compression/distilbert-base-cased/exponential_dedup/f32/ -n 65783040\n```\n### Parameter Explanation\n| Parameter | Explanation |\n|:----------|:----------|\n| -i | input binary file |\n| -p | parameter type: f16 for floating 16, f32 for floating 32, f64 for floating 64 |\n| -n | parameter number |\n| -c | compression      |\n| -d | decompression    |\n| -o | output folder specified |\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Felf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fds2-lab%2Felf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fds2-lab%2Felf/lists"}