https://github.com/mswart/dit
Efficient storage system for container images
https://github.com/mswart/dit
container-image deduplication git leveldb postgresql
Last synced: 3 months ago
JSON representation
Efficient storage system for container images
- Host: GitHub
- URL: https://github.com/mswart/dit
- Owner: mswart
- License: bsd-3-clause
- Created: 2017-11-03T15:19:40.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-11-03T15:19:47.000Z (over 8 years ago)
- Last Synced: 2025-10-24T14:42:04.963Z (9 months ago)
- Topics: container-image, deduplication, git, leveldb, postgresql
- Language: Haskell
- Size: 13.7 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Dit -- Efficient image storaging inspired by Git
Dit is optimized to store directory contains container images.
File and directory metadata are stored within a PostgreSQL database for simple management.
File content is deduplicated and stored within a local LevelDB.
Dit supports currently two major commands:
* **commit**: recording the files and their content from a given directory
* **checkout**: rebuild a recorded system into the wanted directory
Furthermore, Dit can store additional artifacts for systems that are deduplicated, too.
### Why?
Containers build by Continuous Integration systems a very similar. Only some files change between different builds.
To store and built systems efficiently at least the file content should be deduplicated.
Git provides a good storage design for this, but lacks support for file metadata like user, group ... This prevent using Git for storaging system images.
Filesystems like OverlayFS, ZFS or Btrfs support some kind of deduplication but are designed for a different usage case and deduplication not all content or in an inefficient manner.
### Status
Dit is used internally for a continuous integration system.
Besides some known limitations it works well.
### Limitations
* Imroved database schema: the current schema does not duplicate file metadata. Storing many systems requires some PostgreSQL space and reduces performance
* LevelDB sync: to support usage between multiple hosts LevelDB entries must be transfered between multiple hosts
### Dependencies
* haskell
* cabal recommanded
* postgresql
* leveldb