{"id":16957755,"url":"https://github.com/mswart/dit","last_synced_at":"2026-04-14T00:01:15.564Z","repository":{"id":139751261,"uuid":"109410884","full_name":"mswart/dit","owner":"mswart","description":"Efficient storage system for container images","archived":false,"fork":false,"pushed_at":"2017-11-03T15:19:47.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-24T14:42:04.963Z","etag":null,"topics":["container-image","deduplication","git","leveldb","postgresql"],"latest_commit_sha":null,"homepage":null,"language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mswart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-03T15:19:40.000Z","updated_at":"2019-12-09T10:19:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"a03fd6fc-f5c3-4ccb-8671-7ffdd61a3c5f","html_url":"https://github.com/mswart/dit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mswart/dit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mswart%2Fdit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mswart%2Fdit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mswart%2Fdit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mswart%2Fdit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mswart","download_url":"https://codeload.github.com/mswart/dit/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mswart%2Fdit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31776013,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T20:17:16.280Z","status":"ssl_error","status_checked_at":"2026-04-13T20:17:08.216Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["container-image","deduplication","git","leveldb","postgresql"],"created_at":"2024-10-13T22:19:50.854Z","updated_at":"2026-04-14T00:01:15.541Z","avatar_url":"https://github.com/mswart.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dit -- Efficient image storaging inspired by Git\n\nDit is optimized to store directory contains container images.\nFile and directory metadata are stored within a PostgreSQL database for simple management.\nFile content is deduplicated and stored within a local LevelDB.\n\nDit supports currently two major commands: \n\n* **commit**: recording the files and their content from a given directory\n* **checkout**: rebuild a recorded system into the wanted directory\n\nFurthermore, Dit can store additional artifacts for systems that are deduplicated, too.\n\n\n### Why?\n\nContainers build by Continuous Integration systems a very similar. Only some files change between different builds.\nTo store and built systems efficiently at least the file content should be deduplicated.\nGit provides a good storage design for this, but lacks support for file metadata like user, group ... This prevent using Git for storaging system images.\nFilesystems like OverlayFS, ZFS or Btrfs support some kind of deduplication but are designed for a different usage case and deduplication not all content or in an inefficient manner.\n\n\n### Status\n\nDit is used internally for a continuous integration system.\nBesides some known limitations it works well.\n\n\n\n### Limitations\n\n* Imroved database schema: the current schema does not duplicate file metadata. Storing many systems requires some PostgreSQL space and reduces performance\n* LevelDB sync: to support usage between multiple hosts LevelDB entries must be transfered between multiple hosts\n\n\n### Dependencies\n\n* haskell\n* cabal recommanded\n* postgresql\n* leveldb\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmswart%2Fdit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmswart%2Fdit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmswart%2Fdit/lists"}