{"id":19767155,"url":"https://github.com/green-sky/solanaceae_object_store","last_synced_at":"2025-02-28T04:18:14.477Z","repository":{"id":232948698,"uuid":"785452251","full_name":"Green-Sky/solanaceae_object_store","owner":"Green-Sky","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-13T19:06:09.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2024-04-14T08:55:11.781Z","etag":null,"topics":["objectstore","solanaceae"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Green-Sky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-11T23:01:24.000Z","updated_at":"2024-06-25T12:03:24.440Z","dependencies_parsed_at":"2024-04-12T16:34:25.382Z","dependency_job_id":"46ef8d35-7a42-4f6e-a3ec-a09cb78504c2","html_url":"https://github.com/Green-Sky/solanaceae_object_store","commit_stats":null,"previous_names":["green-sky/solanaceae_object_store"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Green-Sky%2Fsolanaceae_object_store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Green-Sky%2Fsolanaceae_object_store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Green-Sky%2Fsolanaceae_object_store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Green-Sky%2Fsolanaceae_object_store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Green-Sky","download_url":"https://codeload.github.com/Green-Sky/solanaceae_object_store/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241099998,"owners_count":19909634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["objectstore","solanaceae"],"created_at":"2024-11-12T04:28:11.066Z","updated_at":"2025-02-28T04:18:14.461Z","avatar_url":"https://github.com/Green-Sky.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Object Store\n\nObjects are are pieces of information split into Metadata and Data.\nThey can be stored seperated or together.\nThey can be used as a Transport protocol/logic too.\n\n[see wikipedia](https://en.wikipedia.org/wiki/Object_store)\n\n# Store types\n\n### Filesystem Store\n\nFragment files are stored with the first 2 hex chars as sub folders:\neg:\n`objects/` (object store root)\n  - `5f/` (first 2hex subfolder)\n    - `4fffffff` (the fragment file without the first 2 hexchars)\n\nMetadata and data stored in seperate files.\nMetadata files have the `.meta` suffix. They also have a filetype specific suffix, like `.json`, `.msgpack` etc.\n(single file objects are planned but no ETA)\n\n### Memory Store\n\nJust keeps the Fragments in memory.\n\n# File formats\n\nFiles can be compressed and encrypted. Since compression needs the data's structure to work properly, it is applied before it is encrypted.\n\n### Text Json\n\nText json only makes sense for metadata if it's neither compressed nor encrypted. (otherwise its binary on disk anyway, so why waste bytes).\nSince the content of data is not looked at, nothing stops you from using text json and ecrypt it, but atleast basic compression is advised.\n\nA Metadata json object can have arbitrary keys, some are predefined:\n- `FragComp::DataEncryptionType` (uint) Encryption type of the data, if any\n- `FragComp::DataCompressionType` (uint) Compression type of the data, if any\n\n### Binary msgpack\n\nmsgpack array:\n\n- `[0]`: file magic string `SOLMET` (6 bytes)\n- `[1]`: uint8 encryption type (`0x00` is none)\n- `[2]`: uint8 compression type (`0x00` is none, `0x01` is zstd)\n- `[3]`: binary metadata (optionally compressed and encrypted)\n\nnote that the encryption and compression are for the metadata only.\nThe metadata itself contains encryption and compression info about the data.\n\n### Split Data\n\nAll the metadata is in the metadata file. (like encryption and compression)\nThis is mostly to allow direct storage for files in the Fragment store without excessive duplication.\nKeep in mind to not use the actual file name as the data/meta file name.\n\n## Compression types\n\n- `0x00` none\n- `0x01` zstd (without dict)\n\nzstd with hardcoded dictionaries will come later.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreen-sky%2Fsolanaceae_object_store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgreen-sky%2Fsolanaceae_object_store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgreen-sky%2Fsolanaceae_object_store/lists"}