{"id":19577134,"url":"https://github.com/halforbit/data-stores","last_synced_at":"2025-04-27T06:32:02.796Z","repository":{"id":25358854,"uuid":"103807211","full_name":"halforbit/data-stores","owner":"halforbit","description":"Data Stores lets you use simple cloud storage to create serverless, distributed, highly available, low-cost databases and caches with sophisticated keying, richly structured values, and compression at-rest.  Data Stores integrates with a wide variety of storage providers and formats, all using a single, simple abstraction.","archived":false,"fork":false,"pushed_at":"2023-08-21T05:18:49.000Z","size":644,"stargazers_count":13,"open_issues_count":3,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-06-17T17:11:50.243Z","etag":null,"topics":["azure","azure-storage","blob-storage","caching","compression","csv","database","distributed","document-database","files","images","json","local-storage","nosql","nosql-database","serialization","serverless","storage","tsv","yaml"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/halforbit.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-17T06:09:01.000Z","updated_at":"2024-06-21T05:45:02.971Z","dependencies_parsed_at":"2024-06-21T05:45:00.911Z","dependency_job_id":"dacb032d-bac4-487f-bf1f-2bfbdb1ecd66","html_url":"https://github.com/halforbit/data-stores","commit_stats":{"total_commits":270,"total_committers":7,"mean_commits":38.57142857142857,"dds":0.6111111111111112,"last_synced_commit":"e1b0cf986c2c12afd6198daa68f80bfd0d283adb"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halforbit%2Fdata-stores","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halforbit%2Fdata-stores/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halforbit%2Fdata-stores/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/halforbit%2Fdata-stores/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/halforbit","download_url":"https://codeload.github.com/halforbit/data-stores/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224024056,"owners_count":17243009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-storage","blob-storage","caching","compression","csv","database","distributed","document-database","files","images","json","local-storage","nosql","nosql-database","serialization","serverless","storage","tsv","yaml"],"created_at":"2024-11-11T07:04:40.815Z","updated_at":"2024-11-11T07:04:41.935Z","avatar_url":"https://github.com/halforbit.png","language":"C#","readme":"# Halforbit Data Stores\r\n\r\n[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) \u0026nbsp;[![Build status](https://ci.appveyor.com/api/projects/status/w8tliyvw96obytai?svg=true)](https://ci.appveyor.com/project/halforbit/data-stores) \u0026nbsp;[![Nuget Package](https://img.shields.io/nuget/v/Halforbit.DataStores.svg)](#nuget-packages)\r\n\r\nData Stores lets you use simple cloud storage to create serverless, distributed, highly available, low-cost databases and caches with sophisticated keying, richly structured values, and compression at-rest. \r\n\r\nData Stores integrates with a wide variety of storage providers and formats, all using a single, simple abstraction. \r\n\r\n## Features\r\n\r\n- **Inexpensive Serverless Databases:** Use simple cloud storage to create serverless, distributed, highly available, low-cost databases. \r\n\r\n- **Easy Distributed Caches:** Create persistent caches to increase performance of backend API controllers, or reduce the number of calls to expensive third-party web APIs.\r\n\r\n- **Many Supported Storage Providers:** Including **Azure Blobs**, **Azure Tables**, **Amazon S3**, **FTP**, **SFTP**, and **Local File Storage**.\r\n\r\n- **Many Supported Formats:** Including human-readable structured data like **JSON** and **YAML**, efficient structured binary data with **Bond** or **Protobuf**, delimited tabular data like **CSV** and **TSV**, and binary data such as **images** and **raw text**. \r\n\r\n- **Optional Compression:** Flip a switch and compress data payloads up to 10x with **GZip** or **LZMA**, improving response times and reducing storage and bandwidth costs.\r\n\r\n- **One Easy Abstraction:** Trade in your error prone bare metal integrations and high ceremony repository patterns for a single, elegant `IDataStore` interface that is uniform across storage platforms and formats.\r\n\r\n## Getting Started\r\n\r\n1. **Install NuGet Packages:** Install the NuGet packages for your desired storage providers and formats:\r\n    ```powershell\r\n    Install-Package Halforbit.DataStores\r\n    ```\r\n    See the [NuGet Packages](#nuget-packages) section below for a list of available NuGet packages and what storage providers and formats they support.\r\n\r\n2. **Define Your Stores:** Use the `DataStore` type to create ad-hoc stores, or define them as properties on a data context.\r\n   \r\n3. **Use Your Stores:** Persist and retrieve data with your stores, or inspect write times, sizes, metadata and more with the stores' `Context` property.\r\n\r\nThere are many options to choose from when creating a store. A store can:\r\n- be defined as a property of a **data context**, or created **ad-hoc** and stored in a local variable. \r\n- be **keyed** with **simple** types like `Guid` or `string`, with the **properties** of a `class`, `record`, or `struct`, or be a keyless **singleton** storing a single value.\r\n- store **values** as **simple** types like `string` or `byte[]`, or **structured** `class`, `record`, or `struct` types serialized as e.g. JSON or TSV.\r\n\r\n\r\n## Example Usage\r\n\r\n\r\nLet's make a simple ad-hoc store to persist a `record` value using Azure Blobs, formatted as JSON, and key it with the properties of a `record` key.\r\n\r\nFirst, we define a type to represent our **value**:\r\n\r\n```cs\r\npublic record Person(\r\n    string Department,\r\n    Guid PersonId,\r\n    string FirstName,\r\n    string LastName);\r\n```\r\n\r\nNext we define a type to represent our **key**:\r\n\r\n```cs\r\npublic record PersonKey(\r\n    string Department,\r\n    Guid PersonId);\r\n```\r\n\r\nNext we create a store ad-hoc with a key type of `PersonKey`, and a value type of `Person`:\r\n\r\n```cs\r\nvar store = DataStore\r\n    .Describe()\r\n    .BlobStorage()\r\n    .ConnectionString(\"\u003cconnection-string-here\u003e\")\r\n    .Container(\"\u003ccontainer-name-here\u003e\")\r\n    .ContentType(\"application/json\")\r\n    .DefaultContentEncoding()\r\n    .JsonSerialization()\r\n    .NoCompression()\r\n    .FileExtension(\".json\")\r\n    .Map\u003cPersonKey, Person\u003e(k =\u003e $\"people/{k.Department}/{k.PersonId}\")\r\n    .Build();\r\n```\r\n\r\nThe `Map\u003c,\u003e` method lets us define a bi-directional mapping between a `PersonKey` and a `string`. \r\n\r\nNote that `people` here is acting similarly to a table or collection name, and the `Department` property is acting as a kind of partition key.\r\n\r\nNow we can put a `Person` in the store with the `Upsert` method:\r\n\r\n```cs\r\nvar key = new PersonKey(\r\n    Department: \"development\",\r\n    PersonId: Guid.NewGuid());\r\n\r\nvar value = new Person(\r\n    Department: key.Department,\r\n    PersonId: key.PersonId,\r\n    FirstName: \"Steve\",\r\n    LastName: \"Smith\");\r\n\r\nawait store.Upsert(key, value);\r\n```\r\n\r\nWe can get a `Person` by their `PersonKey`:\r\n\r\n```cs\r\nvar value = await store.Get(key);\r\n```\r\n\r\nWe can delete a `Person` by their `PersonKey`:\r\n\r\n```cs\r\nawait store.Delete(key);\r\n```\r\n\r\nWe can list the `PersonKey`s in the store, optionally filtered by `PersonKey.Department`:\r\n\r\n```cs\r\nvar keys = await store.ListKeys(k =\u003e k.Department == \"development\");\r\n```\r\n\r\nWe can get all of the `Person`s in the store, optionally filtered by `PersonKey.Department`:\r\n\r\n```cs\r\nvar values = await store.ListValues(k =\u003e k.Department == \"development\");\r\n```\r\n\r\n\u003ca name=\"nuget-packages\"\u003e\u003c/a\u003e\r\n## NuGet Packages\r\n\r\nThe following NuGet packages are provided, parted out by their dependencies. Install the ones that contain the storage providers and formats you wish to use.\r\n\r\n| Storage Provider or Format | NuGet Package |\r\n|----------------------------|---------------|\r\n| Local File Storage, JSON Serialization, Raw Byte Serialization, GZip Compression | [`Halforbit.DataStores`](https://www.nuget.org/packages/Halforbit.DataStores) |\r\n| Azure Blob Storage | [`Halforbit.DataStores.FileStores.BlobStorage`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.BlobStorage) |\r\n| Azure Table Storage | [`Halforbit.DataStores.TableStores.AzureTables`](https://www.nuget.org/packages/Halforbit.DataStores.TableStores.AzureTables) |\r\n| Amazon S3 Storage | [`Halforbit.DataStores.FileStores.AmazonS3`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.AmazonS3) |\r\n| YAML Serialization | [`Halforbit.DataStores.FileStores.Serialization.Yaml`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.Serialization.Yaml) |\r\n| Protobuf Serialization | [`Halforbit.DataStores.FileStores.Serialization.Protobuf`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.Serialization.Protobuf) |\r\n| Bond Serialization | [`Halforbit.DataStores.FileStores.Serialization.Bond`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.Serialization.Bond) |\r\n| LZMA Compression | [`Halforbit.DataStores.FileStores.Compression.Lzma`](https://www.nuget.org/packages/Halforbit.DataStores.FileStores.Compression.Lzma) |\r\n\r\n## License \r\n\r\nData Stores is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalforbit%2Fdata-stores","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhalforbit%2Fdata-stores","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhalforbit%2Fdata-stores/lists"}