{"id":13415090,"url":"https://github.com/aloneguid/stowage","last_synced_at":"2025-04-08T02:43:04.347Z","repository":{"id":42991174,"uuid":"373509113","full_name":"aloneguid/stowage","owner":"aloneguid","description":"Bloat-free, no BS cloud storage SDK.","archived":false,"fork":false,"pushed_at":"2024-09-27T15:04:47.000Z","size":582,"stargazers_count":166,"open_issues_count":2,"forks_count":13,"subscribers_count":9,"default_branch":"master","last_synced_at":"2024-10-13T04:10:55.714Z","etag":null,"topics":["aws-s3","azure-storage","databricks","gcp-storage"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aloneguid.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-03T13:02:07.000Z","updated_at":"2024-08-14T00:05:00.000Z","dependencies_parsed_at":"2024-01-02T22:35:31.401Z","dependency_job_id":"0c383a54-4b77-4dfe-ad17-179fcb26de2f","html_url":"https://github.com/aloneguid/stowage","commit_stats":{"total_commits":86,"total_committers":5,"mean_commits":17.2,"dds":0.08139534883720934,"last_synced_commit":"3b83e2af3925def45763a6ca052ae3f54a65cd55"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aloneguid%2Fstowage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aloneguid%2Fstowage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aloneguid%2Fstowage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aloneguid%2Fstowage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aloneguid","download_url":"https://codeload.github.com/aloneguid/stowage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247767232,"owners_count":20992538,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws-s3","azure-storage","databricks","gcp-storage"],"created_at":"2024-07-30T21:00:43.116Z","updated_at":"2025-04-08T02:43:04.330Z","avatar_url":"https://github.com/aloneguid.png","language":"C#","funding_links":[],"categories":["Database Drivers","Cloud Storage"],"sub_categories":[],"readme":"# Stowage\n\n[![Nuget](https://img.shields.io/nuget/v/Stowage?style=for-the-badge\u0026label=stable)](https://www.nuget.org/packages/Stowage) [![Nuget (with prereleases)](https://img.shields.io/nuget/vpre/Stowage?style=for-the-badge\u0026label=pre-release)](https://www.nuget.org/packages/Stowage) [![Nuget](https://img.shields.io/nuget/dt/Stowage?style=for-the-badge)](https://www.nuget.org/stats/packages/Stowage?groupby=Version)\n\n![logo](../media/icon/icon-256.png)\n\n\u003e This documentation is for Stowage v2 which is a major redesign. Version 1 documentation can be found [here](README.v1.md).\n\n**Stowage** is a **bloat-free .NET cloud storage kit** that supports at minimum THE major ☁ providers.\n\n- **Independent** 🆓. Provides an independent implementation of the ☁ storage APIs. Because you can't just have official corporate SDKs as a single source of truth.\n- **Readable**. Official SDKs like the ones for [AWS](https://github.com/aws/aws-sdk-net), [Google](https://github.com/googleapis/google-cloud-dotnet), or [Azure](https://github.com/Azure/azure-storage-net) are overengineered and unreadable. Some are autogenerated and look just bad and foreign to .NET ecosystem. Some won't even compile without some custom rituals.\n- **Beautiful** 🦋. Designed to fit into .NET ecosystem, not the other way around.\n- **Rich** 💰. Provide maximum functionality. However, in addition to that, provide humanly possible way to easily extend it with new functionality, without waiting for new SDK releases.\n- **Embeddable** 🔱. Has zero external dependencies, relies only on built-in .NET API. Often official SDKs have a very deep dependency tree causing a large binary sizes and endless conflicts during runtime. This one is a single .NET .dll with no dependencies whatsoever.\n- **Cross Cloud** 🌥. Same API. Any cloud. Best decisions made for you. It's like iPhone vs Windows Phone.\n- **Cross Tested** ❎. It's not just cross cloud but also cross tested (I don't know how to call this). It tests that all cloud providers behave absolutely the same on various method calls. They should validate arguments the same, throw same exceptions in the same situations, and support the same set of functionality. Sounds simple, but it's rare to find in a library. And it important, otherwise what's the point of a generic API if you need to write a lot of `if()`s? (or pattern matching).\n\n\u003e This library originally came out from being frustrated on working on my another library - [Storage.Net](https://github.com/aloneguid/storage). While it's OK, most of the time I had to deal with SDK incompatibilities, breaking changes, oddnesses, and slowness, whereas most of the time users needs something simple that just works.\n\n## Getting Started\n\nRight, time to gear up. We'll do it [step by step](https://www.oxfordlearnersdictionaries.com/definition/english/step_1?q=step). First, you need to [install](https://docs.microsoft.com/en-us/nuget/quickstart/install-and-use-a-package-using-the-dotnet-cli) the [![Nuget](https://img.shields.io/nuget/v/Stowage?style=social)](https://www.nuget.org/packages/Stowage) package.\n\n\nSimplest case, using the local 💽 and writing text \"I'm a page!!!\" to a file called \"pagefile.sys\" at the root of disk C::\n\n```csharp\nusing Stowage;\n\nusing(IFileStorage fs = Files.Of.LocalDisk(\"c:\\\\\")) {\n   await fs.WriteText(\"pagefile.sys\", \"I'm a page!!!!\");\n}\n```\n\nThis is local disk, yeah? But what about cloud storage, like Azure Blob Storage? Piece of cake:\n\n```csharp\nusing Stowage;\n\nusing(IFileStorage fs = Files.Of.AzureBlobStorage(\"accountName\", \"accountKey\", \"containerName\")) {\n   var entries = await fs.Ls();\n}\n```\n\n\n\n## ♒ \u003cspan style=\"color:red\"\u003eS\u003c/span\u003etreaming\n\nStreaming is a *first-class feature*. This means the streaming is real with no workarounds or in-memory buffers, so you can upload/download files of virtually unlimited sizes. Most official SDKs *do not support streaming* at all - surprisingly even the cloud leader's .NET SDK doesn't. Each requires some sort of crippled down version of stream - either knowing length beforehand, or will buffer it all in memory. I don't. I stream like a stream.\n\nProper streaming support also means that you can transform streams as you write to them or read from them - something that is not available in the native SDKs. For instance gzipping, encryption, anything else.\n\nStreaming is also truly compatible with synchronous and asynchronous API.\n\n## Details/Documentation\n\nWhenever a method appears here, I assume it belongs to `IFileStorage` interface, unless specified.\n\n### Listing/Browsing\n\nUse `.Ls()` ([short for list](https://en.wikipedia.org/wiki/Ls)) - very easy to remember! Everyone knows what **ls** does, right? Optionally allows to list entries recursively.\n\n### Reading\n\nThe core method for reading is `Stream OpenRead(IOPath path)` - this returns a stream from file path. Stream is the lowest level data structure. There are other helper methods that by default rely on this method, like `ReadText` etc. Just have a quick look:\n\n```csharp\nIFileStorage fs = ...;\nStream target = ...;\n\n// copy to another stream\nusing Stream s = await fs.OpenRead(\"/myfile.txt\");\n\n// synchronous copy:\ns.CopyTo(target);\n\n// or alternatively, asynchronous copy (preferred):\nawait s.CopyToAsync(target);\n\n// if you just need text:\nstring content = await fs.ReadText(\"/myfile.txt\");\n```\n\nOf course there are more overloaded methods you can take advantage of.\n\n### Writing\n\nThe main method `Stream OpenWrite(IOPath path, ...)` opens(/creates?) a file for writing. It returns a real writeable stream you can write to and close afterwards. It behaves like a stream and is a stream.\n\nThere are other overloads which support writing text etc.\n\n### Destroying 🧨\n\n`Rm(IOPath path)` trashes files or folders (or both) with options to do it recursively!\n\n### Other\n\nThere are other useful utility methods:\n\n- `bool Exists(IOPath path)` that checks for file existence. It supposed to be really efficient, hence a separate method.\n- `Ren` renames files and folders.\n- and more are coming - check `IFileStorage` interface to be up to date.\n\n## Supported Storage Systems (Built-In)\n\n- Local Disk Directory (`Files.Of.LocalDisk(...)`).\n- In-Memory (`Files.Of.InternalMemory(...)`).\n- [AWS S3](https://aws.amazon.com/s3/) (`Files.Of.AmazonS3(...)`).\n  - [Minio](https://min.io/) (`Files.Of.Minio(...)`).\n  - [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces) (`Files.Of.DigitalOceanSpaces(...)`).\n\n- [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) / [Data Lake Gen 2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction#key-features-of-data-lake-storage-gen2) (`Files.Of.AzureBlobStorage(...)`).\n- [Google Cloud Storage](https://cloud.google.com/storage) (`Files.Of.GoogleCloudStorage(...)`).\n- [Databricks DBFS](https://docs.databricks.com/data/databricks-file-system.html) (`Files.Of.DatabricksDbfs(...)`).\n\nInstantiation instructions are in the code documentation ([IntelliSense](https://docs.microsoft.com/en-us/visualstudio/ide/using-intellisense?view=vs-2019)?) - I prefer this to writing out here locally.\n\nBelow are some details worth mentioning.\n\n### Local disk\n\nIt's just what it says - local disk, which by default maps to an entire filesystem. In Windows, which has drive letters, it will map to an entire disk of where the application's current directory's drive is (i.e. if you are in `c:/my/app` it it will map to `c:/`).\n\n```csharp\nIFileStorage storage = Files.Of.LocalDisk();\n// or\nIFileStorage storage = Storage.Of.ConnectionString(\"disk://\")\n```\n\nOptionally, you can specify the root directory:\n\n```csharp\nIFileStorage storage = Files.Of.LocalDisk(\"/a/folder\");\n// or\nIFileStorage storage = Storage.Of.ConnectionString(\"disk://path=/a/folder\")\n```\n\n\u003e do not forget there is `path` keyword in the connection string, it's so tempting to write \"disk://a/folder\" instead of \"disk://path=/a/folder\"!\n\n### AWS S3\n\nIn AWS, the path addressing style is the following:\n\n```\n/bucket/path/object\n```\n\n`Ls` on the root folder returns *list of buckets* in the AWS account, whether you do have access to them or not.\n\n#### Authentication\n\n##### Key/Secret\n\n The most usual way to authenticate with S3 is to use the following method:\n\n```csharp\nIFileStorage storage = Files.Of.AmazonS3(key, secret, region);\n// or\nIFileStorage storage = Storage.Of.ConnectionString(\"s3://keyId=\u003ckey\u003e;key=\u003csecret\u003e;region=\u003cregion\u003e\")\n```\n\nThese are what Amazon calls \"long-term\" credentials. If you are using STS, the same method overload allows you to pass `sessionToken`.\n\n```csharp\nIFileStorage storage = Files.Of.AmazonS3(key, secret, region, sessionToken);\n//or\nIFileStorage storage = Storage.Of.ConnectionString(\"s3://keyId=\u003ckey\u003e;key=\u003csecret\u003e;region=\u003cregion\u003e;sessionToken=\u003csession token\u003e\")\n```\n\n##### CLI Profile\n\nAnother way to authenticate is using CLI profile. This is useful when you machine is already authenticated using [aws cli](https://aws.amazon.com/cli/), [awsume](https://awsu.me/) or similar tools that write credentials and configuration to `~/.aws/credentials` and `~/.aws/config`. \n\nYou only need to pass the profile name (and only if it's not a default one):\n\n```csharp\n// ---- using default profile ----\n\nIFileStorage storage = Files.Of.AmazonS3FromCliProfile();\n//or\nIFileStorage storage = Storage.Of.ConnectionString(\"s3://\");\n\n// ----- using specific profile, like \"myprofile\" ----\nIFileStorage storage = Files.Of.AmazonS3FromCliProfile(\"myprofile\");\n//or\nIFileStorage storage = Storage.Of.ConnectionString(\"s3://profile=myprofile\");\n\n```\n\nThis method has other default parameters, such as `regionName` which can be specified or overridden if not found in CLI configuration, i.e. `Files.Of.AmazonS3FromCliProfile();` has optional `region` parameters, and connection string has optional `region=` keyword.\n\n##### Minio\n\nMinio is essentially using the standard S3 protocol, but addressing style is slightly different. There is a helper extension that somewhat simplifies Minio authentication:\n\n```csharp\nIFileStorage storage = Files.Of.Minio(endpoint, key, secret);\n```\n\n### Azure Blob Storage\n\nIn Azure Blob Storage, path addressing style is the following:\n\n```\n/container/path/object\n```\n\nNote that there is no storage account in the path, mostly because *Shared Key authentication is storage account scoped*, not tenant scoped.\n\n`Ls` on the root folder returns *list of containers* in the storage account.\n\n#### Authentication\n\n##### Shared Key\n\nAzure provider supports authentication with [Shared Key](https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key):\n\n```csharp\nIFileStorage storage = Files.Of.AzureBlobStorage(accountName, sharedKey);\n// or\nIFileStorage storage = Storage.Of.ConnectionString(\"az://account=\u003caccount name\u003e;key=\u003cshared key\u003e\")\n```\n\n##### Entra Id service principals\n\n```csharp\nIFileStorage storage = Files.Of.AzureBlobStorage(\n    accountName,\n    new ClientSecretCredential(tenantId, clientId, clientSecret));\n```\n\nManaged identities are not yet supported due to low demand, but watch this space.\n\n#### Emulator\n\nAzure emulator is supported, just use `AzureBlobStorageWithLocalEmulator()` method to connect to it.\n\n### Exotic providers\n\n#### Local disk cache\n\nThis storage essentially wraps around another storage to provide content caching capabilities. Example:\n\n```csharp\nIFileStorage storage = Files.Of.AzureBlobStorage(accountName, sharedKey);\nIFileStorage cachedStorage = Files.Of.LocalDiskCacheStorage(storage);\n```\n\nWhen using `cachedStorage`, all the operations are forwarded to `storage` as is, except for `OpenRead` which downloads content locally and opens a stream to the local file.\n\n\n## 📈 Extending\n\nThere are many ways to extend functionality:\n\n1. **Documentation**. You might think it's not extending anything, however if user is not aware for some functionality it doesn't exist. Documenting it is making it available, hence extending. You must be slightly mad to follow my style of writing though.\n2. **New functionality**. Adding utility methods like copying files inside or between accounts, automatic JSON serialisation etc. is always good. Look [`IFileStorage`](src/Stowage/IFileStorage.cs) interface and [`PolyfilledFileStorage`](src/Stowage/PolyfilledFileStorage.cs). In most cases these two files are enough to add pure business logic. Not counting unit tests. Which you must write. Otherwise it's easier to do the whole thing by myself. Which is what will happen according to my experience.\n3. **Native optimisations**. Some functionality is generic, and some depends on a specific cloud provider. For instance, one can copy a file by downloading it locally, and uploading with a new name. Or utilise a native REST call that accepts source and target file name, if it exists. Involves digging deeper into specific provider's API.\n\nWhen contributing a new provider, it's way more preferrable to embed it's code in the library, provided that:\n\n- there are no extra nuget dependencies.\n- it's cross-platform.\n\nI'm a strong advocate of simplicity and not going to repeat the mistake of turning this into a nuget tree dependency hell!\n\n## ❔ Who?\n\n- Featured in [The .NET MAUI Podcast, episode 98](https://www.dotnetmauipodcast.com/98).\n- Blog post [Exploring Stowage](https://developingdane.com/exploring-stowage/).\n\nRaise a PR to appear here.\n\n## Related Projects\n\n- [R**CLONE**](https://rclone.org/) - cross-platform open-source cloud sync tool.\n- [Storage.Net](https://www.aloneguid.uk/projects/storagenet/) - the roots of this project.\n\n## 💰 Contributing\n\nYou are welcome to contribute in any form, however I wouldn't bother, especially financially. Don't bother buying me a ☕, I can do it myself real cheap. During my years of OSS development everyone I know (including myself) have only lost money. Why I'm still doing this? Probably because *it's just cool and I'm enjoying it*.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faloneguid%2Fstowage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faloneguid%2Fstowage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faloneguid%2Fstowage/lists"}