{"id":20290658,"url":"https://github.com/gocardless/bucket-store","last_synced_at":"2026-03-11T17:02:07.577Z","repository":{"id":39617072,"uuid":"281087158","full_name":"gocardless/bucket-store","owner":"gocardless","description":"Helper library to access cloud storage services","archived":false,"fork":false,"pushed_at":"2025-03-20T18:22:21.000Z","size":177,"stargazers_count":3,"open_issues_count":4,"forks_count":0,"subscribers_count":46,"default_branch":"master","last_synced_at":"2025-08-25T09:53:05.701Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gocardless.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-20T10:33:55.000Z","updated_at":"2025-03-20T18:22:24.000Z","dependencies_parsed_at":"2023-02-01T00:30:26.250Z","dependency_job_id":"6c60983c-0ca9-42ad-b7ef-63681e76a609","html_url":"https://github.com/gocardless/bucket-store","commit_stats":{"total_commits":113,"total_committers":5,"mean_commits":22.6,"dds":"0.16814159292035402","last_synced_commit":"f158a0c7a30b4d9e695bf3e07e363e26af8b479d"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/gocardless/bucket-store","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fbucket-store","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fbucket-store/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fbucket-store/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fbucket-store/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gocardless","download_url":"https://codeload.github.com/gocardless/bucket-store/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gocardless%2Fbucket-store/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278779755,"owners_count":26044441,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T15:08:37.518Z","updated_at":"2025-10-13T18:04:13.124Z","avatar_url":"https://github.com/gocardless.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# BucketStore\n\nAn abstraction layer on the top of file cloud storage systems such as Google Cloud\nStorage or S3. This module exposes a generic interface that allows interoperability\nbetween different storage options. Callers don't need to worry about the specifics\nof where and how a file is stored and retrieved as long as the given key is valid.\n\nKeys within the `BucketStorage` are URI strings that can universally locate an object\nin the given provider. A valid key example would be\n`gs://a-gcs-bucket/file/path.json`.\n\n## Usage\nThis library is distributed as a Ruby gem, and we recommend adding it to your Gemfile:\n\n```ruby\ngem \"bucket_store\"\n```\n\nSome attributes can be configured via `BucketStore.configure`. If using Rails, you want to\nadd a new initializer for `BucketStore`. Example:\n\n```ruby\nBucketStore.configure do |config|\n  config.logger = Logger.new($stderr)\nend\n```\n\nIf using RSpec, you'll probably want to add this line to RSpec's config block (see\nthe *Adapters* section for more details):\n\n```ruby\nconfig.before { BucketStore::InMemory.reset! }\n```\n\nFor our policy on compatibility with Ruby versions, see [COMPATIBILITY.md](docs/COMPATIBILITY.md).\n\n## Design and Architecture\nThe main principle behind `BucketStore` is that each resource or group of resources must\nbe unequivocally identifiable by a URI. The URI is always composed of three parts:\n\n- the \"adapter\" used to fetch the resource (see \"adapters\" below)\n- the \"bucket\" where the resource lives\n- the path to the resource(s)\n\nAs an example, all the following are valid URIs:\n\n- `gs://gcs-bucket/path/to/file.xml`\n- `inmemory://bucket/separator/file.xml`\n- `disk://hello/path/to/file.json`\n\nEven though `BucketStore`'s main goal is to be an abstraction layer on top of systems such\nas S3 or Google Cloud Storage where the \"path\" to a resource is in practice a unique\nidentifier as a whole (i.e. the `/` is not a directory separator but rather part of the\nkey's name), we assume that clients will actually want some sort of hierarchical\nseparation of resources and assume that such separation is achieved by defining each\npart of the hierarchy via `/`.\n\nThis means that the following are also valid URIs in `BucketStore` but they refer to\nall the resources under that specific hierarchy:\n\n- `gs://gcs-bucket/path/subpath/`\n- `inmemory://bucket/separator/`\n- `disk://hello/path`\n\n## Configuration\n`BucketStore` exposes some configurable attributes via `BucketStore.configure`. If\nnecessary this should be called at startup time before any other method is invoked.\n\n- `logger`: custom logger class. By default, logs will be sent to stdout.\n\n## Adapters\n\n`BucketStore` comes with 4 built-in adapters:\n\n- `gs`: the Google Cloud Storage adapter\n- `s3`: the S3 adapter\n- `disk`: a disk-based adapter\n- `inmemory`: an in-memory store\n\n### GS adapter\nThis is the adapter for Google Cloud Storage. `BucketStore` assumes that the  authorisation\nfor accessing the resources has been set up outside of the gem.\n\n### S3 adapter\nThis is the adapter for S3. `BucketStore` assumes that the authorisation for accessing\nthe resources has been set up outside of the gem (see also\nhttps://docs.aws.amazon.com/sdk-for-ruby/v3/api/index.html#Configuration).\n\n### Disk adapter\nA disk-backed key-value store. This adapter will create a temporary directory where\nall the files will be written into/read from. The base directory can be explicitly\ndefined by setting the `DISK_ADAPTER_BASE_DIR` environment variable, otherwise a temporary\ndirectory will be created.\n\n### In-memory adapter\nAn in-memory key-value storage. This works just like the disk adapter, except that\nthe content of all the files is stored in memory, which is particularly useful for\ntesting. Note that content added to this adapter will persist for the lifetime of\nthe application as it's not possible to create different instances of the same adapter.\nIn general, this is not what's expected during testing where the content of the bucket\nshould be reset between different tests. The adapter provides a way to easily reset the\ncontent though via a `.reset!` method. In RSpec this would translate to adding this line\nin the `spec_helper`:\n\n```ruby\nconfig.before { BucketStore::InMemory.reset! }\n```\n\n## BucketStore vs ActiveStorage\n\nActiveStorage is a common framework to access cloud storage systems that is included in\nthe ActiveSupport library. In general, ActiveStorage provides a lot more than BucketStore\ndoes (including many more adapters) however the two libraries have different use cases\nin mind:\n\n- ActiveStorage requires you to define every possible bucket you're planning to use\n  ahead of time in a YAML file. This works well for most cases, however if you plan to\n  use a lot of buckets this soon becomes impractical. We think that BucketStore approach\n  works much better in this case.\n- BucketStore does not provide ways to manipulate the content whereas ActiveStorage does.\n  If you plan to apply transformations to the content before uploading or after\n  downloading them, then probably ActiveStorage is the library for you. With that said,\n  it's still possible to do these transformations outside of BucketStore and in fact we've\n  found the explicitness of this approach a desirable property.\n- BucketStore approach makes any resource on a cloud storage system uniquely identifiable\n  via a single URI, which means normally it's enough to pass that string around different\n  systems to be able to access the resource without ambiguity. As the URI also includes\n  the adapter, it's possible for example to download a `disk://dir/input_file` and\n  upload it to a `gs://bucket/output_file` all going through a single interface.\n  ActiveStorage is instead focused on persisting an equivalent reference on a Rails model.\n  If your application does not use Rails, or does not need to persist the reference or\n  just requires more flexibility in general, then BucketStore is probably the library for\n  you.\n\n\n## Examples\n\n### Uploading a string to a bucket\n```ruby\nBucketStore.for(\"inmemory://bucket/path/file.xml\").upload!(\"hello world\")\n=\u003e \"inmemory://bucket/path/file.xml\"\n```\n\n### Accessing a string in a bucket\n```ruby\nBucketStore.for(\"inmemory://bucket/path/file.xml\").download\n=\u003e {:bucket=\u003e\"bucket\", :key=\u003e\"path/file.xml\", :content=\u003e\"hello world\"}\n```\n\n### Uploading a file-like object to a bucket\n```ruby\nbuffer = StringIO.new(\"This could also be an actual file\")\nBucketStore.for(\"inmemory://bucket/path/file.xml\").stream.upload!(file: buffer)\n=\u003e \"inmemory://bucket/path/file.xml\"\n```\n\n### Downloading to a file-like object from a bucket\n```ruby\nbuffer = StringIO.new\nBucketStore.for(\"inmemory://bucket/path/file.xml\").stream.download(file: buffer)\n=\u003e {:bucket=\u003e\"bucket\", :key=\u003e\"path/file.xml\", :file=\u003ebuffer}\nbuffer.string \n=\u003e \"This could also be an actual file\"\n```\n\n### Listing all keys under a prefix\n```ruby\nBucketStore.for(\"inmemory://bucket/path/\").list\n=\u003e [\"inmemory://bucket/path/file.xml\"]\n```\n\n### Delete a file\n```ruby\nBucketStore.for(\"inmemory://bucket/path/file.xml\").delete!\n=\u003e true\n```\n\n## Development\n\n### Running tests\nBucketStore comes with both unit and integration tests. While unit tests can be run by simply\nexecuting `bundle exec rspec`, integration tests require running minio locally. We provide a\ndocker-compose file that spins up pre-configured simulator instances for S3 and GCS with\ntest buckets. Running the integration tests is as easy as:\n\n```\ndocker-compose up\nbundle exec rspec --tag integration\n```\n\n## License \u0026 Contributing\n\n* BucketStore is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n* Bug reports and pull requests are welcome on GitHub at https://github.com/gocardless/bucket-store.\n\nGoCardless ♥ open source. If you do too, come [join us](https://gocardless.com/about/careers/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fbucket-store","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgocardless%2Fbucket-store","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgocardless%2Fbucket-store/lists"}