{"id":18438705,"url":"https://github.com/sajari/storage","last_synced_at":"2025-04-07T21:31:59.000Z","repository":{"id":57483168,"uuid":"63216788","full_name":"sajari/storage","owner":"sajari","description":"Go package for abstracting local, in-memory, and remote (Google Cloud Storage/S3) filesystems","archived":false,"fork":false,"pushed_at":"2018-08-30T01:07:45.000Z","size":17,"stargazers_count":52,"open_issues_count":1,"forks_count":11,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-23T00:41:52.997Z","etag":null,"topics":["cache","filesystem","go","google-cloud-storage","in-memory-fs","s3"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sajari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-07-13T05:24:40.000Z","updated_at":"2023-11-02T21:46:32.000Z","dependencies_parsed_at":"2022-08-27T21:02:30.029Z","dependency_job_id":null,"html_url":"https://github.com/sajari/storage","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sajari%2Fstorage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sajari%2Fstorage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sajari%2Fstorage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sajari%2Fstorage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sajari","download_url":"https://codeload.github.com/sajari/storage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247732583,"owners_count":20986891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","filesystem","go","google-cloud-storage","in-memory-fs","s3"],"created_at":"2024-11-06T06:21:18.260Z","updated_at":"2025-04-07T21:31:58.715Z","avatar_url":"https://github.com/sajari.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Storage\n\n[![Build Status](https://travis-ci.org/sajari/storage.svg?branch=master)](https://travis-ci.org/sajari/storage)\n[![GoDoc](https://godoc.org/code.sajari.com/storage?status.svg)](https://godoc.org/code.sajari.com/storage)\n\nstorage is a Go package which abstracts file systems (local, in-memory, Google Cloud Storage, S3) into a few interfaces.  It includes convenience wrappers for simplifying common file system use cases such as caching, prefix isolation and more!\n\n# Requirements\n\n- [Go 1.6+](http://golang.org/dl/)\n\n# Installation\n\n```console\n$ go get code.sajari.com/storage\n```\n\n# Usage\n\nFor full documentation see: [http://godoc.org/code.sajari.com/storage/](http://godoc.org/code.sajari.com/storage/).\n\nAll storage in this package follow two simple interfaces designed for using file systems.\n\n```go\ntype FS interface {\n\tWalker\n\n\t// Open opens an existing file at path in the filesystem.  Callers must close the\n\t// File when done to release all underlying resources.\n\tOpen(ctx context.Context, path string) (*File, error)\n\n\t// Create makes a new file in the filesystem.  Callers must close the\n\t// returned WriteCloser and check the error to be sure that the file\n\t// was successfully written.\n\tCreate(ctx context.Context, path string) (io.WriteCloser, error)\n\n\t// Delete removes a file from the filesystem.\n\tDelete(ctx context.Context, path string) error\n}\n\n// WalkFn is a function type which is passed to Walk.\ntype WalkFn func(path string) error\n\n// Walker is an interface which defines the Walk method.\ntype Walker interface {\n\t// Walk traverses a path listing by prefix, calling fn with each object path rewritten\n\t// to be relative to the underlying filesystem and provided path.\n\tWalk(ctx context.Context, path string, fn WalkFn) error\n}\n```\n\n## Local\n\nLocal is the default implementation of a local file system (i.e. using `os.Open` etc).\n\n```go\nlocal := storage.Local(\"/some/root/path\")\nf, err := local.Open(context.Background(), \"file.json\") // will open \"/some/root/path/file.json\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\n## Memory\n\nMem is the default in-memory implementation of a file system.\n\n```go\nmem := storage.Mem()\nwc, err := mem.Create(context.Background(), \"file.txt\")\nif err != nil {\n\t// ...\n}\nif _, err := io.WriteString(wc, \"Hello World!\"); err != nil {\n\t// ...\n}\nif err := wc.Close(); err != nil {\n\t// ...\n}\n```\n\nAnd now:\n\n```go\nf, err := mem.Open(context.Background(), \"file.txt\")\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\n## Google Cloud Storage\n\nCloudStorage is the default implementation of Google Cloud Storage.  This uses [https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource](`google.DefaultTokenSource`) for autentication.\n\n```go\nstore := storage.CloudStorage{Bucket:\"some-bucket\"}\nf, err := store.Open(context.Background(), \"file.json\") // will fetch \"gs://some-bucket/file.json\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\n## S3\n\nNot yet implemented!  Watch this space.\n\n## Wrappers and Helpers\n\n### Simple Caching\n\nTo use Cloud Storage as a source file system, but cache all opened files in a local filesystem:\n\n```go\nsrc := storage.CloudStorage{Bucket:\"some-bucket\"}\nlocal := storage.Local(\"/scratch-space\")\n\nfs := storage.Cache(src, local)\nf, err := fs.Open(context.Background(), \"file.json\") // will try src then jump to cache (\"gs://some-bucket/file.json\")\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n\nf, err := fs.Open(context.Background(), \"file.json\") // should now be cached (\"/scratch-space/file.json\")\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\nThis is particularly useful when distributing files across multiple regions or between cloud providers.  For instance, we could add the following code to the previous example:\n\n```go\nmainSrc := storage.CloudStorage{Bucket:\"some-bucket-in-another-region\"}\nfs2 := storage.Cache(mainSrc, fs) // fs is from previous snippet\n\n// Open will:\n// 1. Try local (see above)\n// 2. Try gs://some-bucket\n// 3. Try gs://some-bucket-in-another-region, which will be cached in gs://some-bucket and then local on its\n//    way back to the caller.\nf, err := fs2.Open(context.Background(), \"file.json\") // will fetch \"gs://some-bucket-in-another-region/file.json\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n\nf, err := fs2.Open(context.Background(), \"file.json\") // will fetch \"/scratch-space/file.json\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\n### Adding prefixes to paths\n\nIf you're writing code that relies on a set directory structure, it can be very messy to have to pass path-patterns around.  You can avoid this by wrapping `storage.FS` implementations with `storage.Prefix` that rewrites all incoming paths.\n\n```go\nmodelFS := storage.Prefix(rootFS, \"models/\")\nf, err := modelFS.Open(context.Background(), \"file.json\") // will call rootFS.Open with path \"models/file.json\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n\nIt's also now simple to write wrapper functions to abstract out more complex directory structures.\n\n```go\nfunc UserFS(fs storage.FS, userID, mediaType string) FS {\n\treturn storage.Prefix(fs, fmt.Sprintf(\"%v/%v\", userID, userType))\n}\n\nuserFS := UserFS(rootFS, \"1111\", \"pics\")\nf, err := userFS.Open(context.Background(), \"beach.png\") // will call rootFS.Open with path \"1111/pics/beach.png\"\nif err != nil {\n\t// ...\n}\n// ...\nf.Close()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsajari%2Fstorage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsajari%2Fstorage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsajari%2Fstorage/lists"}