{"id":13561574,"url":"https://github.com/beyondstorage/go-storage","last_synced_at":"2025-12-15T00:22:55.509Z","repository":{"id":37080342,"uuid":"211792310","full_name":"beyondstorage/go-storage","owner":"beyondstorage","description":"A vendor-neutral storage library for Golang: Write once, run on every storage service.","archived":false,"fork":false,"pushed_at":"2025-01-07T16:33:54.000Z","size":5326,"stargazers_count":601,"open_issues_count":60,"forks_count":56,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-05-12T22:59:45.033Z","etag":null,"topics":["azblob","cloud-storage","dropbox","files","fs","gcs","golang","qingstor","s3","storage"],"latest_commit_sha":null,"homepage":"https://beyondstorage.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/beyondstorage.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-30T06:38:31.000Z","updated_at":"2025-05-02T08:13:48.000Z","dependencies_parsed_at":"2023-09-23T17:48:50.342Z","dependency_job_id":"bb590665-2af3-425a-839b-f90d286ec69e","html_url":"https://github.com/beyondstorage/go-storage","commit_stats":{"total_commits":1393,"total_committers":22,"mean_commits":63.31818181818182,"dds":0.4694903086862886,"last_synced_commit":"d9eb2017e4c2ca92bcbc551c5a80107532cb5729"},"previous_names":["aos-dev/go-storage","xuanwo/storage"],"tags_count":61,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondstorage%2Fgo-storage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondstorage%2Fgo-storage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondstorage%2Fgo-storage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/beyondstorage%2Fgo-storage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/beyondstorage","download_url":"https://codeload.github.com/beyondstorage/go-storage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253837405,"owners_count":21971982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azblob","cloud-storage","dropbox","files","fs","gcs","golang","qingstor","s3","storage"],"created_at":"2024-08-01T13:00:58.525Z","updated_at":"2025-12-15T00:22:55.461Z","avatar_url":"https://github.com/beyondstorage.png","language":"Go","funding_links":[],"categories":["Go","HarmonyOS"],"sub_categories":["Windows Manager"],"readme":"# go-storage\n\n## [Website](https://beyondstorage.io) | [Documentation](https://beyondstorage.io/docs/go-storage) | [Community](https://beyondstorage.io/community)\n\n[![Go dev](https://pkg.go.dev/badge/go.beyondstorage.io/v5)](https://pkg.go.dev/go.beyondstorage.io/v5)\n[![License](https://img.shields.io/badge/license-apache%20v2-blue.svg)](https://github.com/beyondstorage/go-storage/blob/master/LICENSE)\n\n[![Matrix](https://img.shields.io/matrix/beyondstorage@go-storage:matrix.org.svg?label=go-storage\u0026logo=matrix)](https://matrix.to/#/#beyondstorage@go-storage:matrix.org)\n[![Slack](https://img.shields.io/badge/slack-4A154B?logo=slack)](https://join.slack.com/t/beyond-storage/shared_invite/zt-pf4yk96u-7znnJVYpQvY57C3rRrbDPg)\n[![Telegram](https://img.shields.io/badge/telegram-2CA5E0?logo=telegram)](https://t.me/beyondstorage_go_storage)\n\n[![Build Test](https://github.com/beyondstorage/go-storage/actions/workflows/build-test.yml/badge.svg)](https://github.com/beyondstorage/go-storage/actions/workflows/build-test.yml)\n[![Cross Build](https://github.com/beyondstorage/go-storage/actions/workflows/cross-build.yml/badge.svg)](https://github.com/beyondstorage/go-storage/actions/workflows/cross-build.yml)\n[![Unit Test](https://github.com/beyondstorage/go-storage/actions/workflows/unit-test.yml/badge.svg)](https://github.com/beyondstorage/go-storage/actions/workflows/unit-test.yml)\n\nA **vendor-neutral** storage library for Golang.\n\n## Vision\n\n**Write once, run on every storage service.**\n\n## Goal\n\n- Vendor agnostic\n- Production ready\n- High performance\n\n## Examples\n\n```go\npackage main\n\nimport (\n    \"log\"\n\n    \"github.com/beyondstorage/go-storage/v5/services\"\n    \"github.com/beyondstorage/go-storage/v5/types\"\n\n    // Add fs support\n    _ \"github.com/beyondstorage/go-storage/services/fs/v4\"\n    // Add s3 support\n    _ \"github.com/beyondstorage/go-storage/services/s3/v3\"\n    // Add gcs support\n    _ \"github.com/beyondstorage/go-storage/services/gcs/v3\"\n    // Add azblob support\n    _ \"github.com/beyondstorage/go-storage/services/azblob/v3\"\n    // More support could be found under BeyondStorage.\n    _ \"github.com/beyondstorage/go-storage/services/xxx\" \n)\n\nfunc main() {\n    // Init a Storager from connection string. \n    store, err := services.NewStoragerFromString(\"s3://bucket_name/path/to/workdir\")\n    if err != nil {\n        log.Fatalf(\"service init failed: %v\", err)\n    }\n\n    // Write data from io.Reader into hello.txt\n    n, err := store.Write(\"hello.txt\", r, length)\n\n    // Read data from hello.txt to io.Writer\n    n, err := store.Read(\"hello.txt\", w)\n\n    // Stat hello.txt to check existence or get its metadata\n    o, err := store.Stat(\"hello.txt\")\n\n    // Use object's functions to get metadata\n    length, ok := o.GetContentLength()\n    \n    // List will create an iterator of object under path.\n    it, err := store.List(\"path\")\n    \n    for {\n    \t// Use iterator.Next to retrieve next object until we meet IterateDone.\n    \to, err := it.Next()\n    \tif errors.Is(err, types.IterateDone) {\n    \t\tbreak\n        }\n    }\n\n    // Delete hello.txt\n    err = store.Delete(\"hello.txt\")\n}\n```\n\nMore examples could be found at [go-storage-example](https://github.com/beyondstorage/go-storage-example).\n\n## Features\n\n### Widely native services support\n\n**16** stable services that have passed all [integration tests](https://github.com/beyondstorage/go-integration-test).\n\n- [azblob](https://github.com/beyondstorage/go-service-azblob/): [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/)\n- [bos](https://github.com/beyondstorage/go-service-bos): [Baidu Object Storage](https://cloud.baidu.com/product/bos.html)\n- [cos](https://github.com/beyondstorage/go-service-cos/): [Tencent Cloud Object Storage](https://cloud.tencent.com/product/cos)\n- [dropbox](https://github.com/beyondstorage/go-service-dropbox/): [Dropbox](https://www.dropbox.com)\n- [fs](https://github.com/beyondstorage/go-service-fs/): Local file system\n- [ftp](https://github.com/beyondstorage/go-service-ftp/): FTP\n- [gcs](https://github.com/beyondstorage/go-service-gcs/): [Google Cloud Storage](https://cloud.google.com/storage/)\n- [gdrive](https://github.com/beyondstorage/go-service-gdrive): [Google Drive](https://www.google.com/drive/)\n- [ipfs](https://github.com/beyondstorage/go-service-ipfs): [InterPlanetary File System](https://ipfs.io)\n- [kodo](https://github.com/beyondstorage/go-service-kodo/): [qiniu kodo](https://www.qiniu.com/products/kodo)\n- [memory](https://github.com/beyondstorage/go-service-memory): data that only in memory\n- [minio](https://github.com/beyondstorage/go-service-minio): [MinIO](https://min.io)\n- [obs](https://github.com/beyondstorage/go-service-obs): [Huawei Object Storage Service](https://www.huaweicloud.com/product/obs.html)\n- [oss](https://github.com/beyondstorage/go-service-oss/): [Aliyun Object Storage](https://www.aliyun.com/product/oss)\n- [qingstor](https://github.com/beyondstorage/go-service-qingstor/): [QingStor Object Storage](https://www.qingcloud.com/products/qingstor/)\n- [s3](https://github.com/beyondstorage/go-service-s3/): [Amazon S3](https://aws.amazon.com/s3/)\n\n**3** beta services that implemented required functions, but not passed [integration tests](https://github.com/beyondstorage/go-integration-test).\n\n- [hdfs](https://github.com/beyondstorage/go-service-hdfs): [Hadoop Distributed File System](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#Introduction)\n- [tar](https://github.com/beyondstorage/go-service-tar): tar files\n- [uss](https://github.com/beyondstorage/go-service-uss/): [UPYUN Storage Service](https://www.upyun.com/products/file-storage)\n\n**4** alpha services that still under development.\n\n- [onedrive](https://github.com/beyondstorage/go-service-onedrive): [Microsoft OneDrive](https://www.microsoft.com/en-ww/microsoft-365/onedrive/online-cloud-storage)\n- [storj](https://github.com/beyondstorage/go-service-storj): [StorJ](https://www.storj.io/)\n- [webdav](https://github.com/beyondstorage/go-service-webdav): [WebDAV](http://www.webdav.org/)\n- [zip](https://github.com/beyondstorage/go-service-zip): zip files\n\nMore service ideas could be found at [Service Integration Tracking](https://github.com/beyondstorage/go-storage/issues/536).\n\n### Complete and easily extensible interface\n\nBasic operations\n\n- Metadata: get `Storager` metadata\n```go\nmeta := store.Metadata()\n_ := meta.GetWorkDir() // Get object WorkDir\n_, ok := meta.GetWriteSizeMaximum() // Get the maximum size for write operation\n```\n- Read: read `Object` content\n```go\n// Read 2048 byte at the offset 1024 into the io.Writer.\nn, err := store.Read(\"path\", w, pairs.WithOffset(1024), pairs.WithSize(2048))\n```\n- Write: write content into `Object`\n```go\n// Write 2048 byte from io.Reader\nn, err := store.Write(\"path\", r, 2048)\n```\n- Stat: get `Object` metadata or check existences\n```go\no, err := store.Stat(\"path\")\nif errors.Is(err, services.ErrObjectNotExist) {\n\t// object is not exist\n}\nlength, ok := o.GetContentLength() // get the object content length.\n```\n- Delete: delete an `Object`\n```go\nerr := store.Delete(\"path\") // Delete the object \"path\"\n```\n- List: list `Object` in given prefix or dir\n```go\nit, err := store.List(\"path\")\nfor {\n\to, err := it.Next()\n\tif err != nil \u0026\u0026 errors.Is(err, types.IterateDone) {\n        // the list is over \n    }\n    length, ok := o.GetContentLength() // get the object content length.\n}\n```\n\nExtended operations\n\n- Copy: copy a `Object` inside storager\n```go\nerr := store.(Copier).Copy(src, dst) // Copy an object from src to dst.\n```\n- Move: move a `Object` inside storager\n```go\nerr := store.(Mover).Move(src, dst) // Move an object from src to dst.\n```\n- Reach: generate a public accessible url to an `Object`\n```go\nurl, err := store.(Reacher).Reach(\"path\") // Generate an url to the object.\n```\n- Dir: Dir `Object` support\n```go\no, err := store.(Direr).CreateDir(\"path\") // Create a dir object.\n```\n\nLarge file manipulation\n\n- Multipart: allow doing multipart uploads\n```go\nms := store.(Multiparter)\n\n// Create a multipart object.\no, err := ms.CreateMultipart(\"path\")\n// Write 1024 bytes from io.Reader into a multipart at index 1\nn, part, err := ms.WriteMultipart(o, r, 1024, 1)\n// Complete a multipart object.\nerr := ms.CompleteMultipart(o, []*Part{part})\n```\n- Append: allow appending to an object\n```go\nas := store.(Appender)\n\n// Create an appendable object.\no, err := as.CreateAppend(\"path\")\n// Write 1024 bytes from io.Reader.\nn, err := as.WriteAppend(o, r, 1024)\n// Commit an append object.\nerr = as.CommitAppend(o)\n```\n- Block: allow combining an object with block ids\n```go\nbs := store.(Blocker)\n\n// Create a block object.\no, err := bs.CreateBlock(\"path\")\n// Write 1024 bytes from io.Reader with block id \"id-abc\"\nn, err := bs.WriteBlock(o, r, 1024, \"id-abc\")\n// Combine block via block ids.\nerr := bs.CombineBlock(o, []string{\"id-abc\"})\n```\n- Page: allow doing random writes\n```go\nps := store.(Pager)\n\n// Create a page object.\no, err := ps.CreatePage(\"path\")\n// Write 1024 bytes from io.Reader at offset 2048\nn, err := ps.WritePage(o, r, 1024, 2048)\n```\n\n### Comprehensive metadata\n\nGlobal object metadata\n\n- `id`: unique key in service\n- `name`: relative path towards service's work dir\n- `mode`: object mode can be a combination of `read`, `dir`, `part` and [more](https://github.com/beyondstorage/go-storage/blob/master/types/object.go#L11) \n- `etag`: entity tag as defined in [rfc2616](https://tools.ietf.org/html/rfc2616#section-14.19)\n- `content-length`: object's content size.\n- `content-md5`: md5 digest as defined in [rfc2616](https://tools.ietf.org/html/rfc2616#section-14.15)\n- `content-type`: media type as defined in [rfc2616](https://tools.ietf.org/html/rfc2616#section-14.17)\n- `last-modified`: object's last updated time.\n\nSystem object metadata\n\nService system object metadata like `storage-class` and so on.\n\n```go\no, err := store.Stat(\"path\")\n\n// Get service system metadata via API provides by go-service-s3.\nom := s3.GetObjectSystemMetadata(o)\n_ = om.StorageClass // this object's storage class\n_ = om.ServerSideEncryptionCustomerAlgorithm // this object's sse algorithm\n```\n\n### Strong Typing Everywhere\n\nSelf maintained codegen [definitions](https://github.com/beyondstorage/go-storage/tree/master/cmd/definitions) helps to generate all our APIs, pairs and metadata.\n\nGenerated pairs which can be used as API optional arguments.\n\n```go\nfunc WithContentMd5(v string) Pair {\n    return Pair{\n        Key:   \"content_md5\",\n        Value: v,\n    }\n}\n```\n\nGenerated object metadata which can be used to get content md5 from object.\n\n```go\nfunc (o *Object) GetContentMd5() (string, bool) {\n    o.stat()\n    \n    if o.bit\u0026objectIndexContentMd5 != 0 {\n        return o.contentMd5, true\n    }\n    \n    return \"\", false\n}\n```\n\n### Server-Side Encrypt\n\nServer-Side Encrypt supports via system pair and system metadata, and we can use [Default Pairs](https://beyondstorage.io/docs/go-storage/pairs#default-pairs) to simplify the job.\n\n```go\n\nfunc NewS3SseC(key []byte) (types.Storager, error) {\n    defaultPairs := s3.DefaultStoragePairs{\n        Write: []types.Pair{\n            // Required, must be AES256\n            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),\n            // Required, your AES-256 key, a 32-byte binary value\n            s3.WithServerSideEncryptionCustomerKey(key),\n        },\n        // Now you have to provide customer key to read encrypted data\n        Read: []types.Pair{\n            // Required, must be AES256\n            s3.WithServerSideEncryptionCustomerAlgorithm(s3.ServerSideEncryptionAes256),\n            // Required, your AES-256 key, a 32-byte binary value\n            s3.WithServerSideEncryptionCustomerKey(key),\n        }}\n    \n    return s3.NewStorager(..., s3.WithDefaultStoragePairs(defaultPairs))\n}\n```\n\n## Sponsor\n\n\u003ca href=\"https://vercel.com?utm_source=beyondstorage\u0026utm_campaign=oss\"\u003e\n    \u003cimg src=\"./docs/images/vercel.svg\"\u003e\n\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeyondstorage%2Fgo-storage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbeyondstorage%2Fgo-storage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbeyondstorage%2Fgo-storage/lists"}