{"id":29572694,"url":"https://github.com/accelbyte/common-blob-go","last_synced_at":"2025-07-19T05:10:54.289Z","repository":{"id":37349793,"uuid":"270555360","full_name":"AccelByte/common-blob-go","owner":"AccelByte","description":"Go library to work with AWS(amazon web services) S3 and GCP(google cloud platform) cloud storage","archived":false,"fork":false,"pushed_at":"2024-07-10T01:28:49.000Z","size":179,"stargazers_count":2,"open_issues_count":0,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-07-10T04:44:12.087Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AccelByte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-08T06:25:53.000Z","updated_at":"2024-07-10T01:27:44.000Z","dependencies_parsed_at":"2023-12-01T02:36:24.149Z","dependency_job_id":"3f057bcc-9cc8-4729-96fc-dda1e3bf81aa","html_url":"https://github.com/AccelByte/common-blob-go","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/AccelByte/common-blob-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccelByte%2Fcommon-blob-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccelByte%2Fcommon-blob-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccelByte%2Fcommon-blob-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccelByte%2Fcommon-blob-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AccelByte","download_url":"https://codeload.github.com/AccelByte/common-blob-go/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AccelByte%2Fcommon-blob-go/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265892514,"owners_count":23845034,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-19T05:10:53.579Z","updated_at":"2025-07-19T05:10:54.281Z","avatar_url":"https://github.com/AccelByte.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/AccelByte/common-blob-go.svg?branch=master)](https://travis-ci.com/AccelByte/common-blob-go)\n\n# common-blob-go\nGo library to work with AWS(amazon web services) S3 and GCP(google cloud platform) cloud storage\n\n## Usage\n\n### Install\n\n```\ngo get -u github.com/AccelByte/common-blob-go\n```\n\n### Importing\n\n```go\ncommonblobgo \"github.com/AccelByte/common-blob-go\"\n```\n\nTo create a new storage client, use this function:\n\n```go\nstorage, err := storage, err := NewCloudStorage(\n    ctx,\n    isTesting,\n    bucketProvider,\n    bucketName,\n    awsS3Endpoint,\n    awsS3Region,\n    awsS3AccessKeyID,\n    awsS3SecretAccessKey,\n    gcpCredentialsJSON,\n    gcpStorageEmulatorHost,\n)\n```\n\n``NewCloudStorage`` requires such parameters :\n * ctx context.Context : a context that could be cancelled to force-stop the initialization\n * isTesting bool : a flag to switch between external and in-docker-compose dependencies. Used from tests\n * bucketProvider string : provider type. Could be `aws` or `gcp`\n * bucketName string : the name of a bucket\n\n * awsS3Endpoint string : S3 endpoint. Used only from tests(required if bucketProvider==`aws` and isTesting == `true`)\n * awsS3Region string : S3 region(required if bucketProvider==`aws`)\n * awsS3AccessKeyID string : S3 Access key(required if bucketProvider==`aws`)\n * awsS3SecretAccessKey string : S3 secret key(required if bucketProvider==`aws`)\n\n * gcpCredentialsJSON string : GCP JSON credentials(optional if bucketProvider==`gcp`). \n   * If empty - the library will attempt to self-configure from k8s GCP API. The Client(service account) should have the role \"Service Account Token Creator\"\n     \u003cdetails\u003e\n       \u003csummary\u003eClick to expand\u003c/summary\u003e\n\n       ![alt text](images/2020-06-25-10_59_01_720.png)\n     \u003c/details\u003e\n \n * gcpStorageEmulatorHost string : GCP storage host. Used only from tests(required if bucketProvider==`gcp` and isTesting == `true`)\n\nTo enable cloud storage additional features:   \n```go\nstorage, err := storage, err := NewCloudStorageWithOption(\n    ctx,\n    isTesting,\n    bucketProvider,\n    bucketName,\n    opts,\n)\n```\nCloud-specific parameter such as `awsRegion`, `gcpStorageEmulatorHost`, etc. has been moved to `opts CloudStorageOption`.\n\nSupported additional cloud storage feature:\n* `opts.AWSEnableS3Accelerate` (default: false) : a boolean that indicate S3 bucket use accelerate endpoint. **Not available in testing using localstack or using path-style S3 endpoint**.\nNote: make sure to enable transfer accelerate in S3 bucket, please refer to [this documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html).\n\n\n\n### Available methods :\n```go\ntype CloudStorage interface {\n\tList(ctx context.Context, prefix string) *ListIterator // iterate over all objects in the folder\n\tListWithOptions(ctx context.Context, options *ListOptions) *ListIterator // iterate over objects in the folder based on ListOptions criteria \n\tGet(ctx context.Context, key string) ([]byte, error) // get the object by a name\n\tGetReader(ctx context.Context, key string) (io.ReadCloser, error) // get reader to operate with io.ReadCloser\n\tDelete(ctx context.Context, key string) error // delete the object by a name\n\tCreateBucket(ctx context.Context, bucketPrefix string, expirationTimeDays int64) error // create a bucket. Used only from tests\n\tClose() // close connection\n\tGetSignedURL(ctx context.Context, key string, expiry time.Duration) (string, error) // create signed URL\n\tWrite(ctx context.Context, key string, body []byte, contentType *string) error // write the object a file-name\n\tGetWriter(ctx context.Context, key string) (io.WriteCloser, error) // get writer to operate with io.WriteCloser\n\tAttributes(ctx context.Context, key string) (*Attributes, error) // get object attributes\n\tExists(ctx context.Context, key string) (bool, error) // check object existence\n}\n```\n\n#### Examples:\n\n##### List(ctx context.Context, prefix string) *ListIterator\n```go\n    list := storage.List(ctx, bucketPrefix)\n\n    for {\n        item, err := list.Next(ctx)\n        if err == io.EOF {\n            break // no more object\n        }\n\n        // ...\n\n        if item.Key == fileName {\n            fileFound = true\n        }\n    }\n```\n\n##### ListWithOptions(ctx context.Context, options *ListOptions) *ListIterator\n```go\n    list := storage.CloudStorage.ListWithOptions(parentCtx, \u0026commonblobgo.ListOptions{\n        Prefix:    bucketPrefix,\n\n        // An empty delimiter means that the bucket is treated as a single flat namespace.\n        //\n        // A non-empty delimiter means that any result with the delimiter in its key after Prefix is stripped will be returned with ListObject.IsDir = true,\n        // ListObject.Key truncated after the delimiter. These results represent \"directories\"\n        Delimiter: \"/\", \n    })\n\n    var directories []string\n    for {\n        item, err := list.Next(ctx)\n        if err == io.EOF {\n            break // no more object\n        }\n\n        // ...\n\n        if item.IsDir {\n            directories = append(directories, item.Key)\n        }\n    }\n```\n\n##### Get(ctx context.Context, key string) ([]byte, error)\n```go\n    storedBody, err := storage.Get(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }   \n\n    fmt.Println(string(storedBody))\n```\n\n##### GetReader(ctx context.Context, key string) (io.ReadCloser, error)\n```go\n    reader, err := storage.GetReader(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }\n    defer reader.Close() // Important to prevent memory leaks\n\n    storedBody, err := ioutil.ReadAll(reader)\n    fmt.Println(string(storedBody))\n```\n\n##### GetRangeReader(ctx context.Context, key string, offset int64, length int64) (io.ReadCloser, error)\n```go\n    reader, err := storage.GetRangeReader(ctx, fileName, offset, length)\n    if err != nil { \n        return nil, err\n    }\n    defer reader.Close() // Important to prevent memory leaks\n\n    storedBody, err := ioutil.ReadAll(reader)\n    fmt.Println(string(storedBody))\n```\n\n##### Delete(ctx context.Context, key string) error\n```go\n    err = storage.Delete(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }   \n```\n\n##### CreateBucket(ctx context.Context, bucketPrefix string, expirationTimeDays int64) error\n```go\n    err = storage.CreateBucket(ctx, bucketPrefix, 1)\n    if err != nil { \n        return nil, err\n    }   \n```\n\n##### Close()\n```go\n    storage, err := storage, err := NewCloudStorage(\n        ctx,\n        isTesting,\n        bucketProvider,\n        bucketName,\n        awsS3Endpoint,\n        awsS3Region,\n        awsS3AccessKeyID,\n        awsS3SecretAccessKey,\n        gcpCredentialsJSON,\n        gcpStorageEmulatorHost,\n    )\n\n    defer storage.Close()\n```\n\n##### GetSignedURL(ctx context.Context, key string, expiry time.Duration) (string, error)\n```go\n    url, err := storage.GetSignedURL(ctx, fileName, time.Hour)\n    if err != nil { \n        return nil, err\n    }   \n\n    fmt.Println(url)\n```\n\n##### Write(ctx context.Context, key string, body []byte, contentType *string) error\n```go\n    err := storage.Write(ctx, fileName, bodyBytes, nil)\n    if err != nil { \n        return nil, err\n    }   \n```\n\n##### \tGetWriter(ctx context.Context, key string) (io.WriteCloser, error)\n```go\n\tbody := []byte(`{\"key\": \"value\", \"key2\": \"value2\"}`)\n\n\twriter, err := storage.GetWriter(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }   \n\n\t_, err = writer.Write(body[:10])\n    if err != nil { \n        return nil, err\n    }   \n\n\t_, err = writer.Write(body[10:20])\n    if err != nil { \n        return nil, err\n    }   \n\n\t_, err = writer.Write(body[20:])\n    if err != nil { \n        return nil, err\n    }   \n\n\terr = writer.Close()\n    if err != nil { \n        return nil, err\n    }   \n```\n\n##### Attributes(ctx context.Context, key string) (*Attributes, error)\n```go\n    attrs, err := storage.Attributes(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }   \n    \n    fmt.Println(attrs.Size)\n```\n\n##### Exists(ctx context.Context, key string) (bool, error)\n```go\n    isExists, err := storage.Exists(ctx, fileName)\n    if err != nil { \n        return nil, err\n    }   \n\n    if isExists {\n        fmt.Println(\"found\")\n    }\n```\n\n### License\n    Copyright © 2020, AccelByte Inc. Released under the Apache License, Version 2.0\n        ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccelbyte%2Fcommon-blob-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faccelbyte%2Fcommon-blob-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faccelbyte%2Fcommon-blob-go/lists"}