{"id":37200821,"url":"https://github.com/openframebox/gostorage","last_synced_at":"2026-01-14T23:09:34.936Z","repository":{"id":321430711,"uuid":"979900208","full_name":"openframebox/gostorage","owner":"openframebox","description":"A flexible, extensible storage abstraction layer for Go that supports multiple backends including local filesystem and AWS S3.","archived":false,"fork":false,"pushed_at":"2025-11-15T17:36:12.000Z","size":19,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-15T19:23:52.281Z","etag":null,"topics":["golang","storage"],"latest_commit_sha":null,"homepage":"https://github.com/openframebox/gostorage","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openframebox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security.go","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-05-08T08:45:25.000Z","updated_at":"2025-11-15T17:36:15.000Z","dependencies_parsed_at":"2025-10-29T17:24:21.352Z","dependency_job_id":null,"html_url":"https://github.com/openframebox/gostorage","commit_stats":null,"previous_names":["openframebox/gostorage"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/openframebox/gostorage","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openframebox%2Fgostorage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openframebox%2Fgostorage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openframebox%2Fgostorage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openframebox%2Fgostorage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openframebox","download_url":"https://codeload.github.com/openframebox/gostorage/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openframebox%2Fgostorage/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28437930,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T22:37:52.437Z","status":"ssl_error","status_checked_at":"2026-01-14T22:37:31.496Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","storage"],"created_at":"2026-01-14T23:09:34.388Z","updated_at":"2026-01-14T23:09:34.890Z","avatar_url":"https://github.com/openframebox.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoStorage\n\nA flexible, extensible storage abstraction layer for Go that supports multiple backends including local filesystem and AWS S3.\n\n## Features\n\n- **Multiple Storage Backends**\n  - Local filesystem storage\n  - AWS S3 storage\n  - Easy to extend with custom backends\n\n- **Rich File Operations**\n  - Basic operations: Put, Get, Delete\n  - Streaming support for large files\n  - File management: Copy, Move, List, Exists, Size\n  - Metadata handling with custom headers\n\n- **Security**\n  - Path validation to prevent directory traversal attacks\n  - Sanitization of file paths\n  - Protection against malicious path patterns\n\n- **Developer Friendly**\n  - Context support for all operations\n  - Comprehensive error handling\n  - Clean, idiomatic Go API\n\n## Installation\n\n```bash\ngo get github.com/openframebox/gostorage\n```\n\nAll dependencies (including AWS SDK v2 for S3/MinIO support) are automatically handled by Go modules.\n\n## Quick Start\n\n### Local Filesystem Storage\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"github.com/openframebox/gostorage\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    storage := gostorage.NewStorage()\n\n    // Add a local disk - Simple configuration!\n    localDisk, err := gostorage.NewLocalDisk(\u0026gostorage.LocalDiskConfig{\n        Path: \"./storage\",\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n    storage.AddDisk(\"local\", localDisk)\n\n    // Put a file\n    content := []byte(\"Hello, World!\")\n    err = storage.Put(ctx, \"local\", \"hello.txt\", content)\n    if err != nil {\n        panic(err)\n    }\n\n    // Get a file\n    data, err := storage.Get(ctx, \"local\", \"hello.txt\")\n    if err != nil {\n        panic(err)\n    }\n    println(string(data)) // Output: Hello, World!\n\n    // Delete a file\n    err = storage.Delete(ctx, \"local\", \"hello.txt\")\n    if err != nil {\n        panic(err)\n    }\n}\n```\n\n### AWS S3 Storage\n\n```go\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n    \"github.com/openframebox/gostorage\"\n)\n\nfunc main() {\n    ctx := context.Background()\n    storage := gostorage.NewStorage()\n\n    // Configure AWS S3 - Simple!\n    s3Disk, err := gostorage.NewS3Disk(\u0026gostorage.S3Config{\n        Region:    \"us-west-2\",\n        AccessKey: \"your-access-key\",\n        SecretKey: \"your-secret-key\",\n        Bucket:    \"my-bucket\",\n        Prefix:    \"my-prefix/\", // Optional\n    })\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    storage.AddDisk(\"s3\", s3Disk)\n\n    // Use it just like local storage\n    err = storage.Put(ctx, \"s3\", \"document.txt\", []byte(\"S3 content\"))\n    if err != nil {\n        panic(err)\n    }\n}\n```\n\n### MinIO Storage\n\n```go\n// Configure MinIO - Just specify the endpoint!\ns3Disk, err := gostorage.NewS3Disk(\u0026gostorage.S3Config{\n    Endpoint:     \"https://minio.example.com\",\n    Region:       \"us-east-1\",\n    AccessKey:    \"minio-access-key\",\n    SecretKey:    \"minio-secret-key\",\n    Bucket:       \"my-bucket\",\n    UsePathStyle: true, // Required for MinIO\n})\nif err != nil {\n    log.Fatal(err)\n}\n\nstorage.AddDisk(\"minio\", s3Disk)\n```\n\n## Configuration\n\n### S3Config Reference\n\n```go\ntype S3Config struct {\n    // Endpoint is the S3 endpoint URL\n    // For AWS S3: leave empty (uses default)\n    // For MinIO: \"https://minio.example.com\"\n    Endpoint string\n\n    // Region is the AWS region (default: \"us-east-1\")\n    Region string\n\n    // AccessKey is the AWS access key ID or MinIO access key\n    AccessKey string\n\n    // SecretKey is the AWS secret access key or MinIO secret key\n    SecretKey string\n\n    // Bucket is the S3 bucket name (required)\n    Bucket string\n\n    // Prefix is an optional prefix for all keys (useful for multi-tenancy)\n    // Example: \"tenant1/\" will prefix all paths\n    Prefix string\n\n    // UsePathStyle forces path-style addressing (required for MinIO)\n    // true:  https://endpoint/bucket/key\n    // false: https://bucket.endpoint/key (default for AWS S3)\n    UsePathStyle bool\n\n    // SessionToken is optional for temporary AWS credentials\n    SessionToken string\n}\n```\n\n### LocalDiskConfig Reference\n\n```go\ntype LocalDiskConfig struct {\n    // Path is the base directory for file storage (required)\n    Path string\n\n    // CreateIfNotExist will create the directory if it doesn't exist\n    // Default: true\n    CreateIfNotExist bool\n\n    // Permissions for created directories\n    // Default: 0755\n    DirPermissions os.FileMode\n\n    // Permissions for created files\n    // Default: 0644\n    FilePermissions os.FileMode\n}\n```\n\n**Example with custom permissions:**\n```go\nlocalDisk, err := gostorage.NewLocalDisk(\u0026gostorage.LocalDiskConfig{\n    Path:            \"/var/app/storage\",\n    DirPermissions:  0750, // rwxr-x---\n    FilePermissions: 0640, // rw-r-----\n})\n```\n\n## API Reference\n\n### Storage Manager\n\n```go\n// Create a new storage manager\nstorage := gostorage.NewStorage()\n\n// Manage disks\nstorage.AddDisk(name string, disk Disk)\nstorage.RemoveDisk(name string)\nstorage.HasDisk(name string) bool\nstorage.DiskNames() []string\n```\n\n### Basic Operations\n\n```go\n// Put a file\nerr := storage.Put(ctx, \"disk\", \"path/to/file.txt\", content)\n\n// Get a file\ndata, err := storage.Get(ctx, \"disk\", \"path/to/file.txt\")\n\n// Delete a file\nerr := storage.Delete(ctx, \"disk\", \"path/to/file.txt\")\n\n// Check if file exists\nexists, err := storage.Exists(ctx, \"disk\", \"path/to/file.txt\")\n\n// Get file size\nsize, err := storage.Size(ctx, \"disk\", \"path/to/file.txt\")\n```\n\n### Streaming Operations\n\nFor large files, use streaming to avoid loading everything into memory:\n\n```go\n// Upload large file using streaming\nfile, _ := os.Open(\"large-file.mp4\")\ndefer file.Close()\n\nerr := storage.PutStream(ctx, \"disk\", \"videos/large.mp4\", file, \u0026gostorage.Metadata{\n    ContentType: \"video/mp4\",\n})\n\n// Download large file using streaming\nreader, err := storage.GetStream(ctx, \"disk\", \"videos/large.mp4\")\nif err != nil {\n    panic(err)\n}\ndefer reader.Close()\n\n// Process the stream\nio.Copy(outputFile, reader)\n```\n\n### File Operations\n\n```go\n// Copy a file within the same disk\nerr := storage.Copy(ctx, \"disk\", \"source.txt\", \"destination.txt\")\n\n// Move a file within the same disk\nerr := storage.Move(ctx, \"disk\", \"old-path.txt\", \"new-path.txt\")\n\n// Copy between different disks\nerr := storage.CopyBetweenDisks(ctx, \"local\", \"s3\", \"local-file.txt\", \"s3-file.txt\")\n\n// Move between different disks\nerr := storage.MoveBetweenDisks(ctx, \"local\", \"s3\", \"local-file.txt\", \"s3-file.txt\")\n\n// List files with a prefix\nfiles, err := storage.List(ctx, \"disk\", \"documents/\")\nfor _, file := range files {\n    fmt.Printf(\"%s - %d bytes\\n\", file.Path, file.Size)\n}\n```\n\n### Metadata Operations\n\nStore custom metadata with your files:\n\n```go\n// Put file with metadata\nmetadata := \u0026gostorage.Metadata{\n    ContentType: \"application/json\",\n    CustomHeaders: map[string]string{\n        \"author\":  \"John Doe\",\n        \"version\": \"1.0\",\n    },\n}\nerr := storage.PutWithMetadata(ctx, \"disk\", \"data.json\", content, metadata)\n\n// Get metadata\nmeta, err := storage.GetMetadata(ctx, \"disk\", \"data.json\")\nif meta != nil {\n    fmt.Println(\"Content-Type:\", meta.ContentType)\n    fmt.Println(\"Author:\", meta.CustomHeaders[\"author\"])\n}\n\n// Update metadata\nnewMeta := \u0026gostorage.Metadata{\n    ContentType: \"application/json\",\n    CustomHeaders: map[string]string{\n        \"author\":  \"Jane Doe\",\n        \"version\": \"2.0\",\n    },\n}\nerr = storage.SetMetadata(ctx, \"disk\", \"data.json\", newMeta)\n```\n\n## File Information\n\nThe `List` operation returns detailed file information:\n\n```go\ntype FileInfo struct {\n    Path         string      // File path\n    Size         int64       // File size in bytes\n    LastModified time.Time   // Last modification time\n    IsDir        bool        // Whether it's a directory\n    Metadata     *Metadata   // File metadata (if available)\n}\n```\n\n## Error Handling\n\nThe package provides structured errors:\n\n```go\ndata, err := storage.Get(ctx, \"disk\", \"file.txt\")\nif err != nil {\n    var pathErr *gostorage.PathError\n    if errors.As(err, \u0026pathErr) {\n        fmt.Printf(\"Operation: %s, Path: %s, Error: %v\\n\",\n            pathErr.Op, pathErr.Path, pathErr.Err)\n    }\n\n    if errors.Is(err, gostorage.ErrFileNotFound) {\n        fmt.Println(\"File not found\")\n    }\n}\n```\n\nAvailable errors:\n- `ErrFileNotFound` - File doesn't exist\n- `ErrInvalidPath` - Invalid path provided\n- `ErrOperationNotSupported` - Operation not supported by disk\n- `DiskNotFoundError` - Disk not found\n\n## Security\n\nAll paths are automatically validated and sanitized to prevent:\n- Directory traversal attacks (e.g., `../../../etc/passwd`)\n- Null byte injection\n- Invalid path characters\n\nPaths are automatically converted to relative paths and cleaned.\n\n## Creating Custom Disk Backends\n\nImplement the `Disk` interface to create custom backends:\n\n```go\ntype Disk interface {\n    // Basic operations\n    put(ctx context.Context, path string, content []byte) error\n    get(ctx context.Context, path string) ([]byte, error)\n    delete(ctx context.Context, path string) error\n\n    // Streaming operations\n    putStream(ctx context.Context, path string, reader io.Reader, metadata *Metadata) error\n    getStream(ctx context.Context, path string) (io.ReadCloser, error)\n\n    // File operations\n    exists(ctx context.Context, path string) (bool, error)\n    size(ctx context.Context, path string) (int64, error)\n    list(ctx context.Context, prefix string) ([]FileInfo, error)\n    copy(ctx context.Context, sourcePath, destPath string) error\n    move(ctx context.Context, sourcePath, destPath string) error\n\n    // Metadata operations\n    putWithMetadata(ctx context.Context, path string, content []byte, metadata *Metadata) error\n    getMetadata(ctx context.Context, path string) (*Metadata, error)\n    setMetadata(ctx context.Context, path string, metadata *Metadata) error\n}\n```\n\n## Examples\n\nSee the [example](./example/main.go) directory for comprehensive usage examples including:\n- Basic file operations\n- Streaming large files\n- Working with metadata\n- Copying and moving files\n- Listing files with prefixes\n\nRun the example:\n```bash\ncd example\ngo run main.go\n```\n\n## Use Cases\n\n- **Multi-cloud applications**: Abstract storage to easily switch between local and cloud storage\n- **File upload services**: Handle user uploads with consistent API regardless of backend\n- **Backup systems**: Copy files between local and cloud storage\n- **Content management**: Store and retrieve files with metadata\n- **Large file processing**: Stream large files without memory constraints\n\n## License\n\nMIT\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenframebox%2Fgostorage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenframebox%2Fgostorage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenframebox%2Fgostorage/lists"}