{"id":37101329,"url":"https://github.com/gurre/s3streamer","last_synced_at":"2026-01-14T12:18:42.917Z","repository":{"id":299494121,"uuid":"1003209929","full_name":"gurre/s3streamer","owner":"gurre","description":"Byte-stream objects from and to S3 using Golang io.Reader.","archived":false,"fork":false,"pushed_at":"2025-08-20T10:27:32.000Z","size":50,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-20T11:39:05.573Z","etag":null,"topics":["aws","aws-s3","bigdata","golang-library","s3","streaming"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gurre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-16T19:47:33.000Z","updated_at":"2025-08-20T10:07:12.000Z","dependencies_parsed_at":"2025-08-20T11:23:10.643Z","dependency_job_id":"9b37347d-091a-4b7a-9ab8-4e67cc77a9f5","html_url":"https://github.com/gurre/s3streamer","commit_stats":null,"previous_names":["gurre/s3streamer"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/gurre/s3streamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gurre%2Fs3streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gurre%2Fs3streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gurre%2Fs3streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gurre%2Fs3streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gurre","download_url":"https://codeload.github.com/gurre/s3streamer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gurre%2Fs3streamer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28420042,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T10:47:48.104Z","status":"ssl_error","status_checked_at":"2026-01-14T10:46:19.031Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","aws-s3","bigdata","golang-library","s3","streaming"],"created_at":"2026-01-14T12:18:42.208Z","updated_at":"2026-01-14T12:18:42.905Z","avatar_url":"https://github.com/gurre.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# s3streamer\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/gurre/s3streamer.svg)](https://pkg.go.dev/github.com/gurre/s3streamer)\n[![Go Report Card](https://goreportcard.com/badge/github.com/gurre/s3streamer)](https://goreportcard.com/report/github.com/gurre/s3streamer)\n\nA high-performance, memory-efficient Go library for streaming large objects to and from Amazon S3. Built on Go's standard `io.Reader` and `io.Writer` interfaces, s3streamer enables processing of multi-gigabyte files without loading them entirely into memory. Features both reading from S3 with chunked downloads and writing to S3 with multipart uploads.\n\n## Key Features\n\n- **Memory Efficient**: Stream objects of any size with configurable chunk sizes\n- **Bidirectional Streaming**: Both `io.Reader` and `io.Writer` implementations for complete S3 integration\n- **Automatic Compression**: Supports gzip and bzip2 with automatic detection/compression via magic bytes or file extensions\n- **Resume Capability**: Start streaming from any byte offset for resumable processing\n- **Line-by-Line Processing**: Optimized for JSON Lines and other line-delimited formats with offset tracking\n- **Multipart Upload**: Efficient writing to S3 using multipart uploads with configurable part sizes (enforces 5MiB minimum)\n- **High Performance**: Configurable chunking with up to 10MB line buffer support\n- **AWS SDK v2 Compatible**: Works with the latest AWS SDK for Go v2\n\n## Installation\n\n```bash\ngo get github.com/gurre/s3streamer\n```\n\n## Quick Start\n\n### Reading from S3\n\n```go\n// Load AWS configuration\ncfg, err := config.LoadDefaultConfig(context.TODO())\nif err != nil {\n    log.Fatal(err)\n}\n\n// Create S3 client and streamer\nclient := s3.NewFromConfig(cfg)\nstreamer := s3streamer.NewS3Streamer(client)\n\n// Process a large JSON Lines file with offset tracking\nerr = streamer.Stream(context.Background(), \"my-bucket\", \"large-file.jsonl.gz\", 0, \n    func(line []byte, offset int64) error {\n        var record map[string]interface{}\n        if err := json.Unmarshal(line, \u0026record); err != nil {\n            return err\n        }\n        \n        // Process your record here with access to its byte offset\n        fmt.Printf(\"Processing record at offset %d: %v\\n\", offset, record[\"id\"])\n        return nil\n    })\n\nif err != nil {\n    log.Fatal(err)\n}\n```\n\n### Writing to S3\n\n```go\n// Load AWS configuration\ncfg, err := config.LoadDefaultConfig(context.TODO())\nif err != nil {\n    log.Fatal(err)\n}\n\n// Create S3 client and writer\nclient := s3.NewFromConfig(cfg)\nwriter, err := s3streamer.NewCompressedS3Writer(context.Background(), client, \"my-bucket\", \"output.jsonl.gz\", 5*1024*1024, s3streamer.Gzip)\nif err != nil {\n    log.Fatal(err)\n}\ndefer writer.Close()\n\n// Write JSON Lines data (automatically compressed)\nfor i := 0; i \u003c 10000; i++ {\n    record := map[string]interface{}{\n        \"id\":      i,\n        \"message\": fmt.Sprintf(\"Record number %d\", i),\n        \"timestamp\": time.Now().Unix(),\n    }\n    \n    data, _ := json.Marshal(record)\n    data = append(data, '\\n') // Add newline for JSON Lines format\n    \n    if _, err := writer.Write(data); err != nil {\n        log.Fatal(err)\n    }\n}\n\n// Close to finalize the upload\nif err := writer.Close(); err != nil {\n    log.Fatal(err)\n}\n```\n\n## Standard Library Integration\n\n### io.Reader Implementation\n\nThe `ChunkStreamer` implements Go's standard `io.Reader` interface, making it compatible with any library that accepts readers:\n\n```go\n// Stream directly to any io.Writer\nfunc copyToFile(ctx context.Context, client *s3.Client, bucket, key, filename string, fileSize int64) error {\n    // Create chunk streamer\n    streamer := s3streamer.NewChunkStreamer(ctx, client, bucket, key, 0, fileSize, 5*1024*1024)\n    \n    // Use with standard library\n    file, err := os.Create(filename)\n    if err != nil {\n        return err\n    }\n    defer file.Close()\n    \n    // Standard io.Copy works seamlessly\n    _, err = io.Copy(file, streamer)\n    return err\n}\n\n// Use with compression libraries\nfunc processCompressedStream(ctx context.Context, client *s3.Client, bucket, key string, fileSize int64) error {\n    streamer := s3streamer.NewChunkStreamer(ctx, client, bucket, key, 0, fileSize, 1024*1024)\n    \n    // Automatic decompression\n    reader, err := s3streamer.Decompress(streamer)\n    if err != nil {\n        return err\n    }\n    \n    // Use with bufio.Scanner for line processing\n    scanner := bufio.NewScanner(reader)\n    for scanner.Scan() {\n        line := scanner.Text()\n        // Process line\n    }\n    return scanner.Err()\n}\n```\n\n### Pipe Integration\n\nCombine with Go's `io.Pipe` for concurrent processing:\n\n```go\nfunc streamWithPipe(ctx context.Context, client *s3.Client, bucket, key string, fileSize int64) error {\n    pr, pw := io.Pipe()\n    \n    // Stream in background goroutine\n    go func() {\n        defer pw.Close()\n        streamer := s3streamer.NewChunkStreamer(ctx, client, bucket, key, 0, fileSize, 1024*1024)\n        io.Copy(pw, streamer)\n    }()\n    \n    // Process in main goroutine\n    reader, err := s3streamer.Decompress(pr)\n    if err != nil {\n        return err\n    }\n    \n    scanner := bufio.NewScanner(reader)\n    for scanner.Scan() {\n        // Process each line as it arrives\n        processLine(scanner.Bytes())\n    }\n    \n    return scanner.Err()\n}\n```\n\n## Writing to S3\n\n### Basic S3 Writing\n\nThe `S3Writer` implements Go's standard `io.Writer` interface and uses AWS S3's multipart upload API for efficient streaming uploads:\n\n```go\n// Create an S3 writer with 5MiB part size\nwriter, err := s3streamer.NewS3Writer(ctx, client, \"my-bucket\", \"output.json\", 5*1024*1024)\nif err != nil {\n    log.Fatal(err)\n}\ndefer writer.Close() // Important: always close to complete the upload\n\n// Write data (can be called multiple times)\ndata := []byte(\"Hello, World!\\nThis is streaming to S3.\\n\")\nn, err := writer.Write(data)\nif err != nil {\n    log.Fatal(err)\n}\n\n// Close finalizes the multipart upload\nif err := writer.Close(); err != nil {\n    log.Fatal(err)\n}\n```\n\n### Compressed Writing\n\nThe `CompressedS3Writer` applies the specified compression format before uploading:\n\n```go\n// Gzip compression\nwriter, err := s3streamer.NewCompressedS3Writer(ctx, client, \"my-bucket\", \"output.json.gz\", 5*1024*1024, s3streamer.Gzip)\nif err != nil {\n    log.Fatal(err)\n}\ndefer writer.Close()\n\n// Data is compressed before upload\nfor i := 0; i \u003c 1000; i++ {\n    record := fmt.Sprintf(`{\"id\": %d, \"message\": \"Hello, World!\"}` + \"\\n\", i)\n    if _, err := writer.Write([]byte(record)); err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\n### Different Compression Types\n\nYou can specify any supported compression type:\n\n```go\n// Bzip2 compression (slower but better compression ratio)\nwriter, err := s3streamer.NewCompressedS3Writer(ctx, client, \"my-bucket\", \"data.txt.bz2\", \n    5*1024*1024, s3streamer.Bzip2)\nif err != nil {\n    log.Fatal(err)\n}\ndefer writer.Close()\n\n// Uncompressed (no compression)\nuncompressedWriter, err := s3streamer.NewCompressedS3Writer(ctx, client, \"my-bucket\", \"data.txt\", \n    5*1024*1024, s3streamer.Uncompressed)\nif err != nil {\n    log.Fatal(err)\n}\ndefer uncompressedWriter.Close()\n\n// Write data\nlargeData := bytes.Repeat([]byte(\"This data compresses well!\\n\"), 10000)\nwriter.Write(largeData)\n```\n\n### Standard Library Integration\n\nWorks seamlessly with any code that accepts `io.Writer`:\n\n```go\n// Copy from any reader to S3\nfunc uploadFile(ctx context.Context, client *s3.Client, filename, bucket, key string) error {\n    file, err := os.Open(filename)\n    if err != nil {\n        return err\n    }\n    defer file.Close()\n    \n    writer, err := s3streamer.NewCompressedS3Writer(ctx, client, bucket, key+\".gz\", 5*1024*1024, s3streamer.Gzip)\n    if err != nil {\n        return err\n    }\n    defer writer.Close()\n    \n    // Standard io.Copy works seamlessly\n    _, err = io.Copy(writer, file)\n    if err != nil {\n        return err\n    }\n    \n    return writer.Close()\n}\n\n// Use with encoding libraries\nfunc writeJSONLines(ctx context.Context, client *s3.Client, records []Record) error {\n    writer, err := s3streamer.NewCompressedS3Writer(ctx, client, \"data-bucket\", \"records.jsonl.gz\", 10*1024*1024, s3streamer.Gzip)\n    if err != nil {\n        return err\n    }\n    defer writer.Close()\n    \n    encoder := json.NewEncoder(writer)\n    for _, record := range records {\n        if err := encoder.Encode(record); err != nil {\n            return err\n        }\n    }\n    \n    return writer.Close()\n}\n```\n\n### Error Handling and Cleanup\n\nAlways handle errors properly and use the abort functionality when needed:\n\n```go\nfunc safeCopyToS3(ctx context.Context, client *s3.Client, src io.Reader, bucket, key string) error {\n    writer, err := s3streamer.NewCompressedS3Writer(ctx, client, bucket, key, 5*1024*1024, s3streamer.Gzip)\n    if err != nil {\n        return err\n    }\n    \n    // Use defer to ensure cleanup happens\n    defer func() {\n        if err != nil {\n            // Abort the upload on error to clean up partial uploads\n            writer.Abort()\n        } else {\n            // Complete the upload on success\n            writer.Close()\n        }\n    }()\n    \n    _, err = io.Copy(writer, src)\n    return err\n}\n```\n\n## Advanced Usage\n\n### Line Offset Tracking\n\nEach line callback receives both the line data and its byte offset within the decompressed stream, enabling precise error reporting and resumable processing:\n\n```go\nfunc processWithOffsetTracking(ctx context.Context, client *s3.Client, bucket, key string) error {\n    streamer := s3streamer.NewS3Streamer(client)\n    \n    var processedCount int\n    return streamer.Stream(ctx, bucket, key, 0, func(line []byte, offset int64) error {\n        processedCount++\n        \n        // Use offset for precise error reporting\n        if err := validateRecord(line); err != nil {\n            return fmt.Errorf(\"validation failed at byte offset %d (record %d): %w\", \n                offset, processedCount, err)\n        }\n        \n        // Save checkpoint with exact position for resumable processing\n        if processedCount%1000 == 0 {\n            if err := saveCheckpoint(offset); err != nil {\n                return fmt.Errorf(\"failed to save checkpoint at offset %d: %w\", offset, err)\n            }\n        }\n        \n        return processRecord(line)\n    })\n}\n```\n\n### Resume from Offset\n\nProcess large files in chunks or resume interrupted operations:\n\n\u003e [!NOTE]\n\u003e Resume capability with non-zero offsets is **only supported for uncompressed files**. Compressed files (gzip, bzip2) cannot be resumed from arbitrary byte offsets because compression streams require reading from the beginning to properly decompress. For compressed files, only use `offset = 0`.\n\n```go\nfunc resumableProcessing(ctx context.Context, client *s3.Client, bucket, key string) error {\n    streamer := s3streamer.NewS3Streamer(client)\n    \n    // Calculate offset (e.g., from previous processing state)\n    // NOTE: This only works for uncompressed files!\n    offset := int64(1024 * 1024 * 100) // Skip first 100MB\n    \n    var recordCount int\n    err := streamer.Stream(ctx, bucket, key, offset, func(line []byte, lineOffset int64) error {\n        recordCount++\n        \n        // The lineOffset parameter gives you the exact position of this line\n        // within the decompressed stream (starting from 0)\n        actualFileOffset := offset + lineOffset\n        \n        // Save progress every 1000 records with precise positioning\n        if recordCount%1000 == 0 {\n            saveCheckpoint(actualFileOffset)\n        }\n        \n        return processRecord(line)\n    })\n    \n    return err\n}\n```\n\n**Supported Resume Scenarios:**\n- ✅ Uncompressed files with any offset\n- ✅ Compressed files with offset = 0 only\n- ❌ Compressed files with non-zero offsets (will cause decompression errors)\n\n## Performance Characteristics\n\n### Memory Usage\n\n- **Constant Memory**: Memory usage remains constant regardless of file size for both reading and writing\n- **Configurable Buffers**: Default 5MiB chunks/parts with configurable sizes\n- **Line Buffer**: Up to 10MB for processing extremely long lines (reading)\n- **Part Buffer**: Each writer part is buffered separately, with minimal memory overhead\n\n### Reading Optimization\n\n```go\n// Default configuration uses 5MiB chunks, optimal for most cases\nstreamer := s3streamer.NewS3Streamer(client)\n\n// For high-throughput or custom chunk sizes, use ChunkStreamer directly\nhighThroughputStreamer := s3streamer.NewChunkStreamer(ctx, client, bucket, key, 0, fileSize, 10*1024*1024) // 10MiB chunks\n\n// For low-latency processing of many small files\nlowLatencyStreamer := s3streamer.NewChunkStreamer(ctx, client, bucket, key, 0, fileSize, 1*1024*1024) // 1MB chunks\n```\n\n### Writing Optimization\n\n```go\n// Default 5MiB parts balance performance and memory usage\nwriter, _ := s3streamer.NewS3Writer(ctx, client, bucket, key, 5*1024*1024)\n\n// High-throughput uploads with larger parts (AWS supports up to 5GiB per part)\nhighThroughputWriter, _ := s3streamer.NewS3Writer(ctx, client, bucket, key, 100*1024*1024) // 100MiB parts\n\n// Part size must be at least 5MiB (AWS requirement)\nmemoryEfficientWriter, _ := s3streamer.NewS3Writer(ctx, client, bucket, key, 5*1024*1024) // 5MiB minimum\n\n// Compressed writing (data is compressed before applying part size limits)\ncompressedWriter, _ := s3streamer.NewCompressedS3Writer(ctx, client, bucket, key, 10*1024*1024, s3streamer.Gzip) // 10MiB parts\n```\n\n### Benchmark Results\n\nBased on included benchmarks processing 1000 records (Apple M4 Pro):\n\n#### Reading Performance (Different Chunk Sizes)\n\n| Chunk Size | Time/Operation | Memory/Operation | Allocations/Operation |\n|------------|----------------|------------------|-----------------------|\n| 256KB      | 152.4 μs       | 1.13 MB          | 66                    |\n| 512KB      | 153.5 μs       | 1.13 MB          | 66                    |\n| 1MB        | 155.2 μs       | 1.13 MB          | 66                    |\n| 5MiB       | 150.6 μs       | 1.13 MB          | 66                    |\n\n*Results show consistent performance across chunk sizes with minimal overhead.*\n\n#### Writing Performance (Different Part Sizes)\n\n| Part Size | Time/Operation | Memory/Operation | Allocations/Operation |\n|-----------|----------------|------------------|-----------------------|\n| 5MiB      | 84.0 μs        | 609 KB           | 34                    |\n| 10MiB     | 84.1 μs        | 609 KB           | 34                    |\n| 25MiB     | 84.5 μs        | 609 KB           | 34                    |\n| 50MiB     | 83.0 μs        | 609 KB           | 34                    |\n\n*Performance remains consistent across different part sizes with minimal overhead.*\n\n#### Compression Performance (10MiB Parts)\n\n| Compression | Time/Operation | Memory/Operation | Allocations/Operation |\n|-------------|----------------|------------------|-----------------------|\n| None        | 76.2 μs        | 609 KB           | 33                    |\n| Gzip        | 356.3 μs       | 840 KB           | 40                    |\n| Bzip2       | 3,058.2 μs     | 2.08 MB          | 68                    |\n\n*Compression adds CPU overhead but significantly reduces upload size for compressible data.*\n\n#### Write Pattern Performance (10MiB Parts)\n\n| Write Pattern    | Time/Operation | Memory/Operation | Allocations/Operation |\n|------------------|----------------|------------------|-----------------------|\n| Single Write     | 117.3 μs       | 838 KB           | 53                    |\n| 100 Records/Write| 112.0 μs       | 785 KB           | 143                   |\n| 10 Records/Write | 108.4 μs       | 712 KB           | 533                   |\n| Per Record Write | 97.4 μs        | 559 KB           | 1033                  |\n\n*Smaller writes show better performance due to reduced buffer management overhead.*\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgurre%2Fs3streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgurre%2Fs3streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgurre%2Fs3streamer/lists"}