{"id":34809889,"url":"https://github.com/databricks/zerobus-sdk-go","last_synced_at":"2026-01-13T20:48:13.369Z","repository":{"id":330294736,"uuid":"1099823906","full_name":"databricks/zerobus-sdk-go","owner":"databricks","description":"Zerobus SDK for Go","archived":false,"fork":false,"pushed_at":"2025-12-25T10:08:38.000Z","size":75157,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-26T03:49:45.251Z","etag":null,"topics":["databricks","databricks-sdk","go","golang","zerobus","zerobus-sdk"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":"DCO","cla":null}},"created_at":"2025-11-19T13:44:36.000Z","updated_at":"2025-12-25T09:56:14.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/databricks/zerobus-sdk-go","commit_stats":null,"previous_names":["databricks/zerobus-sdk-go"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/databricks/zerobus-sdk-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/zerobus-sdk-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28400125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks","databricks-sdk","go","golang","zerobus","zerobus-sdk"],"created_at":"2025-12-25T12:36:20.149Z","updated_at":"2026-01-13T20:48:13.359Z","avatar_url":"https://github.com/databricks.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Zerobus Go SDK\n\nA high-performance Go client for streaming data ingestion into Databricks Delta tables using the Zerobus service.\n\n## Disclaimer\n\n[Public Preview](https://docs.databricks.com/release-notes/release-types.html): This SDK is supported for production use cases and is available to all customers. Databricks is actively working on stabilizing the Zerobus Ingest SDK for Go. Minor version updates may include backwards-incompatible changes.\n\nWe are keen to hear feedback from you on this SDK. Please [file issues](https://github.com/databricks/zerobus-sdk-go/issues), and we will address them.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Architecture](#architecture)\n- [Getting Started](#getting-started)\n  - [For SDK Users (Install from pkg.go.dev)](#for-sdk-users-install-from-pkggodev)\n  - [For Contributors (Build from Source)](#for-contributors-build-from-source)\n- [Quick Start](#quick-start)\n- [Repository Structure](#repository-structure)\n- [Usage Guide](#usage-guide)\n  - [1. Initialize the SDK](#1-initialize-the-sdk)\n  - [2. Configure Authentication](#2-configure-authentication)\n  - [3. Create a Stream](#3-create-a-stream)\n  - [4. Ingest Data](#4-ingest-data)\n  - [5. Handle Acknowledgments](#5-handle-acknowledgments)\n  - [6. Close the Stream](#6-close-the-stream)\n- [Configuration Options](#configuration-options)\n- [Error Handling](#error-handling)\n- [Examples](#examples)\n- [Best Practices](#best-practices)\n- [API Reference](#api-reference)\n- [Building from Source](#building-from-source)\n- [Troubleshooting](#troubleshooting)\n- [Community and Contributing](#community-and-contributing)\n- [License](#license)\n\n## Overview\n\nThe Zerobus Go SDK provides a robust, CGO-based wrapper around the high-performance Rust implementation for ingesting large volumes of data into Databricks Delta tables. It abstracts the complexity of the Zerobus service and handles authentication, retries, stream recovery, and acknowledgment tracking automatically.\n\n**What is Zerobus?** Zerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads.\n\nThis SDK wraps the Rust [zerobus-sdk-rs](https://github.com/databricks/zerobus-sdk-rs) using CGO and FFI (Foreign Function Interface), providing an idiomatic Go API while leveraging the performance and reliability of the underlying Rust implementation.\n\n## Features\n\n- **Static Linking** - Self-contained binaries with no runtime dependencies or LD_LIBRARY_PATH configuration\n- **High-throughput streaming ingestion** into Databricks Delta tables\n- **Automatic OAuth 2.0 authentication** with Unity Catalog\n- **Simple JSON ingestion** - No code generation required for basic use cases\n- **Protocol Buffers support** for type-safe, efficient data encoding\n- **Backpressure control** to manage memory usage\n- **Automatic retry and recovery** for transient failures\n- **Configurable timeouts and retry policies**\n- **Async acknowledgments** for ingested records\n- **Graceful stream management** - Proper flushing and acknowledgment tracking\n\n## Architecture\n\n```\n┌─────────────────┐\n│   Go SDK        │  ← Idiomatic Go API (this package)\n│   (zerobus)     │\n└────────┬────────┘\n         │ CGO (Static Linking)\n┌────────▼────────┐\n│  Rust FFI Crate │  ← C-compatible wrapper\n│ (zerobus-ffi)   │\n└────────┬────────┘\n         │\n┌────────▼────────┐\n│  Rust SDK       │  ← Core async implementation (Tokio)\n│(zerobus-sdk-rs) │     - gRPC bidirectional streaming\n└────────┬────────┘     - OAuth 2.0 authentication\n         │              - Automatic recovery\n         ▼\n   Databricks\n Zerobus Service\n```\n\n## Getting Started\n\nChoose your installation path:\n\n| Path | When to Use |\n|------|-------------|\n| **[Standard Installation](#installation)** | You want to use the SDK in your project (via `go get`) |\n| **[Development Setup](#development-setup)** | You want to contribute or build from source (via `git clone`) |\n\n### Installation\n\nThe Zerobus Go SDK is a CGO-based wrapper around a high-performance Rust core. For the best experience, use a tagged release which includes pre-built binaries.\n\n**Prerequisites:**\n\n- **Go 1.21+**\n- **CGO enabled** (enabled by default)\n- **C compiler** (gcc or clang)\n\n**Installation Steps:**\n\n```bash\n# Add the SDK to your project (use a tagged version for pre-built binaries)\ngo get github.com/databricks/zerobus-sdk-go@latest\n```\n\n\u003e **Note:** Tagged releases (e.g., `v1.0.0`) come with pre-built Rust libraries for Linux, macOS, and Windows. If you use `@main` or a commit hash, you will need to have Rust installed and run `go generate` to build the library yourself.\n\n**In your code:**\n\n```go\nimport zerobus \"github.com/databricks/zerobus-sdk-go\"\n\nfunc main() {\n    sdk, err := zerobus.NewZerobusSdk(\n        \"https://your-shard.zerobus.region.cloud.databricks.com\",\n        \"https://your-workspace.cloud.databricks.com\",\n    )\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer sdk.Free()\n    // ... create streams and ingest data\n}\n```\n\n\u003e **Note:** After the initial `go generate` step, regular `go build` works normally. The Rust library is statically linked into your binary.\n\n### Development Setup\n\n**Prerequisites:**\n- Same as above (Go, CGO, Rust, C compiler, Databricks workspace)\n- **Git**\n\n**Setup:**\n\n```bash\n# Clone the repository\ngit clone https://github.com/databricks/zerobus-sdk-go.git\ncd zerobus-sdk-go\ngo generate  # Builds Rust FFI\nmake build   # Builds everything\n```\n\nSee [Building from Source](#building-from-source) for more build options and [Community and Contributing](#community-and-contributing) for contribution guidelines.\n\n## Quick Start\n\nOnce you've completed the [Getting Started](#getting-started) steps, here's how to ingest data:\n\n### JSON Ingestion (Simplest Approach)\n\nPerfect for getting started quickly without code generation:\n\n```go\npackage main\n\nimport (\n    \"log\"\n    zerobus \"github.com/databricks/zerobus-sdk-go\"\n)\n\nfunc main() {\n    // 1. Initialize SDK\n    sdk, err := zerobus.NewZerobusSdk(\n        \"https://your-shard-id.zerobus.region.cloud.databricks.com\",\n        \"https://your-workspace.cloud.databricks.com\",\n    )\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer sdk.Free()\n\n    // 2. Configure for JSON records\n    options := zerobus.DefaultStreamConfigurationOptions()\n    options.RecordType = zerobus.RecordTypeJson\n\n    // 3. Create stream\n    stream, err := sdk.CreateStream(\n        zerobus.TableProperties{\n            TableName: \"catalog.schema.table\",\n        },\n        \"your-client-id\",\n        \"your-client-secret\",\n        options,\n    )\n    if err != nil {\n        log.Fatal(err)\n    }\n    defer stream.Close()\n\n    // 4. Ingest record (blocks until queued, returns ack handle)\n    ack, err := stream.IngestRecord(`{\"id\": 1, \"message\": \"Hello\"}`)\n    if err != nil {\n        log.Fatal(err)\n    }\n\n    // 5. Await acknowledgment to get offset\n    offset, err := ack.Await()\n    if err != nil {\n        log.Fatal(err)\n    }\n    log.Printf(\"Ingested record at offset %d\", offset)\n\n    // 6. Flush to ensure durability\n    if err := stream.Flush(); err != nil {\n        log.Fatal(err)\n    }\n}\n```\n\n### Protocol Buffer Ingestion (Production Recommended)\n\nFor type-safe, efficient ingestion in production:\n\n```go\nimport (\n    \"google.golang.org/protobuf/proto\"\n    \"google.golang.org/protobuf/types/descriptorpb\"\n)\n\n// 1. Load descriptor from generated files\ndescriptorBytes, err := os.ReadFile(\"path/to/schema.descriptor\")\nif err != nil {\n    log.Fatal(err)\n}\n\ndescriptor := \u0026descriptorpb.DescriptorProto{}\nif err := proto.Unmarshal(descriptorBytes, descriptor); err != nil {\n    log.Fatal(err)\n}\n\n// 2. Create stream for Proto records\noptions := zerobus.DefaultStreamConfigurationOptions()\noptions.RecordType = zerobus.RecordTypeProto\n\nstream, err := sdk.CreateStream(\n    zerobus.TableProperties{\n        TableName:       \"catalog.schema.table\",\n        DescriptorProto: descriptorBytes,\n    },\n    clientID,\n    clientSecret,\n    options,\n)\n\n// 3. Ingest proto-encoded record (blocks until queued)\nack, err := stream.IngestRecord(protoBytes)\n\n// 4. Await acknowledgment to get offset\noffset, err := ack.Await()\n```\n\n### Next Steps\n\n- See [Usage Guide](#usage-guide) for detailed step-by-step documentation\n- See [Examples](#examples) for complete working examples you can run\n- See [Configuration Options](#configuration-options) to tune performance\n\n## Repository Structure\n\n```\nzerobus-sdk-go/\n├── zerobus.go                      # Main SDK and stream implementation\n├── ffi.go                          # CGO bindings to Rust FFI\n├── ffi_test.go                     # CGO bindings tests\n├── errors.go                       # Error types\n├── types.go                        # Type definitions\n├── ack.go                          # Acknowledgment handling\n├── build.go                        # Build utilities and go:generate\n├── build_rust.sh                   # Rust build script\n├── zerobus.h                       # C header (generated)\n├── go.mod                          # Go module definition\n├── zerobus-ffi/                    # Rust FFI crate\n│   ├── src/\n│   │   ├── lib.rs                  # FFI wrapper implementation\n│   │   └── tests.rs                # Rust FFI tests\n│   ├── zerobus.h                   # C header for CGO (generated)\n│   ├── Cargo.toml                  # Rust dependencies\n│   ├── Cargo.lock                  # Rust dependency lock file\n│   └── build.rs                    # Build script for cbindgen\n│\n├── examples/                       # Working examples\n│   ├── basic_example_json/         # JSON ingestion example\n│   │   ├── basic_json_usage.go     # JSON-based example\n│   │   └── go.mod                  # Module file\n│   └── basic_example_proto/        # Protocol Buffer example\n│       ├── basic_proto_usage.go    # Proto-based example\n│       ├── air_quality.proto       # Example proto schema\n│       ├── pb/                     # Generated proto code\n│       └── go.mod                  # Module file\n│\n├── Makefile                        # Build automation\n├── README.md                       # This file\n├── CHANGELOG.md                    # Version history\n├── CONTRIBUTING.md                 # Contribution guidelines\n├── SECURITY.md                     # Security policy\n├── DCO                             # Developer Certificate of Origin\n├── NOTICE                          # Third-party attribution\n└── LICENSE                         # License file\n```\n\n### Key Components\n\n- **Root directory** - The main Go SDK library\n- **`zerobus-ffi/`** - Rust FFI wrapper for high-performance ingestion\n- **`examples/`** - Complete working examples demonstrating SDK usage\n- **`Makefile`** - Standard make targets for building, testing, and linting\n\n## Usage Guide\n\nThis section provides detailed step-by-step documentation. For a quick start, see [Quick Start](#quick-start). For working examples, see [Examples](#examples).\n\n### 1. Initialize the SDK\n\nCreate an SDK instance with your Databricks workspace endpoints:\n\n```go\n// For AWS\nsdk, err := zerobus.NewZerobusSdk(\n    \"https://your-shard-id.zerobus.region.cloud.databricks.com\",\n    \"https://your-workspace.cloud.databricks.com\",\n)\n\n// For Azure\nsdk, err := zerobus.NewZerobusSdk(\n    \"https://your-shard-id.zerobus.region.azuredatabricks.net\",\n    \"https://your-workspace.azuredatabricks.net\",\n)\n\nif err != nil {\n    log.Fatal(err)\n}\ndefer sdk.Free()\n```\n\n### 2. Configure Authentication\n\nThe SDK handles authentication automatically. You just need to provide your OAuth credentials:\n\n```go\nclientID := os.Getenv(\"DATABRICKS_CLIENT_ID\")\nclientSecret := os.Getenv(\"DATABRICKS_CLIENT_SECRET\")\n```\n\nSee the examples directory for how to obtain OAuth credentials.\n\n### Custom Authentication\n\nFor advanced use cases, you can implement the `HeadersProvider` interface to supply your own authentication headers. This is useful for integrating with a different OAuth provider, using a centralized token caching service, or implementing alternative authentication mechanisms.\n\n\u003e **Note:** The headers you provide must still conform to the authentication protocol expected by the Zerobus service. The default OAuth implementation serves as the reference for the required headers (`authorization` and `x-databricks-zerobus-table-name`). This feature provides flexibility in *how* you source your credentials, not in changing the authentication protocol itself.\n\n**Example:**\n\n```go\nimport zerobus \"github.com/databricks/zerobus-sdk-go\"\n\n// Implement the HeadersProvider interface.\ntype MyCustomAuthProvider struct {\n    tableName string\n}\n\nfunc (p *MyCustomAuthProvider) GetHeaders() (map[string]string, error) {\n    // Custom logic to fetch and cache a token would go here.\n    return map[string]string{\n        \"authorization\":                    \"Bearer \u003cyour-token\u003e\",\n        \"x-databricks-zerobus-table-name\": p.tableName,\n    }, nil\n}\n\nfunc example(sdk *zerobus.ZerobusSdk, tableProps zerobus.TableProperties) error {\n    customProvider := \u0026MyCustomAuthProvider{tableName: \"catalog.schema.table\"}\n\n    stream, err := sdk.CreateStreamWithHeadersProvider(\n        tableProps,\n        customProvider,\n        nil,\n    )\n    if err != nil {\n        return err\n    }\n    defer stream.Close()\n\n    ack, _ := stream.IngestRecord(`{\"data\": \"value\"}`)\n    offset, _ := ack.Await()\n    return nil\n}\n```\n\n**Common use cases:**\n\n- **Token caching**: Implement custom token refresh logic\n- **Alternative auth mechanisms**: Use different authentication providers\n- **Dynamic credentials**: Fetch credentials on-demand from secret managers\n\n### 3. Create a Stream\n\nConfigure table properties and stream options:\n\n```go\noptions := zerobus.DefaultStreamConfigurationOptions()\noptions.MaxInflightRequests = 10000\noptions.Recovery = true\noptions.RecoveryRetries = 4\noptions.RecordType = zerobus.RecordTypeJson\n\nstream, err := sdk.CreateStream(\n    zerobus.TableProperties{\n        TableName: \"catalog.schema.table\",\n    },\n    clientID,\n    clientSecret,\n    options,\n)\nif err != nil {\n    log.Fatal(err)\n}\ndefer stream.Close()\n```\n\n### 4. Ingest Data\n\n**Single record:**\n\n```go\n// JSON (string) - blocks until queued, handles backpressure\nack, err := stream.IngestRecord(`{\"id\": 1, \"value\": \"hello\"}`)\nif err != nil {\n    log.Fatal(err)\n}\n\n// Await acknowledgment\noffset, err := ack.Await()\nif err != nil {\n    log.Fatal(err)\n}\nlog.Printf(\"Record ingested at offset: %d\", offset)\n```\n\n**Batch ingestion for high throughput:**\n\n```go\n// Queue all records\nacks := make([]*zerobus.RecordAck, 0, 100000)\nfor i := 0; i \u003c 100000; i++ {\n    jsonData := fmt.Sprintf(`{\"id\": %d, \"timestamp\": %d}`, i, time.Now().Unix())\n    ack, err := stream.IngestRecord(jsonData)\n    if err != nil {\n        log.Fatal(err)\n    }\n    acks = append(acks, ack)\n}\n\n// Wait for server acknowledgments\nfor i, ack := range acks {\n    if _, err := ack.Await(); err != nil {\n        log.Printf(\"Record %d failed: %v\", i, err)\n    }\n}\n```\n\n**For non-blocking concurrent ingestion, use goroutines:**\n\n```go\nvar wg sync.WaitGroup\nerrCh := make(chan error, 100)\n\nfor i := 0; i \u003c 100; i++ {\n    wg.Add(1)\n    go func(id int) {\n        defer wg.Done()\n\n        data := fmt.Sprintf(`{\"id\": %d}`, id)\n        ack, err := stream.IngestRecord(data)  // Blocks this goroutine\n        if err != nil {\n            errCh \u003c- err\n            return\n        }\n\n        offset, err := ack.Await()\n        if err != nil {\n            errCh \u003c- err\n            return\n        }\n        log.Printf(\"Record %d acknowledged at offset %d\", id, offset)\n    }(id)\n}\n\nwg.Wait()\nclose(errCh)\n\n// Check for errors\nfor err := range errCh {\n    log.Printf(\"Ingestion error: %v\", err)\n}\n```\n\n**Concurrent ingestion with multiple streams:**\n\n```go\nvar wg sync.WaitGroup\nfor partition := 0; partition \u003c 4; partition++ {\n    wg.Add(1)\n    go func(p int) {\n        defer wg.Done()\n\n        // Each goroutine gets its own stream\n        stream, err := sdk.CreateStream(tableProps, clientID, clientSecret, options)\n        if err != nil {\n            log.Fatal(err)\n        }\n        defer stream.Close()\n\n        for i := p * 25000; i \u003c (p+1)*25000; i++ {\n            data := fmt.Sprintf(`{\"id\": %d}`, i)\n            // Blocks until queued in this goroutine\n            if _, err := stream.IngestRecord(data); err != nil {\n                log.Fatal(err)\n            }\n            // Note: stream.Close() will flush and await all pending acks\n        }\n    }(p)\n}\nwg.Wait()\n```\n\n### 5. Handle Acknowledgments\n\nAfter `IngestRecord()` returns, the record is queued. Use the returned acknowledgment to wait for server confirmation:\n\n```go\noffset, err := stream.IngestRecord(data)\nif err != nil {\n    // Handle ingestion error\n    if zerobusErr, ok := err.(*zerobus.ZerobusError); ok {\n        if zerobusErr.Retryable() {\n            // Retry logic for transient failures\n        }\n    }\n    log.Fatal(err)\n}\n\n// Offset is available immediately\nlog.Printf(\"Record committed at offset: %d\", offset)\n```\n\n### 6. Close the Stream\n\nAlways close streams to ensure data is flushed:\n\n```go\n// Close gracefully (flushes automatically)\nif err := stream.Close(); err != nil {\n    log.Fatal(err)\n}\n```\n\n## Configuration Options\n\n### StreamConfigurationOptions\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `MaxInflightRequests` | `uint64` | 1,000,000 | Maximum number of in-flight requests |\n| `Recovery` | `bool` | true | Enable automatic stream recovery on failure |\n| `RecoveryTimeoutMs` | `uint64` | 15,000 | Timeout for recovery operations (ms) |\n| `RecoveryBackoffMs` | `uint64` | 2,000 | Delay between recovery retry attempts (ms) |\n| `RecoveryRetries` | `uint32` | 4 | Maximum number of recovery attempts |\n| `FlushTimeoutMs` | `uint64` | 300,000 | Timeout for flush operations (ms) |\n| `ServerLackOfAckTimeoutMs` | `uint64` | 60,000 | Timeout waiting for server acks (ms) |\n| `RecordType` | `int` | Proto | Record type: `RecordTypeProto` or `RecordTypeJson` |\n\n**Example:**\n\n```go\noptions := zerobus.DefaultStreamConfigurationOptions()\noptions.MaxInflightRequests = 50000\noptions.RecoveryRetries = 10\noptions.FlushTimeoutMs = 600000\noptions.RecordType = zerobus.RecordTypeJson\n```\n\n## Error Handling\n\nThe SDK categorizes errors as **retryable** or **non-retryable**:\n\n### Retryable Errors\nAuto-recovered if `Recovery` is enabled:\n- Network failures\n- Connection timeouts\n- Temporary server errors\n- Stream closed by server\n\n### Non-Retryable Errors\nRequire manual intervention:\n- Invalid OAuth credentials\n- Invalid table name\n- Schema mismatch\n- Authentication failure\n- Permission denied\n\n**Check if an error is retryable:**\n\n```go\noffset, err := stream.IngestRecord(data)\nif err != nil {\n    if zerobusErr, ok := err.(*zerobus.ZerobusError); ok {\n        if zerobusErr.Retryable() {\n            log.Printf(\"Retryable error, SDK will auto-recover: %v\", err)\n            // Optionally implement custom retry logic\n        } else {\n            log.Fatalf(\"Fatal error, manual intervention needed: %v\", err)\n        }\n    }\n}\n```\n\n## Examples\n\nThe `examples/` directory contains complete, runnable examples:\n\n- **`examples/basic_example_json/`** - Simple JSON-based ingestion\n- **`examples/basic_example_proto/`** - Type-safe Protocol Buffer ingestion\n\n**To run an example:**\n\n```bash\n# Navigate to an example directory\ncd examples/basic_example_json\n\n# Set your credentials\nexport ZEROBUS_SERVER_ENDPOINT=\"https://your-zerobus-endpoint.databricks.com\"\nexport DATABRICKS_WORKSPACE_URL=\"https://your-workspace.databricks.com\"\nexport DATABRICKS_CLIENT_ID=\"your-client-id\"\nexport DATABRICKS_CLIENT_SECRET=\"your-client-secret\"\nexport ZEROBUS_TABLE_NAME=\"catalog.schema.table\"\n\n# Run the example\ngo run basic_json_usage.go\n```\n\nEach example includes detailed comments and demonstrates best practices for production use.\n\n## Best Practices\n\n1. **Reuse SDK Instances** - Create one `ZerobusSdk` per application and reuse for multiple streams\n2. **Always Close Streams** - Use `defer stream.Close()` to ensure all data is flushed\n3. **Tune Inflight Limits** - Adjust `MaxInflightRequests` based on memory and throughput needs\n4. **Enable Recovery** - Always set `Recovery: true` in production environments\n5. **Use Batch Ingestion** - For high throughput, ingest many records before calling `Flush()`\n6. **Monitor Errors** - Log and alert on non-retryable errors\n7. **Use Protocol Buffers for Production** - More efficient than JSON for high-volume scenarios\n8. **Secure Credentials** - Never hardcode secrets; use environment variables or secret managers\n9. **Test Recovery** - Simulate failures to verify your error handling logic\n10. **One Stream Per Goroutine** - Don't share streams across goroutines; create separate streams for concurrent ingestion\n\n## API Reference\n\n### `ZerobusSdk`\n\nMain entry point for the SDK.\n\n#### `NewZerobusSdk(zerobusEndpoint, unityCatalogURL string) (*ZerobusSdk, error)`\n\nCreates a new SDK instance.\n\n- `zerobusEndpoint`: Zerobus gRPC service endpoint\n- `unityCatalogURL`: Unity Catalog URL for OAuth token acquisition\n\n#### `CreateStream(tableProps TableProperties, clientID, clientSecret string, options *StreamConfigurationOptions) (*ZerobusStream, error)`\n\nCreates a new ingestion stream with OAuth authentication.\n\n#### `CreateStreamWithHeadersProvider(tableProps TableProperties, headersProvider HeadersProvider, options *StreamConfigurationOptions) (*ZerobusStream, error)`\n\nCreates a new ingestion stream with a custom headers provider for advanced authentication. Use this when you need custom authentication logic (e.g., custom token caching, or alternative auth providers).\n\n**Example:**\n```go\nprovider := \u0026MyCustomAuthProvider{}\nstream, err := sdk.CreateStreamWithHeadersProvider(\n    tableProps,\n    provider,\n    options,\n)\n```\n\n#### `Free()`\n\nExplicitly releases SDK resources. Called automatically by finalizer.\n\n### `ZerobusStream`\n\nRepresents an active bidirectional streaming connection.\n\n#### `IngestRecord(payload interface{}) (*RecordAck, error)`\n\nIngests a record into the stream. **Blocks until the record is queued** (handles backpressure), then returns an acknowledgment handle for awaiting server confirmation. \n\nAccepts either:\n- `string` for JSON-encoded records\n- `[]byte` for Protocol Buffer-encoded records\n\nReturns a `*RecordAck` that can be awaited to get the logical offset assigned to the record.\n\n**Example:**\n```go\nack, err := stream.IngestRecord(`{\"id\": 1}`)\nif err != nil {\n    // Handle queueing errors\n}\n\n// Wait for server acknowledgment\noffset, err := ack.Await()\nif err != nil {\n    // Handle acknowledgment errors\n}\n```\n\n**For non-blocking ingestion:**\n```go\ngo func() {\n    ack, _ := stream.IngestRecord(data)  // This goroutine blocks\n    offset, _ := ack.Await()\n    // Handle offset\n}()\n```\n\n#### `Flush() error`\n\nBlocks until all pending records are acknowledged by the server.\n\n#### `Close() error`\n\nGracefully closes the stream after flushing pending records.\n\n### `RecordAck`\n\nRepresents a pending acknowledgment for an ingested record.\n\n#### `Await() (int64, error)`\n\nBlocks until the record is acknowledged by the server and returns the offset.\nCan only be called once - subsequent calls return the cached result.\n\n**Example:**\n```go\nack, _ := stream.IngestRecord(data)\noffset, err := ack.Await()\nif err != nil {\n    log.Printf(\"Record failed: %v\", err)\n}\n```\n\n#### `TryGet() (int64, error, bool)`\n\nNon-blocking check for acknowledgment status.\n\nReturns:\n- `(offset, nil, true)` if acknowledgment is ready\n- `(0, nil, false)` if still pending\n- `(0, error, true)` if an error occurred\n\n**Example:**\n```go\nack, _ := stream.IngestRecord(data)\n// Do other work...\nif offset, err, ready := ack.TryGet(); ready {\n    if err != nil {\n        log.Printf(\"Record failed: %v\", err)\n    } else {\n        log.Printf(\"Record acknowledged at offset %d\", offset)\n    }\n} else {\n    log.Println(\"Still waiting for acknowledgment\")\n}\n```\n\n### `HeadersProvider`\n\nInterface for providing custom authentication headers.\n\n```go\ntype HeadersProvider interface {\n    // GetHeaders returns authentication headers.\n    // Called by the SDK when authentication is needed.\n    GetHeaders() (map[string]string, error)\n}\n```\n\n**Example implementation:**\n```go\ntype CustomProvider struct{}\n\nfunc (p *CustomProvider) GetHeaders() (map[string]string, error) {\n    return map[string]string{\n        \"authorization\": \"Bearer token\",\n        \"x-databricks-zerobus-table-name\": \"catalog.schema.table\",\n    }, nil\n}\n```\n\n### `TableProperties`\n\n```go\ntype TableProperties struct {\n    TableName       string\n    DescriptorProto []byte\n}\n```\n\n### `StreamConfigurationOptions`\n\nSee [Configuration Options](#configuration-options) for details.\n\n### `ZerobusError`\n\n```go\ntype ZerobusError struct {\n    Message     string\n    IsRetryable bool\n}\n```\n\n#### `Error() string`\n\nReturns the error message.\n\n#### `Retryable() bool`\n\nReturns `true` if the error can be automatically recovered by the SDK.\n\n## Building from Source\n\nThis section is for contributors and those who need to build the SDK from source. If you just want to use the SDK, see [Getting Started](#getting-started) instead.\n\n### Basic Build\n\n```bash\ngit clone https://github.com/databricks/zerobus-sdk-go.git\ncd zerobus-sdk-go\nmake build\n```\n\n### Build Specific Components\n\n```bash\n# Build only Rust FFI\nmake build-rust\n\n# Build only Go SDK\nmake build-go\n\n# Build examples\nmake examples\n\n# Run tests\nmake test\n\n# Format code\nmake fmt\n\n# Run linters\nmake lint\n```\n\n### Platform-Specific Build Notes\n\n#### Windows\nThe build system automatically compiles Rust with the GNU toolchain (`x86_64-pc-windows-gnu`) for compatibility with Go's MinGW-based CGO. This ensures the MSVC and GNU ABIs don't conflict.\n#### Linux\nStandard GCC toolchain is used. Install build tools with:\n```bash\nsudo apt-get install build-essential  # Ubuntu/Debian\nsudo dnf install gcc                   # Fedora/RHEL\n```\n\n#### macOS\nUses the system Clang toolchain. Install with:\n```bash\nxcode-select --install\n```\n\n## Troubleshooting\n\n### Installing Rust\n\nIf you need to install Rust, use the official installer:\n\n```bash\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\n\nAfter installation, restart your terminal or run:\n```bash\nsource $HOME/.cargo/env\n```\n\nVerify installation:\n```bash\nrustc --version\ncargo --version\n```\n\n### Common Build Issues\n\n**\"cargo not found\"**\n- Ensure Rust is installed and `~/.cargo/bin` is in your PATH\n- Restart your terminal after installing Rust\n\n**\"linker 'cc' not found\"** (Linux)\n- Install build tools: `sudo apt-get install build-essential`\n\n**\"xcrun: error: unable to find utility 'cc'\"** (macOS)\n- Install Xcode tools: `xcode-select --install`\n\n**\"undefined reference to __chkstk\"** (Windows)\n- Ensure you're using MinGW-w64, not just MinGW\n- The SDK automatically builds for the GNU target\n\n## Community and Contributing\n\nThis is an open source project. We welcome contributions, feedback, and bug reports.\n\n- **[Contributing Guide](CONTRIBUTING.md)**: Learn how to contribute, including our development process and coding style\n- **[Changelog](CHANGELOG.md)**: See the history of changes in the SDK\n- **[Security Policy](SECURITY.md)**: Read about our security process and how to report vulnerabilities\n- **[Developer Certificate of Origin (DCO)](DCO)**: Understand the agreement for contributions\n- **[Open Source Attributions](NOTICE)**: See a list of the open source libraries we use\n\n## License\n\nThis SDK is licensed under the Databricks License. See the [LICENSE](LICENSE) file for the full license text. The license is also available online at [https://www.databricks.com/legal/db-license](https://www.databricks.com/legal/db-license).\n\n---\n\nFor issues, questions, or contributions, please visit the [GitHub repository](https://github.com/databricks/zerobus-sdk-go).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fzerobus-sdk-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk-go/lists"}