{"id":46453965,"url":"https://github.com/databricks/zerobus-sdk","last_synced_at":"2026-04-23T17:00:46.886Z","repository":{"id":320190292,"uuid":"1049763140","full_name":"databricks/zerobus-sdk","owner":"databricks","description":"Databricks's Zerobus Ingest SDKs","archived":false,"fork":false,"pushed_at":"2026-04-21T05:06:46.000Z","size":97992,"stargazers_count":73,"open_issues_count":72,"forks_count":12,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-04-21T06:42:01.081Z","etag":null,"topics":["databricks","databricks-sdk","ffi","go","java","jni","napi-rs","pyo3","python","rust","typescript","zerobus","zerobus-sdk"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":"DCO","cla":null}},"created_at":"2025-09-03T13:11:21.000Z","updated_at":"2026-04-20T21:04:50.000Z","dependencies_parsed_at":"2025-11-29T07:03:26.392Z","dependency_job_id":null,"html_url":"https://github.com/databricks/zerobus-sdk","commit_stats":null,"previous_names":["databricks/zerobus-sdk-rs","databricks/zerobus-sdk"],"tags_count":38,"template":false,"template_full_name":null,"purl":"pkg:github/databricks/zerobus-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/zerobus-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32189658,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-23T15:28:30.493Z","status":"ssl_error","status_checked_at":"2026-04-23T15:28:29.972Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks","databricks-sdk","ffi","go","java","jni","napi-rs","pyo3","python","rust","typescript","zerobus","zerobus-sdk"],"created_at":"2026-03-06T01:10:55.150Z","updated_at":"2026-04-23T17:00:46.853Z","avatar_url":"https://github.com/databricks.png","language":"Rust","readme":"# Zerobus SDKs\n\nMonorepo for Databricks Zerobus Ingest SDKs.\n\n## Disclaimer\n\n[GA](https://docs.databricks.com/release-notes/release-types.html): This SDK is generally available and supported for production use cases. Minor and patch version updates will not contain breaking changes. Major version updates may include breaking changes.\n\nWe are keen to hear feedback from you. Please [file issues](https://github.com/databricks/zerobus-sdk/issues), and we will address them.\n\n## What is Zerobus?\n\nZerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads.\n\n## SDKs\n\n| Language | Directory | Package |\n|----------|-----------|---------|\n| Rust | [`rust/`](rust/) | [`databricks-zerobus-ingest-sdk`](https://crates.io/crates/databricks-zerobus-ingest-sdk) |\n| Python | [`python/`](python/) | [`databricks-zerobus-ingest-sdk`](https://pypi.org/project/databricks-zerobus-ingest-sdk/) |\n| Go | [`go/`](go/) | [`github.com/databricks/zerobus-sdk/go`](https://pkg.go.dev/github.com/databricks/zerobus-sdk/go) |\n| TypeScript | [`typescript/`](typescript/) | [`@databricks/zerobus-ingest-sdk`](https://www.npmjs.com/package/@databricks/zerobus-ingest-sdk) |\n| Java | [`java/`](java/) | [`com.databricks:zerobus-ingest-sdk`](https://central.sonatype.com/artifact/com.databricks/zerobus-ingest-sdk) |\n\n## Platform Support\n\nWe try to provide prebuilt native binaries for the following platforms:\n\n| Platform | Architecture |\n|----------|-------------|\n| Linux | x86_64 |\n| Linux | aarch64 |\n| Windows | x86_64 |\n| macOS | x86_64 |\n| macOS | aarch64 (Apple Silicon) |\n\n\u003e **Note:** We do not currently have macOS CI runners, so macOS binaries are built locally and may not be available for every SDK or release. If your platform is not supported or you encounter compatibility issues, you can [build from source](CONTRIBUTING.md) or [file an issue](https://github.com/databricks/zerobus-sdk/issues).\n\n## Prerequisites\n\nBefore using any SDK, you need the following:\n\n### 1. Workspace URL and Workspace ID\n\nAfter logging into your Databricks workspace, look at the browser URL:\n\n```\nhttps://\u003cdatabricks-instance\u003e.cloud.databricks.com/o=\u003cworkspace-id\u003e\n```\n\n- **Workspace URL**: The part before `/o=` (e.g., `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com`)\n- **Workspace ID**: The part after `/o=` (e.g., `1234567890123456`)\n\n\u003e **Note:** The examples above show AWS endpoints (`.cloud.databricks.com`). For Azure deployments, the workspace URL will be `https://\u003cdatabricks-instance\u003e.azuredatabricks.net`.\n\n### 2. Create a Delta Table\n\nCreate a table using Databricks SQL:\n\n```sql\nCREATE TABLE \u003ccatalog_name\u003e.default.\u003ctable_name\u003e (\n    device_name STRING,\n    temp INT,\n    humidity BIGINT\n)\nUSING DELTA;\n```\n\nReplace `\u003ccatalog_name\u003e` with your catalog name (e.g., `main`).\n\n### 3. Create a Service Principal\n\n1. Navigate to **Settings \u003e Identity and Access** in your Databricks workspace\n2. Click **Service principals** and create a new service principal\n3. Generate a new secret for the service principal and save it securely\n4. Grant the following permissions:\n   - `USE_CATALOG` on the catalog (e.g., `main`)\n   - `USE_SCHEMA` on the schema (e.g., `default`)\n   - `MODIFY` and `SELECT` on the table\n\nGrant permissions using SQL:\n\n```sql\n-- Grant catalog permission\nGRANT USE CATALOG ON CATALOG \u003ccatalog_name\u003e TO `\u003cservice-principal-application-id\u003e`;\n\n-- Grant schema permission\nGRANT USE SCHEMA ON SCHEMA \u003ccatalog_name\u003e.default TO `\u003cservice-principal-application-id\u003e`;\n\n-- Grant table permissions\nGRANT SELECT, MODIFY ON TABLE \u003ccatalog_name\u003e.default.\u003ctable_name\u003e TO `\u003cservice-principal-application-id\u003e`;\n```\n\nThe service principal's **Application ID** is your OAuth **Client ID**, and the generated secret is your **Client Secret**.\n\n## Serialization Formats\n\nAll SDKs support two serialization formats:\n\n- **JSON** - Simple, schema-free ingestion. Pass a JSON string or native object (dict, map, etc.) and the SDK serializes it. No compilation step required. Good for getting started or dynamic schemas.\n- **Protocol Buffers** - Strongly-typed, schema-validated ingestion. More efficient over the wire. Recommended for production workloads.\n\n### Protocol Buffers\n\nUse `proto2` syntax with `optional` fields to correctly represent nullable Delta table columns.\n\n#### Delta → Protobuf Type Mappings\n\n| Delta Type | Proto2 Type |\n|-----------|-------------|\n| TINYINT, BYTE, INT, SMALLINT, SHORT | int32 |\n| BIGINT, LONG | int64 |\n| FLOAT | float |\n| DOUBLE | double |\n| STRING, VARCHAR | string |\n| BOOLEAN | bool |\n| BINARY | bytes |\n| DATE | int32 |\n| TIMESTAMP, TIMESTAMP_NTZ | int64 |\n| ARRAY\\\u003ctype\\\u003e | repeated type |\n| MAP\\\u003ckey, value\\\u003e | map\\\u003ckey, value\\\u003e |\n| STRUCT\\\u003cfields\\\u003e | nested message |\n| VARIANT | string (JSON string) |\n\n### Schema Generation\n\nInstead of writing `.proto` files by hand, each SDK ships a tool to generate protobuf schemas directly from an existing Unity Catalog table. See the individual SDK READMEs for language-specific usage.\n\n## HTTP Proxy Support\n\nAll SDKs support HTTP CONNECT proxies via environment variables, following gRPC core conventions. The first variable found (in order) is used:\n\n| Proxy | No-proxy |\n|-------|----------|\n| `grpc_proxy` / `GRPC_PROXY` | `no_grpc_proxy` / `NO_GRPC_PROXY` |\n| `https_proxy` / `HTTPS_PROXY` | `no_proxy` / `NO_PROXY` |\n| `http_proxy` / `HTTP_PROXY` | |\n\nThe `no_proxy` value is a comma-separated list of hostnames (suffix-matched) or `*` to bypass the proxy entirely.\n\n```bash\nexport https_proxy=http://my-proxy:8080\nexport no_proxy=localhost,127.0.0.1\n```\n\nThe SDK establishes a plaintext HTTP CONNECT tunnel through the proxy, then performs a TLS handshake end-to-end with the Databricks server. The proxy never sees decrypted traffic.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md). Each SDK also has its own contributing guide with language-specific setup instructions.\n\n## License\n\nThis project is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for the full text.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fzerobus-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk/lists"}