{"id":46453965,"url":"https://github.com/databricks/zerobus-sdk","last_synced_at":"2026-03-06T01:10:55.661Z","repository":{"id":320190292,"uuid":"1049763140","full_name":"databricks/zerobus-sdk","owner":"databricks","description":"Databricks's Zerobus Ingest SDKs","archived":false,"fork":false,"pushed_at":"2026-03-02T14:51:53.000Z","size":42481,"stargazers_count":24,"open_issues_count":17,"forks_count":6,"subscribers_count":4,"default_branch":"main","last_synced_at":"2026-03-02T14:53:12.888Z","etag":null,"topics":["databricks","databricks-sdk","rust","zerobus","zerobus-sdk"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":"DCO","cla":null}},"created_at":"2025-09-03T13:11:21.000Z","updated_at":"2026-03-01T21:09:18.000Z","dependencies_parsed_at":"2025-11-29T07:03:26.392Z","dependency_job_id":null,"html_url":"https://github.com/databricks/zerobus-sdk","commit_stats":null,"previous_names":["databricks/zerobus-sdk-rs","databricks/zerobus-sdk"],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/databricks/zerobus-sdk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/zerobus-sdk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fzerobus-sdk/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30157067,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T22:39:40.138Z","status":"ssl_error","status_checked_at":"2026-03-05T22:39:24.771Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["databricks","databricks-sdk","rust","zerobus","zerobus-sdk"],"created_at":"2026-03-06T01:10:55.150Z","updated_at":"2026-03-06T01:10:55.646Z","avatar_url":"https://github.com/databricks.png","language":"Rust","readme":"# Zerobus SDKs\n\nMonorepo for Databricks Zerobus Ingest SDKs.\n\n## Disclaimer\n\n[Public Preview](https://docs.databricks.com/release-notes/release-types.html): These SDKs are supported for production use cases and are available to all customers. Databricks is actively working on stabilizing the Zerobus Ingest SDKs. Minor version updates may include backwards-incompatible changes.\n\nWe are keen to hear feedback from you. Please [file issues](https://github.com/databricks/zerobus-sdk/issues), and we will address them.\n\n## What is Zerobus?\n\nZerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads.\n\n## SDKs\n\n| Language | Directory | Package |\n|----------|-----------|---------|\n| Rust | [`rust/`](rust/) | [`databricks-zerobus-ingest-sdk`](https://crates.io/crates/databricks-zerobus-ingest-sdk) |\n| Python | [`python/`](python/) | [`databricks-zerobus-ingest-sdk`](https://pypi.org/project/databricks-zerobus-ingest-sdk/) |\n| Go | `go/` | *coming soon* |\n| TypeScript | [`typescript/`](typescript/) | [`@databricks/zerobus-ingest-sdk`](https://www.npmjs.com/package/@databricks/zerobus-ingest-sdk) |\n| Java | [`java/`](java/) | [`com.databricks:zerobus-ingest-sdk`](https://central.sonatype.com/artifact/com.databricks/zerobus-ingest-sdk) |\n\n## Prerequisites\n\nBefore using any SDK, you need the following:\n\n### 1. Workspace URL and Workspace ID\n\nAfter logging into your Databricks workspace, look at the browser URL:\n\n```\nhttps://\u003cdatabricks-instance\u003e.cloud.databricks.com/o=\u003cworkspace-id\u003e\n```\n\n- **Workspace URL**: The part before `/o=` (e.g., `https://dbc-a1b2c3d4-e5f6.cloud.databricks.com`)\n- **Workspace ID**: The part after `/o=` (e.g., `1234567890123456`)\n\n\u003e **Note:** The examples above show AWS endpoints (`.cloud.databricks.com`). For Azure deployments, the workspace URL will be `https://\u003cdatabricks-instance\u003e.azuredatabricks.net`.\n\n### 2. Create a Delta Table\n\nCreate a table using Databricks SQL:\n\n```sql\nCREATE TABLE \u003ccatalog_name\u003e.default.\u003ctable_name\u003e (\n    device_name STRING,\n    temp INT,\n    humidity BIGINT\n)\nUSING DELTA;\n```\n\nReplace `\u003ccatalog_name\u003e` with your catalog name (e.g., `main`).\n\n### 3. Create a Service Principal\n\n1. Navigate to **Settings \u003e Identity and Access** in your Databricks workspace\n2. Click **Service principals** and create a new service principal\n3. Generate a new secret for the service principal and save it securely\n4. Grant the following permissions:\n   - `USE_CATALOG` on the catalog (e.g., `main`)\n   - `USE_SCHEMA` on the schema (e.g., `default`)\n   - `MODIFY` and `SELECT` on the table\n\nGrant permissions using SQL:\n\n```sql\n-- Grant catalog permission\nGRANT USE CATALOG ON CATALOG \u003ccatalog_name\u003e TO `\u003cservice-principal-application-id\u003e`;\n\n-- Grant schema permission\nGRANT USE SCHEMA ON SCHEMA \u003ccatalog_name\u003e.default TO `\u003cservice-principal-application-id\u003e`;\n\n-- Grant table permissions\nGRANT SELECT, MODIFY ON TABLE \u003ccatalog_name\u003e.default.\u003ctable_name\u003e TO `\u003cservice-principal-application-id\u003e`;\n```\n\nThe service principal's **Application ID** is your OAuth **Client ID**, and the generated secret is your **Client Secret**.\n\n## Serialization Formats\n\nAll SDKs support two serialization formats:\n\n- **JSON** - Simple, schema-free ingestion. Pass a JSON string or native object (dict, map, etc.) and the SDK serializes it. No compilation step required. Good for getting started or dynamic schemas.\n- **Protocol Buffers** - Strongly-typed, schema-validated ingestion. More efficient over the wire. Recommended for production workloads.\n\n### Protocol Buffers\n\nUse `proto2` syntax with `optional` fields to correctly represent nullable Delta table columns.\n\n#### Delta → Protobuf Type Mappings\n\n| Delta Type | Proto2 Type |\n|-----------|-------------|\n| TINYINT, BYTE, INT, SMALLINT, SHORT | int32 |\n| BIGINT, LONG | int64 |\n| FLOAT | float |\n| DOUBLE | double |\n| STRING, VARCHAR | string |\n| BOOLEAN | bool |\n| BINARY | bytes |\n| DATE | int32 |\n| TIMESTAMP, TIMESTAMP_NTZ | int64 |\n| ARRAY\\\u003ctype\\\u003e | repeated type |\n| MAP\\\u003ckey, value\\\u003e | map\\\u003ckey, value\\\u003e |\n| STRUCT\\\u003cfields\\\u003e | nested message |\n| VARIANT | string (JSON string) |\n\n### Schema Generation\n\nInstead of writing `.proto` files by hand, each SDK ships a tool to generate protobuf schemas directly from an existing Unity Catalog table. See the individual SDK READMEs for language-specific usage.\n\n## HTTP Proxy Support\n\nAll SDKs support HTTP CONNECT proxies via environment variables, following gRPC core conventions. The first variable found (in order) is used:\n\n| Proxy | No-proxy |\n|-------|----------|\n| `grpc_proxy` / `GRPC_PROXY` | `no_grpc_proxy` / `NO_GRPC_PROXY` |\n| `https_proxy` / `HTTPS_PROXY` | `no_proxy` / `NO_PROXY` |\n| `http_proxy` / `HTTP_PROXY` | |\n\nThe `no_proxy` value is a comma-separated list of hostnames (suffix-matched) or `*` to bypass the proxy entirely.\n\n```bash\nexport https_proxy=http://my-proxy:8080\nexport no_proxy=localhost,127.0.0.1\n```\n\nThe SDK establishes a plaintext HTTP CONNECT tunnel through the proxy, then performs a TLS handshake end-to-end with the Databricks server. The proxy never sees decrypted traffic.\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md). Each SDK also has its own contributing guide with language-specific setup instructions.\n\n## License\n\nThis project is licensed under the Databricks License. See [LICENSE](LICENSE) for the full text.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fzerobus-sdk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fzerobus-sdk/lists"}