{"id":30687618,"url":"https://github.com/query-farm/tributary","last_synced_at":"2025-09-02T00:04:35.249Z","repository":{"id":298631460,"uuid":"998167896","full_name":"Query-farm/tributary","owner":"Query-farm","description":"A DuckDB Extension for Kafka","archived":false,"fork":false,"pushed_at":"2025-06-12T03:35:06.000Z","size":24,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-12T04:31:07.025Z","etag":null,"topics":["apache-kafka","apache-kafka-consumer","apache-kafka-producer","duckdb","duckdb-extension","kafka","streaming"],"latest_commit_sha":null,"homepage":"https://query.farm/duckdb_extension_tributary.html","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Query-farm.png","metadata":{"files":{"readme":"docs/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-08T02:28:11.000Z","updated_at":"2025-06-12T03:50:13.000Z","dependencies_parsed_at":"2025-06-12T04:31:27.288Z","dependency_job_id":"93028e90-03c0-49e4-9e78-14de050c2ba1","html_url":"https://github.com/Query-farm/tributary","commit_stats":null,"previous_names":["query-farm/tributary"],"tags_count":0,"template":false,"template_full_name":"duckdb/extension-template","purl":"pkg:github/Query-farm/tributary","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Ftributary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Ftributary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Ftributary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Ftributary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Query-farm","download_url":"https://codeload.github.com/Query-farm/tributary/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Query-farm%2Ftributary/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273208777,"owners_count":25064204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","apache-kafka-consumer","apache-kafka-producer","duckdb","duckdb-extension","kafka","streaming"],"created_at":"2025-09-02T00:03:34.501Z","updated_at":"2025-09-02T00:04:35.237Z","avatar_url":"https://github.com/Query-farm.png","language":"C++","readme":"# DuckDB Tributary Extension\n\nThe **Tributary** extension provides seamless integration between DuckDB and [Apache Kafka](https://kafka.apache.org/), enabling real-time querying and analysis of streaming data. With this extension, users can consume messages directly from Kafka topics into DuckDB for immediate processing, as well as write processed data back to Kafka streams.\n\n## Key Features\n\n- **Direct Kafka Ingestion:** Stream records from Kafka topics directly into DuckDB tables using SQL.\n- **Flexible Topic Consumption:** Supports consuming from specific partitions, offsets, or continuously from the latest messages.\n- **Real-Time Analytics:** Perform analytical queries on streaming data as it arrives.\n- **Kafka Output:** Optionally write results or processed data back to Kafka topics.\n- **SQL-Native Interface:** Kafka integration is fully accessible via SQL, enabling easy adoption for data engineers and analysts.\n\n## Example Usage\n```sql\n\n-- Scan an entire Kafka topic.\nSELECT *\nFROM tributary_scan_topic('test-topic',\n  \"bootstrap.servers\" := 'localhost:9092'\n);\n```\n\n# Documentation\n\nSee the [extension documentation](https://query.farm/duckdb_extension_tributary.html).\n\n## Building\n### Managing dependencies\nDuckDB extensions uses VCPKG for dependency management. Enabling VCPKG is very simple: follow the [installation instructions](https://vcpkg.io/en/getting-started) or just run the following:\n```shell\ncd \u003cyour-working-dir-not-the-plugin-repo\u003e\ngit clone https://github.com/Microsoft/vcpkg.git\nsh ./vcpkg/scripts/bootstrap.sh -disableMetrics\nexport VCPKG_TOOLCHAIN_PATH=`pwd`/vcpkg/scripts/buildsystems/vcpkg.cmake\n```\nNote: VCPKG is only required for extensions that want to rely on it for dependency management. If you want to develop an extension without dependencies, or want to do your own dependency management, just skip this step. Note that the example extension uses VCPKG to build with a dependency for instructive purposes, so when skipping this step the build may not work without removing the dependency.\n\n### Build steps\nNow to build the extension, run:\n```sh\nmake\n```\nThe main binaries that will be built are:\n```sh\n./build/release/duckdb\n./build/release/test/unittest\n./build/release/extension/\u003cextension_name\u003e/\u003cextension_name\u003e.duckdb_extension\n```\n- `duckdb` is the binary for the duckdb shell with the extension code automatically loaded.\n- `unittest` is the test runner of duckdb. Again, the extension is already linked into the binary.\n- `\u003cextension_name\u003e.duckdb_extension` is the loadable binary as it would be distributed.\n\n### Tips for speedy builds\nDuckDB extensions currently rely on DuckDB's build system to provide easy testing and distributing. This does however come at the downside of requiring the template to build DuckDB and its unittest binary every time you build your extension. To mitigate this, we highly recommend installing [ccache](https://ccache.dev/) and [ninja](https://ninja-build.org/). This will ensure you only need to build core DuckDB once and allows for rapid rebuilds.\n\nTo build using ninja and ccache ensure both are installed and run:\n\n```sh\nGEN=ninja make\n```\n\n## Running the extension\nTo run the extension code, simply start the shell with `./build/release/duckdb`. This shell will have the extension pre-loaded.\n\n```\n\n## Running the tests\nDifferent tests can be created for DuckDB extensions. The primary way of testing DuckDB extensions should be the SQL tests in `./test/sql`. These SQL tests can be run using:\n```sh\nmake test\n```\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquery-farm%2Ftributary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquery-farm%2Ftributary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquery-farm%2Ftributary/lists"}