{"id":15024853,"url":"https://github.com/tansu-io/tansu","last_synced_at":"2026-01-16T02:37:16.712Z","repository":{"id":254148032,"uuid":"845594676","full_name":"tansu-io/tansu","owner":"tansu-io","description":"Apache Kafka® compatible broker with S3 or PostgreSQL storage","archived":false,"fork":false,"pushed_at":"2025-04-02T17:02:45.000Z","size":1941,"stargazers_count":323,"open_issues_count":26,"forks_count":6,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-03T22:06:54.489Z","etag":null,"topics":["built-with-rust","kafka","postgres","postgresql","s3"],"latest_commit_sha":null,"homepage":"https://tansu.io","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tansu-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-21T14:47:15.000Z","updated_at":"2025-04-03T08:22:53.000Z","dependencies_parsed_at":"2024-09-13T04:52:30.421Z","dependency_job_id":"4df393e1-7124-4ea4-a358-e6aa5ee53743","html_url":"https://github.com/tansu-io/tansu","commit_stats":{"total_commits":155,"total_committers":2,"mean_commits":77.5,"dds":0.3354838709677419,"last_synced_commit":"88b7a2ec156f1c14f057e24faa6f0035ed62e5ae"},"previous_names":["tansu-io/tansu"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tansu-io%2Ftansu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tansu-io%2Ftansu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tansu-io%2Ftansu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tansu-io%2Ftansu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tansu-io","download_url":"https://codeload.github.com/tansu-io/tansu/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248590751,"owners_count":21129885,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["built-with-rust","kafka","postgres","postgresql","s3"],"created_at":"2024-09-24T20:01:04.219Z","updated_at":"2026-01-16T02:37:16.698Z","avatar_url":"https://github.com/tansu-io.png","language":"Rust","funding_links":[],"categories":["Rust","\u003ca name=\"Rust\"\u003e\u003c/a\u003eRust","Kafka-Compatible Alternatives"],"sub_categories":["Other Notable KIPs"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# Tansu 🗃️\nstateless Kafka-compatible broker with pluggable storage (PostgreSQL, SQLite, S3, memory)\n\n\u003cbr\u003e\n\n[![License](https://img.shields.io/badge/License-Apache-165dfc.svg)](https://github.com/tansu-io/tansu/blob/main/LICENSE)\n\u0026nbsp;\n[![Built with Rust](https://img.shields.io/badge/built_with-Rust-165dfc.svg?logo=rust)](https://www.rust-lang.org/)\n\u0026nbsp;\n\u003cbr\u003e\n[![Docs](https://img.shields.io/badge/📖%20docs-docs.tansu.io-165dfc.svg)](https://docs.tansu.io/)\n\u0026nbsp;\n[![Blog](https://img.shields.io/badge/%F0%9F%93%98%20blog-blog.tansu.io-165dfc.svg)](https://blog.tansu.io/articles)\n\u0026nbsp;\n\u003cbr\u003e\n[![GitHub stars](https://img.shields.io/github/stars/tansu-io/tansu?style=social)](https://github.com/tansu-io/tansu)\n\u0026nbsp;\n[![Bluesky](https://img.shields.io/bluesky/followers/tansu.io)](https://bsky.app/profile/tansu.io)\n\n\u003cbr\u003e\n\n\u003c/div\u003e\n\n# What is Tansu?\n\n[Tansu][github-com-tansu-io] is a drop-in replacement for\nApache Kafka with PostgreSQL, libSQL (SQLite), S3 or memory storage engines.\nSchema backed topics (Avro, JSON or Protocol buffers) can\nbe written as [Apache Iceberg](https://iceberg.apache.org) or [Delta Lake](https://delta.io) tables.\n\nFeatures:\n\n- Apache Kafka API compatible\n- Available with [PostgreSQL](https://www.postgresql.org), [libSQL](https://docs.turso.tech/libsql), [S3](https://en.wikipedia.org/wiki/Amazon_S3) or memory storage engines\n- Topics [validated](docs/schema-registry.md) by [JSON Schema][json-schema-org], [Apache Avro](https://avro.apache.org)\n  or [Protocol buffers](protocol-buffers) can be written as [Apache Iceberg](https://iceberg.apache.org) or [Delta Lake](https://delta.io) tables\n\nSee [examples using pyiceberg](https://github.com/tansu-io/example-pyiceberg), [examples using Apache Spark](https://github.com/tansu-io/example-spark) or 🆕 [examples using Delta Lake](https://github.com/tansu-io/example-delta-lake).\n\nFor data durability:\n\n- S3 is designed to exceed [99.999999999% (11 nines)][aws-s3-storage-classes]\n- PostgreSQL with [continuous archiving][continuous-archiving]\n  streaming transaction logs files to an archive\n- The memory storage engine is designed for ephemeral non-production environments\n\nTansu is a single statically linked binary containing the following:\n\n- **broker** an Apache Kafka API compatible broker and schema registry\n- **topic** a CLI to create/delete Topics\n- **cat** a CLI to consume or produce Avro, JSON or Protobuf messages to a topic\n- **proxy** an Apache Kafka compatible proxy\n\n## broker\n\nThe broker subcommand is default if no other command is supplied.\n\n```shell\nUsage: tansu [OPTIONS]\n       tansu \u003cCOMMAND\u003e\n\nCommands:\n  broker  Apache Kafka compatible broker with Avro, JSON, Protobuf schema validation [default if no command supplied]\n  cat     Easily consume or produce Avro, JSON or Protobuf messages to a topic\n  topic   Create or delete topics managed by the broker\n  proxy   Apache Kafka compatible proxy\n  help    Print this message or the help of the given subcommand(s)\n\nOptions:\n      --kafka-cluster-id \u003cKAFKA_CLUSTER_ID\u003e\n          All members of the same cluster should use the same id [env: CLUSTER_ID=RvQwrYegSUCkIPkaiAZQlQ] [default: tansu_cluster]\n      --kafka-listener-url \u003cKAFKA_LISTENER_URL\u003e\n          The broker will listen on this address [env: LISTENER_URL=] [default: tcp://[::]:9092]\n      --kafka-advertised-listener-url \u003cKAFKA_ADVERTISED_LISTENER_URL\u003e\n          This location is advertised to clients in metadata [env: ADVERTISED_LISTENER_URL=tcp://localhost:9092] [default: tcp://localhost:9092]\n      --storage-engine \u003cSTORAGE_ENGINE\u003e\n          Storage engine examples are: postgres://postgres:postgres@localhost, memory://tansu/ or s3://tansu/ [env: STORAGE_ENGINE=s3://tansu/] [default: memory://tansu/]\n      --schema-registry \u003cSCHEMA_REGISTRY\u003e\n          Schema registry examples are: file://./etc/schema or s3://tansu/, containing: topic.json, topic.proto or topic.avsc [env: SCHEMA_REGISTRY=file://./etc/schema]\n      --data-lake \u003cDATA_LAKE\u003e\n          Apache Parquet files are written to this location, examples are: file://./lake or s3://lake/ [env: DATA_LAKE=s3://lake/]\n      --iceberg-catalog \u003cICEBERG_CATALOG\u003e\n          Apache Iceberg Catalog, examples are: http://localhost:8181/ [env: ICEBERG_CATALOG=http://localhost:8181/]\n      --iceberg-namespace \u003cICEBERG_NAMESPACE\u003e\n          Iceberg namespace [env: ICEBERG_NAMESPACE=] [default: tansu]\n      --prometheus-listener-url \u003cPROMETHEUS_LISTENER_URL\u003e\n          Broker metrics can be scraped by Prometheus from this URL [env: PROMETHEUS_LISTENER_URL=tcp://0.0.0.0:9100] [default: tcp://[::]:9100]\n  -h, --help\n          Print help\n  -V, --version\n          Print version\n```\n\nA broker can be started by simply running `tansu`, all options have defaults. Tansu pickup any existing environment,\nloading any found in `.env`. An [example.env](example.env) is provided as part of the distribution\nand can be copied into `.env` for local modification. Sample schemas can be found in [etc/schema](etc/schema), used in the examples.\n\nIf an Apache Avro, Protobuf or JSON schema has been assigned to a topic, the\nbroker will reject any messages that are invalid. Schema backed topics are written\nas Apache Parquet when the `-data-lake` option is provided.\n\n## topic\n\nThe `tansu topic` command has the following subcommands:\n\n```shell\nCreate or delete topics managed by the broker\n\nUsage: tansu topic \u003cCOMMAND\u003e\n\nCommands:\n  create  Create a topic\n  delete  Delete an existing topic\n  help    Print this message or the help of the given subcommand(s)\n\nOptions:\n  -h, --help  Print help\n```\n\nTo create a topic use:\n\n```shell\ntansu topic create taxi\n```\n\n## cat\n\nThe `tansu cat` command, has the following subcommands:\n\n```shell\ntansu cat --help\nEasily consume or produce Avro, JSON or Protobuf messages to a topic\n\nUsage: tansu cat \u003cCOMMAND\u003e\n\nCommands:\n  produce  Produce Avro/JSON/Protobuf messages to a topic\n  consume  Consume Avro/JSON/Protobuf messages from a topic\n  help     Print this message or the help of the given subcommand(s)\n\nOptions:\n  -h, --help  Print help\n```\n\nThe `produce` subcommand reads JSON formatted messages encoding them into\nApache Avro, Protobuf or JSON depending on the schema used by the topic.\n\nFor example, the `taxi` topic is backed by [taxi.proto](etc/schema/taxi.proto).\nUsing [trips.json](etc/data/trips.json) containing a JSON array of objects,\n`tansu cat produce` encodes each message into protobuf into the broker:\n\n```\ntansu cat produce taxi etc/data/trips.json\n```\n\nUsing [duckdb](https://duckdb.org) we can read the\n[Apache Parquet](https://parquet.apache.org) files\ncreated by the broker:\n\n```shell\nduckdb :memory: \"SELECT * FROM 'data/taxi/*/*.parquet'\"\n```\n\nResults in the following output:\n\n```shell\n|-----------+---------+---------------+-------------+---------------|\n| vendor_id | trip_id | trip_distance | fare_amount | store_and_fwd |\n|     int64 |   int64 |         float |      double |         int32 |\n|-----------+---------+---------------+-------------+---------------|\n|         1 | 1000371 |           1.8 |       15.32 |             0 |\n|         2 | 1000372 |           2.5 |       22.15 |             0 |\n|         2 | 1000373 |           0.9 |        9.01 |             0 |\n|         1 | 1000374 |           8.4 |       42.13 |             1 |\n|-----------+---------+---------------+-------------+---------------|\n```\n\n\n### s3\n\nThe following will configure a S3 storage engine\nusing the \"tansu\" bucket (full context is in\n[compose.yaml](compose.yaml) and [example.env](example.env)):\n\nCopy `example.env` into `.env` so that you have a local working copy:\n\n```shell\ncp example.env .env\n```\n\nEdit `.env` so that `STORAGE_ENGINE` is defined as:\n\n```shell\nSTORAGE_ENGINE=\"s3://tansu/\"\n```\n\nFirst time startup, you'll need to create a bucket, an access key\nand a secret in minio.\n\nJust bring minio up, without tansu:\n\n```shell\ndocker compose up -d minio\n```\n\nCreate a minio `local` alias representing `http://localhost:9000` with the default credentials of `minioadmin`:\n\n```shell\ndocker compose exec minio \\\n   /usr/bin/mc \\\n   alias \\\n   set \\\n   local \\\n   http://localhost:9000 \\\n   minioadmin \\\n   minioadmin\n```\n\nCreate a `tansu` bucket in minio using the `local` alias:\n\n```shell\ndocker compose exec minio \\\n   /usr/bin/mc mb local/tansu\n```\n\nOnce this is done, you can start tansu with:\n\n```shell\ndocker compose up -d tansu\n```\n\nUsing the regular Apache Kafka CLI you can create topics, produce and consume\nmessages with Tansu:\n\n```shell\nkafka-topics \\\n  --bootstrap-server localhost:9092 \\\n  --partitions=3 \\\n  --replication-factor=1 \\\n  --create --topic test\n```\n\nDescribe the `test` topic:\n\n```shell\nkafka-topics \\\n  --bootstrap-server localhost:9092 \\\n  --describe \\\n  --topic test\n```\n\nNote that node 111 is the leader and ISR for each topic partition.\nThis node represents the broker handling your request. All brokers are node 111.\n\nProducer:\n\n```shell\necho \"hello world\" | kafka-console-producer \\\n    --bootstrap-server localhost:9092 \\\n    --topic test\n```\n\nGroup consumer using `test-consumer-group`:\n\n```shell\nkafka-console-consumer \\\n  --bootstrap-server localhost:9092 \\\n  --group test-consumer-group \\\n  --topic test \\\n  --from-beginning \\\n  --property print.timestamp=true \\\n  --property print.key=true \\\n  --property print.offset=true \\\n  --property print.partition=true \\\n  --property print.headers=true \\\n  --property print.value=true\n```\n\nDescribe the consumer `test-consumer-group` group:\n\n```shell\nkafka-consumer-groups \\\n  --bootstrap-server localhost:9092 \\\n  --group test-consumer-group \\\n  --describe\n```\n\n### PostgreSQL\n\nTo switch between the minio and PostgreSQL examples, firstly\nshutdown Tansu:\n\n```shell\ndocker compose down tansu\n```\n\nSwitch to the PostgreSQL storage engine by updating [.env](.env):\n\n```env\n# minio storage engine\n# STORAGE_ENGINE=\"s3://tansu/\"\n\n# PostgreSQL storage engine -- NB: @db and NOT @localhost :)\nSTORAGE_ENGINE=\"postgres://postgres:postgres@db\"\n```\n\nStart PostgreSQL:\n\n```shell\ndocker compose up -d db\n```\n\nBring Tansu back up:\n\n```shell\ndocker compose up -d tansu\n```\n\nUsing the regular Apache Kafka CLI you can create topics, produce and consume\nmessages with Tansu:\n\n```shell\nkafka-topics \\\n  --bootstrap-server localhost:9092 \\\n  --partitions=3 \\\n  --replication-factor=1 \\\n  --create --topic test\n```\n\nProducer:\n\n```shell\necho \"hello world\" | kafka-console-producer \\\n    --bootstrap-server localhost:9092 \\\n    --topic test\n```\n\nConsumer:\n\n```shell\nkafka-console-consumer \\\n  --bootstrap-server localhost:9092 \\\n  --group test-consumer-group \\\n  --topic test \\\n  --from-beginning \\\n  --property print.timestamp=true \\\n  --property print.key=true \\\n  --property print.offset=true \\\n  --property print.partition=true \\\n  --property print.headers=true \\\n  --property print.value=true\n```\n\nOr using [librdkafka][librdkafka] to produce:\n\n```shell\necho \"Lorem ipsum dolor...\" | \\\n  ./examples/rdkafka_example -P \\\n  -t test -p 1 \\\n  -b localhost:9092 \\\n  -z gzip\n```\n\nConsumer:\n\n```shell\n./examples/rdkafka_example \\\n  -C \\\n  -t test -p 1 \\\n  -b localhost:9092\n```\n\n## Feedback\n\nPlease [raise an issue][tansu-issues] if you encounter a problem.\n\n## License\n\nTansu is licensed under [Apache 2.0][apache-license].\n\n[apache-license]: https://www.apache.org/licenses/LICENSE-2.0\n[apache-zookeeper]: https://en.wikipedia.org/wiki/Apache_ZooKeeper\n[aws-s3-conditional-requests]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html\n[aws-s3-conditional-writes]: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/\n[aws-s3-storage-classes]: https://aws.amazon.com/s3/storage-classes/\n[cloudflare-r2]: https://developers.cloudflare.com/r2/\n[continuous-archiving]: https://www.postgresql.org/docs/current/continuous-archiving.html\n[crates-io-object-store]: https://crates.io/crates/object_store\n[github-com-tansu-io]: https://github.com/tansu-io/tansu\n[json-schema-org]: https://json-schema.org/\n[librdkafka]: https://github.com/confluentinc/librdkafka\n[min-io]: https://min.io\n[minio-create-access-key]: https://min.io/docs/minio/container/administration/console/security-and-access.html#id1\n[minio-create-bucket]: https://min.io/docs/minio/container/administration/console/managing-objects.html#creating-buckets\n[object-store-dynamo-conditional-put]: https://docs.rs/object_store/0.11.0/object_store/aws/struct.DynamoCommit.html\n[protocol-buffers]: https://protobuf.dev\n[raft-consensus]: https://raft.github.io\n[rust-lang-org]: https://www.rust-lang.org\n[tansu-issues]: https://github.com/tansu-io/tansu/issues\n[tigris-conditional-writes]: https://www.tigrisdata.com/blog/s3-conditional-writes/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftansu-io%2Ftansu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftansu-io%2Ftansu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftansu-io%2Ftansu/lists"}