{"id":19670956,"url":"https://github.com/emc-mongoose/mongoose-storage-driver-pulsar","last_synced_at":"2025-06-13T17:32:35.568Z","repository":{"id":144519249,"uuid":"212817891","full_name":"emc-mongoose/mongoose-storage-driver-pulsar","owner":"emc-mongoose","description":"Apache Pulsar extension for Mongoose","archived":false,"fork":false,"pushed_at":"2023-03-02T17:20:34.000Z","size":111,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-29T00:41:01.996Z","etag":null,"topics":["apache-pulsar","benchmark","mongoose","performance","streaming-data"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emc-mongoose.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-10-04T13:10:38.000Z","updated_at":"2023-02-22T19:42:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"65abd866-f551-45e2-9262-d24dd379e781","html_url":"https://github.com/emc-mongoose/mongoose-storage-driver-pulsar","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/emc-mongoose/mongoose-storage-driver-pulsar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emc-mongoose%2Fmongoose-storage-driver-pulsar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emc-mongoose%2Fmongoose-storage-driver-pulsar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emc-mongoose%2Fmongoose-storage-driver-pulsar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emc-mongoose%2Fmongoose-storage-driver-pulsar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emc-mongoose","download_url":"https://codeload.github.com/emc-mongoose/mongoose-storage-driver-pulsar/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emc-mongoose%2Fmongoose-storage-driver-pulsar/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259688365,"owners_count":22896392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-pulsar","benchmark","mongoose","performance","streaming-data"],"created_at":"2024-11-11T17:07:38.738Z","updated_at":"2025-06-13T17:32:35.560Z","avatar_url":"https://github.com/emc-mongoose.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Gitter chat](https://badges.gitter.im/emc-mongoose.png)](https://gitter.im/emc-mongoose)\n[![Issue Tracker](https://img.shields.io/badge/Issue-Tracker-red.svg)](https://mongoose-issues.atlassian.net/projects/PULSAR)\n[![CI status](https://gitlab.com/emc-mongoose/mongoose-storage-driver-pulsar/badges/master/pipeline.svg)](https://travis-ci.org/emc-mongoose/mongoose-storage-driver-pravega/builds)\n[![Maven metadata URL](https://img.shields.io/maven-metadata/v/http/central.maven.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-pulsar/maven-metadata.xml.svg)](http://central.maven.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-pulsar)\n[![Docker Pulls](https://img.shields.io/docker/pulls/emcmongoose/mongoose-storage-driver-pulsar.svg)](https://hub.docker.com/r/emcmongoose/mongoose-storage-driver-pulsar/)\n\nApache Pulsar extension for Mongoose\n\n# Content\n\n1. [Introduction](#1-introduction)\u003cbr/\u003e\n2. [Features](#2-features)\u003cbr/\u003e\n3. [Deployment](#3-deployment)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;3.1. [Basic](#31-basic)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;3.2. [Docker](#32-docker)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.1. [Standalone](#321-standalone)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.2. [Distributed](#322-distributed)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.2.1. [Additional Node](#3221-additional-node)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;3.2.2.2. [Entry Node](#3222-entry-node)\u003cbr/\u003e\n4. [Configuration](#4-configuration)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;4.1. [Specific Options](#41-specific-options)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;4.2. [Tuning](#42-tuning)\u003cbr/\u003e\n5. [Usage](#5-usage)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;5.1. [Message Operations](#51-message-operations)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.1.1. [Create](#511-create)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.1.2. [Read](#512-read)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.1.2.1. [Basic](#5121-basic)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.1.2.2. [Tail](#5122-tail)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.1.2.3. [End-to-end Latency](#5123-end-to-end-latency)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;5.2. [Topic Operations](#52-topic-operations)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.2.1. [Create](#521-create)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.2.2. [Read](#522-read)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.2.3. [Update](#523-update)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;5.2.4. [Delete](#524-delete)\u003cbr/\u003e\n6. [Open Issues](#6-open-issues)\u003cbr/\u003e\n7. [Development](#7-development)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;7.1. [Build](#71-build)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;7.2. [Test](#72-test)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;7.2.1. [Automated](#721-automated)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;7.2.1.1. [Unit](#7211-unit)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;7.2.1.2. [Integration](#7212-integration)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;7.2.1.3. [Functional](#7213-functional)\u003cbr/\u003e\n\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;7.2.2. [Manual](#722-manual)\u003cbr/\u003e\n\n# 1. Introduction\n\n| Pulsar | [Mongoose](https://github.com/emc-mongoose/mongoose-base/tree/master/doc/design/architecture#1-basic-terms) |\n|---------|----------|\n| Message | *Data Item* |\n| Topic | *Item Path* or *Data Item* |\n\n# 2. Features\n\n* Authentication: https://mongoose-issues.atlassian.net/browse/PULSAR-4\n* SSL/TLS: https://mongoose-issues.atlassian.net/browse/PULSAR-1\n* Item Types:\n    * `data`: corresponds to a ***message*** \n    * `path`: corresponds to a ***topic***\n    * `token`: not supported\n* Supported load operations:\n    * `create` (messages)\n    * `read` (messages)\n    * `updae` (topics appending, TODO)\n    * `delete` (topics, TODO)\n* Storage-specific:\n    * [End-to-end latency measurement](#5123-end-to-end-latency)\n\n# 3. Deployment\n\n## 3.1. Basic\n\nJava 11+ is required to build/run.\n\n1. Get the latest `mongoose-base` jar from the \n[maven repo](http://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-base/)\nand put it to your working directory. Note the particular version, which is referred as *BASE_VERSION* below.\n\n2. Get the latest `mongoose-storage-driver-coop` jar from the\n[maven repo](http://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-coop/)\nand put it to the `~/.mongoose/\u003cBASE_VERSION\u003e/ext` directory.\n\n3. Get the latest `mongoose-storage-driver-pulsar` jar from the\n[maven repo](http://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-pulsar/)\nand put it to the `~/.mongoose/\u003cBASE_VERSION\u003e/ext` directory.\n\n```bash\njava -jar mongoose-base-\u003cBASE_VERSION\u003e.jar \\\n    --storage-driver-type=pulsar \\\n    --storage-net-node-addrs=\u003cNODE_IP_ADDRS\u003e \\\n    --storage-net-node-port=6650 \\\n    --load-batch-size=1000 \\\n    --storage-driver-limit-concurrency=1000 \\\n    ...\n```\n\n## 3.2. Docker\n\n### 3.2.1. Standalone\n\n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --storage-net-node-addrs=\u003cNODE_IP_ADDRS\u003e \\\n    --load-batch-size=1000 \\\n    --storage-driver-limit-concurrency=1000 \\\n    ...\n```\n\n### 3.2.2. Distributed\n\n#### 3.2.2.1. Additional Node\n\n```bash\ndocker run \\\n    --network host \\\n    --expose 1099 \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --run-node\n```\n\n#### 3.2.2.2. Entry Node\n\n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --load-step-node-addrs=\u003cADDR1,ADDR2,...\u003e \\\n    --storage-net-node-addrs=\u003cNODE_IP_ADDRS\u003e \\\n    --load-batch-size=1000 \\\n    --storage-driver-limit-concurrency=1000 \\\n    ...\n```\n\n# 4. Configuration\n\nReference\n\n## 4.1. Specific Options\n\n| Name                               | Type            | Default Value | Description                                      |\n|:-----------------------------------|:----------------|:--------------|:-------------------------------------------------|\n| storage-driver-create-batch-enabled| boolean         | `true`        | [Producer batching](https://pulsar.apache.org/docs/en/concepts-messaging/#batching)\n| storage-driver-create-batch-delayMicros | long integer | 1000        | Maximum publish latency (microseconds)\n| storage-driver-create-compression  | enum            | `none`        | Should compress data upon messages create or not (default). The available compression types are defined by the [Pulsar client](https://github.com/apache/pulsar/blob/be6a102511560349701dbe6b83fabf831dc81340/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/CompressionType.java#L24)\n| storage-driver-create-timestamp    | boolean         | `false`       | If enabled, will record the message creation timestamp into the message's metadata. Required for [end-to-end latency measurement](#5123-end-to-end-latency)\n| storage-driver-read-tail           | boolean         | `false`       | Should read the latest message of the topic or read from the topic beginning (default). Should be `true` for [end-to-end latency measurement](#5123-end-to-end-latency)\n| storage-net-tcpNoDelay             | boolean         | `true`        | The option specifies whether the server disables the delay of sending successive small packets on the network.\n| storage-net-timeoutMilliSec        | integer         | `0`           | Connection timeout. `0` means no timeout\n| storage-net-node-addrs             | list of strings | `127.0.0.1`   | The list of the storage node IPs or hostnames to use for HTTP load. May include port numbers.\n| storage-net-node-port              | integer         | `6650`        | The common port number to access the storage nodes, may be overriden adding the port number to the storage-driver-addrs, for example: \"127.0.0.1:6650,127.0.0.1:6651,...\"\n\n## 4.2. Tuning\n\n* `load-batch-size`\nDetermines how many operations the driver will try to submit at once. Large values may increase both the throughput and \nthe memory consumption. \n\n* `storage-driver-concurrency-limit`\nDetermines how many operations may be in flight at every moment of the time. Specifying `0` will cause the \"*burst \nmode*\", when the driver submits as many operations as possible regardless their completion. The unlimited concurrency\nis useful for the sustained rate measurement but may cause either Pulsar or Mongoose inconsistent state.\n\n* `storage-driver-threads`\nDetermines the Pulsar client IO worker threads count.\n\n# 5. Usage\n\n## 5.1. Message Operations\n\n### 5.1.1. Create\n\nExample, write 1KB messages to the topic \"topic1\" in the Pulsar instance w/ address 12.34.56.78: \n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --storage-net-node-addrs=12.34.56.78 \\\n    --load-batch-size=1000 \\\n    --storage-driver-limit-concurrency=1000 \\\n    --item-data-size=1KB \\\n    --item-output-path=topic1\n```\n\n### 5.1.2. Read\n\n***Notes***: \n1. Generally, `load-op-recycle` option should be set to `true` to make the messages reading working. \n2. Mongoose couldn't determine the end of the topic(s), so it's mandatory to specify the count/time limit.\n\n#### 5.1.2.1. Basic\n\nExample, read 1M messages from the beginning of the topic \"topic1\":\n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --storage-net-node-addrs=12.34.56.78 \\\n    --load-batch-size=100 \\\n    --storage-driver-limit-concurrency=100 \\\n    --read \\ \n    --item-input-path=topic1 \\\n    --load-op-recycle \\\n    --load-op-limit-count=1000000\n```\n\n#### 5.1.2.2. Tail\n\nExample, read all new messages from the topic \"topic1\" during the 1 minute:\n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --storage-net-node-addrs=12.34.56.78 \\\n    --load-batch-size=100 \\\n    --storage-driver-limit-concurrency=100 \\\n    --read \\ \n    --item-input-path=topic1 \\\n    --storage-driver-read-tail \\\n    --load-op-recycle \\\n    --load-step-limit-time=1m\n```\n\n### 5.1.2.3. End-To-End Latency\n\n1. Start writing the messages to some topic with enabled timestamps recording. Example command:\n\n```bash\ndocker run \\\n    --network host \\\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --item-data-size=1KB \\\n    --item-output-path=topic1 \\\n    --load-batch-size=1000 \\\n    --storage-driver-create-timestamp\n```\n\n2. Start the tail read from the same topic:\n\n```bash\ndocker run \\\n    --network host \\\n    --volume $PWD/log:/root/.mongoose/4.2.16/log\n    emcmongoose/mongoose-storage-driver-pulsar \\\n    --load-batch-size=1000 \\\n    --read \\\n    --item-input-path=topic1 \\\n    --storage-driver-read-tail \\\n    --load-op-recycle \\\n    --load-step-id=e2e_test\n```\n\n3. Check the end-to-end time data in the `log/e2e_test/op.trace.csv` log file. \nThe data is in the CSV format with 3 columns:\n* internal message id\n* message payload size\n* end-to-end time *in milliseconds*\n\n***Note***: the end-to-end time data will not be aggregated in the distributed mode.\n\n## 5.2. Topic Operations\n\nTODO: https://mongoose-issues.atlassian.net/browse/PULSAR-2\n\n### 5.2.1. Create\n\nTODO\n\n### 5.2.2. Read\n\nTODO\n\n### 5.2.3. Update\n\nTODO\n\n### 5.2.4. Delete\n\nTODO\n\n# 6. Open Issues\n\nPlease refer to the [issue tracker](https://mongoose-issues.atlassian.net/projects/PULSAR)\n# 7. Development\n\n## 7.1. Build\n\n```bash\n./gradlew clean jar\n```\n\n## 7.2. Test\n\n### 7.2.1. Automated\n\n#### 7.2.1.1. Unit\n\n```bash\n./gradlew clean test\n```\n\n#### 7.2.1.2. Integration\n```bash\ndocker run -it \\\n    -p 6650:6650 \\\n    -p 8080:8080 \\\n    apachepulsar/pulsar:\u003cPULSAR_VERSION\u003e \\\n    bin/pulsar standalone\n./gradlew integrationTest\n```\n\n#### 7.2.1.3. Functional\n```bash\n./gradlew jar\nexport SUITE=api.storage\nTEST=\u003cTODO_TEST_NAME\u003e ./gradlew robotest\n```\n\n### 7.2.2. Manual\n\n1. [Build the storage driver](#71-build)\n2. Copy the storage driver's jar file into the mongoose's `ext` directory:\n```bash\ncp -f build/libs/mongoose-storage-driver-pulsar-*.jar ~/.mongoose/\u003cMONGOOSE_BASE_VERSION\u003e/ext/\n```\nNote that the Pulsar storage driver depends on the \n[Coop Storage Driver](http://repo.maven.apache.org/maven2/com/github/emc-mongoose/mongoose-storage-driver-coop/) \nextension so it should be also put into the `ext` directory\n3. Install and run the Apache Pulsar\n```bash\ndocker run -it \\\n    -p 6650:6650 \\\n    -p 8080:8080 \\\n    apachepulsar/pulsar:\u003cPULSAR_VERSION\u003e \\\n    bin/pulsar standalone\n```\n4. Run Mongoose's default scenario with some specific command-line arguments:\n```bash\njava -jar mongoose-\u003cMONGOOSE_BASE_VERSION\u003e.jar \\\n    --storage-driver-type=pulsar \\\n    --storage-net-node-port=6650 \\\n    --storage-driver-limit-concurrency=10 \\\n    --item-output-path=topic-0\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femc-mongoose%2Fmongoose-storage-driver-pulsar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femc-mongoose%2Fmongoose-storage-driver-pulsar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femc-mongoose%2Fmongoose-storage-driver-pulsar/lists"}