{"id":19704689,"url":"https://github.com/datastax/starlight-for-rabbitmq","last_synced_at":"2025-10-30T16:03:12.390Z","repository":{"id":37860952,"uuid":"371291318","full_name":"datastax/starlight-for-rabbitmq","owner":"datastax","description":"Starlight for RabbitMQ, a proxy layer between RabbitMQ/AMQP0.9.1 clients and Apache Pulsar","archived":false,"fork":false,"pushed_at":"2025-04-25T08:03:23.000Z","size":527,"stargazers_count":29,"open_issues_count":7,"forks_count":8,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-25T08:28:44.383Z","etag":null,"topics":["amqp","apache-pulsar","broker","messaging","pulsar","rabbitmq"],"latest_commit_sha":null,"homepage":"https://docs.datastax.com/en/starlight-rabbitmq/docs/1.0/index.html","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-05-27T07:53:32.000Z","updated_at":"2024-07-03T18:27:40.000Z","dependencies_parsed_at":"2024-11-11T21:24:36.109Z","dependency_job_id":"c48eb25a-faf7-4e07-b3aa-626a6ba7d42b","html_url":"https://github.com/datastax/starlight-for-rabbitmq","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fstarlight-for-rabbitmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fstarlight-for-rabbitmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fstarlight-for-rabbitmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fstarlight-for-rabbitmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastax","download_url":"https://codeload.github.com/datastax/starlight-for-rabbitmq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251519194,"owners_count":21602297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amqp","apache-pulsar","broker","messaging","pulsar","rabbitmq"],"created_at":"2024-11-11T21:24:04.864Z","updated_at":"2025-10-30T16:03:07.350Z","avatar_url":"https://github.com/datastax.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Starlight for RabbitMQ\n\nStarlight for RabbitMQ acts as a proxy between your RabbitMQ application and Apache Pulsar.\nIt implements the AMQP 0.9.1 protocol used by RabbitMQ clients and translates AMQP frames and concepts to Pulsar ones.\nThe proxy can be run as a standalone jar, a Pulsar\n[Pluggable Protocol Handler](https://github.com/apache/pulsar/wiki/PIP-41%3A-Pluggable-Protocol-Handler) \n or a Pulsar [Proxy extension](https://github.com/apache/pulsar/wiki/PIP-99%3A-Pulsar-Proxy-Extensions).\n\n## Limitations\n\nThis is currently not implemented but on the roadmap:\n* Headers exchanges\n* Exclusive consumers\n* Non durable exchanges and queues\n* Transient messages (all messages are persisted)\n\nRabbitMQ and Pulsar work in a pretty different way.\nStarlight for RabbitMQ was designed to make the most benefit from Pulsar's scalability.\nThis results in some differences of behavior:\n* Canceling an AMQP consumer will requeue the messages that were received through it since it also closes\nthe associated Pulsar consumers.\n\n## Get started\n\n### Download and build Starlight for RabbitMQ\n\nTo build from code, complete the following steps:\n1. Clone the project from GitHub.\n\n```bash\ngit clone https://github.com/datastax/starlight-for-rabbitmq.git\ncd starlight-for-rabbitmq\n```\n\n2. Build the project.\n```bash\nmvn clean install -DskipTests\n```\n\nYou can find the executable jar file in the following directory.\n```bash\n./starlight-rabbitmq/target/starlight-rabbitmq-${version}-jar-with-dependencies.jar\n```\nYou can find the nar file in the following directory.\n```bash\n./starlight-rabbitmq/target/starlight-rabbitmq-${version}.nar\n```\n\n### Running Starlight for RabbitMQ as a standalone executable jar\n\n1. Set the URLs of the Pulsar brokers and the ZooKeeper configuration store in a configuration file. Eg:\n   ```properties\n   brokerServiceURL=pulsar://localhost:6650\n   brokerWebServiceURL=http://localhost:8080\n   configurationStoreServers=localhost:2181\n   ```\n2. Run as a Java application and provide the configuration file path in the `-c/--config` option:\n   ```bash\n   java -jar ./starlight-rabbitmq/target/starlight-rabbitmq-${version}-jar-with-dependencies.jar -c conf/starlight-rabbitmq.conf\n   ```\n\n### Running Starlight for RabbitMQ as a protocol handler\n\nStarlight for RabbitMQ can be embedded directly into the Pulsar brokers by loading it as a protocol handler.\n\n1. Set the configuration of the Starlight for RabbitMQ protocol handler in the broker configuration file (generally `broker.conf` or `standalone.conf`).\n   Example where the NAR file was copied into the `./protocols` directory:\n    ```properties\n   messagingProtocols=rabbitmq\n   protocolHandlerDirectory=./protocols\n    ```\n\n2. Set the AMQP service listeners. Note that the hostname value in listeners is the same as Pulsar broker's `advertisedAddress`. \n   The following is an example.\n   ```properties\n   amqpListeners=amqp://127.0.0.1:5672\n   advertisedAddress=127.0.0.1\n   ```\n\n3. Start the Pulsar broker\n\n### Running Starlight for RabbitMQ as a Pulsar Proxy extension\n\nStarlight for RabbitMQ can be embedded into the Pulsar Proxy by loading it as a proxy extension.\n\n1. Set the configuration of the Starlight for RabbitMQ proxy extension in the proxy configuration file (generally `proxy.conf` or `standalone.conf`).\n   Example where the NAR file was copied into the `./proxyextensions` directory:\n    ```properties\n   proxyExtensions=rabbitmq\n   proxyExtensionsDirectory=./proxyextensions\n    ```\n\n2. Set the AMQP service listeners. Note that the hostname value in listeners is the same as Pulsar proxy's `advertisedAddress`.\n   The following is an example.\n   ```properties\n   amqpListeners=amqp://127.0.0.1:5672\n   advertisedAddress=127.0.0.1\n   ```\n\n3. Start the Pulsar Proxy\n\n### Checking that it works\n\nYou can use a RabbitMQ/AMQP-0.9.1 client or a tool such as [RabbitMQ PerfTest](https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/)\nto check that everything works correctly.\n\nFor instance the following Python script creates a queue, publishes a message that will be routed to this queue, reads the message from the queue and deletes the queue\n```python\n#!/usr/bin/env python\nimport pika\n\nconnection = pika.BlockingConnection(pika.ConnectionParameters(port=5672))\nchannel = connection.channel()\n\ntry:\n    channel.queue_declare(\"test-queue\")\n    print(\"created test-queue queue\")\n    channel.basic_publish(exchange=\"\", routing_key=\"test-queue\", body=\"test\".encode('utf-8'))\n    print(\"published message test\")\n    _, _, res = channel.basic_get(queue=\"test-queue\", auto_ack=True)\n    assert res is not None, \"should have received a message\"\n    print(\"received message: \" + res.decode())\n    channel.queue_delete(\"test-queue\")\n    print(\"deleted test-queue queue\")\nfinally:\n    connection.close()\n```\n\n## Configuration\n\n### Generic configuration\n|Name|Description|Default|\n|---|---|---|\n|configurationStoreServers|Zookeeper configuration store connection string (as a comma-separated list)|\n|amqpListeners|Used to specify multiple advertised listeners for the proxy. The value must format as `amqp[s]://\u003chost\u003e:\u003cport\u003e`, multiple listeners should be separated with commas.|amqp://127.0.0.1:5672\n|amqpSessionCountLimit|The maximum number of sessions which can exist concurrently on an AMQP connection.|256\n|amqpHeartbeatDelay|The default period with which Broker and client will exchange heartbeat messages (in seconds) when using AMQP. Clients may negotiate a different heartbeat frequency or disable it altogether.|0\n|amqpHeartbeatTimeoutFactor|Factor to determine the maximum length of that may elapse between heartbeats being received from the peer before an AMQP0.9 connection is deemed to have been broken.|2\n|amqpNetworkBufferSize|AMQP Network buffer size.|2097152 (2MB)\n|amqpMaxMessageSize|AMQP Max message size.|104857600 (100MB)\n|amqpDebugBinaryDataLength|AMQP Length of binary data sent to debug log.|80\n|amqpConnectionCloseTimeout|Timeout in ms after which the AMQP connection closes even if a ConnectionCloseOk frame is not received|2000\n|amqpBatchingEnabled|Whether batching messages is enabled in AMQP|true\n\n### Authentication configuration\n|Name|Description|Default|\n|---|---|---|\n|authenticationEnabled| Whether authentication is enabled for the proxy  |false|\n|amqpAuthenticationMechanisms|Authentication mechanism name list for AMQP (a comma-separated list of mecanisms. Eg: PLAIN,EXTERNAL)|PLAIN\n|tokenSecretKey| Configure the secret key to be used to validate auth tokens. The key can be specified like: `tokenSecretKey=data:;base64,xxxxxxxxx` or `tokenSecretKey=file:///my/secret.key`.  Note: key file must be DER-encoded.||\n|tokenPublicKey| Configure the public key to be used to validate auth tokens. The key can be specified like: `tokenPublicKey=data:;base64,xxxxxxxxx` or `tokenPublicKey=file:///my/secret.key`. Note: key file must be DER-encoded.||\n|tokenAuthClaim| Specify the token claim that will be used as the authentication \"principal\" or \"role\". The \"subject\" field will be used if this is left blank ||\n|tokenAudienceClaim| The token audience \"claim\" name, e.g. \"aud\". It is used to get the audience from token. If it is not set, the audience is not verified. ||\n|tokenAudience| The token audience stands for this broker. The field `tokenAudienceClaim` of a valid token need contains this parameter.| |\n\n### Broker client configuration\n|Name|Description|Default|\n|---|---|---|\n|brokerServiceURL| The service URL pointing to the broker cluster. | |\n|brokerWebServiceURL| The Web service URL pointing to the broker cluster | |\n|brokerClientAuthenticationPlugin|  The authentication plugin used by the Pulsar proxy to authenticate with Pulsar brokers  ||\n|brokerClientAuthenticationParameters|  The authentication parameters used by the Pulsar proxy to authenticate with Pulsar brokers  ||\n|amqpBrokerClientAuthenticationParameters|If set, the RabbitMQ service will use these parameters to authenticate on Pulsar's brokers. If not set, the brokerClientAuthenticationParameters setting will be used. This setting allows to have different credentials for the Pulsar proxy and for the RabbitMQ service|\n|tlsEnabledWithBroker|  Whether TLS is enabled when communicating with Pulsar brokers. |false|\n|brokerClientTrustCertsFilePath|  The path to trusted certificates used by the Pulsar proxy to authenticate with Pulsar brokers ||\n|brokerClientTlsEnabledWithKeyStore| Whether the proxy use KeyStore type to authenticate with Pulsar brokers ||\n|brokerClientTlsTrustStoreType| TLS TrustStore type configuration for proxy: JKS, PKCS12  used by the proxy to authenticate with Pulsar brokers ||\n|brokerClientTlsTrustStore| TLS TrustStore path for proxy,  used by the Pulsar proxy to authenticate with Pulsar brokers ||\n|brokerClientTlsTrustStorePassword| TLS TrustStore password for proxy,  used by the Pulsar proxy to authenticate with Pulsar brokers ||\n\n### TLS configuration\n|Name|Description|Default|\n|---|---|---|\n|tlsCertRefreshCheckDurationSec| TLS certificate refresh duration in seconds. If the value is set 0, check TLS certificate every new connection. | 300 |\n|tlsCertificateFilePath|  Path for the TLS certificate file ||\n|tlsKeyFilePath|  Path for the TLS private key file ||\n|tlsTrustCertsFilePath| Path for the trusted TLS certificate pem file ||\n|tlsAllowInsecureConnection| Accept untrusted TLS certificate from client. If true, a client with a cert which cannot be verified with the `tlsTrustCertsFilePath` cert will be allowed to connect to the server, though the cert will not be used for client authentication ||\n|tlsHostnameVerificationEnabled|  Whether the hostname is validated when the proxy creates a TLS connection with brokers  |false|\n|tlsRequireTrustedClientCertOnConnect|  Whether client certificates are required for TLS. Connections are rejected if the client certificate isn’t trusted. |false|\n|tlsProtocols|Specify the tls protocols the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLSv1.3```, ```TLSv1.2``` ||\n|tlsCiphers|Specify the tls cipher the broker will use to negotiate during TLS Handshake. Multiple values can be specified, separated by commas. Example:- ```TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256```||\n|tlsRequireTrustedClientCertOnConnect| Whether client certificates are required for TLS. Connections are rejected if the client certificate isn't trusted ||\n|tlsEnabledWithKeyStore| Enable TLS with KeyStore type configuration for proxy ||\n|tlsProvider| TLS Provider ||\n|tlsKeyStoreType| TLS KeyStore type configuration for proxy: JKS, PKCS12 ||\n|tlsKeyStore| TLS KeyStore path for proxy ||\n|tlsKeyStorePassword| TLS KeyStore password for proxy ||\n|tlsTrustStoreType| TLS TrustStore type configuration for proxy: JKS, PKCS12 ||\n|tlsTrustStore| TLS TrustStore path for proxy ||\n|tlsTrustStorePassword| TLS TrustStore password for proxy ||\n\n## Under the hood\n\nAMQP 0.9.1 (the protocol used by RabbitMQ) employs the concepts of `Exchanges`, `Queues` and `Bindings` to provide basic routing capabilities inside the message broker.\nThese concepts are mapped to Pulsar topics and features.\nOne important architectural decision is that Starlight for RabbitMQ doesn’t interact directly with the managed ledger as in other RabbitMQ integrations for Pulsar.\nInteracting with the ledger has the advantage of being performant, but the disadvantage is that the broker which interacts with the ledger must have ownership of the topic.\nSince in AMQP 0.9.1 there is a many-to-many relationship between `Exchanges` and `Queues` for a given `Virtual host`, all `Exchanges` and `Queues` and related topics would have to be owned by the same broker.\nThere are techniques to do this using Topic bundles, but the result is that a full AMQP `Virtual host` `can be handled by only one broker at a time. This is an issue for scalability.\nSo instead, Starlight for RabbitMQ acts as a proxy and uses the Pulsar binary protocol to communicate with the brokers.\nThis way it can leverage Pulsar features such as load balancing of the topics on the brokers, batching of messages, partitioning of topics, and load balancing of the data on the consumers.\n\nOn the publishing side, an AMQP exchange is mapped to a topic. Depending on the type of exchange, the publishing routing key may also be included in the topic name.\n\nOn the consumer side, Pulsar shared subscriptions are used to represent the AMQP `Bindings` from an `Exchange` to a `Queue`.\nWhen creating an AMQP Queue consumer, the proxy creates Pulsar consumers for all the `Bindings` of the `Queue`.\n\nWhen you unbind the `Queue`, the Pulsar subscription isn’t deleted right away since the consumer may be lagging.\nMessages are still received from the subscription and filtered if their position is past the end of the binding.\nWhen all messages from the binding have been acknowledged, then the corresponding subscription can finally be removed by the `SubscriptionCleaner` task.\n\n### Consistent metadata store\n\nStarlight for RabbitMQ uses Apache Zookeeper to store the AMQP entities metadata consistently.\nThe existing ZooKeeper configuration store can be reused for this, and Starlight for RabbitMQ will employ the /starlight-rabbitmq prefix to write its entries into ZooKeeper.\n\n### Security and authentication\n\nStarlight for RabbitMQ supports connections using TLS/mTLS to ensure privacy and security of the communication.\nIt also supports the PLAIN and EXTERNAL mechanisms used by RabbitMQ.\nInternally, it uses the same `AuthenticationService` as Apache Pulsar and maps the AMQP mechanisms to existing Pulsar authentication modes.\nAt the moment there is no support for authorization so an authenticated user has full access to all `Virtual hosts`.\nStarlight for RabbitMQ can connect to brokers that have TLS and/or authentication, and/or authorization enabled.\nTo perform its operations, Starlight for RabbitMQ currently needs to use an “admin role”.\nFuture versions will relay the principal authenticated to the proxy and use a “proxy role” so operations on the broker will have permissions from the originating application.\n\n#### PLAIN authentication mechanism\n\nThe PLAIN mechanism is mapped to the AuthenticationProviderToken mode of authentication. The username is ignored and the password is used as the JSON Web Token (JWT).\n\n#### EXTERNAL authentication mechanism\n\nThe EXTERNAL mechanism is mapped to the AuthenticationProviderTls mode of authentication.\nThis is the equivalent of the [rabbitmq-auth-mechanism-ssl plugin](https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_auth_mechanism_ssl) with `ssl_cert_login_from` parameter set to `common_name`.\n\n### Clustering\n\nMultiple Starlight for RabbitMQ proxies can be launched at the same time for scalability and high availability needs.\nThe proxies are stateless and can be started and stopped at will.\nThey share their configuration in Zookeeper so you can create/delete/bind/unbind exchanges and queues on any proxy, and the configuration will be synchronized on the other proxies.\nPublishing messages can be done on any proxy.\nOn the receiving side, messages will be dispatched evenly to all connected AMQP consumers since the Pulsar subscriptions are shared ones.\n\n### Multi-tenancy\n\nStarlight for RabbitMQ offers support for multi-tenancy by mapping an AMQP `Virtual host` to a Pulsar `tenant` and `namespace`.\nThe mapping depends on the following configuration parameters\n\n|Name|Description|Default|\n|---|---|---|\n|amqpDefaultTenant|Default Pulsar tenant used to map short or empty VHosts|public\n|amqpDefaultNamespace|Default Pulsar namespace used to map short or empty VHosts|default\n|amqpMapShortVhostToTenant|By default short VHosts (not containing a slash character) will be mapped to a namespace on the default tenant. Set this parameter to true to map short VHosts to tenants with the default namespace instead.|false\n\nThe mapping is done as follows:\n* AMQP vhost `/` is mapped to Pulsar namespace `\u003camqpDefaultTenant\u003e/\u003camqpDefaultNamespace\u003e`\n* AMQP vhost `/\u003cvhost\u003e` or `\u003cvhost\u003e` is mapped to Pulsar namespace `\u003camqpDefaultTenant\u003e/\u003cvhost\u003e` if `amqpMapShortVhostToTenant`  is false (default) or to `\u003cvhost\u003e/\u003camqpDefaultNamespace\u003e` if `amqpMapShortVhostToTenant` is true\n* AMQP vhost `/\u003ctenant\u003e/\u003cnamespace\u003e` or `\u003ctenant\u003e/\u003cnamespace\u003e` is mapped to Pulsar namespace `\u003ctenant\u003e/\u003cnamespace\u003e`\n\nThis means that AMQP vhosts must only contain characters that are accepted in Pulsar tenant and namespace names (ie. `a-zA-Z0-9_-=:.`)\n\n# Contribute\n## Release\n\n```\nmvn release:prepare -DautoVersionSubmodules -Prelease -Darguments='-DskipTests'\n```\n\nThe GitHub release is handled by a GitHub action whenever a tag is being pushed\n\n## System tests\n\nSystem tests is a test suite which runs a RabbitMQ client against a real Pulsar Cluster and tests the protocol handler is working correctly.\n\n```\nmvn -f rabbitmq-tests integration-test failsafe:verify -Dgroups=com.datastax.oss.starlight.rabbitmqtests.SystemTest \\\n   -Dtests.systemtests.enabled=true \\\n   -Dtests.systemtests.pulsar.host=your-pulsar-broker-or-proxy-hostname \\\n   -Dtests.systemtests.ampqlistener.port=5672\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fstarlight-for-rabbitmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastax%2Fstarlight-for-rabbitmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fstarlight-for-rabbitmq/lists"}