https://github.com/spotify/async-google-pubsub-client
[SUNSET] Async Google Pubsub Client
https://github.com/spotify/async-google-pubsub-client
Last synced: 9 months ago
JSON representation
[SUNSET] Async Google Pubsub Client
- Host: GitHub
- URL: https://github.com/spotify/async-google-pubsub-client
- Owner: spotify
- License: apache-2.0
- Created: 2015-06-29T03:24:39.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2023-03-18T19:53:00.000Z (about 3 years ago)
- Last Synced: 2025-04-08T01:51:16.766Z (12 months ago)
- Language: Java
- Homepage:
- Size: 351 KB
- Stars: 158
- Watchers: 105
- Forks: 36
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-java - Async Google Pub/Sub Client
README
**NOTE: This library is sunsetting. Please consider using the official [Google Cloud Java Client for Pub/Sub](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-clients/google-cloud-pubsub).**
async-google-pubsub-client
==========================
[](https://maven-badges.herokuapp.com/maven-central/com.spotify/async-google-pubsub-client) [](https://travis-ci.org/spotify/async-google-pubsub-client) [](http://codecov.io/github/spotify/async-google-pubsub-client?branch=master)
A performant [Google Cloud Pub/Sub](https://cloud.google.com/pubsub/) client and batch publisher.
What
----
A low level Pub/Sub client and a concurrent per-topic batching Publisher.
The client uses async-http-client with the Netty provider for making efficient and async HTTP requests to the Google Cloud Pub/Sub api.
The publisher is implemented on top of the async Pub/Sub client and concurrently gathers individual messages into per-topic batches which are then pushed to Google Cloud Pub/Sub at a specified desired request concurrency level in order to achieve both low-latency and high throughput.
Why
---
The official Google Cloud Pub/Sub client library was not performant enough for our purposes due to blocking I/O etc.
Usage
-----
### Pubsub Client
```java
// Create a topic
pubsub.createTopic("my-google-cloud-project", "the-topic").get();
// Create a subscription
pubsub.createSubscription("my-google-cloud-project", "the-subscription-name", "the-topic").get();
// Create a batch of messages
final List messages = asList(
Message.builder()
.attributes("type", "foo")
.data(encode("hello foo"))
.build(),
Message.builder()
.attributes("type", "bar")
.data(encode("hello foo"))
.build());
// Publish the messages
final List messageIds = pubsub.publish("my-google-cloud-project", "the-topic", messages).get();
System.out.println("Message IDs: " + messageIds);
// Pull the message
final List received = pubsub.pull("my-google-cloud-project", "the-subscription").get();
System.out.println("Received Messages: " + received);
// Ack the received messages
final List ackIds = received.stream().map(ReceivedMessage::ackId).collect(Collectors.toList());
pubsub.acknowledge("my-google-cloud-project", "the-subscription", ackIds).get();
```
### Publisher
```java
final Pubsub pubsub = Pubsub.builder()
.build();
final Publisher publisher = Publisher.builder()
.pubsub(pubsub)
.project("my-google-cloud-project")
.concurrency(128)
.build();
// A never ending stream of messages...
final Iterable messageStream = incomingMessages();
// Publish incoming messages
messageStream.forEach(m -> publisher.publish(m.topic, m.message));
```
### Puller
```java
final Pubsub pubsub = Pubsub.builder()
.build();
final MessageHandler handler = (puller, subscription, message, ackId) -> {
System.out.println("got message: " + message);
return CompletableFuture.completedFuture(ackId);
};
final Puller puller = builder()
.pubsub(pubsub)
.project("my-google-cloud-project")
.subscription("my-subscription")
.concurrency(32)
.messageHandler(handler)
.build();
```
### `pom.xml`
```xml
com.spotify
async-google-pubsub-client
1.31
```
Publisher Benchmark
-------------------
Note: This benchmark uses a lot of quota and network bandwidth.
```
$ mvn exec:exec -Dexec.executable="java" -Dexec.classpathScope="test" -Dexec.args="-cp %classpath com.spotify.google.cloud.pubsub.client.integration.PublisherBenchmark"
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building async-google-pubsub-client 1.13-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ async-google-pubsub-client ---
2015-09-29 18:31:44 (1 s)
----------------------------------------------------------------------------------------------------------------
publishes 1,235 ( 1,235 avg) messages/s 934.888 ( 934.888 avg) ms latency 1,237 total
... warmup ...
2015-09-29 18:31:53 (10 s)
----------------------------------------------------------------------------------------------------------------
publishes 198,902 ( 156,137 avg) messages/s 503.912 ( 620.260 avg) ms latency 1,565,391 total
2015-09-29 18:31:54 (11 s)
----------------------------------------------------------------------------------------------------------------
publishes 212,755 ( 177,264 avg) messages/s 475.023 ( 602.638 avg) ms latency 1,778,331 total
...
```
End To End Benchmark
-------------------
Note: This benchmark uses a lot of quota and network bandwidth.
```
$ mvn exec:exec -Dexec.executable="java" -Dexec.classpathScope="test" -Dexec.args="-cp %classpath com.spotify.google.cloud.pubsub.client.integration.EndToEndBenchmark"
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO] ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building async-google-pubsub-client 1.13-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ async-google-pubsub-client ---
2015-09-29 18:29:12 (1 s)
----------------------------------------------------------------------------------------------------------------
publishes 15,230 ( 15,230 avg) messages/s 650.532 ( 650.532 avg) ms latency 15,224 total
receives 0 ( 0 avg) messages/s 0.000 ( 0.000 avg) ms latency 0 total
... warmup ...
2015-09-29 18:29:27 (16 s)
----------------------------------------------------------------------------------------------------------------
publishes 85,455 ( 79,706 avg) messages/s 588.480 ( 659.066 avg) ms latency 980,385 total
receives 78,382 ( 81,186 avg) messages/s 1,112.738 ( 1,319.294 avg) ms latency 939,036 total
2015-09-29 18:29:28 (17 s)
----------------------------------------------------------------------------------------------------------------
publishes 107,998 ( 84,596 avg) messages/s 597.299 ( 645.375 avg) ms latency 1,088,490 total
receives 103,196 ( 83,902 avg) messages/s 1,071.667 ( 1,212.498 avg) ms latency 1,042,383 total
...
```
Pulling Benchmark
-------------------
Note: This benchmark uses a lot of quota and network bandwidth.
Set the `GOOGLE_PUBSUB_SUBSCRIPTION` env var to the name of a subscription to consume from.
```
$ mvn exec:exec -Dexec.executable="java" -Dexec.classpathScope="test" -Dexec.args="-cp %classpath com.spotify.google.cloud.pubsub.client.integration.PullerBenchmark"
```
Releasing
---------
We tag releases on github and publish release jars to maven central hosted by
Sonatype:
### Prerequisites
1. Sonatype credentials for publishing to maven central. Apply for permission
to publish jars on the `com.spotify` group id.
See .
2. Add the sonatype credentials to `~/.m2/settings.xml`
ossrh
YOUR_SONATYPE_USER
YOUR_SONATYPE_PASS
3. Set up GnuPG. See .
Make sure that you've distributed your public key to a key server.
### Performing a Release
Have your GnuPG password ready. Both prepare and perform steps will ask you for it.
*Note:* The current tests run during both `prepare` and `perform` include
integration tests against the real Google Pub/Sub API. Verify
that you have a suitable default project and credentials
configured with the `gcloud` cli.
1. Tag and push a new release to github:
mvn release:prepare
2. Publish the signed jar to maven central:
mvn release:perform
Todo
----
* Implement a high level consumer (raw pull/ack support is there)
* Implement retries on auth failure