{"id":13449041,"url":"https://github.com/spotify/async-google-pubsub-client","last_synced_at":"2025-07-02T07:06:07.141Z","repository":{"id":34308851,"uuid":"38225634","full_name":"spotify/async-google-pubsub-client","owner":"spotify","description":"[SUNSET] Async Google Pubsub Client","archived":false,"fork":false,"pushed_at":"2023-03-18T19:53:00.000Z","size":359,"stargazers_count":158,"open_issues_count":8,"forks_count":36,"subscribers_count":105,"default_branch":"master","last_synced_at":"2025-04-08T01:51:16.766Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spotify.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-06-29T03:24:39.000Z","updated_at":"2024-10-15T02:54:39.000Z","dependencies_parsed_at":"2023-02-14T22:31:42.344Z","dependency_job_id":"1bbc31c1-bd4e-48f6-81c5-832aa01d4600","html_url":"https://github.com/spotify/async-google-pubsub-client","commit_stats":null,"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"purl":"pkg:github/spotify/async-google-pubsub-client","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fasync-google-pubsub-client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fasync-google-pubsub-client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fasync-google-pubsub-client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fasync-google-pubsub-client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spotify","download_url":"https://codeload.github.com/spotify/async-google-pubsub-client/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spotify%2Fasync-google-pubsub-client/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263091026,"owners_count":23412343,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T06:00:29.317Z","updated_at":"2025-07-02T07:06:07.094Z","avatar_url":"https://github.com/spotify.png","language":"Java","funding_links":[],"categories":["Java","进程间通信"],"sub_categories":[],"readme":"**NOTE: This library is sunsetting. Please consider using the official [Google Cloud Java Client for Pub/Sub](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-clients/google-cloud-pubsub).**\n\nasync-google-pubsub-client\n==========================\n\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.spotify/async-google-pubsub-client/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.spotify/async-google-pubsub-client) [![Build Status](https://travis-ci.org/spotify/async-google-pubsub-client.svg?branch=master)](https://travis-ci.org/spotify/async-google-pubsub-client) [![codecov.io](http://codecov.io/github/spotify/async-google-pubsub-client/coverage.svg?branch=master)](http://codecov.io/github/spotify/async-google-pubsub-client?branch=master)\n\nA performant [Google Cloud Pub/Sub](https://cloud.google.com/pubsub/) client and batch publisher.\n\nWhat\n----\n\nA low level Pub/Sub client and a concurrent per-topic batching Publisher.\n\nThe client uses async-http-client with the Netty provider for making efficient and async HTTP requests to the Google Cloud Pub/Sub api. \n\nThe publisher is implemented on top of the async Pub/Sub client and concurrently gathers individual messages into per-topic batches which are then pushed to Google Cloud Pub/Sub at a specified desired request concurrency level in order to achieve both low-latency and high throughput.\n\nWhy\n---\nThe official Google Cloud Pub/Sub client library was not performant enough for our purposes due to blocking I/O etc.\n\nUsage\n-----\n\n### Pubsub Client\n\n```java\n// Create a topic\npubsub.createTopic(\"my-google-cloud-project\", \"the-topic\").get();\n\n// Create a subscription\npubsub.createSubscription(\"my-google-cloud-project\", \"the-subscription-name\", \"the-topic\").get();\n\n// Create a batch of messages\nfinal List\u003cMessage\u003e messages = asList(\n    Message.builder()\n        .attributes(\"type\", \"foo\")\n        .data(encode(\"hello foo\"))\n        .build(),\n    Message.builder()\n        .attributes(\"type\", \"bar\")\n        .data(encode(\"hello foo\"))\n        .build());\n\n// Publish the messages\nfinal List\u003cString\u003e messageIds = pubsub.publish(\"my-google-cloud-project\", \"the-topic\", messages).get();\nSystem.out.println(\"Message IDs: \" + messageIds);\n\n// Pull the message\nfinal List\u003cReceivedMessage\u003e received = pubsub.pull(\"my-google-cloud-project\", \"the-subscription\").get();\nSystem.out.println(\"Received Messages: \" + received);\n\n// Ack the received messages\nfinal List\u003cString\u003e ackIds = received.stream().map(ReceivedMessage::ackId).collect(Collectors.toList());\npubsub.acknowledge(\"my-google-cloud-project\", \"the-subscription\", ackIds).get();\n```\n\n### Publisher\n\n```java\nfinal Pubsub pubsub = Pubsub.builder()\n    .build();\n\nfinal Publisher publisher = Publisher.builder()\n    .pubsub(pubsub)\n    .project(\"my-google-cloud-project\")\n    .concurrency(128)\n    .build();\n\n// A never ending stream of messages...\nfinal Iterable\u003cMessageAndTopic\u003e messageStream = incomingMessages();\n\n// Publish incoming messages\nmessageStream.forEach(m -\u003e publisher.publish(m.topic, m.message));\n```\n\n### Puller\n\n```java\nfinal Pubsub pubsub = Pubsub.builder()\n    .build();\n\nfinal MessageHandler handler = (puller, subscription, message, ackId) -\u003e {\n  System.out.println(\"got message: \" + message);\n  return CompletableFuture.completedFuture(ackId);\n};\n\nfinal Puller puller = builder()\n    .pubsub(pubsub)\n    .project(\"my-google-cloud-project\")\n    .subscription(\"my-subscription\")\n    .concurrency(32)\n    .messageHandler(handler)\n    .build();\n```\n\n### `pom.xml`\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.spotify\u003c/groupId\u003e\n  \u003cartifactId\u003easync-google-pubsub-client\u003c/artifactId\u003e\n  \u003cversion\u003e1.31\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\nPublisher Benchmark\n-------------------\n\nNote: This benchmark uses a lot of quota and network bandwidth.\n\n```\n$ mvn exec:exec -Dexec.executable=\"java\" -Dexec.classpathScope=\"test\" -Dexec.args=\"-cp %classpath com.spotify.google.cloud.pubsub.client.integration.PublisherBenchmark\"\n[INFO] Scanning for projects...\n[INFO] Inspecting build with total of 1 modules...\n[INFO] Installing Nexus Staging features:\n[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin\n[INFO]\n[INFO] ------------------------------------------------------------------------\n[INFO] Building async-google-pubsub-client 1.13-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ async-google-pubsub-client ---\n2015-09-29 18:31:44 (1 s)\n----------------------------------------------------------------------------------------------------------------\npublishes         1,235 (    1,235 avg) messages/s      934.888 (     934.888 avg) ms latency        1,237 total\n\n... warmup ...\n\n2015-09-29 18:31:53 (10 s)\n----------------------------------------------------------------------------------------------------------------\npublishes       198,902 (  156,137 avg) messages/s      503.912 (     620.260 avg) ms latency    1,565,391 total\n\n2015-09-29 18:31:54 (11 s)\n----------------------------------------------------------------------------------------------------------------\npublishes       212,755 (  177,264 avg) messages/s      475.023 (     602.638 avg) ms latency    1,778,331 total\n\n...\n```\n\nEnd To End Benchmark\n-------------------\n\nNote: This benchmark uses a lot of quota and network bandwidth.\n\n```\n$ mvn exec:exec -Dexec.executable=\"java\" -Dexec.classpathScope=\"test\" -Dexec.args=\"-cp %classpath com.spotify.google.cloud.pubsub.client.integration.EndToEndBenchmark\"\n[INFO] Scanning for projects...\n[INFO] Inspecting build with total of 1 modules...\n[INFO] Installing Nexus Staging features:\n[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin\n[INFO]\n[INFO] ------------------------------------------------------------------------\n[INFO] Building async-google-pubsub-client 1.13-SNAPSHOT\n[INFO] ------------------------------------------------------------------------\n[INFO]\n[INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ async-google-pubsub-client ---\n2015-09-29 18:29:12 (1 s)\n----------------------------------------------------------------------------------------------------------------\npublishes        15,230 (   15,230 avg) messages/s      650.532 (     650.532 avg) ms latency       15,224 total\n receives             0 (        0 avg) messages/s        0.000 (       0.000 avg) ms latency            0 total\n\n... warmup ...\n\n2015-09-29 18:29:27 (16 s)\n----------------------------------------------------------------------------------------------------------------\npublishes        85,455 (   79,706 avg) messages/s      588.480 (     659.066 avg) ms latency      980,385 total\n receives        78,382 (   81,186 avg) messages/s    1,112.738 (   1,319.294 avg) ms latency      939,036 total\n\n2015-09-29 18:29:28 (17 s)\n----------------------------------------------------------------------------------------------------------------\npublishes       107,998 (   84,596 avg) messages/s      597.299 (     645.375 avg) ms latency    1,088,490 total\n receives       103,196 (   83,902 avg) messages/s    1,071.667 (   1,212.498 avg) ms latency    1,042,383 total\n...\n```\n\nPulling Benchmark\n-------------------\n\nNote: This benchmark uses a lot of quota and network bandwidth.\n\nSet the `GOOGLE_PUBSUB_SUBSCRIPTION` env var to the name of a subscription to consume from.\n\n```\n$ mvn exec:exec -Dexec.executable=\"java\" -Dexec.classpathScope=\"test\" -Dexec.args=\"-cp %classpath com.spotify.google.cloud.pubsub.client.integration.PullerBenchmark\"\n```\n\nReleasing\n---------\n\nWe tag releases on github and publish release jars to maven central hosted by\nSonatype: \u003chttp://central.sonatype.org\u003e\n\n### Prerequisites\n\n\n1. Sonatype credentials for publishing to maven central. Apply for permission\n   to publish jars on the `com.spotify` group id.\n   See \u003chttp://central.sonatype.org/pages/ossrh-guide.html\u003e.\n\n2. Add the sonatype credentials to `~/.m2/settings.xml`\n\n        \u003cserver\u003e\n          \u003cid\u003eossrh\u003c/id\u003e\n          \u003cusername\u003eYOUR_SONATYPE_USER\u003c/username\u003e\n          \u003cpassword\u003eYOUR_SONATYPE_PASS\u003c/password\u003e\n        \u003c/server\u003e\n\n3. Set up GnuPG. See \u003chttp://central.sonatype.org/pages/working-with-pgp-signatures.html\u003e.\n   Make sure that you've distributed your public key to a key server.\n\n\n### Performing a Release\n\nHave your GnuPG password ready. Both prepare and perform steps will ask you for it.\n\n*Note:* The current tests run during both `prepare` and `perform` include\n        integration tests against the real Google Pub/Sub API. Verify\n        that you have a suitable default project and credentials\n        configured with the `gcloud` cli.\n\n1. Tag and push a new release to github:\n\n        mvn release:prepare\n\n2. Publish the signed jar to maven central:\n\n        mvn release:perform\n\n\nTodo\n----\n* Implement a high level consumer (raw pull/ack support is there)\n* Implement retries on auth failure\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fasync-google-pubsub-client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fasync-google-pubsub-client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspotify%2Fasync-google-pubsub-client/lists"}