{"id":18941432,"url":"https://github.com/mchmarny/kpush","last_synced_at":"2026-03-23T19:30:16.974Z","repository":{"id":64306946,"uuid":"162717831","full_name":"mchmarny/kpush","owner":"mchmarny","description":"Demo of signed message push from PubSub to Knative service","archived":false,"fork":false,"pushed_at":"2023-07-05T20:39:23.000Z","size":461,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-31T22:42:50.602Z","etag":null,"topics":["gcp","golang","knative","pubsub","security"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-21T13:32:53.000Z","updated_at":"2019-04-14T18:33:10.000Z","dependencies_parsed_at":"2024-06-20T14:43:14.438Z","dependency_job_id":"5c9e586b-6085-43db-905c-fd7d56adcf7d","html_url":"https://github.com/mchmarny/kpush","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fkpush","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fkpush/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fkpush/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fkpush/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/kpush/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239940402,"owners_count":19722014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gcp","golang","knative","pubsub","security"],"created_at":"2024-11-08T12:28:02.972Z","updated_at":"2026-03-23T19:30:16.909Z","avatar_url":"https://github.com/mchmarny.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kpush - push signed messages from PubSub to Knative service\n\nThe use of [GCP Cloud PubSub](https://cloud.google.com/pubsub/) is common component in data flow pipelines. It creates global and elastic separation between the data provider and data consumers. One of its features is the ability to [push](https://cloud.google.com/pubsub/docs/push) subscription which sends messages to the configured `webhook`. Consider the following solution:\n\n![kpush flow](img/kpush-flow.png)\n\nIn this case, the `kpush` client publishes messages to PubSub `topic` which the `subscription` subsequently pushes to target processing service, an app hosted on an instance of `Knative`. PubSub provides [instructions](https://cloud.google.com/pubsub/docs/push) on how to include [token](https://cloud.google.com/pubsub/docs/faq#security) in POST to ensure that only submissions from valid senders are processed by target service.\n\nSince this token can leak (e.g. logs), `kpush` demonstrates how to add an additional level of validation by signing each message on the client and validating that signature on the service side. The below PubSub message sample illustrates the `sig` attribute holding [SHA-1](https://en.wikipedia.org/wiki/SHA-1) of the `message.data` payload which is appended to each message.\n\n```json\n{\n    \"message\": {\n        \"attributes\": {\n            \"sig\": \"sha1=22c477fd1269c9d3bab8591b371a66976f10006e\"\n        },\n        \"data\": \"eyJpZC...\",\n        \"messageId\": \"333651121184341\",\n        \"publishTime\": \"2018-12-22T19:05:01.067Z\",\n    }\n}\n```\n\n## Setup\n\nThe `kpush` flow includes two components: `client`, generating, signing, and publishing mocked up messages, and `server` which receives PubSub pushed messages and validates their signature.\n\n\u003e Assuming `gcloud` SDK already configured. See [gcloud docs](https://cloud.google.com/sdk/gcloud/) for instructions if you need assistance\n\n### PubSub topic\n\nLet's start by creating the PubSub topic named `kpush` to which our client will be publishing messages.\n\n```shell\ngcloud pubsub topics create kpush\n```\n\nThe response should be\n\n```shell\nCreated topic [projects/YOUR_PROJECT_ID/topics/kpush].\n```\n\n### Knative Service\n\nThe installation and configuration of [Knative](https://github.com/knative) are beyond the scope of this readme, but, you can find detailed instructions on how to configure it on [Kubernetes](https://kubernetes.io/) service offered by most Cloud Service Providers [here](https://github.com/knative/docs/tree/master/install). In this example I will be using Google's [Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) (GKE) which now can be easily configured with the validated version of Knative with a single checkbox.\n\nQuickest way to build your service image is through [GCP Build](https://cloud.google.com/cloud-build/). Just submit the build request from within the `kpush` directory:\n\n```shell\ngcloud builds submit \\\n    --project ${GCP_PROJECT_ID} \\\n\t--tag gcr.io/${GCP_PROJECT_ID}/kpush-server:latest\n```\n\nThe build service is pretty verbose in output but eventually you should see something like this\n\n```shell\nID           CREATE_TIME          DURATION  SOURCE                                   IMAGES                      STATUS\n6905dd3a...  2018-12-23T03:48...  1M43S     gs://PROJECT_cloudbuild/source/15...tgz  gcr.io/PROJECT/kpush-server SUCCESS\n```\n\nThe `IMAGE` column will have the repository URI for your new image (e.g. `gcr.io/PROJECT/kpush-server`)\n\nBefore we can deploy that service to Knative, we just need to update the `deploy/server.yaml` file\n\nFirst, update the container `image` to the URI from the build `IMAGE` column value. Then update the two environment variables values:\n\n* `KNOWN_PUBLISHER_TOKENS` which holds the token we will use in PubSub URL\n* `MSG_SIG_KEY` which should be the key used by your clients to sign published messages\n\n\u003e Note, `KNOWN_PUBLISHER_TOKENS` may include many tokens separated by comma\n\nFinally, to deploy `kpush` service to Knative apply the updated deployment using following command:\n\n```shell\nkubectl apply -f deploy/server.yaml\n```\n\nThe response should be\n\n```shell\nservice.serving.knative.dev \"pushme\" configured\n```\n\nTo check if the service was deployed successfully you can check the status using `kubectl get pods` command. The response should look something like this (e.g. Ready `3/3` and Status `Running`).\n\n```shell\nNAME                                          READY     STATUS    RESTARTS   AGE\npushme-00002-deployment-5645f48b4d-mb24j      3/3       Running   0          4h\n```\n\nKnative uses convention to build serving URL by combining the deployment name (e.g. `pushme`), namespace name (e.g. `demo`), and the pre-configured domain name (e.g. `knative.tech`). The resulting URL should look something like this\n\n```shell\nhttps://pushme.default.knative.tech\n```\n\nGo ahead, test it in browser, you should following JSON response:\n\n```json\n{ \"handlers\": [ \"POST: /post\" ] }\n```\n\nTarget of PubSub push must also be an HTTPS server with non-self-signed certificate. The instructions on how to configure Knative with SSL certificate are located [here](https://github.com/knative/docs/blob/master/serving/using-an-ssl-cert.md).\n\n\u003e Note, PubSub will publish only to registering endpoints which will require you to A) Verify you have access to the domain, and B) Register your domain in APIs \u0026 services. Instructions for both can be set up [here](https://cloud.google.com/pubsub/docs/push)\n\n### PubSub Subscription\n\nThe final component we need to set up is the PubSub subscription which will push messages to our Knative hosted service. the `MYTOKEN` should be set to one of the tokens you defined in the Knative service manifest environment variables (`KNOWN_PUBLISHER_TOKENS`).\n\n```shell\ngcloud pubsub subscriptions create kpush-sub \\\n    --topic kpush \\\n    --push-endpoint https://pushme.demo.knative.tech/push?publisherToken=${MYTOKEN} \\\n    --ack-deadline 30\n```\n\n\u003e Note, cold-starts on Knative can be sometimes slow so we added the `ack-deadline=30` parameter. For additional info on PubSub subscription options see this [doc](* https://cloud.google.com/pubsub/docs/subscriber)\n\n\n## Run\n\nTo demo the entire pipeline, `kpush` includes a client which will mock up a few messages, sign them, and submit them to the configured topic.\n\nNavigate to the `client` directory...\n\n```shell\ncd cmd/client\n```\n\n...and run the following command:\n\n```shell\ngo run main.go --project ${PROJECT_ID} \\\n               --key ${MSG_SIG_KEY} \\\n               --topic kpush\n```\n\nThe response should look something like this\n\n```shell\n2018/12/23 13:58:44 Using topic publisher: YOUR-GCP-PROJECT-NAME:kpush\n   status: published msg[0] eacc9be0-06fd-11e9-868f-acde48001122\n   status: published msg[1] eb2e0cae-06fd-11e9-868f-acde48001122\n   status: published msg[2] eb39aaa0-06fd-11e9-868f-acde48001122\nsent 3 messages\n```\n\n### Monitoring\n\nA successful end-to-end run of the pipeline will result in `202` status code from the Knative hosted service. To validate you can either query the Knative logs or navigate to the [GCP Stackdriver](https://cloud.google.com/stackdriver/) console to review the PubSub subscription panel (Resources \u003e PubSub \u003e Subscriptions)\n\n![kpush flow](img/kpush-chart.png)\n\n## Direct Push\n\n`kpush` also includes an `HTTPPublisher` which can skip the PubSub entirely and submit signed messages directly to the Knative service. To do that include `url` parameter in the `kpush client` command like this\n\n```shell\ngo run main.go --project ${PROJECT_ID} \\\n               --key ${MSG_SIG_KEY} \\\n               --url https://pushme.demo.knative.tech/push?publisherToken=${MYTOKEN} \\\n               --messages 10\n```\n\n## How to use kpush in your project\n\nYou can import `kpush` into your project as a package\n\n```shell\ngo get -u github.com/mchmarny/kpush\n```\n\nFor demonstration how you can use `kpush` in your project see provided [example](https://github.com/mchmarny/kpush/blob/master/example/main.go).\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fkpush","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Fkpush","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fkpush/lists"}