{"id":18941475,"url":"https://github.com/mchmarny/xstreams","last_synced_at":"2026-05-17T09:46:01.166Z","repository":{"id":77051958,"uuid":"191859941","full_name":"mchmarny/xstreams","owner":"mchmarny","description":"Stream processing using Cloud PubSub and Dataflow SQL in BigQuery","archived":false,"fork":false,"pushed_at":"2019-06-16T02:11:35.000Z","size":3061,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-08-24T12:45:09.559Z","etag":null,"topics":["bigquery","dataflow","gce","gcp","golang","pubsub","stream"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mchmarny.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-14T02:01:15.000Z","updated_at":"2021-11-04T21:55:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"efa7038d-bc6c-458a-8f36-0bccf27ca28a","html_url":"https://github.com/mchmarny/xstreams","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/mchmarny/xstreams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fxstreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fxstreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fxstreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fxstreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mchmarny","download_url":"https://codeload.github.com/mchmarny/xstreams/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mchmarny%2Fxstreams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279016268,"owners_count":26085827,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-13T02:00:06.723Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","dataflow","gce","gcp","golang","pubsub","stream"],"created_at":"2024-11-08T12:28:12.478Z","updated_at":"2025-10-13T17:02:26.944Z","avatar_url":"https://github.com/mchmarny.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# xstreams\n\nHow to joins multiple event streams using Cloud PubSub and Cloud Dataflow SQL in BigQuery.\n\nThis demo will cover:\n\n* [Generate Events](#generate-events) how to generator synthetic temperature (Celsius) and vibration (mm/s2) metrics and publish them onto Cloud PubSub topic\n* [Process Events](#process-events) how to process the two event streams and use Cloud Dataflow windowing function to join the two unbounded event streams\n* [Analyze Data](#analyze-data) how to use BigQuery SQL to analyze the resulting event data\n\nAdditionally, we will visualize the time-series data using the new data source plugin for BigQuery\n\n![alt text](image/grafana.png \"Grafana Chart\")\n\n## Generate Events\n\nThe included `xstreams` data generation utility publishes mocked up events onto PubSub topic in form of a JSON payload for both temperature and friction:\n\n```json\n{\n    \"source_id\": \"device1\",\n    \"event_id\": \"eid-243303fe-bc7c-4f3e-b9fd-7dc1929f9ae9\",\n    \"event_time\": 1560484054,\n    \"metric\": \"temperature\",\n    \"value\": 17.48582634964145\n}\n```\n\nWhile you can run the `xstreams` data generation utility locally, the simplest way to generate sustained stream of events is by using Cloud compute resources. To do that we will use the `glcoud` CLI to spin up a small VM with the `xstreams` container.\n\n\u003e For more information how to install and configure `glcoud` see [here](https://cloud.google.com/sdk/install)\n\n\n```shell\ngcloud compute instances create-with-container xstreams \\\n    --container-image=gcr.io/cloudylabs-public/xstreams:0.1.1 \\\n    --zone=us-central1-c \\\n    --image-family=cos-stable \\\n    --image-project=cos-cloud \\\n    --scopes=cloud-platform \\\n    --container-env=\"GOOGLE_APPLICATION_CREDENTIALS=/tmp/xstreams.pem\" \\\n    --container-mount-host-path=mount-path=/tmp,host-path=/tmp,mode=rw\n```\n\nThis command will deploy the prebuilt image into a VM on GCE. To complete the install you will have to also upload also a GCP service account file so that the container can start.\n\n```shell\ngcloud compute scp $GCP_KEYS/xstreams.pem xstreams:/tmp/xstreams.pem\n```\n\n\u003e Your default GCP credentials can do a lot more than `xstreams` needs. To follow the least privilege principle, you should create a brand new service account that has only the necessary `Pub/Sub Editor` role. For more information on how to crate a service account and generate its keys see [here](https://cloud.google.com/iam/docs/creating-managing-service-accounts)\n\nTo monitor logs output from `xstreams` container you will need to capture the instance ID first\n\n```shell\nINSTANCE_ID=$(gcloud compute instances describe xstreams --zone us-central1-c --format=\"value(id)\")\n```\n\nThen you can output the logs using this command:\n\n```shell\ngcloud logging read \"resource.type=gce_instance \\\n    AND logName=projects/cloudylabs/logs/cos_containers \\\n    AND resource.labels.instance_id=${INSTANCE_ID}\"\n```\n\n\n## Process Events\n\n\n### Raw Events\n\nFirst we are going to load the raw event data from the two PubSub topics into single BigQuery table. Let's create the BigQuery dataset and table:\n\n```shell\nbq mk xstreams\nbq query --use_legacy_sql=false \"\nCREATE OR REPLACE TABLE xstreams.raw_events (\n  source_id STRING NOT NULL,\n  event_id STRING NOT NULL,\n  event_time INT64 NOT NULL,\n  metric STRING NOT NULL,\n  value FLOAT64 NOT NULL\n)\"\n```\n\nOnce the table is created, we can create Dataflow job to drain the payloads from the `eventmakertemp` and `eventmakervibe` topics to `raw_events` table in BigQuery\n\n\n```shell\nPROJECT=$(gcloud config get-value project)\n\ngcloud dataflow jobs run xstreams-raw-tepm-events \\\n  --gcs-location gs://dataflow-templates/latest/PubSub_to_BigQuery \\\n  --parameters \"inputTopic=projects/${PROJECT}/topics/eventmakertemp,outputTableSpec=${PROJECT}:xstreams.raw_events\"\n\ngcloud dataflow jobs run xstreams-raw-vibe-events \\\n  --gcs-location gs://dataflow-templates/latest/PubSub_to_BigQuery \\\n  --parameters \"inputTopic=projects/${PROJECT}/topics/eventmakervibe,outputTableSpec=${PROJECT}:xstreams.raw_events\"\n```\n\nDataflow will take a couple of minutes to create the necessary resources. When done, you will see data in the `kadvice.raw_events` table in BigQuery.\n\n### Windowing\n\nOne of the most common operations in unbounded event stream processing is grouping events by slicing them into period window based on the timestamp of each event. The simplest form of windowing is tumbling which uses consistent duration to group events into non-overlapping time interval. Since our synthetic events have only a one event time, the `tumbling window` is a natural fit.\n\n\u003e You can read more about Windowing [here](https://cloud.google.com/dataflow/docs/guides/sql/streaming-pipeline-basics)\n\nIn the Alpha release of Cloud Dataflow SQL in BigQuery joining of multiple unbounded event streams doesn't seem to be yet supported. So, we are going to create two separate jobs: `temperature` and `vibration`.\n\n#### Temperature\n\nExecute the following query with the Dataflow engine and save the results to `xstreams.temp_tumble_30`.\n\nNotice the use of `TUMBLE_START` function which will return the starting timestamp of each one of our events. The use of event payload timestamps is not supported yet, so for now, we are going to use the time when this event was published onto PubSub topic (`event_timestamp` of the PubSub wrapper).\n\n```sql\nSELECT\n   t.payload.source_id,\n   TUMBLE_START(\"INTERVAL 30 SECOND\") AS period_start,\n\n   MIN(t.payload.value) as min_temp_period_val,\n   AVG(t.payload.value) as avg_temp_period_val,\n   MAX(t.payload.value) as max_temp_period_val,\n   MIN(t.payload.event_time) as min_temp_event_time,\n   MAX(t.payload.event_time) as max_temp_event_time\n\n FROM pubsub.topic.cloudylabs.eventmakertemp t\n GROUP BY\n   t.payload.source_id,\n   TUMBLE(t.event_timestamp, \"INTERVAL 30 SECOND\")\n```\n\n#### Vibration\n\nSimilarly execute the following with the Dataflow engine and save the results to `xstreams.vibe_tumble_30`.\n\n```sql\nSELECT\n   v.payload.source_id,\n   TUMBLE_START(\"INTERVAL 30 SECOND\") AS period_start,\n\n   MIN(v.payload.value) as min_vibe_period_val,\n   AVG(v.payload.value) as avg_vibe_period_val,\n   MAX(v.payload.value) as max_vibe_period_val,\n   MIN(v.payload.event_time) as min_vibe_event_time,\n   MAX(v.payload.event_time) as max_vibe_event_time\n\n FROM pubsub.topic.cloudylabs.eventmakervibe v\n GROUP BY\n   v.payload.source_id,\n   TUMBLE(v.event_timestamp, \"INTERVAL 30 SECOND\")\n```\n\n## Analyze Data\n\nNow that we have both `temp_tumble_30` and `vibe_tumble_30` tables created, we can switch back to BigQuery query engine to analyze the data using SQL.\n\nHere is an \"uber query\" example that joins temperature and vibrations using `source_id` and `period_start` to:\n\n* find `min`, `avg`, and `max` values for each event in each `30 sec` period\n* compare `period_start` to the event time to derive `min` and `max` delta\n* find `min`, `max` event time within each processing period\n\n```sql\nSELECT\n  t.period_start,\n  t.min_temp_period_val,\n  t.avg_temp_period_val,\n  t.max_temp_period_val,\n  TIMESTAMP_SECONDS(t.min_temp_event_time) as min_temp_event_time,\n  TIMESTAMP_SECONDS(t.max_temp_event_time) AS max_temp_event_time,\n  TIMESTAMP_DIFF(TIMESTAMP_SECONDS(t.min_temp_event_time), t.period_start, SECOND) as first_temp_period_event_time,\n  TIMESTAMP_DIFF(TIMESTAMP_SECONDS(t.max_temp_event_time), t.period_start, SECOND) as last_temp_period_event_time,\n  v.min_vibe_period_val,\n  v.avg_vibe_period_val,\n  v.max_vibe_period_val,\n  TIMESTAMP_SECONDS(v.min_vibe_event_time) as min_vibe_event_time,\n  TIMESTAMP_SECONDS(v.max_vibe_event_time) as max_vibe_event_time,\n  TIMESTAMP_DIFF(TIMESTAMP_SECONDS(v.min_vibe_event_time), t.period_start, SECOND) as first_vibe_period_event_time,\n  TIMESTAMP_DIFF(TIMESTAMP_SECONDS(v.max_vibe_event_time), t.period_start, SECOND) as last_vibe_period_event_time\nFROM xstreams.vibe_tumble_30 v\nINNER JOIN xstreams.temp_tumble_30 t ON\n  t.source_id = v.source_id AND\n  t.period_start = v.period_start\nORDER BY v.period_start DESC\n```\n\nSimilarly, to identify temperatures in current period that are above the 10 minute average:\n\n```sql\nSELECT\n  TIMESTAMP_SECONDS(event_time) as event_time,\n  source_id,\n  value\nFROM xstreams.raw_events\nWHERE\n  metric = 'temperature'\n  AND TIMESTAMP_SECONDS(event_time) \u003e (\n    SELECT MAX(period_start)\n    FROM xstreams.temp_tumble_30\n  )\n  AND value \u003e (\n    SELECT AVG(avg_temp_period_val)\n    FROM xstreams.temp_tumble_30\n    WHERE period_start \u003e TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 10 MINUTE)\n  )\n```\n\nAnd finally to identify outliers for either temperature of vibration for a given period within last 5 minutes you can run this query:\n\n```sql\nSELECT\n  TIMESTAMP_SECONDS(e.event_time) as event_time,\n  e.source_id,\n  e.metric,\n  e.value,\n  CASE\n    WHEN e.metric = 'temperature' THEN AVG(t.avg_temp_period_val)\n    WHEN e.metric = 'vibration' THEN AVG(v.avg_vibe_period_val)\n    ELSE 0\n  END as avg_period_value,\n  CASE\n    WHEN e.metric = 'temperature' THEN e.value - AVG(t.avg_temp_period_val)\n    WHEN e.metric = 'vibration' THEN e.value - AVG(v.avg_vibe_period_val)\n    ELSE 0\n  END as avg_period_delta\nFROM xstreams.raw_events e\nINNER JOIN xstreams.temp_tumble_30 t ON e.source_id = t.source_id\n  AND e.event_time BETWEEN t.min_temp_event_time AND t.max_temp_event_time\nINNER JOIN xstreams.vibe_tumble_30 v ON e.source_id = v.source_id\n  AND e.event_time BETWEEN v.min_vibe_event_time AND v.max_vibe_event_time\nWHERE\n  TIMESTAMP_SECONDS(e.event_time) \u003e TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 MINUTE)\nGROUP BY\n  e.event_time,\n  e.source_id,\n  e.metric,\n  e.value\nORDER BY 1 DESC\n```\n\nYou can obviously customize each one of these or write your own, more interesting, SQL queries.\n\n## Visualization\n\nTo use show the temperature and vibrations time-series data in BigQuery you will need to install the plugin. To do so, follow one of the ways outlined [here](https://github.com/doitintl/bigquery-grafana/blob/master/INSTALL.md). After that, you can follow the data source configuration steps outlined [here](https://github.com/doitintl/bigquery-grafana).\n\nWhen creating query in Grafana, you can either follow the wizard or just paste the following select statements into each SQL query editor window\n\nTemperature Query\n\n```sql\n#standardSQL\nSELECT\n period_start AS time,\n  source_id AS metric,\n  avg_temp_period_val\nFROM `cloudylabs.xstreams.temp_tumble_30`\nWHERE\n  $__timeFilter(period_start)\nORDER BY 1,2\n```\n\nVibration Query\n\n```sql\n#standardSQL\nSELECT\n period_start AS time,\n  source_id AS metric,\n  avg_vibe_period_val\nFROM `cloudylabs.xstreams.vibe_tumble_30`\nWHERE\n  $__timeFilter(period_start)\nORDER BY 1,2\n```\n\nThen save by clicking the save button at the top right side of the screen.\n\n![alt text](image/grafana-query.png \"Grafana Query\")\n\n## Cost\n\nThere is a pretty generous [free tier](https://cloud.google.com/free/) on GCP\n\n* PubSub - is priced based on data volume transmitted in a calendar month (first 10GB free). For more information see [PubSub Pricing](https://cloud.google.com/pubsub/pricing)\n* GCE - the `n1-standard-1` VM used in this example is $0.0475/hr. For more information see [Compute Pricing](https://cloud.google.com/compute/pricing)\n* Dataflow an BigQuery - pricing for Dataflow an BigQuery is more complicated as it is based on usage of vCPU, RAM, Storage, and data processing per GB. For details see [Cloud Dataflow pricing](https://cloud.google.com/dataflow/pricing) and [BigQuery pricing](https://cloud.google.com/bigquery/pricing)\n\n## Cleanup\n\nTo delete the data generation VM\n\n```shell\ngcloud compute instances delete xstreams --zone=us-central1-c\n```\n\nTo delete the two topics created by `xstreams`\n\n```shell\ngcloud pubsub topics delete eventmakertemp\ngcloud pubsub topics delete eventmakervibe\n```\n\nTo delete BigQuery dataset created by `xstreams`\n\n\n```shell\nbq rm -r -f xstreams\n```\n\nTo stop all the Dataflow jobs created in this demo (unless you named them explicitly), you will have to go to the Google Cloud Console and delete them manually\n\n\n## Disclaimer\n\nThis is my personal project and it does not represent my employer. I take no responsibility for issues caused by this code. I do my best to ensure that everything works, but if something goes wrong, my apologies is all you will get.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fxstreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmchmarny%2Fxstreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmchmarny%2Fxstreams/lists"}