{"id":22374886,"url":"https://github.com/qubitproducts/mflowd","last_synced_at":"2026-02-25T16:03:25.675Z","repository":{"id":66964754,"uuid":"82174381","full_name":"QubitProducts/mflowd","owner":"QubitProducts","description":"A go daemon that collects monitoring metrics from Google Dataflow workers and exposes them to Prometheus","archived":false,"fork":false,"pushed_at":"2018-04-16T14:10:02.000Z","size":30,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-07-30T23:11:26.583Z","etag":null,"topics":["ceh","go","google-dataflow","google-pubsub","infra","monitoring","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QubitProducts.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-02-16T11:39:04.000Z","updated_at":"2023-06-02T11:48:33.000Z","dependencies_parsed_at":"2023-03-01T16:15:43.474Z","dependency_job_id":null,"html_url":"https://github.com/QubitProducts/mflowd","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/QubitProducts/mflowd","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QubitProducts%2Fmflowd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QubitProducts%2Fmflowd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QubitProducts%2Fmflowd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QubitProducts%2Fmflowd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QubitProducts","download_url":"https://codeload.github.com/QubitProducts/mflowd/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QubitProducts%2Fmflowd/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29829409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T15:41:19.027Z","status":"ssl_error","status_checked_at":"2026-02-25T15:40:47.150Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceh","go","google-dataflow","google-pubsub","infra","monitoring","prometheus"],"created_at":"2024-12-04T21:18:51.200Z","updated_at":"2026-02-25T16:03:25.670Z","avatar_url":"https://github.com/QubitProducts.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Metrics Flow Daemon ![Build Status](https://travis-ci.org/QubitProducts/mflowd.svg) ![Go Report](https://goreportcard.com/badge/github.com/QubitProducts/mflowd)\n\nIt is a small daemon that aggregates custom monitoring metrics collected from Google Dataflow workers that use [metrics-flow](https://github.com/QubitProducts/metrics-flow) library and exposes them to [Prometheus](https://prometheus.io/).\n\n# How it works\n\nGoogle Dataflow workers with `metrics-flow` plugged in pre-aggregate custom monitoring metrics using pipeline windowing functions and dump\nthe results into a Google Pub/Sub topic. The metrics flow daemon polls a subscription to the topic, converts received metric update events to `Prometheus` format and exposes them through `/metrics` endpoint.\n\n\n    +------------+                                                      \n    | Dataflow 1 +----+                                                 \n    +------------+    |                                                 \n                      |                                                 \n    +------------+    |                       +------------+            \n    | Dataflow i |----|-\u003e(Google Pub/Sub)----\u003e|   mflowd   |            \n    +------------+    |                       +------------+            \n                      |                              ^                  \n    +------------+    |                              |                  \n    | Dataflow N |----+                       +-------------+           \n    +------------+                            | Prometheus  |           \n                                              +-------------+           \n                                                                   \n                                                                    \n                                                                                                                  \n# Installation\n\n    % go get github.com/QubitProducts/mflowd\n\n## Build it from scratch\n\n1. Make sure you have `dep` installed (if you don't know how to install it, follow [this](https://github.com/golang/dep#installation) link)\n2. `make bootstrap`\n3. `make test`\n4. `make mflowd`\n\n# Running\n\n1. Create a [pub/sub topic](https://cloud.google.com/pubsub/docs/publisher#create) you will use for publishing metrics from your Dataflow workers (if you don't have one already).\n2. Create a [pull subscription](https://cloud.google.com/pubsub/docs/pull) to the topic\n3. Make sure you are authorized to use the subscription (if not sure, use [gcloud auth login](https://cloud.google.com/sdk/gcloud/reference/auth/login))\n4. Run the daemon\n\n       % ./mflowd [-v] -p \u003cport\u003e -s pubsub \u003csubscription_id\u003e\n\nWhere\n* `port` is a port where `/metrics` endpoint will be exposed\n* `subscription_path` is a subscription identifier which usually looks like `projects/\u003cproject_name\u003e/subscriptions/\u003csubcsciption_name\u003e`\n* use optional `-v` flag to run the daemon in verbose mode\n\n# Using Docker \n\nYou can easily build a \"containerized\" version of `mflowd` and run it on `mesos` or `kubernetes`. \n\n## Building\n\n    % make docker\n    % docker images | grep mflowd\n    mflowd                                             latest                                            e9cbac93f703\n    ...\n\n## Running\n\nBefore you can run the image you need to set up a Google Cloud API service account to allow `mflowd` use the subscription you have created. So\n\n1. [Create a service account](https://cloud.google.com/vision/docs/common/auth) for mflowd\n2. Create an empty directory on your host machine (say, `% mkdir ~/.mflowd`)\n3. Download the service account key in JSON format and put it to the created directory\n4. Finally run the countainer:\n\n       % docker run -e \"MFLOWD_SUB=\u003csubscription_id\u003e\" -v $HOME/.mflowd:/etc/mflowd 'mflowd:latest'\n\n## Using docker-compose\n\nYou can also run both `mflowd` and `prometheus` docker images using `docker-compose`:\n\n    % cd ~/go/src/github.com/QubitProducts/mflowd\n    % mkdir gcp\n    # download your service account JSON key to gcp directory\n    % cat \u003e .env\n    MFLOWD_SUB=\u003csubscription_id\u003e\n    MFLOWD_VERBOSE=0 # set to 1 to turn verbose mode on\n    ^C\n    % docker-compose up\n\nFollow http://localhost:9090 to get to `Prometheus` UI\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubitproducts%2Fmflowd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqubitproducts%2Fmflowd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqubitproducts%2Fmflowd/lists"}