{"id":24087911,"url":"https://github.com/tinybirdco/tinybird-org-metrics-exporter","last_synced_at":"2025-05-05T21:41:01.672Z","repository":{"id":267314929,"uuid":"897213351","full_name":"tinybirdco/tinybird-org-metrics-exporter","owner":"tinybirdco","description":null,"archived":false,"fork":false,"pushed_at":"2025-01-08T16:10:17.000Z","size":954,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-01T18:58:20.949Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-02T08:37:45.000Z","updated_at":"2025-02-09T03:10:59.000Z","dependencies_parsed_at":"2024-12-09T17:24:15.658Z","dependency_job_id":"076159dd-8b4a-4e7b-be1c-fa6dc96ac60f","html_url":"https://github.com/tinybirdco/tinybird-org-metrics-exporter","commit_stats":null,"previous_names":["tinybirdco/tinybird-org-metrics-exporter"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Ftinybird-org-metrics-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Ftinybird-org-metrics-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Ftinybird-org-metrics-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Ftinybird-org-metrics-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/tinybird-org-metrics-exporter/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252581811,"owners_count":21771572,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:55:54.684Z","updated_at":"2025-05-05T21:41:01.650Z","avatar_url":"https://github.com/tinybirdco.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tinybird Organization Metrics Exporter\n\nThis project uses Tinybird's [Service Data Sources](https://www.tinybird.co/docs/monitoring/organizations#organization-service-data-sources) to aggregate and publish organizational metrics as endpoints in [Prometheus format](https://www.tinybird.co/docs/guides/integrations/consume-api-endpoints-in-prometheus-format) for quick integration with common monitoring tools.\n\n## Features\n\n- Includes a set of Pipes that consolidate and process organizational metrics.  \n- Exposes all key metrics through a single Pipe Endpoint in Prometheus format for streamlined monitoring.  \n- Provides essential insights to help you monitor your usage, detect anomalies, and set up alerts in critical areas such as:  \n  - Data ingestion: Track the volume and frequency of ingested data.  \n  - Pipe endpoint usage: Analyze requests and identify unusual traffic patterns.  \n  - Storage: Monitor storage usage to optimize resources and avoid limits.  \n  - Jobs: Keep track of the status and performance of scheduled jobs.  \n\n## Get started\n\n### Prerequisites\n\n1. You must have a [Tinybird organization](https://www.tinybird.co/docs/monitoring/organizations).\n2. You need to be an admin of the organization to access [organization Service Data Sources](https://www.tinybird.co/docs/monitoring/organizations#organization-service-data-sources).\n\n### Setup\n\n#### Deploy to a Tinybird Workspace\n\nTo deploy this template on Tinybird, click the following button:\n\n[![Deploy to Tinybird](https://cdn.tinybird.co/static/images/Tinybird-Deploy-Button.svg)](https://app.tinybird.co?starter_kit=https://github.com/tinybirdco/tinybird-org-metrics-exporter)\n\n#### Use the Prometheus endpoint  \n\nAccess the Prometheus metrics endpoint at `https://api.tinybird.co/v0/pipes/organization_metrics.prometheus`.\n\n- Replace `api.tinybird.co` with your Tinybird host if the workspace is in a different region. See [Regions and endpoints](https://www.tinybird.co/docs/api-reference#regions-and-endpoints).\n- **Token**: for a quick check in development, you can use the pre-existing *admin `user@domain` Token* of an Organization admin to authenticate requests. But for production, the creation of a *new token* with less permissions is recommended, including these scopes:\n```\n  PIPES:READ:organization_metrics\n  ORG_DATASOURCES:READ\n```\n- As of this writing, the `ORG_DATASOURCES:READ` is not available from UI, but you can create this new **prometheus_org_access** token using tokens API like this:\n```shell\nPOST https://api.tinybird.co/v0/tokens/?name=prometheus_org_access\u0026description=optional\u0026scope=PIPES:READ:organization_metrics\u0026scope=ORG_DATASOURCES:READ\n```\n\nTo scrape the Tinybird metrics endpoint, you can configure your `prometheus.yml` file as follows:\n\n```yaml\nscrape_configs:\n  - job_name: tinybird_org_metrics\n    scrape_interval: 15s  # Adjust the scrape interval as needed\n    scheme: 'https'\n    static_configs:\n      - targets: \n        - 'api.tinybird.co'  # Adjust this for your region if necessary\n    metrics_path: '/v0/pipes/organization_metrics.prometheus'\n    bearer_token: '\u003cadmin-user-token\u003e'  # From an Organization admin\n```\n\nIf you use a different setup, such as Datadog and OpenMetrics, you can use the following configuration:\n\n```yaml\ninstances:\n  - prometheus_url: https://api.tinybird.co/v0/pipes/organization_metrics.prometheus?token=\u003cadmin-user-token\u003e\n    namespace: tinybird\n    metrics:\n      - \"*\"\n    tags:\n      - tinybird\n    max_returned_metrics: 10000 # Adjust this value as needed\n```\n\nRemember to replace `api.tinybird.co` with your Tinybird host if the Workspace is in a different region. See [Regions and endpoints](https://www.tinybird.co/docs/api-reference#regions-and-endpoints).\n\n### Grafana quick start\n\nWe've included a sample dashboard config for Grafana to help you get started. See the [JSON file](grafana/tinybird_org_metrics.json).\n\nImport the dashboard with the following steps:\n\n1. Go to Grafana and click on the Dashboards menu.\n2. Select **New** \u003e **Import**.\n3. Click on Upload dashboard JSON file.\n4. Select the [JSON file](grafana/tinybird_org_metrics.json) and select **Upload**.\n5. Select the Prometheus datasource.\n6. Click **Import**.\n\nThe dashboard uses the Prometheus configuration in this repo, with the job name `tinybird_org_metrics`.\n\n![Grafana dashboard example](./assets/img/grafana.png)\n\n### Datadog quick start\n\nAdd the following configuration to your OpenMetrics Datadog agent `conf.yaml` file:\n\n```yaml\ninstances:\n  - prometheus_url: 'https://api.tinybird.co/v0/pipes/organization_metrics.prometheus?token=\u003cadmin-user-token\u003e'\n    namespace: tinybird_org_metrics\n    metrics:\n      - \"*\"\n    max_returned_metrics: 700000\n```\n\n- Replace `api.tinybird.co` with your Tinybird host if the workspace is in a different region. See [Regions and endpoints](https://www.tinybird.co/docs/api-reference#regions-and-endpoints).\n- Use the `user@domain` admin token of an Organization admin to authenticate requests.\n\nWe've included a sample dashboard config for Datadog that you can use to get started. See the [JSON file](datadog/tinybird_org_metrics.json).\n\nImport the dashboard with the following steps:\n\n1. Go to Datadog and click on the Dashboards menu.\n2. Select **New Dashboard**.\n3. Select **New Dashboard** in the dialog.\n4. Select the settings gear icon (⚙️).\n5. Select **Import Dashboard**.\n6. Browse to the [JSON file](datadog/tinybird_org_metrics.json).\n7. Select **Yes, replace**.\n\n![Datadog dashboard example](./assets/img/datadog.png)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Ftinybird-org-metrics-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Ftinybird-org-metrics-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Ftinybird-org-metrics-exporter/lists"}