{"id":24087933,"url":"https://github.com/tinybirdco/datadog-integration","last_synced_at":"2025-02-27T05:21:13.265Z","repository":{"id":37793082,"uuid":"495345539","full_name":"tinybirdco/datadog-integration","owner":"tinybirdco","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-02T14:48:52.000Z","size":15,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-10T03:55:59.340Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-23T09:40:16.000Z","updated_at":"2024-10-02T14:48:57.000Z","dependencies_parsed_at":"2022-08-18T15:01:16.973Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/datadog-integration","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdatadog-integration","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdatadog-integration/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdatadog-integration/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdatadog-integration/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/datadog-integration/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240982501,"owners_count":19888513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:56:15.236Z","updated_at":"2025-02-27T05:21:13.246Z","avatar_url":"https://github.com/tinybirdco.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tinybird \u003c\u003e Datadog integration\n\nThis data project demonstrates how users can integrate [Tinybird](https://tinybird.co) with [Datadog](https://datadog.com/) bulding endpoints on top of the [Tinybird Service Data Sources](https://docs.tinybird.co/monitoring/service-datasources.html) and using [vector.dev](https://vector.dev).\n\n\u003e Note: This example uses vector.dev version 0.22.2. We're not actively maintaining this example. If you want to use a newer version, you would have to update the configuration from `vector-ops-log.toml` and `vector-pipes-stats.toml`.\n\nThe project is defined by:\n\n- **Data project**: The Tinybird data project\n- **Vector.dev files**: Files describing vector.dev workflows\n- **GitHub Actions files**: Files describing the GitHub Action workflows used to call Tinybird endpoints, and execute vector.dev flows\n\n## Push data project to Tinybird\n\nThe data project is defined in the `/data-project` folder. To push the project to Tinybird, you just have to install and configure the Tinybird CLI following [these instructions](https://docs.tinybird.co/cli.html), and then execute\n\n```bash\ncd data-project\ntb push\n```\n\nThis will create two endpoints:\n\n- `endpoints/ep_datadog_ops_log.pipe`: Used to extract datasources operations metrics from `tinybird.datasources_ops_log`\n- `endpoints/ep_datadog_pipes_stats.pipe`: Used to extract endpoints metrics from `tinybird.pipe_stats_rt`\n\nAnd a datasource:\n\n- `datasources/datadog_integration.datasource`: Used to keep track of the last execution to avoid duplicating data in Datadog\n\nThe process will also create a token named `datadog_integration_token` with read access to the endpoints and append access to the datasource.\n\n## Configure GitHub Actions\n\n\u003e :warning: Note: This repository illustrates how to do the integration using GitHub Action but you can use the job scheduler best fits your current architecture.\n\nTo configure the GitHub Actions, you'll have to create the following environment variables:\n\n- `TB_TOKEN`: Tinybird token named `datadog_integration_token`\n- `TB_HOST`: Tinybird host: \"api.us-east\" for us or \"api\" for eu\n- `DATADOG_API_KEY`: Datadog API Key\n- `DATADOG_REGION`: Datadog region: \"us|eu\" [Context](https://vector.dev/docs/reference/configuration/sinks/datadog_metrics/#region)\n\n## How everything works?\n\nThe process basically uses vector.dev to read data from Tinybird API endpoints, makes basic transformations to generate metrics, and sends data to Datadog. The jobs are scheduled to run every 10 minutes, running the following commands:\n\n```bash\ncurl \"https://${TB_HOST}.tinybird.co/v0/pipes/ep_datadog_pipes_stats.ndjson?token=${TB_TOKEN}\" | ~/.vector/bin/vector --config ./vector-pipes-stats.toml\ncurl \"https://${TB_HOST}.tinybird.co/v0/pipes/ep_datadog_ops_log.ndjson?token=${TB_TOKEN}\" | ~/.vector/bin/vector --config ./vector-ops-log.toml\n```\n\n## Metrics\n\n| metric_name             | metric_type | interval | unit_name  |\n|-------------------------|-------------|----------|------------|\n| tb.pipes.count          | count       | 1min     | requests   |\n| tb.pipes.duration       | gauge       | 1min     | seconds    |\n| tb.pipes.duration_p99   | gauge       | 1min     | seconds    |\n| tb.pipes.read_bytes     | count       | 1min     | bytes      |\n| tb.datasources.count    | count       | 1min     | operations |\n| tb.datasources.rows     | count       | 1min     | rows       |\n| tb.datasources.duration | gauge       | 1min     | seconds    |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdatadog-integration","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Fdatadog-integration","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdatadog-integration/lists"}