{"id":15687363,"url":"https://github.com/grantbirki/fastly-tempo","last_synced_at":"2025-05-07T19:41:59.951Z","repository":{"id":48632878,"uuid":"332003534","full_name":"GrantBirki/fastly-tempo","owner":"GrantBirki","description":"🚀 Fastly Tempo - Aggregate log streaming using Docker and Python.","archived":false,"fork":false,"pushed_at":"2023-05-23T01:08:49.000Z","size":3456,"stargazers_count":9,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-05-04T21:46:03.385Z","etag":null,"topics":["dashboards","docker","docker-container","fastly","insights","insights-image","logging","metrics","newrelic","newrelic-insights","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GrantBirki.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-22T16:37:04.000Z","updated_at":"2022-11-22T07:27:51.000Z","dependencies_parsed_at":"2024-10-23T20:37:02.996Z","dependency_job_id":"76d11964-5d3e-4f3e-ae93-1c26a1b11e46","html_url":"https://github.com/GrantBirki/fastly-tempo","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GrantBirki%2Ffastly-tempo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GrantBirki%2Ffastly-tempo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GrantBirki%2Ffastly-tempo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GrantBirki%2Ffastly-tempo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GrantBirki","download_url":"https://codeload.github.com/GrantBirki/fastly-tempo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252945727,"owners_count":21829644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dashboards","docker","docker-container","fastly","insights","insights-image","logging","metrics","newrelic","newrelic-insights","python"],"created_at":"2024-10-03T17:47:56.828Z","updated_at":"2025-05-07T19:41:59.929Z","avatar_url":"https://github.com/GrantBirki.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fastly Tempo 🚀\n\n[![security-checks Actions Status](https://github.com/grantbirki/fastly-tempo/workflows/security-checks/badge.svg)](https://github.com/grantbirki/fastly-tempo/actions) [![python-tests Actions Status](https://github.com/grantbirki/fastly-tempo/workflows/python-tests/badge.svg)](https://github.com/grantbirki/fastly-tempo/actions) [![docker-build Actions Status](https://github.com/grantbirki/fastly-tempo/workflows/docker-build/badge.svg)](https://github.com/grantbirki/fastly-tempo/actions) [![codeQL Actions Status](https://github.com/grantbirki/fastly-tempo/workflows/codeQL/badge.svg)](https://github.com/grantbirki/fastly-tempo/actions)\n\n[![Docker Pulls](https://img.shields.io/docker/pulls/grantbirki/fastly-tempo.svg)](https://img.shields.io/docker/pulls/grantbirki/fastly-tempo.svg)\n\nThis is based off the New Relic blessed way to get your Fastly metrics into New Relic Insights, packaged as a Docker container image for ease of use!\n\nIn order to use the Fastly Tempo Docker image, you will need an active New Relic account with Insights, an active Fastly account with Read access, a New Relic Insights Insert key and a Fastly API Key.\n\n[🗨️ Fastly's tweet about this project!](https://twitter.com/fastly/status/1410007456256774152)\n\n## Dashboard 🗺️\n\nHere is an example of a 4xx alert dashboard that can be created in New Relic from the `Fastly Tempo` container:\n\n\u003c!-- markdownlint-disable no-inline-html --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/img/panel.png\"/\u003e\n\u003c/p\u003e\n\u003c!-- markdownlint-enable no-inline-html --\u003e\n\n\u003c!-- markdownlint-disable no-inline-html --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/img/carbon.png\"/\u003e\n\u003c/p\u003e\n\u003c!-- markdownlint-enable no-inline-html --\u003e\n\n## Supported Backends 🧰\n\nCurrently, there is only one support backend to send metrics to: `New Relic`\n\nHowever, there are plans to integrate with other providers/backends in the future for greater integrations.\n\n* New Relic: Supported ✔️\n* Datadog: Planned\n* Prometheus: Planned\n* Graphite: Planned\n* Splunk: Planned\n\n## Quick Start ⭐\n\nGrab the image from [DockerHub](https://hub.docker.com/r/grantbirki/fastly-tempo) 🐳\n\n### Using Docker-Compose (Preferred)\n\nUsing Docker-Compose to run this image is extremely easy.\n\n1. Create a file named `creds.env` at the root of this repo with the following contents:\n\n    ```ini\n    ACCOUNT_ID=#######\n    INSERT_KEY=XXXXXXXXXXXXXX\n    FASTLY_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXX\n    ```\n\n2. Add your services to `config.env`\n\n    ```ini\n    ...\n    SERVICES=ServiceName1:ServiceId1 ServiceName2:ServiceId2 ...\n    ```\n\n3. Run: `make run`\n\n4. ✔️ Log into New Relic and view your logs!\n\n### Using Docker\n\n1. Build: `docker build -t fastly-tempo .`\n2. Run:\n\n    ```bash\n    docker run \\\n      -e ACCOUNT_ID='yourNewRelicAccountId' \\\n      -e FASTLY_KEY='yourFastlyKey' \\\n      -e INSERT_KEY='yourNewRelicInsertKey' \\\n      -e SERVICES='ServiceId1 ServiceId2 ...' \\\n      fastly-tempo\n    ```\n\n## How to use this image 📚\n\nBefore you get started, make sure that you have a [Fastly API Key](https://docs.fastly.com/guides/account-management-and-security/using-api-tokens) and a [New Relic Insert Key](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/insert-custom-events-insights-api#register).\n\nThe Fastly Tempo image is configured by environment variables.\n\nThese are required:\n\n* `ACCOUNT_ID`\n* `FASTLY_KEY`\n* `INSERT_KEY`\n* `SERVICES`\n\nThese are optional:\n\n* `SILENT` - Print output True/False. Default: `False`\n* `INTERVAL` - Interval in seconds to poll data from Fastly. Default: `1`\n\nNote: It is suggested to use a minimum `INTERVAL` of `1` second to avoid publishing metrics for the same second twice.\n\n### Fastly Services\n\nYou have two options for passing in Fastly Services to the Docker container.\n\n1. Passing in only the **ServiceId**\n2. Passing in both the **ServiceName** and the **ServiceId**\n\nNote: In both examples, one serivce has millions of 200s and the other has none.\n\n#### 1. ServiceId\n\nIf you are just using the ServiceId method, `SERVICES` needs to be a string with the ids of the Fastly services you want to see data for in Insights, separated by a space. I know that's not ideal. A limitation of Fastly is that you have to query one service at a time, so I chose to create an array of service ids and loop through them to query Fastly. A limitation of Docker is that you can't pass an array via the command line, so I chose to split a string on \" \". If you have a better idea, I would love to hear it - please contribute!\n\nExample:\n\n```bash\ndocker run \\\n  -e ACCOUNT_ID='yourNewRelicAccountId' \\\n  -e FASTLY_KEY='yourFastlyKey' \\\n  -e INSERT_KEY='yourNewRelicInsertKey' \\\n  -e SERVICES='ServiceId1 ServiceId2 ...' \\\n  fastly-tempo\n```\n\nYou may optionally add `-e SILENT=True` or `-e INTERVAL=\u003ctime in seconds\u003e` for custom configuration.\n\nHere is what this will look like in New Relic:\n\n![ServieId Only Image](assets/img/service-id-only.png)\n\n#### 2. ServiceId + ServiceName\n\nIf you want to map your ServiceId to a friendly name use this method. This method is the same as `1. ServiceId` above with one minor change. With this method `SERVICES` needs to be a string with the `\u003cNameOfService\u003e:\u003cServiceId\u003e` of the Fastly services you want to see data for in Insights, separated by a space (Example below).\n\nThe benefit to using this method is you can name the service whatever you want. It could be a friendly name, the actual name of the service in Fastly, or your favorite planet.\n\nExample:\n\n```bash\ndocker run \\\n  -e ACCOUNT_ID='yourNewRelicAccountId' \\\n  -e FASTLY_KEY='yourFastlyKey' \\\n  -e INSERT_KEY='yourNewRelicInsertKey' \\\n  -e SERVICES='NameOfService1:ServiceId1 NameOfService2:ServiceId2 ...' \\\n  fastly-tempo\n```\n\nYou may optionally add `-e SILENT=True` or `-e INTERVAL=\u003ctime in seconds\u003e` for custom configuration.\n\nHere is what this will look like in New Relic:\n\n![ServieId Only Image](assets/img/friendly-service-name.png)\n\nNote: You can mix `method 1` and `method 2` together. In the `SERVICES` variable. I would not recommend doing this though.\n\n## Dashboard Examples 🗺️\n\nBelow are a few screenshots of dashboards that can be created with this project:\n\nNote: *example.com would be replaced by the `serviceName` which you specify with the `SERVICES` env var*\n\n![3xx and 4xx Dashboard](assets/img/3xx-and-4xx.png)\n\n![Cache Dashboard](assets/img/cache.png)\n\n![Total Reqs Dashboard](assets/img/total-reqs.png)\n\n![2xx Dashboard](assets/img/2xx.png)\n\n## New Relic Queries to View Data 📊\n\nHere are some helpful queries in New Relic to start viewing your metrics:\n\nHint: *You may also use the pre-made Dashboard JSON template below*\n\n* 2xx Status codes by service\n\n    ```sql\n    SELECT average(status_2xx) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* 3xx Status codes by service\n\n    ```sql\n        SELECT average(status_3xx) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* 4xx Status codes by service\n\n    ```sql\n    SELECT average(status_4xx) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* 5xx Status codes by service\n\n    ```sql\n    SELECT average(status_5xx) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* The number of cache hits by service\n\n    ```sql\n    SELECT average(hits) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* The number of cache misses by service\n\n    ```sql\n    SELECT average(miss) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\n* The total amount of time spent processing cache misses (in seconds)\n\n    ```sql\n    SELECT average(miss_time) FROM FastlyLogAggregate since 30 minutes ago TIMESERIES 15 minutes facet service\n    ```\n\nTo see more info on these queries, check out the blog post by New Relic [here](https://blog.newrelic.com/engineering/monitor-fastly-data/).\n\n## Dashboard JSON Template 📋\n\nRather than build out a monitoring Dashboard by hand (using the commands above), this repo comes with a pre-made one to get you going!\n\nYou can simply `copy and paste` the JSON template located [here](assets/dashboards/new_relic.json) into New Relic and it will be created for you.\n\nNote: You **will** need to **change** all occurrences of the `\"accountId\": 1234567` value in the `.json` file to your own New Relic accountId.\n\nNew Relic Docs in-case you need them for the import - [documentation](https://docs.newrelic.com/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards/#get-started)\n\n## Exported Values 🧮\n\nAll of the following values are aggregated and accessible right away in the New Relic console!\n\n```text\nservice\nnum_requests\nnum_tls\nnum_http2\nnum_logs\nnum_pci\nnum_video\nipv6\npipe\nuncacheable\nshield\nshield_resp_header_bytes\nshield_resp_body_bytes\notfp\notfp_shield_time\notfp_deliver_time\notfp_manifests\notfp_shield_resp_header_bytes\notfp_shield_resp_body_bytes\notfp_resp_header_bytes\notfp_resp_body_bytes\nbandwidth\nresp_header_bytes\nheader_size\nresp_body_bytes\nbody_size\nreq_body_bytes\nreq_header_bytes\nbereq_header_bytes\nbereq_body_bytes\nbilled_header_bytes\nbilled_body_bytes\nstatus_2xx\nstatus_3xx\nstatus_4xx\nstatus_5xx\nstatus_200\nstatus_204\nstatus_301\nstatus_304\nstatus_400\nstatus_401\nstatus_403\nstatus_404\nstatus_500\nstatus_501\nstatus_502\nstatus_503\nstatus_504\nstatus_505\nstatus_1xx\nwaf_logged\nwaf_blocked\nwaf_passed\nattack_req_body_bytes\nattack_req_header_bytes\nattack_logged_req_body_bytes\nattack_logged_req_header_bytes\nattack_blocked_req_body_bytes\nattack_blocked_req_header_bytes\nattack_passed_req_body_bytes\nattack_passed_req_header_bytes\nattack_resp_synth_bytes\nhits\nhit_ratio\nmiss\npass\npass_time\nsynth\nerrors\nrestarts\nhits_time\nmiss_time\ntls\ntls_v10\ntls_v11\ntls_v12\ntls_v13\nimgopto\nimgopto_resp_body_bytes\nimgopto_resp_header_bytes\nimgopto_shield_resp_body_bytes\nimgopto_shield_resp_header_bytes\nobject_size_1k\nobject_size_10k\nobject_size_100k\nobject_size_1m\nobject_size_10m\nobject_size_100m\nobject_size_1g\nrecv_sub_time\nrecv_sub_count\nhash_sub_time\nhash_sub_count\ndeliver_sub_time\ndeliver_sub_count\nhit_sub_time\nhit_sub_count\nprehash_sub_time\nprehash_sub_count\npredeliver_sub_time\npredeliver_sub_count\n```\n\n## Contributing 👩‍💻\n\nYou are welcome to send pull requests to this repo. All and any contributors are welcome.\n\n## More Information 📖\n\nFor more information on the Fastly Real-Time Analytics API, look [here](https://docs.fastly.com/api/analytics).\n\nFor more information on the New Relic Insights API, look [here](https://docs.newrelic.com/docs/insights/insights-data-sources/custom-data/insert-custom-events-insights-api).\n\nThis project is provided AS-IS WITHOUT WARRANTY OR SUPPORT, although you can report issues and contribute to the project.\n\n### Medium\n\nFeel free to checkout a write-up on this project on [Medium.com](https://birki.medium.com/fastly-tempo-a-real-time-data-pipeline-for-fastlys-cdn-f1d5831a169c) for more information as well.\n\n## JavaScript version 🔗\n\nThis project is the python implementation of the original [fastly-to-insights](https://github.com/newrelic/fastly-to-insights) project. Check out the source project to see how this one differs and if the Python version is right for you.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrantbirki%2Ffastly-tempo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgrantbirki%2Ffastly-tempo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgrantbirki%2Ffastly-tempo/lists"}