{"id":19450807,"url":"https://github.com/clearlinux/telemetrics-backend","last_synced_at":"2025-06-27T02:36:34.468Z","repository":{"id":51353799,"uuid":"99628785","full_name":"clearlinux/telemetrics-backend","owner":"clearlinux","description":"Collector and WebUI for https://github.com/clearlinux/telemetrics-client","archived":false,"fork":false,"pushed_at":"2025-01-06T18:27:04.000Z","size":898,"stargazers_count":13,"open_issues_count":8,"forks_count":10,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-04-25T03:37:18.571Z","etag":null,"topics":["telemetry","telemetry-server"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clearlinux.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-07T23:17:59.000Z","updated_at":"2025-01-06T18:27:10.000Z","dependencies_parsed_at":"2025-04-25T03:42:19.145Z","dependency_job_id":null,"html_url":"https://github.com/clearlinux/telemetrics-backend","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/clearlinux/telemetrics-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clearlinux%2Ftelemetrics-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clearlinux%2Ftelemetrics-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clearlinux%2Ftelemetrics-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clearlinux%2Ftelemetrics-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clearlinux","download_url":"https://codeload.github.com/clearlinux/telemetrics-backend/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clearlinux%2Ftelemetrics-backend/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262179147,"owners_count":23271172,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["telemetry","telemetry-server"],"created_at":"2024-11-10T16:39:07.499Z","updated_at":"2025-06-27T02:36:34.448Z","avatar_url":"https://github.com/clearlinux.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# telemetrics-backend\n\n## Overview\n\nThis project provides the server-side component of a complete telemetrics\n(telemetry + analytics) solution for Linux-based operating systems. The\nclient-side component source repository lives at\nhttps://github.com/clearlinux/telemetrics-client.\n\nIt consists of a Flask application `telemetryui`,\nthat exposes several views to visualize the telemetry data. The `telemetryui`\napp also provides a REST API to perform queries on the data.\n\nThe Flask apps have several dependencies listed [here](services/collector/requirements.txt)\nand [here](deployments/services/telemetryui/requirements.txt). The testing infrastructure\nis described by [this](deployments/docker-compose.yaml) docker-compose.yaml and production by\n[this](deployments/docker-compose.prod.yaml) docker-compose.prod.yaml\n\n## Security considerations\n\nThe telemetrics-backend was written with a particular deployment scenario in\nmind: internal LAN (e.g. a corporate network not exposed to the public internet).\nAlso, no identity management, user authentication, or role-based access controls\nhave yet been implemented for the applications.\n\nTo control access to the applications, it is recommended that system\nadministrators leverage web server authentication.\n\nTo enable HTTPS connections replace the placeholders files [here](services/nginx/telemetry.cert.pem)\nand [here](services/nginx/telemetry.key.pem) with a certificate and private key\nfor your server. In addition uncomment lines 3, 9, and 10 in the [sites](services/nginx/sites.conf).\nconfiguration file.\n\n## Deployment\n\nThe application is containerized to simplify the deployment. \n\n```\n# checkout latest tag source\ngit clone --branch latest https://github.com/clearlinux/telemetrics-backend.git\ncd telemetrics-backend\n```\n\nFor a deployment in production make sure to update the value for POSTGRES_PASSWORD otherwise\nthe build step will fail.\n\n```\n# update services/production.env\nvi services/production.env\n\n# build production environment\nsudo -E docker-compose --file ./deployments/docker-compose.prod.yaml build --force-rm\n```\n\nOnce the images are build successfully the environment can be started using:\n\n```\n# start environment on the background\nsudo -E docker-compose --file ./deployments/docker-compose.prod.yaml up --detach\n```\n\n### Deploying as systemd service\n\nTo deploy the environment as a systemd service use the following example:\n\n```\n[Unit]\nDescription=Telemetry Backend\nRequires=docker.service\nAfter=docker.service\n\n[Service]\nRestart=always\nWorkingDirectory=/srv/telemetrics-backend\nExecStart=/usr/local/bin/docker-compose --file deployments/docker-compose.prod.yaml up\nExecStop=/usr/local/bin/docker-compose --file deployments/docker-compose.prod.yaml down -v\n\n[Install]\nWantedBy=multi-user.target\n```\n\n\n## `telemetryui` views\n\nThe `telemetryui` app is a web app that exposes several views to visualize the\ntelemetry data and also provides a REST API to perform queries on record data.\n\nThe current views are:\n\n* Records view - a paginated list of all records in the `telemetry` database that\n  have been accepted by the `collector`. The records are presented in tabular\n  format and the columns map to select fields from the `records` database table.\n  At the top of the view, an HTML form can be used for \"advanced searches\",\n  filtering the list of records to display.\n\n* Builds view - a basic column chart that displays how many records have been\n  received for each OS build. Note that the view is optimized for Clear Linux\n  OS, since the chart only displays data for records when their build numbers are\n  integers. For example, records with non-integer build numbers, like \"16.04\" for\n  Ubuntu, are not displayed in this view.\n\n* Stats view - two pie charts displaying the statistical breakdown of\n  classifications and platforms for all records in the database. The\n  \"classification\" field is used to identify the type of record sent by a\n  specific client probe; classifications use the format DOMAIN/PROBE/REST, where\n  DOMAIN is the vendor of the probe, PROBE is the probe name, and REST is a\n  probe-defined field to classify what is contained in the payload. The\n  \"platform\" field is a formatted string,\n  `\"sys_vendor|product_name|product_version\"`, where the values are taken from\n  files with those names in the `/sys/class/dmi/id/` directory; if any of these\n  files are empty or contain only space characters, the client library\n  substitutes \"blank\" for their value.\n\n* Crashes view - features a table displaying the top 10 crash reports from\n  crash records received in the past week. It only consumes records from the\n  telemetrics-client `crashprobe`, which extracts backtrace information from core\n  files and creates/sends telemetry records containing this data. The crash\n  reports are grouped by \"guilties\"; a guilty is a frame from a crash backtrace\n  chosen as the best candidate for the cause of the crash. The logic for\n  determining crash record guilty frames accepts user input; the user can\n  identify which frames in a backtrace are never guilty.\n\n* MCE view - charts that display MCE (machine check exception) data from a\n  patched version of `mcelog` that uses libtelemetry to create and send\n  telemetry records. The mcelog patch is available from\n  https://github.com/clearlinux-pkgs/mcelog.\n\n* Thermal view - similar to the MCE view, but it only displays a chart for MCE\n  Thermal event records, also received from the patched `mcelog`.\n\n* Population view - contains column charts that display the number of unique\n  systems that are running each version of an OS over a specific range of time.\n  The \"uniqueness\" of a system is determined by its \"machine ID\" field, managed\n  by the telemetrics-client daemon, which by default rotates the value every 3\n  days. Thus, the analysis presented in this view is *fuzzy* due to the machine\n  ID rotation.\n\n## Custom `telemetryui` views\n\nTo provided users with an extensible framework a concept of \"plugin views\" was\nimplemented to add views without the need to make changes to the core of the\napplication. To read more about [plugin view](/telemetryui/telemetryui/plugins/README.md)\ngo to relevant documentation.\n\n## Using the REST API\n\nA REST API for querying records is available at \"/api/records\". The API returns\na JSON response that contains a list of records keyed on \"records\".\n\nSeveral parameters are available for filtering queries, similar to the filters\navailable through the `telemetryui` Records view.\n\n* `classification`: The classification of the record. Right now this is\n  restricted at 140 characters. If a classification with more that 140\n  characters is supplied as a query parameter, an HTTP response 400 is sent back.\n* `severity`: The severity of the record. Restricted to integer value.\n* `machine_id`: The id of the machine where this record was generated on.\n  Should be 32 characters in length.\n* `build`: The build on which the record was generated. Restricted to 256\n  characters.\n* `created_in_days`: This should be an integer value. It causes the query to\n  return records created after the last given days. Note: the server timestamp\n  is used as a reference point.\n* `created_in_sec`: This should be an integer again. If returns the records\n  created after the last given seconds. This is used only if the previous\n  parameter is absent. Note: the server timestamp is used as a reference point.\n* `limit`: The maximum number of records to be returned.\n\n### Example queries\n\nTo query for records, simply make a GET call to the endpoint.\n\n* `GET /api/records` - Returns a maximum of 10000 most recent records in the\n  backend database ordered by the record id (descending).\n* `GET /api/records?classification=org.clearlinux%2Fkernel%2Fwarning\u0026severity=2\u0026build=2980\u0026created_in_sec=5\u0026limit=100` - Returns at most 100 records with the classification\n  \"org.clearlinux/kernel/warning\", severity 2, build 2980, and created in the\n  last 5 seconds. As shown the query parameters need to be [URL encoded](https://en.wikipedia.org/wiki/Percent-encoding).\n\n### Response object\n\nThe response is a JSON object that contains a list of objects keyed on\n\"records\". This list is empty in case no records match the query parameters.\nExample response:\n\n```\n{\n    \"records\": [\n        {\n            \"arch\": \"x86_64\",\n            \"build\": \"2980\",\n            \"classification\": \"org.clearlinux/hello/world\",\n            \"kernel_version\": \"4.2.0-120\",\n            \"machine_id\": \"66c196ce4222dd761470da5e7e35f6f1\",\n            \"machine_type\": \"blank|blank|blank\",\n            \"payload\": \"hello\\n\\n\",\n            \"record_format_version\": 2,\n            \"severity\": 1,\n            \"ts_capture\": \"2015-09-30 00:39:35 UTC\",\n            \"ts_reception\": \"2015-09-30 00:56:59 UTC\"\n        },\n        {\n            \"arch\": \"x86_64\",\n            \"build\": \"2980\",\n            \"classification\": \"org.clearlinux/hello/world\",\n            \"kernel_version\": \"4.2.0-120\",\n            \"machine_id\": \"66c196ce4222dd761470da5e7e35f6f1\",\n            \"machine_type\": \"blank|blank|blank\",\n            \"payload\": \"hello\\n\",\n            \"record_format_version\": 2,\n            \"severity\": 1,\n            \"ts_capture\": \"2015-09-30 00:36:22 UTC\",\n            \"ts_reception\": \"2015-09-30 00:38:45 UTC\"\n        }\n    ]\n}\n```\n\n## Creating new database migrations\n\nDatabase migrations are managed using\n[Flask-Migrate](https://flask-migrate.readthedocs.io/en/latest/). Upon initial\ninstall of telemetrics-backend, the first migration will be applied, and any\nadditional migrations in the `telemetryui/migrations/versions/` directory will\nbe applied in sequence and upgrade the `telemetry` schema to the latest version.\n\nAny new migration from a new realease will be applied when the environment is\nstarted, this applies for both production and testing configurations.\n\n## Development\n\n```\n# Build\nsudo -E docker-compose --file deployments/docker-compose.yaml build --force-rm\n\n# Launch\nsudo -E docker-compose --file deployments/docker-compose.yaml up\n```\n\n## Software License\n\nThe telemetrics-backend project is licensed under the Apache License, Version\n2.0. The full license text is found in the LICENSE file, and individual source\nfiles contain the boilerplate notice described in the appendix of the LICENSE\nfile.\n\n\n## Security Disclosures\n\nTo report a security issue or receive security advisories please follow procedures\nin this [link](https://01.org/security).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclearlinux%2Ftelemetrics-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclearlinux%2Ftelemetrics-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclearlinux%2Ftelemetrics-backend/lists"}