{"id":50496861,"url":"https://github.com/hubzero/hzmetrics","last_synced_at":"2026-06-02T08:04:06.253Z","repository":{"id":358157635,"uuid":"1240262267","full_name":"hubzero/hzmetrics","owner":"hubzero","description":"HUBzero metrics pipeline — Python rewrite of the legacy PHP/Perl/Bash pipeline","archived":false,"fork":false,"pushed_at":"2026-05-31T21:41:57.000Z","size":3651,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-31T22:14:11.093Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://hubzero.github.io/hzmetrics/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hubzero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":"docs/roadmap.md","authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-15T23:59:06.000Z","updated_at":"2026-05-31T21:42:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hubzero/hzmetrics","commit_stats":null,"previous_names":["hubzero/hzmetrics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hubzero/hzmetrics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hubzero%2Fhzmetrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hubzero%2Fhzmetrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hubzero%2Fhzmetrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hubzero%2Fhzmetrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hubzero","download_url":"https://codeload.github.com/hubzero/hzmetrics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hubzero%2Fhzmetrics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33812205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-02T08:04:05.238Z","updated_at":"2026-06-02T08:04:06.237Z","avatar_url":"https://github.com/hubzero.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"gh-pages/assets/logo.svg\" alt=\"\" width=\"120\" height=\"120\"\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eHubzero Metrics Pipeline\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cem\u003eApache logs \u0026rarr; MariaDB analytics, in one Python file.\u003c/em\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/hubzero/hzmetrics/actions/workflows/tests.yml\"\u003e\u003cimg alt=\"tests\" src=\"https://github.com/hubzero/hzmetrics/actions/workflows/tests.yml/badge.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/hubzero/hzmetrics/actions/workflows/docs.yml\"\u003e\u003cimg alt=\"docs CI\" src=\"https://github.com/hubzero/hzmetrics/actions/workflows/docs.yml/badge.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://hubzero.github.io/hzmetrics/\"\u003e\u003cimg alt=\"documentation\" src=\"https://img.shields.io/badge/docs-hubzero.github.io%2Fhzmetrics-2456c2?logo=github\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"Python 3.10+\" src=\"https://img.shields.io/badge/python-3.10%2B-3776ab?logo=python\u0026logoColor=white\"\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg alt=\"license: MIT\" src=\"https://img.shields.io/badge/license-MIT-007ec6\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"status: beta\" src=\"https://img.shields.io/badge/status-beta-d54a3c\"\u003e\n\u003c/p\u003e\n\n---\n\n`hzmetrics.py` is the analytics pipeline for a HUBzero-based science\ngateway. It ingests Apache access logs and CMS authentication logs,\nenriches them (reverse DNS, domain classification, GeoIP, session\ncoalescing), and produces monthly summary statistics in a MariaDB\nmetrics database. Those statistics drive the hub's usage reporting\npages and grant reporting.\n\nOne Python file (~8000 lines) replaces the decade-plus accumulation\nof PHP, Perl, and Bash scripts that previously lived at\n`/opt/hubzero/bin/metrics/`. The legacy reference implementation is\npreserved verbatim under [`tests/legacy/`](tests/legacy/) and is the\nbug-for-bug parity target the A/B test harness compares against.\n\n## Quickstart\n\n```sh\n# 1. Deps + /opt tree + scripts (root; idempotent).\nsudo make install\n\n# 2. Drop the unified per-tenant config in place (DB creds + DNS settings).\nsudo install -o apache -g apache -m 0600 hzmetrics.conf \\\n    /opt/hubzero/metrics/conf/hzmetrics.conf\n\n# 3. Create the metrics DB, run baseline DDL, apply migrations.\nsudo -u apache python3 /opt/hubzero/metrics/bin/hzmetrics.py init\n\n# 4. Confirm everything is healthy.\nsudo -u apache python3 /opt/hubzero/metrics/bin/hzmetrics.py doctor\n\n# 5. Register the cron line.\nsudo -u apache crontab /opt/hubzero/metrics/conf/hzmetrics.cron.apache.sample\n```\n\n`make install`, `init`, and `doctor` are idempotent. The same `init`\nmachinery also runs automatically on the first cron tick when invoked\nas `apache` / `www-data`, so if you skip step 3 the next tick will\ncatch up — see\n[`docs/architecture.md → Self-bootstrap`](docs/architecture.md#self-bootstrap).\n\nThe cron line is one entry, every five minutes:\n\n```\n*/5 * * * * python3 /opt/hubzero/metrics/bin/hzmetrics.py tick\n```\n\n`tick` refreshes the whoisonline map every invocation; at `:30` past\neach hour it also opportunistically runs the metrics pipeline under a\nPID lock. The pipeline is a three-mode state machine (`normal`,\n`catchup`, `rebuild`) — a multi-year backlog drains autonomously\nwithout operator intervention.\n\nFor everything else, `hzmetrics.py --help` and the\n[full documentation](https://hubzero.github.io/hzmetrics/).\n\n## Source layout\n\n```\n.\n├── hzmetrics.py                              the entire pipeline\n├── Makefile                                  install / uninstall / test / lint\n├── conf/                                     templates: hzmetrics.conf.sample, cron\n├── docs/                                     plain-markdown documentation\n├── gh-pages/                                 static-site templates + builder\n└── tests/\n    ├── legacy/                               pre-rewrite PHP/Perl/Bash baseline\n    └── ab/                                   A/B + golden + defensive harness\n                                              (44 ports — see docs/testing.md)\n```\n\n## Documentation\n\nStart at [`docs/README.md`](docs/README.md) (or the\n[rendered site](https://hubzero.github.io/hzmetrics/)). Most-touched\noperational pages:\n\n- [`docs/deployment.md`](docs/deployment.md) — install, cron,\n  logrotate, hzmetrics.conf.\n- [`docs/operations.md`](docs/operations.md) — runbook: catch-up,\n  stuck lock, bot inflation, DNS issues, crash recovery,\n  ANALYZE TABLE, etc.\n- [`docs/architecture.md`](docs/architecture.md) — pipeline phases,\n  tables, scheduling, the catchup state machine, self-bootstrap.\n- [`docs/testing.md`](docs/testing.md) — A/B + golden + defensive\n  test modes.\n\n## Acknowledgments\n\nThe HUBzero metrics subsystem was originally written in Perl by\nSwaroop Shivarajapura and later ported to PHP by Nicholas J.\nKisseberth. Long-term stewardship of the codebase has been carried\nby J.M. Sperhac (SDSC), among others. This Python rewrite builds\ndirectly on their work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhubzero%2Fhzmetrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhubzero%2Fhzmetrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhubzero%2Fhzmetrics/lists"}