{"id":13610232,"url":"https://github.com/m-lab/stats-pipeline","last_synced_at":"2025-05-02T23:30:28.961Z","repository":{"id":37798142,"uuid":"274486640","full_name":"m-lab/stats-pipeline","owner":"m-lab","description":"Contains code that processes M-Lab data and provides it in various formats for other use.","archived":false,"fork":false,"pushed_at":"2024-06-08T17:02:44.000Z","size":491,"stargazers_count":14,"open_issues_count":26,"forks_count":6,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-10-29T14:53:30.617Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-06-23T19:02:39.000Z","updated_at":"2024-03-15T12:45:44.000Z","dependencies_parsed_at":"2024-11-06T16:38:28.665Z","dependency_job_id":"50987415-e341-45e8-9552-5516c6be1f1e","html_url":"https://github.com/m-lab/stats-pipeline","commit_stats":{"total_commits":294,"total_committers":10,"mean_commits":29.4,"dds":0.45578231292517,"last_synced_commit":"e83928737959e5713e097f33c55f32f74b23e955"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fstats-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fstats-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fstats-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m-lab%2Fstats-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m-lab","download_url":"https://codeload.github.com/m-lab/stats-pipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224003718,"owners_count":17239495,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:01:42.702Z","updated_at":"2024-11-12T20:11:09.129Z","avatar_url":"https://github.com/m-lab.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![Version](https://img.shields.io/github/tag/m-lab/stats-pipeline.svg)](https://github.com/m-lab/stats-pipeline/releases) [![Build Status](https://travis-ci.com/m-lab/stats-pipeline.svg?branch=master)](https://travis-ci.com/m-lab/stats-pipeline) [![Coverage Status](https://coveralls.io/repos/github/m-lab/stats-pipeline/badge.svg?branch=master)](https://coveralls.io/github/m-lab/stats-pipeline?branch=master) [![GoDoc](https://godoc.org/github.com/m-lab/stats-pipeline?status.svg)](https://godoc.org/github.com/m-lab/stats-pipeline) [![Go Report Card](https://goreportcard.com/badge/github.com/m-lab/stats-pipeline)](https://goreportcard.com/report/github.com/m-lab/stats-pipeline)\n\n# Statistics Pipeline Service\nThis repository contains code that processes NDT data and provides aggregate\nmetrics by day for standard global, and some national geographies. The resulting\naggregations are made available in JSON format, for use by other applications.\n\nThe `stats-pipeline` service is written in Go, runs on GKE, and generates and\nupdates daily aggregate statistics. Access is provided in public BigQuery tables\nand in per-year JSON formatted files hosted on GCS.\n\n## Documentation Provided for the Statistics Pipeline Service\n* (This document) Overview of the `stats-pipeline` service, fields provided\n  (schema), output formats, available geographies, and API URL structure.\n* [What Statistics are Provided by stats-pipeline, and How are They Calculated?][stats-overview]\n* [Geographic Precision in stats-pipeline][geo-precision]\n* [Statistics Output Format, Schema, and Field Descriptions][format-schema]\n* [Statistics API URL Structure, Available Geographies \u0026 Aggregations][api-structure]\n\n[stats-overview]: docs/stats-overview.md\n[geo-precision]: docs/geo-precision.md\n[format-schema]: docs/format-schema.md\n[api-structure]: docs/api-structure.md\n\n## General Recommendations for All Aggregations of NDT data\nIn general, [our recommendations][recommendations] for research aggregating NDT data are:\n\n* Don't oversimplify\n* Aggregate by ASN in addition to time/date and location\n* Be aware of, and illustrate multimodal distributions\n* Use histogram and logarithmic scales\n* Take into account, and compensate for, client bias and population drift\n\n[recommendations]: upcoming-blog-post\n\n## Roadmap\nBelow we list additional features, methods, geographies, etc. which may be\nconsidered for future versioned releases of `stats-pipeline`.\n\n### Geographies\n* US Zip Codes, US Congressional Districts, Block Groups, Blocks\n\n### Output Formats\n* histogram_daily_stats.csv - Same data as the JSON, but in CSV. Useful for importing into a spreadsheet.\n* histogram_daily_stats.sql - A SQL query which returns the same rows in the corresponding .json and .csv. Useful for verifying the exported data against the source and to tweak the query as needed by different use cases. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fstats-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm-lab%2Fstats-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm-lab%2Fstats-pipeline/lists"}