{"id":13561523,"url":"https://github.com/cncf/devstats","last_synced_at":"2025-04-05T16:06:35.796Z","repository":{"id":154220503,"uuid":"621207364","full_name":"cncf/devstats","owner":"cncf","description":"📈CNCF-created tool for analyzing and graphing developer contributions","archived":false,"fork":false,"pushed_at":"2025-03-28T06:54:38.000Z","size":25165,"stargazers_count":87,"open_issues_count":8,"forks_count":31,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-04T20:03:07.908Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://devstats.cncf.io","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cncf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-30T07:49:21.000Z","updated_at":"2025-04-04T12:13:28.000Z","dependencies_parsed_at":"2023-10-30T07:32:02.230Z","dependency_job_id":"ad325ccf-6754-4256-81ed-d1191e9552cb","html_url":"https://github.com/cncf/devstats","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cncf%2Fdevstats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cncf%2Fdevstats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cncf%2Fdevstats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cncf%2Fdevstats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cncf","download_url":"https://codeload.github.com/cncf/devstats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361686,"owners_count":20926643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:00:57.889Z","updated_at":"2025-04-05T16:06:30.783Z","avatar_url":"https://github.com/cncf.png","language":"Shell","funding_links":[],"categories":["GitHub Metrics and Dashboards"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/cncf/devstats.svg?branch=master)](https://travis-ci.org/cncf/devstats)\n[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1357/badge)](https://bestpractices.coreinfrastructure.org/projects/1357)\n\n# GitHub archives and git Grafana visualization dashboards\n\nAuthors: Łukasz Gryglicki \u003clgryglicki@cncf.io\u003e, Justyna Gryglicka \u003cjgryglicka@cncf.io\u003e.\n\nThis is a toolset to visualize GitHub [archives](https://www.gharchive.org) using Grafana dashboards.\n\nGHA2DB stands for **G**it**H**ub **A**rchives to **D**ash**B**oards.\n\nMore information about Kubernetes dashboards [here](https://github.com/cncf/devstats/blob/master/README_K8s.md).\n\n\n# Kubernetes and Helm\n\nPlease see [example Helm chart](https://github.com/cncf/devstats-helm-example) for an example Helm deployment.\n\nPlease see [Helm chart](https://github.com/cncf/devstats-helm) for a full Helm deployment.\n\nPlease see [LF Helm chart](https://github.com/cncf/devstats-helm-lf) for the LF Helm deployment (it is a data deployment, has no Grafana).\n\nPlease see [GraphQL Helm chart](https://github.com/cncf/devstats-helm-graphql) for GraphQL foundation DevStats deployment.\n\nPlease see [Kubernetes dashboard](https://github.com/cncf/devstats-kubernetes-dashboard) if you want to enable a local dashboard to explore the cluster state.\n\nPlease see [bare metal example](https://github.com/cncf/devstats-example) to see an example of bare metal deployment.\n\nThe rest of this document describes the current bare metal deployment on metal.equinix.com used by CNCF projects.\n\n\n# Presentations\n\n- Presentations are available [here](https://github.com/cncf/devstats/blob/master/docs/presentation).\n- Direct [link](https://docs.google.com/presentation/d/1v5zuSFQkwcthWXgS2p9vs9x5e4fnavMR8HdykS7aWYA/edit?usp=sharing).\n- Another direct [link](https://docs.google.com/presentation/d/1LLv4kio_KGP36cjkpeSMHZrNl0IYJ7B2pKGU0aHWqx8/edit?usp=sharing).\n\n\n# Talks\n\n- [Who What How: Understanding Kubernetes Development through DevStats](https://www.youtube.com/watch?v=D3CMuxQymR8).\n- [A Kubernetes Application End-to-End: DevStats](https://www.youtube.com/watch?v=U2PTifzzKNE\u0026t=58s).\n\n\n# Architecture\n\nDevStats is deployed using [Helm](https://helm.sh) on [Kubernetes](https://kubernetes.io) running on bare metal servers provided by [Equinix](https://www.equinix.com).\n\nDevStats is written in [Go](https://go.dev), it uses [GitHub archives](https://www.gharchive.org), [GitHub API](https://docs.github.com/en/rest) and [git](https://git-scm.com) as its main data sources.\n\nUnder the hood, DevStats uses the following CNCF projects:\n- Helm (for deployment).\n- containerd (as a Kubernetes container runtime, CRI).\n- cert-manager (for HTTPS/SSL certificates).\n- OpenEBS (for local storage volumes support).\n- MetalLB (as a load balancer for bare metal servers).\n- CoreDNS (Kubernetes internal DNS).\n\nAnd other projects, including:\n- Equinix (bare metal servers provider).\n- Ubuntu (containers base operating system).\n- kubeadm (for installing Kubernetes).\n- NFS (for shared write network volumes support).\n- NGINX (for ingress).\n- Calico (as networking for Kubernetes, CNI).\n- Golang (DevStats is written in Go).\n- PostgreSQL (DevStats database is Postgres).\n- patroni (HA deployment of PostgreSQL database, tweaked for DevStats).\n- GitHub archives (main data source).\n- GitHub API (data source).\n- git (data source).\n- Grafana (UI).\n- Let's Encrypt (provides HTTPS/SSL certificates).\n- Travis CI (continuous integration \u0026 testing).\n\nPlease check [this](https://github.com/cncf/devstats-helm#architecture) for a detailed architecture description.\n\n\n# Deploying on your own project(s)\n\nSee the simple [DevStats example](https://github.com/cncf/devstats-example) repository for single project deployment (Homebrew), follow [instructions](https://github.com/cncf/devstats-example/blob/master/SETUP_OTHER_PROJECT.md) to deploy for your own project.\n\n# Goal\n\nWe want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).\n\nEverything is open source so that it can be used by other CNCF and non-CNCF open source projects.\n\nThe only requirement is that project must be hosted on a public GitHub repository/repositories.\n\n# Data hiding\n\nIf you want to hide your data (replace with anon-#) please follow the instructions [here](https://github.com/cncf/devstats/blob/master/HIDE_DATA.md).\n\n# Forking and installing locally\n\nThis toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases, and multiple Grafana instances.\nIt is written in Go and can be forked and installed by anyone.\n\nContributions and PRs are welcome.\nIf you see a bug or want to add a new metric please create an [issue](https://github.com/cncf/devstats/issues) and/or [PR](https://github.com/cncf/devstats/pulls).\n\nTo work on this project locally please fork the original [repository](https://github.com/cncf/devstats), and:\n- [Compiling and running on Linux Ubuntu 18 LTS](./INSTALL_UBUNTU18.md).\n- [Compiling and running on Linux Ubuntu 17](./INSTALL_UBUNTU17.md).\n- [Compiling and running on Linux Ubuntu 16 LTS](./INSTALL_UBUNTU16.md).\n- [Compiling and running on macOS](./INSTALL_MAC.md).\n- [Compiling and running on FreeBSD](./INSTALL_FREEBSD.md).\n\nPlease see [Development](https://github.com/cncf/devstats/blob/master/DEVELOPMENT.md) for local development guide.\n\nFor more detailed description of all environment variables, tools, switches, etc, please see [Usage](https://github.com/cncf/devstats/blob/master/USAGE.md).\n\n# Metrics\n\nWe want to support all kinds of metrics, including historical ones.\nPlease see [requested metrics](https://docs.google.com/document/d/1o5ncrY6lVX3qSNJGWtJXx2aAC2MEqSjnML4VJDrNpmE/edit?usp=sharing) to see what kind of metrics are needed.\nMany of them cannot be computed based on the data sources currently used.\n\n# Repository groups\n\nThere are some groups of repositories that are grouped together as a repository groups.\nThey are defined in [scripts/kubernetes/repo_groups.sql](https://github.com/cncf/devstats/blob/master/scripts/kubernetes/repo_groups.sql).\n\nTo setup default repository groups:\n- `PG_PASS=pwd ./kubernetes/setup_repo_groups.sh`.\n\nThis is a part of `kubernetes/psql.sh` script and [kubernetes psql dump](https://devstats.cncf.io/backups/gha.dump) already has groups configured.\n\nIn an [All CNCF project](https://all.teststats.cncf.io) repository groups are mapped to individual CNCF projects [scripts/all/repo_groups.sql](https://github.com/cncf/devstats/blob/master/scripts/all/repo_groups.sql):\n\n# Company Affiliations\n\nWe also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.\n\nThere is a project that attempts to create such mapping [cncf/gitdm](https://github.com/cncf/gitdm).\n\nDevStats has an import tool that fetches company affiliations from `cncf/gitdm` and allows to create per company metrics/statistics. It also uses `companies.yaml` file to map company acquisitions (any data generated by a company acquired by another company is assigned to the latter using a mapping from `companies.yaml`).\n\nIf you see errors in the company affiliations, please open a pull request on [cncf/gitdm](https://github.com/cncf/gitdm) and the updates will be reflected on [https://k8s.devstats.cncf.io](https://k8s.devstats.cncf.io) a couple of days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.\n\nNew affiliations are imported into DevStats about 1-2 times/month.\n\n# Architecture\n\nFor architecture details please see [architecture](https://github.com/cncf/devstats/blob/master/ARCHITECTURE.md) file.\n\nDetailed usage is [here](https://github.com/cncf/devstats/blob/master/USAGE.md)\n\n# Adding new metrics\n\nPlease see [metrics](https://github.com/cncf/devstats/blob/master/METRICS.md) to see how to add new metrics.\n\n# Adding new projects\n\nTo add a new project on a bare metal deployment follow [adding new project](https://github.com/cncf/devstats/blob/master/ADDING_NEW_PROJECT.md) instructions.\n\nSee `cncf/devstats-helm`:`ADDING_NEW_PROJECTS.md` for information about how to add more projects on Kubernetes/Helm deployment.\n\n# Grafana dashboards\n\nPlease see [dashboards](https://github.com/cncf/devstats/blob/master/DASHBOARDS.md) to see a list of already defined Grafana dashboards.\n\n# Exporting data\n\nPlease see [exporting](https://github.com/cncf/devstats/blob/master/EXPORT.md).\n\n# Detailed Usage instructions\n\n- [USAGE](https://github.com/cncf/devstats/blob/master/USAGE.md)\n\n# Servers\n\nThe servers to run `devstats` are generously provided by [Equinix](https://metal.equinix.com) bare metal hosting as part of CNCF's [Community Infrastructure Lab](https://github.com/cncf/cluster).\n\n# One line run all projects\n\n- Use `GHA2DB_PROJECTS_OVERRIDE=\"+cncf\" PG_PASS=pwd devstats`.\n- Or add this command using `crontab -e` to run every hour HH:08.\n\n# Checking projects activity\n\n- Use: `PG_PASS=... PG_DB=allprj ./devel/activity.sh '1 month,,' \u003e all.txt`.\n- Example results [here](https://teststats.cncf.io) - all CNCF project activity during January 2018, excluding bots.\n\n\n# Troubleshooting\n\nIf you see error like this `pq: row is too big: size 8192, maximum size 8160` and/or `Error result for xyz (took 11m52.048191357s)`:\n\n- Shell into logging database and check:\n- Run on DevStats node: `k exec -itn devstats-prod devstats-postgres-0 -- psql devstats`.\n- Run while on `devstats` database: `select dt, run_dt, msg from gha_logs where msg like '%Error result for%';`.\n```\n             dt             |           run_dt           |                                  msg\n----------------------------+----------------------------+-----------------------------------------------------------------------\n 2024-09-01 00:48:07.079436 | 2024-09-01 00:34:26.426402 | Error result for helm (took 13m36.712884455s): exit status 2\n 2024-09-07 00:16:11.132541 | 2024-09-07 00:04:14.051939 | Error result for prometheus (took 11m52.048191357s): exit status 2\n 2024-09-07 00:26:43.701404 | 2024-09-07 00:05:55.08925  | Error result for fluentd (took 15m1.328366817s): exit status 2\n 2024-09-07 00:16:11.038887 | 2024-09-07 00:08:43.846938 | Error result for grpc (took 7m24.348182232s): exit status 2\n 2024-09-03 13:20:02.682134 | 2024-09-03 12:57:23.220227 | Error result for opentelemetry (took 22m29.324614973s): exit status 2\n 2024-09-03 13:09:56.535074 | 2024-09-03 13:04:43.451026 | Error result for spinnaker (took 5m7.631109092s): exit status 2\n(6 rows)\n```\n- You can investigate each via: `` echo \"select dt, prog, proj, msg from gha_logs where run_dt = '2024-09-01 00:34:26.426402';\" | k exec -itn devstats-prod devstats-postgres-1 -- psql devstats \u003e log.txt ``.\n- Or: `` select distinct proj from gha_logs where msg like '%row is too big%'; ``.\n- Eventually check: `` vim logs_prod.txt logs_test.txt ``.\n- `row is too big` is usually caused by metric: `suser_activity`. You can add this metric to `./devel/test_metrics.yaml` and generate devstats docker images to reinitialize it for given project(s) via:\n- `helm install --generate-name ./devstats-helm --set namespace='devstats-prod',skipSecrets=1,skipPVs=1,skipBackupsPV=1,skipVacuum=1,skipBackups=1,skipBootstrap=1,skipCrons=1,skipAffiliations=1,skipGrafanas=1,skipServices=1,skipPostgres=1,skipIngress=1,skipStatic=1,skipAPI=1,skipNamespaces=1,testServer='',prodServer='1',provisionImage='lukaszgryglicki/devstats-prod',provisionCommand='./devstats-helm/add_metric.sh',nCPUs=8,indexProvisionsFrom=N,indexProvisionsTo=M`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcncf%2Fdevstats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcncf%2Fdevstats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcncf%2Fdevstats/lists"}