Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/mona-actions/gh-repo-stats-plus

TypeScript version of repo-stats to collect statistics from GitHub repositories
https://github.com/mona-actions/gh-repo-stats-plus

gh-extension github-api migrations repo-stats

Last synced: 2 months ago
JSON representation

TypeScript version of repo-stats to collect statistics from GitHub repositories

Awesome Lists containing this project

README 

          
# gh-repo-stats-plus



A GitHub CLI extension for gathering comprehensive repository statistics from GitHub organizations. This TypeScript implementation builds upon the solid foundation of [mona-actions/gh-repo-stats](https://github.com/mona-actions/gh-repo-stats), adding modern features and performance improvements for enterprise-scale repository analysis.



## 🚀 Quick Start



1. **Install the extension**:



   ```bash

   gh extension install mona-actions/gh-repo-stats-plus

   ```



2. **Authenticate with GitHub**:



   ```bash

   gh auth login

   ```



3. **Collect repository statistics**:



   ```bash

   gh repo-stats-plus repo-stats --org-name my-org

   ```



The tool will generate a CSV file with comprehensive repository statistics in the `./output/` directory (or a custom directory you specify).



## Key Features



This TypeScript rewrite offers several advantages:



1. **Octokit SDK Integration**: Built on GitHub's official Octokit.js SDK, providing:

   - Token renewal

   - Built-in retries

   - Rate limit handling

   - Pagination

   - GraphQL and REST API support



2. **Streaming Processing with Async Generators**: Writes results incrementally as they're processed rather than collecting everything up front, resulting in better memory management and reliability.



3. **State Persistence with Multi-Organization Support**: Saves processing state to organization-specific files (e.g., `last_known_state_.json`) after each successful repository, storing the current cursor position and processed repositories. Each organization maintains its own isolated state, allowing sequential or parallel processing of multiple organizations without conflicts.



4. **Resume Capability**: Can resume operations from the last saved state in case of interruptions or failures.



5. **Smart Duplicate Avoidance**: Skips already processed repositories when resuming to prevent duplicates and save processing time.



6. **Advanced Retry Logic**: Implements exponential backoff strategy for retries to gracefully handle rate limits and transient errors.



7. **Enhanced Debugging**: Easier to debug and maintain with modern TypeScript development tools like VS Code.



8. **Comprehensive Logging**: Detailed logs stored in log files for later review and troubleshooting.



9. **Missing Repositories Detection**: Dedicated command to identify repositories that might have been missed during processing.



10. **Configurable Output Directory**: Control where output files and state files are saved with the `--output-dir` option (defaults to `./output/`) for organized file management.



11. **Project Stats Tracking**: Counts unique ProjectsV2 linked to repositories via issues and directly, based on [jcantosz/Count-repo-projects](https://github.com/jcantosz/Count-repo-projects).



12. **Batch Processing**: Split large organizations into parallel batches using `--batch-size` and `--batch-index`, ideal for GitHub Actions matrix strategies. Includes a `combine-stats` command to merge batch results. See the [Batch Processing Guide](docs/batch-processing.md).



13. **CSV Post-Processing**: Transform and standardize CSV data using configurable rules for pattern matching, value replacement, and indicator column generation. Based on [jcantosz/generate-repo-report/post-process](https://github.com/jcantosz/generate-repo-report/tree/main/post-process). See the [Post-Process Command Reference](docs/commands/post-process.md).



14. **Rows-to-Columns Pivot**: Convert rows from an additional CSV (e.g., migration audit data) into new columns in a base CSV by matching rows and pivoting values. Based on [jcantosz/generate-repo-report/rows-to-columns](https://github.com/jcantosz/generate-repo-report/tree/main/rows-to-columns). See the [Rows-to-Columns Command Reference](docs/commands/rows-to-columns.md).



## Technical Implementation



The extension is built using modern TypeScript patterns with:



- **Async Generators** for streaming large datasets

- **Retry Logic** with exponential backoff

- **Rate Limit Handling** via GitHub Octokit SDK

- **State Persistence** for resumable operations

- **Comprehensive Logging** with Winston

- **Type Safety** throughout the codebase

- **On-demand Building** for clean installation without pre-built artifacts



## Documentation



| Guide                                               | Description                                   |

| --------------------------------------------------- | --------------------------------------------- |

| [Installation](docs/installation.md)                | Prerequisites and installation methods        |

| [Usage Guide](docs/usage.md)                        | Authentication and usage examples             |

| [Commands](docs/commands.md)                        | Complete command reference                    |

| [LFS Sizing](docs/lfs-sizing.md)                    | Git LFS storage analysis per repo             |

| [Development](docs/development.md)                  | Setup and development workflow                |

| [Batch Processing](docs/batch-processing.md)        | Parallel batch processing with GitHub Actions |

| [Post-Processing](docs/commands/post-process.md)    | CSV transformation with configurable rules    |

| [Rows-to-Columns](docs/commands/rows-to-columns.md) | Pivot additional CSV rows into columns        |



## Common Usage Examples



### Basic Organization Analysis



```bash

# Generate repository statistics (output saved to ./output/ directory)

gh repo-stats-plus repo-stats --org-name my-org

```



### Multiple Organizations



Process multiple organizations from a single file:



```bash

# Create an org list file (one org per line)

cat > orgs.txt << EOF

Org1

Org2

Org3

EOF



# Process all organizations with a single command

gh repo-stats-plus repo-stats --org-list orgs.txt



# Add delays between organizations (default: 5 seconds)

gh repo-stats-plus repo-stats --org-list orgs.txt --delay-between-orgs 10



# Continue processing other orgs if one fails

gh repo-stats-plus repo-stats --org-list orgs.txt --continue-on-error



# Combine options

gh repo-stats-plus repo-stats \

  --org-list orgs.txt \

  --delay-between-orgs 10 \

  --continue-on-error \

  --output-dir ./reports

```



> [!NOTE]

> Organizations are processed strictly sequentially. This design choice is intentional to respect GitHub API rate limits and provide predictable resource usage. For large organization lists, consider the configurable delay between organizations and the estimated processing time logged at startup.



Or process organizations individually:



```bash

# Process multiple organizations sequentially (each maintains its own state)

gh repo-stats-plus repo-stats --org-name org1

gh repo-stats-plus repo-stats --org-name org2

gh repo-stats-plus repo-stats --org-name org3



# Use custom output directory (state files are stored here too)

gh repo-stats-plus repo-stats --org-name my-org --output-dir ./reports



# Clean up state file after successful completion

gh repo-stats-plus repo-stats --org-name my-org --clean-state

```



### Custom Output Directory



```bash

# Save output files to a custom directory

gh repo-stats-plus repo-stats --org-name my-org --output-dir /path/to/my/reports



# Use relative path from current directory

gh repo-stats-plus repo-stats --org-name my-org --output-dir reports

```



### Resume Long-Running Collection



```bash

gh repo-stats-plus repo-stats --org-name my-org --resume-from-last-save

```



### High-Volume Processing with GitHub App



```bash

gh repo-stats-plus repo-stats \

  --org-name my-org \

  --app-id 12345 \

  --private-key-file app.pem \

  --app-installation-id 67890 \

  --output-dir /path/to/reports

```



### Find and Process Missing Data



```bash

# Check for missing repositories (looks for CSV in ./output/ by default)

gh repo-stats-plus missing-repos --org-name my-org --file results.csv



# Use custom output directory for missing repos check

gh repo-stats-plus missing-repos \

  --org-name my-org \

  --file results.csv \

  --output-dir /path/to/reports



# Auto-process missing repositories

gh repo-stats-plus repo-stats --org-name my-org --auto-process-missing

```



### Batch Processing



Split a large organization into parallel batches (e.g., for GitHub Actions matrix jobs):



```bash

# Use a dedicated directory for this workflow/run to avoid mixing CSVs from other commands

RUN_OUTPUT_DIR="output/run-$(date +%Y%m%d-%H%M%S)"

mkdir -p "$RUN_OUTPUT_DIR"



# Process batch 0 of 50 repos each

gh repo-stats-plus repo-stats \

  --org-name my-org \

  --batch-size 50 \

  --batch-index 0 \

  --output-dir "$RUN_OUTPUT_DIR"



# Combine only this run's batch CSV files after all batches complete

gh repo-stats-plus combine-stats \

  --files "$RUN_OUTPUT_DIR"/*.csv \

  --output-dir "$RUN_OUTPUT_DIR" \

  --output-file-name combined-stats.csv

```



See the [Batch Processing Guide](docs/batch-processing.md) for complete GitHub Actions workflow examples.



### Project Statistics



```bash

# Count ProjectsV2 linked to repositories via issues

gh repo-stats-plus project-stats --org-name my-org



# Process specific repos from a file

gh repo-stats-plus project-stats --org-name my-org --repo-list repos.txt



# Multiple organizations

gh repo-stats-plus project-stats --org-list orgs.txt --continue-on-error



# Resume interrupted processing

gh repo-stats-plus project-stats --org-name my-org --resume-from-last-save

```



### Post-Processing



```bash

# Post-process a combined CSV with rules

gh repo-stats-plus post-process \

  --input output/combined-stats.csv \

  --rules-file post-process.rules.json



# Custom output file and directory

gh repo-stats-plus post-process \

  --input output/combined-stats.csv \

  --rules-file rules.json \

  --output-file-name final-report.csv \

  --output-dir ./reports

```



See the [Post-Process Command Reference](docs/commands/post-process.md) for rules configuration details and examples. A sample rules file is included at [`docs/examples/post-process.rules.json`](docs/examples/post-process.rules.json).



### Rows-to-Columns



```bash

# Combine repo-stats with migration audit data

gh repo-stats-plus rows-to-columns \

  --base-csv-file output/combined-stats.csv \

  --additional-csv-file output/migration-audit.csv \

  --header-column-keys type \

  --header-column-values message



# Custom column mappings and output

gh repo-stats-plus rows-to-columns \

  --base-csv-file output/stats.csv \

  --additional-csv-file output/audit.csv \

  --header-column-keys type \

  --header-column-values message \

  --base-csv-columns Org_Name,Repo_Name \

  --additional-csv-columns owner,name \

  --output-file-name final-report.csv

```



See the [Rows-to-Columns Command Reference](docs/commands/rows-to-columns.md) for details on how values are parsed and examples.



#### Repo Stats Options



**Organization Selection** (one required):



- `-o, --org-name `: Process a single organization

- `--org-list `: Process multiple organizations from a file (one org per line)



**Multi-Organization Options**:



- `--delay-between-orgs `: Delay between processing organizations (Default: 5)

- `--continue-on-error`: Continue processing other organizations if one fails



**Authentication**:



- `-t, --access-token `: GitHub access token

- `--app-id `: GitHub App ID

- `--private-key `: GitHub App private key

- `--private-key-file `: Path to GitHub App private key file

- `--app-installation-id `: GitHub App installation ID



**Processing Options**:



- `--resume-from-last-save`: Resume from the last saved state

- `--repo-list `: Path to file containing list of repositories to process (format: owner/repo_name)

- `--auto-process-missing`: Automatically process any missing repositories when main processing is complete

- `--clean-state`: Remove state file after successful completion



**Batch Processing**:



- `--batch-size `: Number of repositories per batch

- `--batch-index `: Zero-based index of the batch to process

- `--batch-delay `: Delay before starting a batch (multiplied by batch index to stagger parallel runs)



**Configuration**:



- `-u, --base-url `: GitHub API base URL (Default: )

- `--proxy-url `: Proxy URL if required

- `--output-dir `: Output directory for generated files (Default: ./output)

- `-v, --verbose`: Enable verbose logging



**Performance Tuning**:



- `--page-size `: Number of items per page (Default: 10)

- `--extra-page-size `: Extra page size (Default: 50)

- `--rate-limit-check-interval `: Interval for rate limit checks (Default: 60)

- `--retry-max-attempts `: Maximum number of retry attempts (Default: 3)

- `--retry-initial-delay `: Initial delay for retry (Default: 1000)

- `--retry-max-delay `: Maximum delay for retry (Default: 30000)

- `--retry-backoff-factor `: Backoff factor for retry delays (Default: 2)

- `--retry-success-threshold `: Successful operations before resetting retry count (Default: 5)



#### Project Stats Options



The `project-stats` command supports the same authentication, retry, multi-org, and processing options as `repo-stats` above, with these differences:



- `--page-size `: Number of issues per page (Default: 100)

- No `--extra-page-size` or `--auto-process-missing` options



See the [Commands Reference](docs/commands.md) for the complete list of project-stats options.



## Permissions



The permissions needed by repo-stats-ts depends on the authentication method:



### For Personal Access Token (PAT)



- `repo`: Full control of private repositories

- `read:org`: Read organization membership

- `read:project`: Read project information

- `read:user`: Read user information



### For GitHub App



The app requires `Read-only` permissions to the following:



- Repository Administration

- Repository Contents

- Repository Issues

- Repository Metadata

- Repository Projects

- Repository Pull requests

- Organization Members



## Output



The tool generates:



1. A CSV file with repository statistics (or project statistics for the `project-stats` command)

2. A `last_known_state.json` file with the current processing state

3. Log files in the `logs/` directory



### CSV Output Columns



The CSV output includes detailed information about each repository:



- `Org_Name`: Organization login

- `Repo_Name`: Repository name

- `Is_Empty`: Whether the repository is empty

- `Last_Push`: Date/time when a push was last made

- `Last_Update`: Date/time when an update was last made

- `isFork`: Whether the repository is a fork

- `isArchived`: Whether the repository is archived

- `isTemplate`: Whether the repository is a template repository

- `Visibility`: Repository visibility (e.g., PUBLIC, PRIVATE, INTERNAL)

- `Repo_Size_mb`: Size of the repository in megabytes

- `Record_Count`: Total number of database records this repository represents

- `Collaborator_Count`: Number of users who have contributed to this repository

- `Protected_Branch_Count`: Number of branch protection rules on this repository

- `Ruleset_Count`: Number of rulesets that apply to this repository, inclusive of active rulesets defined at the organization level

- `PR_Review_Count`: Number of pull request reviews

- `Milestone_Count`: Number of issue milestones

- `Issue_Count`: Number of issues

- `PR_Count`: Number of pull requests

- `PR_Review_Comment_Count`: Number of pull request review comments

- `Commit_Comment_Count`: Number of commit comments

- `Issue_Comment_Count`: Number of issue comments

- `Issue_Event_Count`: Number of issue events

- `Release_Count`: Number of releases

- `Project_Count`: Number of projects

- `Branch_Count`: Number of branches

- `Tag_Count`: Number of tags

- `Discussion_Count`: Number of discussions

- `Star_Count`: Number of stargazers

- `Fork_Count`: Number of forks

- `Watcher_Count`: Number of watchers

- `Has_Wiki`: Whether the repository has wiki feature enabled

- `Has_LFS`: Whether the repository has Git LFS tracking configured (see [LFS Detection Limitations](#lfs-detection-limitations))

- `Default_Branch`: Name of the default branch

- `Primary_Language`: Primary programming language of the repository

- `Languages`: Semicolon-separated list of languages with usage percentages (e.g., `TypeScript:85.2%;JavaScript:14.8%`)

- `License`: License identifier (SPDX ID or name)

- `Topics`: Semicolon-separated list of repository topics

- `Description`: Repository description

- `Homepage_URL`: Repository homepage URL

- `Auto_Merge_Allowed`: Whether auto-merge is enabled for pull requests

- `Delete_Branch_On_Merge`: Whether branches are automatically deleted after merging

- `Merge_Commit_Allowed`: Whether merge commits are allowed

- `Squash_Merge_Allowed`: Whether squash merging is allowed

- `Rebase_Merge_Allowed`: Whether rebase merging is allowed

- `Full_URL`: Repository URL

- `Migration_Issue`: Indicates whether the repository might have problems during migration due to:

  - 60,000 or more objects being imported

  - 1.5 GB or larger size on disk

- `Created`: Date/time when the repository was created



### LFS Detection Limitations



The `Has_LFS` column indicates whether the repository's `.gitattributes` file on the default branch contains `filter=lfs` entries. This is a lightweight check performed as part of the existing GraphQL query with no additional API calls.



**Limitations to be aware of:**



- **Default branch only**: The check reads `.gitattributes` from `HEAD` (the default branch). LFS tracking configured only on other branches will not be detected.

- **Root `.gitattributes` only**: Nested `.gitattributes` files in subdirectories are not inspected.

- **Detection, not sizing**: This column only indicates whether LFS is configured — it does not report the number or size of LFS objects.

- **Empty repositories**: Empty repositories will always report `FALSE` since there is no `.gitattributes` file to read.



**For actual LFS sizing**, use the standalone `script/lfs-size.sh` script to inspect individual repositories where `Has_LFS` is `TRUE`. This performs a shallow bare clone and reports per-file LFS sizes and totals. See the [LFS Sizing Guide](docs/lfs-sizing.md) for prerequisites and usage.



### Project Stats CSV Output Columns



The `project-stats` command generates a separate CSV file with the following columns:



- `Org_Name`: Organization login

- `Repo_Name`: Repository name

- `Issues_Linked_To_Projects`: Number of issues that have at least one linked ProjectV2

- `Unique_Projects_Linked_By_Issues`: Count of distinct ProjectV2 items found across all issues

- `Projects_Linked_To_Repo`: Total count of projects directly associated with the repository



## 🛠️ Development Quick Start



```bash

git clone https://github.com/mona-actions/gh-repo-stats-plus.git

cd gh-repo-stats-plus

npm install

npm run build

npm test

```



See the [Development Guide](docs/development.md) for detailed setup instructions.



## Requirements



- **Node.js** 20 or later

- **GitHub CLI** (latest version recommended)

- **GitHub Authentication** (personal token, GitHub App, or GitHub CLI)



## Contributing



We welcome contributions! Please see our [Development Guide](docs/development.md) for setup instructions and guidelines.



## License



This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.