Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ryclarke/batch-tool

This tool provides convenient features for working on multiple git repositories simultaneously.
https://github.com/ryclarke/batch-tool

automation batch-processing tooling

Last synced: 3 months ago
JSON representation

This tool provides convenient features for working on multiple git repositories simultaneously.

Awesome Lists containing this project

README 

          
# Batch Tool



This tool provides a suite of functionality for performing common tasks across multiple git repositories, including branch management and pull request creation.



## Features



- **Git Operations**: Create and push branches and commits, and manage history

- **Pull Request Management**: Create, edit, and merge pull requests (supports GitHub and Bitbucket)

- **Repository Catalog**: Cache repository metadata for quick matching and topical grouping

- **Make Target Execution**: Run make targets across multiple repositories

- **Flexible Configuration**: YAML-based configuration with repository aliases and default reviewers



## Getting Started



**TL;DR for new users:**

1. `make install` - Installs batch-tool to your system

2. Create a `batch-tool.yaml` file with your git provider and repositories

3. Run `batch-tool git status ` to get started



For detailed setup instructions, see the sections below.



## Installation



### Pre-built Binaries



Download and unpack the binary for your platform from [the latest release](https://github.com/ryclarke/batch-tool/releases/latest).



### Build and Install from Source



For the fastest setup, use Make to install directly to your Go bin directory:



```bash

git clone git@github.com:ryclarke/batch-tool.git

cd batch-tool



make install

```



You'll need a Go environment set up with GOPATH set. See [the Go getting started docs](https://golang.org/doc/install) for more info.



This will automatically build the tool and install it to `$GOPATH/bin` (or `~/go/bin` if `GOPATH` is not set).



## Quick Start



### 1. Install



If building from source, install the tool to your system:



```bash

make install

```



Make sure `$GOPATH/bin` (or `~/go/bin`) is in your `PATH`.



### 2. Configuration



Create a configuration file `batch-tool.yaml` in your working directory or user config directory (`~/.config`), or specify a file path manually with `--config`:



```yaml

git:

  provider: github

  host: github.com

  project: your-username-or-org

  default-branch: main # fallback for a repo with no default branch

  stash-updates: false # if true, automatically stash/restore changes during git update



channels:

  output-style: tui  # Modern TUI interface (default); use "native" for scripts



repos:

  sort: true  # sort output alphabetically by repository name



  # don't match repos containing the following topics unless explicitly added

  skip-unwanted: true

  unwanted-labels: deprecated,poc



  # aliases act like custom topics for referencing and grouping repoistories

  aliases:

    myproject:

      - repo1

      - repo2

      - repo3



  # list of default reviewers to request for each repository

  reviewers:

    repo1:

      - reviewer1

      - reviewer2

```



The only required field is `git.project`, the rest of the configuration has safe default values.



### 3. Authentication



For repository discovery and pull request operations, you'll need to configure authentication:



- **GitHub**: Set up a [personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)

- **Bitbucket**: Configure an [API token](https://support.atlassian.com/bitbucket-cloud/docs/using-api-tokens/)



The authentication token should be provided via the `AUTH_TOKEN` environment variable (recommended) or the `auth-token` field in the batch-tool config file.



### 4. Basic Usage



Check the status of multiple repositories:

```bash

batch-tool git status repo1 repo2 repo3

```



Repositories can also be referenced by Github [Topics](https://github.com/topics) or Bitbucket [Labels](https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-labels#api-group-labels):

```bash

batch-tool git status '~libraries'

```



You may also use the same syntax to refer to aliases defined locally in the config file.



- To refer to an alias or topic, include `~` in the argument as seen above.

- To invert a match to *exclude* a repository or alias/topic, include a `!`.

- To force selection of a repository or alias/topic, include a `+`.

  - This bypasses unwanted label filtering and ignores exclusions from other arguments.

  - If applied to an alias/topic, _every_ member will be included regardless of other filters.

- To operate on the current working directory only, pass `.` as the sole repository argument.

  - In this mode, argument expansion (aliases/topics/labels) is skipped.

  - No repository batching occurs; the command runs once against the current directory.



-------------------------------



ℹ️ The `~all` alias is defined implicitly and refers to all discovered repositories for the configured namespace (user profile or organization).



-------------------------------



Example:

```bash

# repos.aliases:

#   myservice: [repo1 repo2 repo3 repo4]

#   deprecated: [repo4]

# repos.unwanted_labels: [deprecated]

#

batch-tool git status '~myservice' '!repo3' # matches repo1 and repo2 only

batch-tool git status '~myservice' '+repo4' # forces inclusion of repo4

batch-tool git status '+~myservice' '!~deprecated' # matches all 4 repos

batch-tool pr get . # run against current directory without alias/label expansion

```



⚠️ When using special characters for matching and exclusion, ensure that all arguments are quoted properly to avoid improper shell expansion.



## Interactive Mode



When the `--style=tui` flag is provided, batch-tool launches an interactive terminal user interface (TUI) for command selection and navigation.



### Using Interactive Mode



```bash

# Launch the interactive command selector

batch-tool --style=tui

```



The TUI provides:

- **Visual command navigation**: Browse through available commands and subcommands

- **Help information**: View command descriptions at each level

- **Keyboard navigation**: Use arrow keys, vim-style (hjkl), or Enter to navigate

- **Breadcrumb trail**: See your current location in the command hierarchy



### Keyboard Controls



- `↑/k` - Move cursor up

- `↓/j` - Move cursor down

- `←/h/backspace` - Go back to parent command

- `→/l/enter` - Select command or enter subcommand

- `q/ctrl+c` - Quit



### Output Styles



The `--style` flag also controls how command output is displayed:



- `tui` (default): Modern TUI with interactive progress display, real-time updates, and scrollable output

- `native`: Traditional terminal output with streaming updates



Example with output style:

```bash

# Use interactive TUI for output display

batch-tool git status repo1 repo2 --style=tui

```



## Commands



### Git Operations



```bash

# Check status across repositories

batch-tool git status 



# Create new branches for each repository

batch-tool git branch -b "" 



# Checkout the default branches and pull any upstream changes (destroys uncommitted changes)

batch-tool git update 



# Show diff information in the working trees

batch-tool git diff 



# Commit and push changes

batch-tool git commit -m "commit message" 

```



#### Git Update with Stash



The `git update` command can automatically save and restore uncommitted changes using `git stash`. This is useful when you have local modifications that you want to preserve while updating to the latest default branch.



**Configuration:**

```yaml

git:

  stash-updates: true  # Enable automatic stash by default

```



**Usage:**

```bash

# Enable stash just for this command

batch-tool git update --stash 



# Or use the inverted flag to disable stash if it's enabled in config

batch-tool git update --no-stash 

```



**Safety Features:**

- Only works with stashes created by batch-tool (identified by `batch-tool YYYY-MM-DDTHH:MM:SSZ` message)

- Safely handles clean working directories (no changes to stash)

- Fails explicitly if stashed changes cannot be restored

- Can be globally enabled in config, but can be disabled per-command with `--no-stash`



### Pull Request Management



**Note**: Requires authentication token configuration.



```bash

# Create new pull requests

batch-tool pr new -t "PR Title" -d "Description" 



# Create PRs with user and team reviewers

batch-tool pr new -r reviewer1 -R my-org/platform-team 



# Edit existing pull requests

batch-tool pr edit -t "New Title" -d "New Description" 



# Add requested reviewers by username

batch-tool pr edit -r reviewer1 -r reviewer2 



# Add requested team reviewers by slug (GitHub provider)

batch-tool pr edit -R my-org/backend-team -R my-org/frontend-team 



# Replace existing reviewers (users and teams) instead of appending

batch-tool pr edit -r reviewer1 -R my-org/platform-team --reset-reviewers 



# Merge with explicit method and status check behavior

batch-tool pr merge -m squash --check 

batch-tool pr merge --force 



# Merge all accepted pull requests

batch-tool pr merge 

```



### Miscellaneous



```bash

# Generate autocompletion script for the specified shell

batch-tool completion 



# Execute make targets

batch-tool make -t  



# Execute arbitrary shell commands across repositories

## (DANGEROUS - use with caution) ##

batch-tool sh -c "command to execute" 



# Execute a script file or executable binary across repositories

## (DANGEROUS - use with caution) ##

batch-tool sh -f /path/to/script.sh 



# Pass arguments to the script using -a/--args flag (repeatable)

batch-tool sh -f /path/to/script.sh -a arg1 -a arg2 



# Skip confirmation prompt with -y flag

batch-tool sh -y -c "echo 'safe command'" 



# Test repository filter rules against topics and local aliases

batch-tool labels 



# View local repository metadata

batch-tool catalog



# Run synchronously (useful for computationally-expensive operations)

batch-tool --sync  

```



## Global Flags



- `--config string`: Specify config file (default: `batch-tool.yaml`)

- `--style string` / `-o`: Output style (`tui` or `native`, default: `tui`)

- `--print` / `-p`: Print results to stdout after processing is complete

- `--wait` / `--no-wait`: Wait for user input after processing (auto-disabled in non-interactive output)

- `--env` / `-e` (repeatable): Environment variables to set for command execution (`KEY=VALUE`)

- `--sync`: Execute commands synchronously (alias for `--max-concurrency=1`)

- `--max-concurrency int`: Maximum number of concurrent operations (default: number of logical CPUs)

- `--sort`: Sort repositories (default: true)

- `--skip-unwanted`: Skip repositories with unwanted labels (default: true)



## Configuration Reference



### Git Provider Settings



```yaml

git:

  provider: github | bitbucket

  host: github.com  # or your Bitbucket server

  project: your-org-or-username

  default-branch: main | develop

  stash-updates: false          # Stash/restore local changes during git update by default

  default-merge-method: squash  # PR merge method: merge | squash | rebase

  directory: /path/to/git/repos  # Base directory for repository clones

```



#### Repository Directory Structure



The `git.directory` option configures the base directory where repositories will be cloned. When specified, repositories are automatically organized in subdirectories that mirror the git provider's structure:



```

$GIT_DIRECTORY/

├── github.com/

│   ├── myorg/

│   │   ├── repo1/

│   │   ├── repo2/

│   │   └── repo3/

│   └── anothorg/

│       └── shared-repo/

└── bitbucket.example.com/

    └── myproject/

        └── api-service/

```



**Default behavior**: If not specified, defaults to `$GOPATH/src` if `GOPATH` is set, otherwise uses the current working directory.



### Repository Settings



```yaml

repos:

  sort: true                   # Sort repository output

  skip-unwanted: true          # Skip repos with unwanted labels

  unwanted-labels:             # Labels to skip when skip-unwanted is true

    - deprecated

    - poc

    - archived



  aliases:                     # Group repositories under aliases

    frontend:

      - web-app

      - mobile-app

    backend:

      - api-server

      - worker-service



  reviewers:                   # Default reviewers per repository

    web-app:

      - frontend-team

    api-server:

      - backend-team

```



### Output Style and Concurrency Settings



```yaml

channels:

  output-style: tui          # Output style: "tui" (default) or "native" (fallback)

  buffer-size: 100           # Channel buffer size for console output (default: 100)

  max-concurrency: 8         # Maximum concurrent operations (default: number of logical CPUs)

```



#### Output Styles



The `output-style` setting controls how command output is displayed:



- **`tui`** (default): Modern terminal UI with real-time updates, styled output, and progress indicators. Features include:

  - Live progress tracking with completion status

  - Styled repository names and status indicators

  - Real-time output streaming with full scrolling support

  - Color-coded errors and messages

  - Elapsed time display

  - Keyboard controls for navigation, including support for Vim keybinds!



- **`native`**: Low-complexity fallback with traditional sequential output with repository headers. Suitable for legacy or non-interactive environments, scripts, and CI/CD pipelines.



> **Note**: The TUI style is recommended for interactive use and provides a better experience for operations across many repositories and/or using long-running commands.



#### Concurrency Control



The `max-concurrency` setting controls how many repositories are processed simultaneously. This is useful for:



- **Resource-intensive operations**: Reduce concurrency to avoid overwhelming the system

- **Rate-limited APIs**: Prevent hitting API rate limits when working with pull requests

- **Network-bound operations**: Balance between speed and stability



**Examples:**

- `max-concurrency: 1` - Process repositories one at a time (equivalent to `--sync`)

- `max-concurrency: 5` - Conservative setting for API operations

- `max-concurrency: 20` - Aggressive setting for local git operations



**Tip**: The default concurrency is set to the number of logical CPUs on your system. Start with this default and adjust based on your specific use case and system capabilities.



## Examples



### Daily Workflow



1. **Morning sync**: Update all repositories to latest

```bash

batch-tool git update '~all'

```



2. **Create feature branch**: Start new work across multiple repos

```bash

batch-tool git branch -b feature/new-feature '~frontend' '~backend'

```



3. **Check status**: See what's changed

```bash

batch-tool git status '~frontend' '~backend'

```



4. **Create pull requests**: Submit your changes

```bash

batch-tool pr new -t "Add new feature" -d "Detailed description" '~frontend' '~backend'

```



### Maintenance Tasks



1. **Run tests across projects**:

```bash

batch-tool make -t test '~myproject'

```



2. **Format code**:

```bash

batch-tool make -t format '~myproject'

```



3. **Synchronous operations** (when needed):

```bash

batch-tool --sync make -t build '~myproject'

```



4. **Execute custom commands** (use with caution):

```bash

# Example: Check Go version across repositories

batch-tool sh -c "go version" '~myproject'



# Example: Clean up temporary files

batch-tool sh -c "rm -f *.tmp" '~myproject'

```



## Tips



- Use repository aliases in your config to group related repositories

- The `--sync` flag is useful for operations that must run sequentially

- Repository labels help organize and filter your catalog

- Default reviewers in config save time when creating pull requests

- The tool works from any directory - it will find repositories based on your configuration

- **⚠️ Shell Command Safety**: The `sh` command is powerful but dangerous. It will prompt for confirmation before executing any command across multiple repositories. Use with extreme caution, especially with destructive commands.



## Troubleshooting



- **Authentication errors**: Ensure your API token is properly configured

- **Repository not found**: Check that repository names match your git provider

- **Sync issues**: Use `--sync` flag for operations that need to run sequentially

- **Config issues**: Verify your `batch-tool.yaml` syntax and paths



For more detailed help on any command, use:

```bash

batch-tool [command] --help

```

## Development



### Prerequisites



- Go 1.25 or later

- Make



### Setup



1. Clone the repository:

```bash

git clone git@github.com:ryclarke/batch-tool.git

cd batch-tool

```



2. Install development tools:

```bash

make deps

```



This installs:

- `gotestsum` - Enhanced test output

- `goreleaser` - Release automation

- `golangci-lint` - Code quality and style enforcement



### Building



Build for your current platform:

```bash

make build

```



### Testing



Run tests:

```bash

make test

```



Run tests with coverage:

```bash

make cover

```



### Linting



Run linters to check code quality:

```bash

make lint

```



Auto-fix linting issues where possible:

```bash

make lint-fix

```



### Code Style



This project uses `golangci-lint` to enforce consistent code style. Key requirements:



- **Import Grouping**: Imports must be grouped in order:

  1. Standard library packages

  2. Third-party dependencies

  3. Internal project packages (github.com/ryclarke/batch-tool/*)



- **Error Handling**: All errors must be checked (use `_ =` to explicitly ignore)

- **Code Quality**: Follow standard Go idioms and best practices



The CI pipeline runs linting automatically on all pull requests.