Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/miztizm/farmore

Farmore is a comprehensive Python CLI tool for backing up GitHub repositories and their associated data. Clone repositories, export issues, download releases, backup wikis, and more — all with a single command.
https://github.com/miztizm/farmore

clone github mirror mirrored-repository pull repository

Last synced: 3 months ago
JSON representation

Farmore is a comprehensive Python CLI tool for backing up GitHub repositories and their associated data. Clone repositories, export issues, download releases, backup wikis, and more — all with a single command.

Awesome Lists containing this project

README 

          
# 🥔 Farmore



> _"Mirror every repo you own — in one command."_



**Farmore** is a comprehensive Python CLI tool for backing up GitHub repositories and their associated data. Clone repositories, export issues, download releases, backup wikis, and more — all with a single command.



**Version:** 0.9.0

**License:** MIT

**Python:** 3.10+



---



## ✨ Features



### 🔄 Repository Management

- **Bulk backups** - Clone all repos for a user or organization in one command

- **Single repo backups** - Backup individual repositories with the `repo` command

- **Repository search** - Search GitHub and clone matching repositories by keyword

- **Smart updates** - Automatically pulls updates for existing repositories

- **Parallel processing** - Fast backups with configurable worker threads

- **SSH/HTTPS support** - Tries SSH first, falls back to HTTPS with token

- **Bare/Mirror clones** - Use `--bare` flag for true 1:1 repository backups

- **Git LFS support** - Use `--lfs` flag for repositories with large files

- **Name regex filtering** - Filter repositories by name pattern with `--name-regex`

- **Incremental backups** - Track backup state with `--incremental` flag



### 📊 Data Export

- **Issues export** - Export all issues to JSON/YAML with optional comments

- **Pull requests export** - Export PRs with metadata and comments

- **Workflows backup** - Backup GitHub Actions workflows and run history

- **Releases download** - Download releases with metadata and binary assets

- **Wiki backup** - Clone repository wikis as git repositories

- **Labels export** - Backup repository labels with colors

- **Milestones export** - Backup milestone configuration and progress

- **Gists backup** - Clone all your GitHub gists

- **Attachments download** - Download images/files from issues and PRs



### 🔐 Access & Security

- **Private repository support** - Full access with GitHub Personal Access Tokens

- **Organization repos** - Backup all organization repositories

- **Starred & watched repos** - Mirror repositories you've starred or are watching

- **Secrets export** - Export repository secret names (values are never exposed)

- **Followers/Following** - Export user's social connections

- **Webhooks export** - Backup webhook configurations



### 🎯 Advanced Features

- **Flexible filtering** - By visibility (public/private/all), forks, archived status

- **Rate limit handling** - Automatic retries with exponential backoff

- **Organized structure** - Clean directory organization separating code from data

- **Cross-platform** - Works on Linux, macOS, and Windows

- **Beautiful CLI** - Powered by Typer and Rich with progress bars

- **GitHub Enterprise** - Support via `--github-host` flag

- **Discussions backup** - Export repository discussions (GraphQL API)

- **Projects backup** - Export Projects v2 data (GraphQL API)



---



## 📦 Installation



### Requirements



**Python Version:** 3.10 or higher is required.



**Additional Requirements:**

- Git installed and available in PATH

- GitHub Personal Access Token (for private repos and higher rate limits)



---



### 🎯 From PyPI (Recommended)



The easiest way to install Farmore is from the Python Package Index (PyPI):



```bash

pip install farmore

```



This is the recommended method for end users. Once installed, the `farmore` command will be available globally.



**Verify installation:**

```bash

farmore --version

farmore --help

```



---



### 🧪 From TestPyPI (Pre-release Testing)



To test pre-release versions before they're published to PyPI:



```bash

pip install --index-url https://test.pypi.org/simple/ farmore

```



**When to use this:**

- Testing new features before official release

- Helping with beta testing

- Verifying bug fixes in development versions



**Note:** TestPyPI packages may not have all dependencies available. You might need to install dependencies from regular PyPI separately.



---



### 📥 From GitHub Releases (Specific Versions)



Download a specific version from the [GitHub Releases page](https://github.com/miztizm/farmore/releases):



1. **Download the `.whl` file** from the release you want (e.g., `farmore-0.3.0-py3-none-any.whl`)

2. **Install the downloaded file:**



```bash

pip install farmore-0.3.0-py3-none-any.whl

```



**When to use this:**

- You need a specific version

- You want to verify package integrity

- You're installing in an offline environment (download first, install later)



**Alternative - Source distribution:**

```bash

# Download the .tar.gz file instead

pip install farmore-0.3.0.tar.gz

```



---



### 🔧 From Source (Development)



For developers who want to contribute or modify the code:



```bash

# Clone the repository

git clone https://github.com/miztizm/farmore.git

cd farmore



# Install in editable mode (changes to code take effect immediately)

pip install -e .



# Or install with development dependencies (recommended for contributors)

pip install -e ".[dev]"

```



**Development dependencies include:**

- `pytest` - Testing framework

- `pytest-cov` - Code coverage

- `ruff` - Linting and formatting

- `mypy` - Type checking

- Additional testing utilities



**Verify installation:**

```bash

farmore --version

farmore --help

```



---



### 🔄 Upgrading



To upgrade to the latest version:



```bash

# From PyPI

pip install --upgrade farmore



# From TestPyPI

pip install --upgrade --index-url https://test.pypi.org/simple/ farmore

```



---



### 🗑️ Uninstalling



To remove Farmore:



```bash

pip uninstall farmore

```



---



## 🚀 Quick Start



```bash

# Backup all repos for a user

farmore user miztizm



# Backup a single repository

farmore repo microsoft/vscode



# Search and clone repositories by keyword

farmore search "nuxt laravel" --limit 10



# Backup with issues and pull requests

farmore repo miztizm/hello-world --include-issues --include-pulls



# Backup everything for a repository

farmore repo python/cpython --all



# Backup all repos for an organization

farmore org github --include-issues --include-pulls



# With authentication (recommended)

export GITHUB_TOKEN=ghp_your_token_here

farmore user miztizm

```



_"They say privacy is dead. Prove them wrong. Use a token."_ — schema.cx



---



## 🔑 Authentication



Farmore uses GitHub Personal Access Tokens (PAT) for authentication. Tokens provide:

- Access to private repositories

- Higher rate limits (5,000 vs 60 requests/hour)

- Organization repository access



### Creating a Token



**⭐ Recommended: Use Classic Personal Access Token**



1. **Create a Classic PAT:** https://github.com/settings/tokens

   - Click **"Tokens (classic)"** → **"Generate new token (classic)"**

   - Give it a name: `farmore-backup`

   - Select scope: ✅ **`repo`** (required for private repositories)

   - Optional: ✅ **`delete_repo`** (only if using `farmore delete` command)

   - Click "Generate token" and **copy it immediately**



2. **Set environment variable:**



```bash

# Linux/macOS

export GITHUB_TOKEN=ghp_your_token_here



# Windows PowerShell

$env:GITHUB_TOKEN="ghp_your_token_here"



# Or create a .env file in the project directory

echo "GITHUB_TOKEN=ghp_your_token_here" > .env

```



### Why Classic PAT?

- ✅ Simple setup (just check `repo` scope)

- ✅ Works with all repository types (personal + organization)

- ✅ Proven reliability

- ✅ Broader API compatibility



### Rate Limits



| Mode | Requests/Hour | Use Case |

|------|---------------|----------|

| ❌ Unauthenticated | 60 | Small public repos only |

| ✅ Authenticated | 5,000 | Production, private repos |



### Security Best Practices



- ✅ Use environment variables or `.env` files

- ✅ Set token expiration (90 days recommended)

- ✅ Use minimal required permissions

- ❌ Never commit tokens to version control

- ❌ Avoid `--token` flag (exposes in shell history)



---



## 📚 Commands



Farmore provides **14 commands** organized into 4 categories:



### 🔄 Repository Backup



#### `farmore user `

Backup all private and public repositories for a GitHub user.



```bash

# Backup all repos

farmore user miztizm



# Backup with data exports

farmore user miztizm --include-issues --include-pulls



# Filter by visibility

farmore user miztizm --visibility public



# Dry run

farmore user miztizm --dry-run

```



**Key Options:** `--visibility`, `--include-forks`, `--include-archived`, `--include-issues`, `--include-pulls`, `--include-workflows`, `--include-releases`, `--include-wikis`, `--max-workers`, `--dry-run`



#### `farmore org `

Backup all repositories for an organization. Same options as `user`.



#### `farmore repo /`

Backup a single repository with optional data exports.



```bash

# Just clone

farmore repo microsoft/vscode



# Clone + data

farmore repo microsoft/vscode --include-issues --include-pulls



# Everything

farmore repo python/cpython --all

```



**Key Options:** `--include-issues`, `--include-pulls`, `--include-workflows`, `--include-releases`, `--include-wikis`, `--all`



#### `farmore search ` 🆕

Search GitHub repositories by keyword and clone matching results.



```bash

# Basic search - clone top 10 results

farmore search "smsbomber"



# Search with filters

farmore search "machine learning" --language python --min-stars 1000 --limit 20



# Search and auto-confirm (skip prompt)

farmore search "react components" --limit 5 --yes



# Custom output directory

farmore search "awesome-python" --output-dir ./my-collections --limit 15



# Sort by stars (descending)

farmore search "cli tools" --language go --sort stars --order desc --limit 25



# Flat structure (no owner subdirectories)

farmore search "react hooks" --flat-structure --limit 10

```



**Key Options:**

- `--limit` (1-100): Maximum repositories to clone (default: 10)

- `--language`: Filter by programming language (e.g., "python", "javascript", "go")

- `--min-stars`: Minimum number of stars required

- `--sort`: Sort order - "best-match" (default), "stars", "forks", or "updated"

- `--order`: Sort direction - "desc" (default) or "asc"

- `--yes` / `-y`: Skip confirmation prompt

- `--output-dir`: Custom output directory (default: `./search-results//`)

- `--flat-structure`: Clone repos directly without owner subdirectories

- `--workers`: Number of parallel workers for cloning (default: 4)



**Output Structure (Default):**

```

search-results/

└── /

    ├── /

    │   └── /

    ├── /

    │   └── /

    └── /

        └── /

```



**Output Structure (Flat - with `--flat-structure`):**

```

search-results/

└── /

    ├── /

    ├── /

    └── /

```



**Note:** When using `--flat-structure`, repositories with duplicate names will have their owner appended (e.g., `repo-owner`) to avoid conflicts.



**Rate Limits:**

- Search API: 30 requests/minute (authenticated users)

- Unauthenticated: 10 requests/minute



**Note:** The search command uses GitHub's search API with support for advanced search qualifiers. You can also use GitHub's native search syntax directly in the query (e.g., `"language:python stars:>1000"`).



---



### 📊 Data Export



#### `farmore issues /`

Export issues to JSON/YAML.



```bash

farmore issues microsoft/vscode

farmore issues miztizm/hello-world --state open --include-comments

```



**Options:** `--format [json|yaml]`, `--state [all|open|closed]`, `--include-comments`



#### `farmore pulls /`

Export pull requests to JSON/YAML.



```bash

farmore pulls microsoft/vscode

farmore pulls miztizm/hello-world --state open --include-comments

```



**Options:** `--format [json|yaml]`, `--state [all|open|closed]`, `--include-comments`



#### `farmore workflows /`

Backup GitHub Actions workflows.



```bash

farmore workflows microsoft/vscode

farmore workflows actions/checkout --include-runs

```



**Options:** `--include-runs`



#### `farmore releases /`

Download releases and assets.



```bash

farmore releases microsoft/vscode

farmore releases nodejs/node --download-assets

```



**Options:** `--download-assets`



#### `farmore wiki /`

Clone repository wiki.



```bash

farmore wiki python/cpython

```



---



### 🔍 Profile & Discovery



#### `farmore profile [username]`

Export user profile.



```bash

farmore profile              # Your profile

farmore profile miztizm      # Another user's profile

```



**Options:** `--format [json|yaml]`



#### `farmore starred [username]`

Mirror starred repositories.



```bash

farmore starred              # Your starred repos

farmore starred miztizm      # Another user's starred repos

```



#### `farmore watched [username]`

Mirror watched repositories.



```bash

farmore watched

```



---



### 🔐 Security & Management



#### `farmore secrets /`

Export repository secret names (values never exposed).



```bash

farmore secrets miztizm/farmore

```



**Options:** `--format [json|yaml]`



#### `farmore delete /`

Delete a repository (requires confirmation).



```bash

farmore delete miztizm/old-project

farmore delete miztizm/test-repo --force  # Skip confirmation

```



**⚠️ Warning:** Requires `delete_repo` scope in your GitHub token.



---



## 📂 Directory Structure



Farmore organizes backups with a clean structure that separates code from metadata:



```

backups/

└── /

    ├── profile.json                          # User profile

    ├── repos/                                # Git repositories

    │   ├── private///

    │   ├── public///

    │   ├── starred///

    │   ├── watched///

    │   ├── organizations///

    │   └── forks///

    └── data/                                 # Metadata

        ├── issues/__issues.json

        ├── pulls/__pulls.json

        ├── workflows/_/

        ├── releases/_/

        ├── wikis/_.wiki/

        └── secrets/__secrets.json

```



**Example:**

```bash

# After: farmore user miztizm --include-issues --include-pulls

backups/miztizm/

├── profile.json

├── repos/

│   ├── public/miztizm/farmore/

│   └── private/miztizm/secret-project/

└── data/

    ├── issues/

    │   ├── miztizm_farmore_issues.json

    │   └── miztizm_secret-project_issues.json

    └── pulls/

        ├── miztizm_farmore_pulls.json

        └── miztizm_secret-project_pulls.json

```



---



## 🔧 How It Works



1. **Discovery** - Uses GitHub API to find repositories

2. **Filtering** - Applies visibility, fork, and archived filters

3. **Parallel Processing** - Processes multiple repos simultaneously

4. **Smart Cloning** - Tries SSH first, falls back to HTTPS

5. **Progress Reporting** - Real-time progress with rich output

6. **Summary** - Final statistics and error reporting



**Key Features:**

- **Incremental backups** - Updates existing repos, clones new ones

- **Error resilience** - Continues even if individual repos fail

- **Rate limit aware** - Handles GitHub API limits automatically

- **Organized output** - Auto-categorizes by type



---



## 🛠️ Development



### Setup



```bash

git clone https://github.com/miztizm/farmore.git

cd farmore

python -m venv venv

source venv/bin/activate  # Windows: venv\Scripts\activate

pip install -e ".[dev]"

```



### Testing



```bash

pytest                                    # Run all tests

pytest --cov=farmore --cov-report=html   # With coverage

pytest tests/test_github_api.py          # Specific test

```



### Code Quality



```bash

black farmore/        # Format

ruff check farmore/   # Lint

mypy farmore/         # Type check

```



---



## 📄 License



MIT License - see [LICENSE](LICENSE) file for details.



---



## 🤝 Contributing



Contributions welcome! Please:

1. Check existing issues and PRs

2. Follow the existing code style

3. Add tests for new features

4. Update documentation



---



## 💬 Support



- 🐛 For bugs, [**Issues**](https://github.com/miztizm/farmore/issues)

- 📩 For questions, [**Email**](mailto:github@miztizm.com)



---



## 🌟 Acknowledgments



Built with [Typer](https://typer.tiangolo.com/), [Rich](https://rich.readthedocs.io/), [Requests](https://requests.readthedocs.io/), and [PyYAML](https://pyyaml.org/).



---



_"Control is an illusion. But backups? Those are real."_ — schema.cx



**Made with 🥔 by miztizm**