{"id":44779027,"url":"https://github.com/hyperi-io/repo-backup","last_synced_at":"2026-03-06T06:02:59.679Z","repository":{"id":320968975,"uuid":"1043667392","full_name":"hyperi-io/repo-backup","owner":"hyperi-io","description":"Repository backup tool for GitHub, GitLab, and Bitbucket platforms","archived":false,"fork":false,"pushed_at":"2025-11-25T07:09:51.000Z","size":190,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-02-16T12:59:45.315Z","etag":null,"topics":["automation","backup","git"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperi-io.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-24T11:13:29.000Z","updated_at":"2026-01-15T08:39:09.000Z","dependencies_parsed_at":"2025-10-27T03:29:20.363Z","dependency_job_id":null,"html_url":"https://github.com/hyperi-io/repo-backup","commit_stats":null,"previous_names":["hypersec-io/repo-backup","hyperi-io/repo-backup"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/hyperi-io/repo-backup","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Frepo-backup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Frepo-backup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Frepo-backup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Frepo-backup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperi-io","download_url":"https://codeload.github.com/hyperi-io/repo-backup/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperi-io%2Frepo-backup/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30164532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T04:43:31.446Z","status":"ssl_error","status_checked_at":"2026-03-06T04:40:30.133Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","backup","git"],"created_at":"2026-02-16T07:19:48.647Z","updated_at":"2026-03-06T06:02:58.943Z","avatar_url":"https://github.com/hyperi-io.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Repository Backup Tool\n\nA straightforward enterprise tool for backing up Git repositories from GitHub, GitLab, and Bitbucket\n\n[![semantic-release: conventional](https://img.shields.io/badge/semantic--release-conventional-e10079?logo=semantic-release)](https://github.com/semantic-release/semantic-release)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n\nWe built this tool because we needed a reliable way to backup all our repositories across different platforms. It handles the heavy lifting of discovering repos, cloning them efficiently, and storing them either locally or in S3. Nothing fancy, just solid backup automation that works.\n\n## What It Does\n\n- Backs up repositories from GitHub, GitLab, and Bitbucket\n- Works with corporate/organization repositories by default\n- Uploads directly to S3 or saves locally\n- Creates git bundles (preserves complete history) or tar archives\n- Processes multiple repos in parallel for speed\n- Filters repos by name patterns if needed\n- Handles multiple accounts per platform\n- Shows progress bars so you know what's happening\n\n## What You'll Need\n\n- Python 3.11 or newer\n- Git installed on your system\n- Access tokens for the platforms you want to backup (or use auto-discovery from CLI tools)\n- AWS account with S3 (optional, for cloud backups)\n\n## Getting Started\n\n### Installation\n\nInstall with uv (recommended - isolated environment):\n\n```bash\n# Install uv first if you don't have it\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# Install repo-backup system-wide (isolated venv)\nuv tool install git+https://gitlab.com/hypersec-repo/repo-backup\n\n# Or from PyPI (when published)\n# uv tool install repo-backup\n\n# Check it worked\nrepo-backup --help\n```\n\nThis installs `repo-backup` in its own isolated virtual environment with the executable available on your PATH.\n\nOr run without installing:\n\n```bash\n# One-off execution\nuvx --from git+https://gitlab.com/hypersec-repo/repo-backup repo-backup --help\n```\n\n### Developer Setup\n\nIf you're planning to contribute or modify the code:\n\n```bash\n# Get the code\ngit clone https://github.com/hypersec-io/repo-backup.git\ncd repo-backup\n\n# Install with dev and test dependencies\nuv sync --extra dev --extra test\n\n# Run the local CI checks\n./scripts/ci\n\n# Try it out\nuv run repo-backup local /tmp/test-backup --test\n```\n\n### Upgrading\n\n```bash\n# Upgrade to latest version\nuv tool upgrade repo-backup\n\n# Or upgrade all tools\nuv tool upgrade --all\n```\n\n### Uninstalling\n\n```bash\nuv tool uninstall repo-backup\n```\n\n## Setting Things Up\n\n### Step 1: S3 Setup (If You Want Cloud Backups)\n\n**Note:** Check out [AWS.md](AWS.md) for the full AWS configuration guide if you need it.\n\nThe tool can set up your S3 bucket automatically. You'll need AWS permissions to create S3 buckets and IAM users - not full admin access, just those specific permissions.\n\nQuick setup:\n\n```bash\n# Basic setup using your current AWS profile\nrepo-backup s3 --setup\n\n# Use a specific AWS profile for setup\nrepo-backup s3 --setup --profile admin-profile\n\n# Enable Glacier for cheaper long-term storage\nrepo-backup s3 --setup --enable-glacier\n\n# Use your own bucket name\nrepo-backup s3 --setup --bucket-name my-backups --region us-east-1\n```\n\nThis creates everything you need:\n- S3 bucket with a unique name (or your chosen name)\n- Versioning enabled for backup history\n- Encryption turned on\n- Public access blocked\n- Lifecycle policies configured\n- Dedicated IAM user with minimal permissions\n- AWS CLI profile set up\n- `.env` file with all the settings\n- Quick test to make sure it works\n\nAfter setup, you'll see the bucket name and profile info. The `.env` file will have your S3 config ready to go.\n\n### Step 2: Add Your Git Platform Tokens\n\nFirst, copy the example config (skip this if S3 setup already created one):\n\n```bash\ncp .env.example .env\n```\n\nThen add your tokens to `.env`:\n\n```bash\n# Platform tokens - get these from your platforms (see below)\nGITHUB_TOKEN=ghp_your_token_here\nGITLAB_TOKEN=glpat_your_token_here\nBITBUCKET_TOKEN=ATCTT_your_token_here\nBITBUCKET_WORKSPACE=your-workspace\n\n# Where to save backups\nLOCAL_BACKUP_PATH=/mnt/backups/repo-backup\nAWS_S3_BUCKET=repo-backup-123456789  # Set by S3 setup\n\n# AWS settings (filled by S3 setup)\nAWS_PROFILE=repo-backup-profile\nAWS_REGION=us-west-2\n\n# Optional tweaks\nPARALLEL_WORKERS=5\nBACKUP_METHOD=direct\n```\n\n### Auto-Discovery of Tokens\n\nThe tool can automatically discover tokens from standard CLI tool configurations if they're not set in `.env`:\n\n- **GitHub**: Reads from `gh` CLI (`gh auth token`) or `GH_TOKEN`/`GITHUB_TOKEN` env vars\n- **GitLab**: Reads from `glab` CLI config (`~/.config/glab-cli/config.yml`) or `GITLAB_TOKEN` env var\n- **Bitbucket**: Reads from `~/.netrc` file or `BITBUCKET_TOKEN` env var\n- **AWS**: Reads from `~/.aws/credentials` file or standard AWS env vars\n\nThis means if you're already authenticated with `gh`, `glab`, or have AWS credentials configured, the tool will use them automatically.\n\n### Getting Your Access Tokens\n\n*Quick note: These instructions are current as of late 2025. Platform UIs change, so check their docs if something looks different.*\n\n#### GitHub Tokens\n\n**Classic Token (Still Works Great):**\n1. Go to Settings → Developer settings → Personal access tokens → Tokens (classic)\n2. Generate new token (classic)\n3. Set expiration (90 days is reasonable)\n4. Check these scopes:\n   - `repo` - Access to private repositories\n   - `read:org` - See organization repos\n5. Generate and copy the token (starts with `ghp_`)\n6. Save it somewhere safe - you won't see it again\n\n**Fine-grained Token (GitHub's New Way):**\n1. Go to Settings → Developer settings → Personal access tokens → Fine-grained tokens\n2. Generate new token\n3. Name it something like \"repo-backup\"\n4. Pick your repositories or \"All repositories\"\n5. Set permissions:\n   - Contents: Read\n   - Metadata: Read (automatic)\n   - Actions: Read (if you backup workflows)\n6. Generate and copy (starts with `github_pat_`)\n\n#### GitLab Tokens\n\nPretty straightforward:\n1. Go to User Settings → Access Tokens\n2. Name it \"repo-backup\"\n3. Set an expiration date\n4. Check these scopes:\n   - `read_repository`\n   - `read_api`\n5. Create and copy the token (starts with `glpat-`)\n\n#### Bitbucket Tokens\n\n**Heads up:** Bitbucket works differently than the others. Each token only works for one workspace, so you'll need separate tokens for different workspaces.\n\n**Workspace Tokens (Current Method):**\n1. Go to your workspace settings\n2. Find Access tokens under Security\n3. Create token with Repositories: Read permission\n4. Copy the token (starts with `ATCTT`)\n5. Remember to set both token and workspace in `.env`:\n   ```bash\n   BITBUCKET_TOKEN=ATCTT_your_token\n   BITBUCKET_WORKSPACE=your-workspace\n   ```\n\n**App Passwords (Old Method, Still Works):**\n1. Go to Personal settings → App passwords\n2. Create app password named \"repo-backup\"\n3. Give it these permissions:\n   - Account: Read\n   - Workspace membership: Read\n   - Repositories: Read\n4. Use with your username in `.env`\n\n## Using the Tool\n\n### Basic Commands\n\n```bash\n# Backup to local directory\nrepo-backup local /path/to/backups\n\n# Backup to S3\nrepo-backup s3\n\n# Do both\nrepo-backup both /path/to/local/backup\n```\n\n### Filtering What to Backup\n\n```bash\n# Just GitHub repos\nrepo-backup local /backup/dir --platform github\n\n# Multiple platforms\nrepo-backup local /backup/dir --platform gitlab,bitbucket\n\n# Specific repositories\nrepo-backup local /backup/dir --repos owner/repo1,owner/repo2\n\n# Pattern matching\nrepo-backup local /backup/dir --pattern \"frontend-*\"\nrepo-backup local /backup/dir --pattern-type regex --pattern \".*-service$\"\n\n# Include forks (normally skipped)\nrepo-backup local /backup/dir --include-forks\n```\n\n### Checking Your Backups\n\n```bash\n# See what's backed up locally\nrepo-backup local /backup/dir --list\n\n# Check S3 backups\nrepo-backup s3 --list\n\n# Filter by platform\nrepo-backup local /backup/dir --list --platform github\n```\n\n### Advanced Stuff\n\n```bash\n# Speed things up with more workers\nrepo-backup local /backup/dir --workers 10\n\n# Or slow down for limited bandwidth\nrepo-backup local /backup/dir --workers 2\n\n# Test mode - just backs up smallest repo\nrepo-backup local /backup/dir --test\n\n# Use tar archives instead of git bundles\nrepo-backup local /backup/dir --archive\n\n# Force re-backup everything\nrepo-backup local /backup/dir --force\n\n# See what's happening\nrepo-backup local /backup/dir --verbose\n\n# Check if everything's configured right\nrepo-backup --health-check\nrepo-backup --validate-config\n```\n\n## How It Actually Works\n\n1. **Discovery**: Connects to each platform and finds all your repos\n2. **Filtering**: Skips personal repos, forks, and applies your patterns\n3. **Backup**: For each repository:\n   - Clones with full history using `git clone --mirror`\n   - Creates a git bundle or tar.gz file\n   - Uploads to S3 or saves locally\n   - Cleans up temp files\n4. **Report**: Shows you what worked and what didn't\n\n## Backup File Formats\n\n### Git Bundles (Default)\n\nThese are like portable git repositories:\n- Path: `repos/{platform}/{owner}/{repo_name}_{timestamp}.bundle`\n- Contains complete history and all branches\n- Restore with: `git clone repo.bundle restored-repo`\n\n### Git LFS Support\n\nFor repositories using Git LFS, the tool creates an additional archive:\n\n- Bundle: `repos/{platform}/{owner}/{repo_name}_{timestamp}.bundle` (git history)\n- LFS: `repos/{platform}/{owner}/{repo_name}_{timestamp}_lfs.tar.gz` (large files)\n\nBoth files are needed for a complete restore of LFS repositories.\n\n### Tar Archives\n\nTraditional compressed archives (use `--backup-method archive`):\n- Path: `repos/{platform}/{owner}/{repo_name}_{timestamp}.tar.gz`\n- Contains the bare git repository including LFS objects\n- Restore with: `tar -xzf repo.tar.gz`\n\n## Restoring Your Backups\n\n### From a Git Bundle\n\nThe easy way:\n\n```bash\n# Get the bundle from S3\naws s3 cp s3://your-bucket/repos/github/org/repo.bundle repo.bundle\n\n# Clone it\ngit clone repo.bundle restored-repo\ncd restored-repo\n\n# Point it back to GitHub (optional)\ngit remote set-url origin https://github.com/org/repo.git\n\n# Check everything's there\ngit log --oneline -5\ngit branch -a\ngit tag -l\n```\n\n### From a Git Bundle with LFS\n\nFor repositories that use Git LFS:\n\n```bash\n# Get both files from S3\naws s3 cp s3://your-bucket/repos/github/org/repo.bundle repo.bundle\naws s3 cp s3://your-bucket/repos/github/org/repo_lfs.tar.gz repo_lfs.tar.gz\n\n# Clone the bundle\ngit clone repo.bundle restored-repo\ncd restored-repo\n\n# Restore LFS objects\nmkdir -p .git/lfs\ntar -xzf ../repo_lfs.tar.gz -C .git/lfs/\n\n# Checkout LFS files (no network needed)\ngit lfs checkout\n\n# Verify LFS files are restored\ngit lfs ls-files\n```\n\n### From an Archive\n\n```bash\n# Download and extract\naws s3 cp s3://your-bucket/repos/github/org/repo.tar.gz repo.tar.gz\ntar -xzf repo.tar.gz\n\n# Convert bare repo to normal repo\ngit clone repo.git restored-repo\n```\n\n## Automating Backups\n\n### Linux/Mac (using cron)\n\nAdd to your crontab (runs daily at 2 AM):\n```bash\n0 2 * * * repo-backup s3 \u003e\u003e /var/log/repo-backup.log 2\u003e\u00261\n```\n\n### Windows (using Task Scheduler)\n\nCreate a batch file:\n```batch\nrepo-backup s3\n```\nThen schedule it in Task Scheduler.\n\n### GitHub Actions\n\n```yaml\nname: Backup Repos\non:\n  schedule:\n    - cron: '0 2 * * *'  # Daily at 2 AM UTC\n  workflow_dispatch:     # Manual trigger\n\njobs:\n  backup:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: astral-sh/setup-uv@v4\n      - run: |\n          uv tool install git+https://github.com/hypersec-io/repo-backup\n          repo-backup s3\n        env:\n          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}\n          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n          GITHUB_TOKEN: ${{ secrets.BACKUP_GITHUB_TOKEN }}\n```\n\n## Security Notes\n\n### Keep Your Tokens Safe\n\n- Never commit tokens to git (seriously, don't)\n- Use environment variables for sensitive stuff:\n  ```bash\n  export GITHUB_TOKEN=ghp_...\n  export AWS_ACCESS_KEY_ID=...\n  ```\n- Rotate tokens regularly\n- Give tokens minimal permissions needed\n\n### AWS Security\n\nCheck out [AWS.md](AWS.md) for the full security guide, including IAM roles, encryption, and cost optimization.\n\n## Troubleshooting Common Issues\n\n### Authentication Problems\n- Double-check your tokens have the right permissions\n- Make sure tokens haven't expired\n- Verify you can reach the git platforms from your network\n\n### S3 Upload Issues\n- See [AWS.md - Troubleshooting](AWS.md#troubleshooting) for AWS-specific problems\n- Check AWS credentials are set correctly\n- Verify the bucket exists and you can access it\n\n### Large Repository Problems\n- Try fewer parallel workers\n- Make sure you have enough disk space (2x the largest repo)\n- Consider backing up huge repos separately\n\n### Running Out of Space\n- The tool uses `./.tmp` for temporary files\n- These get cleaned up automatically\n- Make sure you have enough space for the largest repo × 2\n\n## Performance Tips\n\n### Worker Count\n- Default is 5 (good for most cases)\n- Fast connection? Try 10-20\n- Limited bandwidth? Use 2-3\n\n### Smart Filtering\n- Skip test/demo repositories\n- Focus on production code\n- Use patterns to exclude unnecessary repos\n\n### Timing\n- Run during off-peak hours\n- Stagger backups if you have many repos\n\n## Contributing\n\nWe welcome contributions! Check out [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n### Release Process\n\nWe use semantic-release for automated versioning:\n- Commits to `main` trigger automatic releases\n- Version numbers are determined from commit messages\n- CHANGELOG.md is generated automatically\n- Never manually create tags or edit the changelog\n\n## License\n\nApache License 2.0 - See [LICENSE](LICENSE) for details\n\n## Security Reminder\n\n**About Those Tokens:**\n- Never put real tokens in code you commit\n- Use `.env.local` for your actual credentials (it's git-ignored)\n- The `.env` file in the repo has only examples\n- Rotate tokens regularly\n- Use separate, limited tokens for CI/CD\n\n## Need Help?\n\nIf something's not working:\n1. Check the logs - they're pretty detailed\n2. Verify your `.env` configuration\n3. Make sure you have all prerequisites installed\n4. Review the token setup instructions above\n5. Check our [issues](https://github.com/hypersec-io/repo-backup/issues) page\n\n---\n\nBuilt with practicality in mind. We needed reliable repository backups, so we made this. Hope it helps you too!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperi-io%2Frepo-backup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperi-io%2Frepo-backup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperi-io%2Frepo-backup/lists"}