{"id":29671350,"url":"https://github.com/tinybirdco/forward_migration_checker","last_synced_at":"2026-02-07T00:31:13.454Z","repository":{"id":296193195,"uuid":"992568142","full_name":"tinybirdco/forward_migration_checker","owner":"tinybirdco","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-29T15:08:30.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-22T20:13:24.807Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-29T11:13:59.000Z","updated_at":"2025-05-29T15:08:34.000Z","dependencies_parsed_at":"2025-05-29T13:05:57.729Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/forward_migration_checker","commit_stats":null,"previous_names":["tinybirdco/forward_migration_checker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tinybirdco/forward_migration_checker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fforward_migration_checker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fforward_migration_checker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fforward_migration_checker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fforward_migration_checker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/forward_migration_checker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fforward_migration_checker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29181792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T23:15:33.022Z","status":"ssl_error","status_checked_at":"2026-02-06T23:15:09.128Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-22T20:07:38.865Z","updated_at":"2026-02-07T00:31:13.436Z","avatar_url":"https://github.com/tinybirdco.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tinybird Migration Agent\n\nA powerful AI-powered tool to analyze Tinybird Classic projects and assess their compatibility for migration to Tinybird Forward.\n\n## Features\n\n- **Comprehensive Analysis**: Checks version tags, sinks, shared datasources, DynamoDB connections, endpoint types, and include files\n- **AI-Powered Insights**: Uses Google's Vertex AI to generate detailed migration recommendations\n- **🔧 Automatic Fixes**: Can automatically fix compatible issues with user confirmation\n- **Detailed Reporting**: Generates a comprehensive `migration.md` report with step-by-step migration plans\n- **Safe Operations**: Creates backup files before making any changes\n- **Easy to Use**: Simple command-line interface with progress indicators\n\n## Installation\n\n1. **Install Dependencies**:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n2. **Set up Google Cloud Authentication**:\n   \n   You need to authenticate with Google Cloud to use Vertex AI:\n   \n   ```bash\n   # Option 1: Service Account Key\n   export GOOGLE_APPLICATION_CREDENTIALS=\"path/to/your/service-account-key.json\"\n   \n   # Option 2: gcloud CLI\n   gcloud auth application-default login\n   \n   # Set your project ID\n   export GOOGLE_CLOUD_PROJECT=\"your-project-id\"\n   ```\n\n## Usage\n\n### Basic Usage\n\nRun the migration check on a Tinybird project:\n\n```bash\npython main.py\n```\n\nThis will analyze the `./tinybird` directory by default.\n\n### Custom Project Path\n\nSpecify a custom path to your Tinybird project:\n\n```bash\npython main.py /path/to/your/tinybird/project\n```\n\n### What the Agent Checks\n\nThe migration agent performs these checks:\n\n1. **🏷️ Version Tags**: Looks for version management files (*.mdc, *version*)\n2. **🚰 Sinks**: Checks for `TYPE sink` declarations in .pipe files (**Auto-fixable** ✅)\n3. **🤝 Shared Datasources**: Looks for `SHARED_WITH` declarations and vendor/ directories (**Auto-fixable** ✅)\n4. **🗄️ DynamoDB Connections**: Finds `IMPORT_SERVICE \"dynamodb\"` declarations\n5. **🔗 Endpoint Types**: Validates endpoint file structures (**Auto-fixable** ✅)\n6. **📁 Include Files**: Checks for `INCLUDE` statements and .incl files (**Auto-fixable** ✅)\n\n### Auto-Fix Capabilities\n\nThe agent can automatically fix the following issues with your confirmation:\n\n#### ✅ **Auto-Fixable Issues**\n- **Sinks**: Comments out sink declarations and related export configurations\n- **Shared Datasources**: Removes `SHARED_WITH` declarations and offers to remove vendor directories\n- **Endpoint Types**: Adds missing `NODE` declarations to endpoint files\n- **Include Files**: Comments out `INCLUDE` statements and offers to move .incl files to backup\n\n#### ⚠️ **Manual Fix Required**\n- **DynamoDB Connections**: Requires manual migration strategy\n- **Version Tags**: Needs manual setup of version management\n\n### Migration Compatibility Rules\n\n#### ❌ **BLOCKING Issues** (Must Fix Before Migration)\n- **Sinks**: Not supported in Tinybird Forward (**Auto-fixable** ✅)\n- **Shared Datasources**: Not supported in Tinybird Forward (**Auto-fixable** ✅)\n\n#### ⚠️ **WARNING Issues** (Review Required)\n- **DynamoDB Connections**: May have limitations in Forward\n- **Include Files**: Need to verify compatibility (**Auto-fixable** ✅)\n- **Version Tags**: Recommended for migration tracking\n\n#### ℹ️ **Important Notes**\n- **BI Connector**: Not available in Tinybird Forward - migrate to REST/SQL APIs\n\n## Auto-Fix Process\n\nWhen the agent detects fixable issues, it will:\n\n1. **🔍 Identify** the specific problems\n2. **💬 Ask for confirmation** before making any changes\n3. **📄 Create backup files** with `.backup` extension\n4. **🔧 Apply the fixes** safely\n5. **📋 Report** what was changed\n\n### Example Auto-Fix Session\n\n```\n🔧 Found 2 sink-related issues that can be auto-fixed:\n  - Sink found in tinybird/pipes/s3_sink_example.pipe\n  - Export configurations detected\n\nWould you like to automatically comment out sink declarations? (y/n): y\n\n📄 Created backup: tinybird/pipes/s3_sink_example.pipe.backup\n✅ Fixed 1 sink issues\n```\n\n## Output\n\nThe agent generates:\n\n1. **Console Output**: Real-time progress and summary\n2. **migration.md**: Detailed migration report with:\n   - Executive summary\n   - Detailed check results\n   - Specific recommendations\n   - Step-by-step migration plan\n   - BI Connector migration warning\n\n## Example Output\n\n```\n🚀 Starting Tinybird migration check for: ./tinybird\n\n🔍 Starting Tinybird migration compatibility check...\n📁 Found 4 Tinybird files\n\n📋 Running migration checks:\n  1️⃣ Checking version tags...\n     Status: WARNING\n  2️⃣ Checking for sinks...\n     Status: FAIL\n  3️⃣ Checking shared datasources...\n     Status: FAIL\n  4️⃣ Checking DynamoDB connections...\n     Status: WARNING\n  5️⃣ Checking endpoint types...\n     Status: PASS\n  6️⃣ Checking include files...\n     Status: PASS\n\n🤖 Generating migration analysis...\n\n✅ Migration check complete!\n📄 Report saved to: migration.md\n```\n\n## Sample Project Structure\n\nThe included sample project demonstrates common migration issues and auto-fixes:\n\n```\ntinybird/\n├── datasources/\n│   ├── sample_data.datasource      # DynamoDB connection (WARNING)\n│   └── shared_data.datasource      # Shared datasource (AUTO-FIXABLE)\n├── pipes/\n│   ├── s3_sink_example.pipe        # Sink usage (AUTO-FIXABLE)\n│   └── example_with_include.pipe   # Include statements (AUTO-FIXABLE)\n├── endpoints/\n│   └── sample_endpoint.endpoint    # Regular endpoint (PASS)\n└── includes/\n    └── common_filters.incl         # Include file (AUTO-FIXABLE)\n```\n\n## Backup and Recovery\n\n### Backup Files\nThe agent automatically creates backup files before making changes:\n- Original: `file.datasource`\n- Backup: `file.datasource.backup`\n\n### Restoring from Backup\nTo restore original files:\n\n```bash\n# Restore a specific file\ncp tinybird/datasources/shared_data.datasource.backup tinybird/datasources/shared_data.datasource\n\n# Restore all files from backups (be careful!)\nfind tinybird/ -name \"*.backup\" | while read backup; do\n    original=\"${backup%.backup}\"\n    cp \"$backup\" \"$original\"\ndone\n```\n\n### Include Files Backup\nInclude files are moved to `tinybird/includes_backup/` directory for safe keeping.\n\n## Troubleshooting\n\n### Authentication Issues\n- Ensure Google Cloud credentials are properly set\n- Verify your project has Vertex AI API enabled\n- Check that your service account has the necessary permissions\n\n### File Permission Issues\n- Ensure the agent has read access to your Tinybird project directory\n- Check that it can write to the current directory for the migration.md report\n\n### No Tinybird Files Found\n- Verify the path to your Tinybird project is correct\n- Ensure your project has .datasource, .pipe, or .endpoint files\n\n## Contributing\n\nFeel free to contribute improvements or report issues. The agent is designed to be extensible for additional migration checks.\n\n## License\n\nMIT License - see LICENSE file for details. ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fforward_migration_checker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Fforward_migration_checker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fforward_migration_checker/lists"}