{"id":46305222,"url":"https://github.com/toolsascode/bfm","last_synced_at":"2026-03-04T12:01:21.345Z","repository":{"id":325534041,"uuid":"1100156511","full_name":"toolsascode/bfm","owner":"toolsascode","description":"BfM is a comprehensive database migration system that supports multiple backends (PostgreSQL, GreptimeDB, Etcd) with HTTP and Protobuf APIs.","archived":false,"fork":false,"pushed_at":"2025-12-24T12:16:23.000Z","size":2393,"stargazers_count":0,"open_issues_count":13,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-25T07:14:08.907Z","etag":null,"topics":["automation","devops","devops-tools","migrations","migrations-tool"],"latest_commit_sha":null,"homepage":"https://opsview.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/toolsascode.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-19T23:04:42.000Z","updated_at":"2025-12-13T00:43:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/toolsascode/bfm","commit_stats":null,"previous_names":["toolsascode/bfm"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/toolsascode/bfm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toolsascode%2Fbfm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toolsascode%2Fbfm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toolsascode%2Fbfm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toolsascode%2Fbfm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/toolsascode","download_url":"https://codeload.github.com/toolsascode/bfm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/toolsascode%2Fbfm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30079565,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T11:57:42.557Z","status":"ssl_error","status_checked_at":"2026-03-04T11:56:10.793Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","devops","devops-tools","migrations","migrations-tool"],"created_at":"2026-03-04T12:00:54.514Z","updated_at":"2026-03-04T12:01:21.315Z","avatar_url":"https://github.com/toolsascode.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg src=\"./assets/BfM.svg\" alt=\"drawing\" width=\"150\" /\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003ch1 style=\"font-size: 25px; font-weight: bold; text-align: center; font-family: 'Montserrat', system-ui, sans-serif;\" \u003e\nBackend For Migrations (BfM)\n\u003c/h1\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/toolsascode/bfm/api/migrations.svg)](https://pkg.go.dev/github.com/toolsascode/bfm/api/migrations) [![Release CLI](https://github.com/toolsascode/bfm/actions/workflows/release-cli.yml/badge.svg)](https://github.com/toolsascode/bfm/actions/workflows/release-cli.yml) [![Release Docker](https://github.com/toolsascode/bfm/actions/workflows/release-docker.yml/badge.svg)](https://github.com/toolsascode/bfm/actions/workflows/release-docker.yml) [![codecov](https://codecov.io/github/toolsascode/bfm/graph/badge.svg?token=NTDG468UNG)](https://codecov.io/github/toolsascode/bfm)\n\n\u003c/p\u003e\n\nBfM is a comprehensive database migration system that supports multiple backends (PostgreSQL, GreptimeDB, Etcd) with HTTP and Protobuf APIs.\n\n## Features\n\n- Multi-backend support: PostgreSQL, GreptimeDB, Etcd\n- HTTP REST API with authentication\n- Protobuf/gRPC API (requires code generation)\n- Migration state tracking in PostgreSQL/MySQL\n- Support for fixed and dynamic schemas\n- Embedded SQL scripts in Go files\n- Dry-run mode for testing\n- Idempotent migrations\n\n## Screenshots\n\n| | |\n|---|---|\n| \u003cimg src=\"./assets/screenshots/login.jpeg\" alt=\"Login\" width=\"400\" /\u003e | \u003cimg src=\"./assets/screenshots/dashboard.jpeg\" alt=\"Dashboard\" width=\"400\" /\u003e |\n| \u003cimg src=\"./assets/screenshots/MigrationList.jpeg\" alt=\"Migration List\" width=\"400\" /\u003e | \u003cimg src=\"./assets/screenshots/MigrationDetails.jpeg\" alt=\"Migration Details\" width=\"400\" /\u003e |\n\n## Configuration\n\n### Environment Variables\n\n#### Server Configuration\n\n- `BFM_HTTP_PORT` - HTTP server port (default: 7070)\n- `BFM_GRPC_PORT` - gRPC server port (default: 9090)\n- `BFM_API_TOKEN` - API token for authentication (required)\n\n#### State Database Configuration\n\n- `BFM_STATE_BACKEND` - State database type: \"postgresql\" or \"mysql\" (default: \"postgresql\")\n- `BFM_STATE_DB_HOST` - State database host (default: \"localhost\")\n- `BFM_STATE_DB_PORT` - State database port (default: \"5432\")\n- `BFM_STATE_DB_USERNAME` - State database username (default: \"postgres\")\n- `BFM_STATE_DB_PASSWORD` - State database password (required)\n- `BFM_STATE_DB_NAME` - State database name (default: \"migration_state\")\n- `BFM_STATE_SCHEMA` - State database schema (default: \"public\")\n\n#### Connection Configuration\n\nFor each connection (e.g., \"core\", \"guard\", \"logs\"), set:\n\n- `{CONNECTION}_BACKEND` - Backend type: \"postgresql\", \"greptimedb\", or \"etcd\"\n- `{CONNECTION}_DB_HOST` - Database host\n- `{CONNECTION}_DB_PORT` - Database port\n- `{CONNECTION}_DB_USERNAME` - Database username\n- `{CONNECTION}_DB_PASSWORD` - Database password\n- `{CONNECTION}_DB_NAME` - Database name\n- `{CONNECTION}_SCHEMA` - Schema name (optional, for fixed schemas)\n\nExample:\n\n```bash\nCORE_BACKEND=postgresql\nCORE_DB_HOST=localhost\nCORE_DB_PORT=5432\nCORE_DB_USERNAME=dashcloud\nCORE_DB_PASSWORD=password\nCORE_DB_NAME=dashcloud\nCORE_SCHEMA=core\n```\n\n## Development\n\nFor development environment setup, local development, and hot-reload configuration, see [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md).\n\n## Production Deployment\n\n### Using Docker Image\n\nBfM provides a production-ready Docker image that includes:\n\n- BfM API Server (HTTP on port 7070, gRPC on port 9090)\n- BfM Worker (optional, enabled via `BFM_QUEUE_ENABLED=true`)\n- BfM CLI tool (available inside container)\n- FfM Frontend (served via the API server)\n\n#### Registry\n\nInstall from the command line\n\n``` bash\ndocker pull ghcr.io/toolsascode/bfm:latest\n```\n\n#### Building the Production Image\n\nBuild the standalone production Docker image:\n\n```bash\n# Using Makefile\nmake prod-build\n\n# Or manually\ndocker build -t bfm-production:latest -f docker/Dockerfile .\n```\n\n#### Running with Docker Compose (Recommended)\n\nThe easiest way to run BfM in production is using the standalone Docker Compose configuration:\n\n1. **Create environment file:**\n\n```bash\n# Copy and edit environment variables\ncp .env.example .env\n```\n\n2. **Configure environment variables:**\n\nSet the following required variables in your `.env` file:\n\n```bash\n# API Authentication\nBFM_API_TOKEN=your-secure-random-token-here\n\n# State Database\nBFM_STATE_DB_PASSWORD=your-secure-password\nBFM_STATE_DB_USERNAME=postgres\n\n# Backend Connections (configure as needed)\nCORE_DB_HOST=your-postgres-host\nCORE_DB_PASSWORD=your-password\nCORE_DB_NAME=your-database\n\n# Optional: Queue Configuration\nBFM_QUEUE_ENABLED=false  # Set to true to enable worker\n```\n\n3. **Start the service:**\n\n```bash\n# Using Makefile\nmake standalone-up\n\n# Or manually\ndocker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml up -d --build\n```\n\n4. **Verify the service:**\n\n```bash\n# Check health\ncurl http://localhost:7070/health\n\n# Check service status\nmake standalone-ps\n# or\ndocker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml ps\n```\n\n5. **View logs:**\n\n```bash\n# Using Makefile\nmake standalone-logs\n\n# Or manually\ndocker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml logs -f\n```\n\n6. **Stop the service:**\n\n```bash\n# Using Makefile\nmake standalone-down\n\n# Or manually\ndocker compose -p bfm-standalone -f deploy/docker-compose.standalone.yml down\n```\n\n#### Running with Docker Run\n\nFor more control, you can run the container directly:\n\n```bash\ndocker run -d \\\n  --name bfm-production \\\n  -p 7070:7070 \\\n  -p 9090:9090 \\\n  -e BFM_API_TOKEN=your-secure-token \\\n  -e BFM_STATE_DB_HOST=postgres \\\n  -e BFM_STATE_DB_PASSWORD=your-password \\\n  -e CORE_DB_HOST=your-postgres-host \\\n  -e CORE_DB_PASSWORD=your-password \\\n  -v /path/to/your/sfm:/app/sfm:ro \\\n  bfm-production:latest\n```\n\n#### Accessing the Services\n\nOnce running, you can access:\n\n- **Frontend UI**: \u003chttp://localhost:7070\u003e\n- **HTTP API**: \u003chttp://localhost:7070/api/v1\u003e\n- **OpenAPI Spec (YAML)**: \u003chttp://localhost:7070/api/v1/openapi.yaml\u003e\n- **OpenAPI Spec (JSON)**: \u003chttp://localhost:7070/api/v1/openapi.json\u003e\n- **gRPC API**: localhost:9090\n- **Health Check**: \u003chttp://localhost:7070/health\u003e\n\n#### Using the CLI Inside Container\n\nThe BfM CLI is available inside the production container:\n\n```bash\n# Execute CLI commands\ndocker exec bfm-standalone /app/bin/bfm-cli --help\ndocker exec bfm-standalone /app/bin/bfm-cli version\n\n# Build migration files from SFM directory\ndocker exec bfm-standalone /app/bin/bfm-cli build /app/sfm --verbose\n```\n\n### BfM CLI Tool\n\nThe BfM CLI is a command-line tool for generating migration `.go` files from SQL/JSON scripts.\n\n## How to install?\n\n### Go Install\n\n```shell\ngo install github.com/toolsascode/bfm@latest\n```\n\n### Via Github\n\n- [Latest version](https://github.com/toolsascode/bfm/releases/latest)\n\n```shell\ncurl -fLSs https://raw.githubusercontent.com/toolsascode/bfm/main/scripts/install.sh | bash\n```\n\nOr\n\n```shell\ncurl -fLSs https://raw.githubusercontent.com/toolsascode/bfm/main/scripts/install.sh | sudo bash\n```\n\n### Homebrew\n\n```shell\nbrew install toolsascode/tap/bfm\n```\n\n### Scoop\n\n1. Run **PowerShell as an Administrator** and:\n2. To add this bucket, run `scoop bucket add bfm-scoop https://github.com/toolsascode/scoop-bucket`.\n3. To install, do `scoop install bfm`.\n\n#### Building the CLI\n\n```bash\n# Using Makefile\nmake build-cli\n\n# Or manually\ncd api \u0026\u0026 go build -o ../bfm-cli ./cmd/cli\n```\n\n#### CLI Commands\n\n**Version:**\n\n```bash\n./bfm-cli version\n```\n\n**Build Migration Files:**\n\nGenerate `.go` files from migration scripts in the SFM directory:\n\n```bash\n# Basic usage\n./bfm-cli build examples/sfm\n\n# With verbose output\n./bfm-cli build examples/sfm --verbose\n\n# Dry run (show what would be generated)\n./bfm-cli build examples/sfm --dry-run\n\n# Custom output directory\n./bfm-cli build examples/sfm --output /path/to/output\n\n# Custom path\n./bfm-cli build /path/to/sfm --verbose\n```\n\n#### SFM Directory Structure\n\nThe CLI expects migration scripts in the following structure:\n\n```\n{sfm_path}/\n  {backend}/                    # The first folder after SfM is considered the backend.\n    {connection}/               # The last folder will be considered a connection.\n      {version}_{name}.up.sql\n      {version}_{name}.down.sql\n      # OR for etcd\n      {version}_{name}.up.json\n      {version}_{name}.down.json\n```\n\nExample:\n\n```\nexamples/sfm/\n  postgresql/\n    core/\n      20250101120000_create_users.up.sql\n      20250101120000_create_users.down.sql\n  greptimedb/\n    logs/\n      20250101120000_create_metrics.up.sql\n      20250101120000_create_metrics.down.sql\n  etcd/\n    metadata/\n      20250101120000_init_config.up.json\n      20250101120000_init_config.down.json\n```\n\nThe CLI will generate corresponding `.go` files that embed the SQL/JSON and register migrations in the global registry.\n\n#### Using CLI in Production\n\nIn production environments, you can:\n\n1. **Build migrations as part of CI/CD:**\n\n```bash\n# In your build pipeline\n./bfm-cli build /path/to/migrations --output /path/to/generated\ngo build -o bfm-server ./cmd/server\n```\n\n2. **Use in Docker builds:**\n\n```dockerfile\n# Copy migration scripts\nCOPY migrations/sfm /app/sfm\n\n# Generate .go files\nRUN /app/bin/bfm-cli build /app/sfm\n\n# Build server with generated migrations\nRUN go build -o bfm-server ./cmd/server\n```\n\n3. **Validate migrations before deployment:**\n\n```bash\n# Dry run to check for errors\n./bfm-cli build /path/to/migrations --dry-run --verbose\n```\n\n### Production Best Practices\n\n1. **Security:**\n   - Use strong, randomly generated API tokens\n   - Store credentials in secret management systems (e.g., HashiCorp Vault, AWS Secrets Manager)\n   - Enable TLS/HTTPS via reverse proxy (nginx, Traefik)\n   - Restrict network access to BfM API\n\n2. **High Availability:**\n   - Run multiple BfM instances behind a load balancer\n   - Use PostgreSQL replication for state database\n   - Implement distributed locking to prevent concurrent migrations\n   - Monitor health endpoints\n\n3. **Monitoring:**\n   - Set up health check monitoring (`GET /health`)\n   - Collect logs via centralized logging (ELK, Loki, etc.)\n   - Track migration execution metrics\n   - Alert on migration failures\n\n4. **Backup:**\n   - Regularly backup the state database\n   - Version control all migration scripts\n   - Test restore procedures\n\n5. **Migration Management:**\n   - Always test migrations in staging first\n   - Use dry-run mode before applying migrations\n   - Keep migration scripts idempotent\n   - Document complex migrations\n\nSee `docs/DEPLOYMENT.md` for more detailed deployment instructions.\n\n## Usage\n\n### OpenAPI Specification\n\nBfM provides a complete OpenAPI v3.2.0 specification for the HTTP API. The specification is available at:\n\n- **YAML format**: `http://localhost:7070/api/v1/openapi.yaml`\n- **JSON format**: `http://localhost:7070/api/v1/openapi.json`\n\nThe OpenAPI specification includes:\n- Complete API endpoint documentation\n- Request/response schemas\n- Authentication requirements\n- Example requests and responses\n- Error response formats\n\nYou can use the OpenAPI spec with tools like:\n\n- [Swagger UI](https://swagger.io/tools/swagger-ui/) - Interactive API documentation\n- [Postman](https://www.postman.com/) - API testing and development\n- [OpenAPI Generator](https://openapi-generator.tech/) - Generate client SDKs\n- [Redoc](https://github.com/Redocly/redoc) - Beautiful API documentation\n\n**Example: View OpenAPI spec in Swagger UI**\n\n```bash\n# Using Docker\ndocker run -p 8080:8080 -e SWAGGER_JSON=/openapi.yaml -v $(pwd)/api/internal/api/http/openapi.yaml:/openapi.yaml swaggerapi/swagger-ui\n\n# Or use online Swagger Editor\n# Paste the content from http://localhost:7070/api/v1/openapi.yaml\n```\n\n### HTTP API\n\n#### Migrate Endpoint\n\n```bash\nPOST /api/v1/migrate\nAuthorization: Bearer {BFM_API_TOKEN}\nContent-Type: application/json\n\n{\n  \"target\": {\n    \"backend\": \"postgresql\",\n    \"schema\": \"core\",\n    \"tables\": [],\n    \"version\": \"\",\n    \"connection\": \"core\"\n  },\n  \"connection\": \"core\",\n  \"schema\": \"core\",\n  \"environment\": \"\",\n  \"dry_run\": false\n}\n```\n\nResponse:\n\n```json\n{\n  \"success\": true,\n  \"applied\": [\"core_users_20250101120000_create_users\"],\n  \"skipped\": [],\n  \"errors\": []\n}\n```\n\n**Note**: For complete API documentation, see the [OpenAPI Specification](#openapi-specification) section above.\n\n### Health Check\n\n```bash\nGET /health\n```\n\n## Migration Scripts\n\nMigration scripts are located in `sfm/{backend}/{connection}/` and follow the naming convention:\n`{version}_{name}.up.sql` and `{version}_{name}.down.sql`\n\nThe BfM CLI generates corresponding `.go` files with the format `{version}_{name}.go` that:\n\n1. Embed SQL/JSON files using `//go:embed`\n2. Register themselves in the global registry via `init()`\n3. Include both up and down migrations\n\nExample structure:\n\n```\nsfm/\n  postgresql/\n    core/\n      20250101120000_create_users.up.sql\n      20250101120000_create_users.down.sql\n      20250101120000_create_users.go  # Generated by CLI\n  greptimedb/\n    logs/\n      20250101120000_create_metrics.up.sql\n      20250101120000_create_metrics.down.sql\n      20250101120000_create_metrics.go  # Generated by CLI\n  etcd/\n    metadata/\n      20250101120000_init_config.up.json\n      20250101120000_init_config.down.json\n      20250101120000_init_config.go  # Generated by CLI\n```\n\n## Migration from Existing System\n\nTo migrate from the existing GORM AutoMigrate system:\n\n1. Extract table definitions from GORM models\n2. Create SQL migration scripts following the naming convention\n3. Place scripts in appropriate `sfm/{backend}/{connection}/` directory\n4. Register migrations via `init()` functions\n5. Run migrations via HTTP API or Protobuf API\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoolsascode%2Fbfm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoolsascode%2Fbfm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoolsascode%2Fbfm/lists"}