{"id":31033035,"url":"https://github.com/open-metadata/openmetadata-dbt-action","last_synced_at":"2026-02-21T23:02:02.078Z","repository":{"id":310479307,"uuid":"1039993241","full_name":"open-metadata/openmetadata-dbt-action","owner":"open-metadata","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-18T11:16:42.000Z","size":82,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-20T07:53:41.674Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/open-metadata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-18T09:47:39.000Z","updated_at":"2025-08-18T11:16:46.000Z","dependencies_parsed_at":"2025-08-18T12:23:42.567Z","dependency_job_id":"bfa2c335-c710-4a55-bba8-070e732a51b9","html_url":"https://github.com/open-metadata/openmetadata-dbt-action","commit_stats":null,"previous_names":["open-metadata/openmetadata-dbt-action"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/open-metadata/openmetadata-dbt-action","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-metadata%2Fopenmetadata-dbt-action","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-metadata%2Fopenmetadata-dbt-action/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-metadata%2Fopenmetadata-dbt-action/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-metadata%2Fopenmetadata-dbt-action/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/open-metadata","download_url":"https://codeload.github.com/open-metadata/openmetadata-dbt-action/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/open-metadata%2Fopenmetadata-dbt-action/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29696785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T18:18:25.093Z","status":"ssl_error","status_checked_at":"2026-02-21T18:18:22.435Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T01:07:45.018Z","updated_at":"2026-02-21T23:02:02.042Z","avatar_url":"https://github.com/open-metadata.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenMetadata DBT Ingestion with GitHub Actions\n\nThis repository demonstrates how to automate DBT metadata ingestion into OpenMetadata using GitHub Actions. It provides a complete example of running DBT workflows externally and ingesting the generated artifacts into OpenMetadata.\n\n## Overview\n\nThis solution enables you to:\n- Automatically ingest DBT metadata into OpenMetadata on every push\n- Run ingestion workflows without direct access to the OpenMetadata server\n- Store DBT artifacts locally and process them through GitHub Actions\n- Maintain version control of your DBT metadata configuration\n\n## Architecture\n\n```\nDBT Project → Generate Artifacts → GitHub Actions → OpenMetadata Ingestion → OpenMetadata Server\n```\n\n## Prerequisites\n\n- An OpenMetadata instance (version 1.0+)\n- DBT artifacts (manifest.json, catalog.json, run_results.json)\n- GitHub repository with Actions enabled\n- OpenMetadata JWT token for authentication\n\n\u003e **⚠️ Version Compatibility:** The `openmetadata-ingestion` package version MUST match your OpenMetadata server version. For example, if your OpenMetadata server is version 1.5.11, you must install `openmetadata-ingestion[dbt]==1.5.11`\n\n## Repository Structure\n\n```\n.\n├── .github/\n│   └── workflows/\n│       └── openmetadata-ingestion.yml    # GitHub Action workflow\n├── artifacts/                            # Example location (can be anywhere in repo)\n│   ├── catalog.json                      # DBT catalog artifact\n│   ├── manifest.json                     # DBT manifest artifact\n│   └── run_results.json                  # DBT run results artifact\n└── openmetadata_dbt_config.yaml          # OpenMetadata ingestion configuration\n```\n\n\u003e **Note:** DBT artifacts can be stored anywhere within your repository. The `artifacts/` folder is just an example. You can organize them in any directory structure that suits your project (e.g., `dbt/target/`, `build/dbt/`, etc.)\n\n## Setup Instructions\n\n### 1. Generate DBT Artifacts\n\nFirst, generate your DBT artifacts by running these commands in your DBT project:\n\n```bash\n# Generate manifest.json\ndbt compile\n\n# Generate catalog.json\ndbt docs generate\n\n# Generate run_results.json\ndbt run\n```\n\nPlace these files in your repository. They can be stored in any directory structure you prefer (e.g., `artifacts/`, `dbt/target/`, `build/`, etc.).\n\n### 2. Configure GitHub Secrets\n\nYou have two options for setting up secrets:\n\n#### Option A: Environment Secrets (Recommended)\n1. Go to Settings → Environments → New environment\n2. Create an environment named `test` (or modify the workflow to use your environment name)\n3. Add these secrets to the environment:\n\n| Secret Name | Description | Example | Required |\n|------------|-------------|---------|----------|\n| `OPENMETADATA_HOST` | Your OpenMetadata API endpoint | `http://your-server:8585/api` | ✅ |\n| `OPENMETADATA_JWT_TOKEN` | JWT token for authentication | `eyJhbGciOiJIUzI1NiIsInR5cCI6...` | ✅ |\n| `OPENMETADATA_VERSION` | OpenMetadata server version | `1.9.1.0` | ✅ |\n\n#### Option B: Repository Secrets\nAdd the secrets directly to Settings → Secrets and variables → Actions → Repository secrets (then remove the `environment: test` line from the workflow)\n\n\u003e **Important:** \n\u003e - The `OPENMETADATA_VERSION` must exactly match your OpenMetadata server version\n\u003e - Check your server version in OpenMetadata UI → Settings → About\n\u003e - Common versions: `1.5.11`, `1.6.2`, `1.7.3`, `1.8.0`, `1.9.1.0`\n\n### 3. Configuration File\n\nThe `openmetadata_dbt_config.yaml` file contains the ingestion configuration. **You need to update the following fields:**\n\n#### Required Updates:\n\n1. **`serviceName`**: Must match an existing database service name in your OpenMetadata instance\n2. **File paths**: Update to match your actual artifact locations in the repository\n\n```yaml\nsource:\n  type: dbt\n  serviceName: dbt_git_test  # ⚠️ UPDATE: Must match existing service in OpenMetadata\n  sourceConfig:\n    config:\n      type: DBT\n  dbtConfigSource:\n    dbtConfigType: local\n    # ⚠️ UPDATE: Change these paths to match your artifact locations\n    dbtCatalogFilePath: /home/runner/work/openmetadata-dbt-action/openmetadata-dbt-action/artifacts/catalog.json\n    dbtManifestFilePath: /home/runner/work/openmetadata-dbt-action/openmetadata-dbt-action/artifacts/manifest.json\n    dbtRunResultsFilePath: /home/runner/work/openmetadata-dbt-action/openmetadata-dbt-action/artifacts/run_results.json\n\nsink:\n  type: metadata-rest\n  config: {}\n\nworkflowConfig:\n  loggerLevel: INFO\n  openMetadataServerConfig:\n    hostPort: $OPENMETADATA_HOST\n    authProvider: openmetadata\n    securityConfig:\n      jwtToken: $OPENMETADATA_JWT_TOKEN\n```\n\n\u003e **Important:** \n\u003e - The `serviceName` must exactly match a DBT service that already exists in your OpenMetadata instance\n\u003e - If the service doesn't exist, you'll need to create it first in OpenMetadata\n\u003e - File paths are relative to the repository root\n\n### 4. GitHub Action Workflow\n\nThe workflow (`.github/workflows/openmetadata-ingestion.yml`) automatically:\n- Installs the OpenMetadata ingestion package\n- Validates DBT artifacts exist\n- Runs the metadata ingestion\n- Uploads logs as artifacts for debugging\n\n## Usage\n\n### Automatic Ingestion\n\nThe ingestion runs automatically when:\n- You push changes to `main` or `master` branches\n- DBT artifacts are updated\n- The configuration file is modified\n- A pull request is opened\n\n### Manual Ingestion\n\nYou can manually trigger the workflow:\n1. Go to Actions tab in your GitHub repository\n2. Select \"OpenMetadata DBT Ingestion\"\n3. Click \"Run workflow\"\n\n### Local Testing\n\nTo test the ingestion locally:\n\n```bash\n# Install the package (version must match your OpenMetadata server)\n# Replace X.X.X with your OpenMetadata version (e.g., 1.5.11)\npip install \"openmetadata-ingestion[dbt]==X.X.X\"\n\n# Set environment variables\nexport OPENMETADATA_HOST=\"http://localhost:8585/api\"\nexport OPENMETADATA_JWT_TOKEN=\"your-jwt-token\"\n\n# Run ingestion\nmetadata ingest -c openmetadata_dbt_config.yaml\n```\n\n\u003e **Note:** Always ensure the `openmetadata-ingestion` package version matches your OpenMetadata server version to avoid compatibility issues.\n\n## Monitoring\n\n- Check the Actions tab in GitHub for workflow runs\n- Download ingestion logs from workflow artifacts\n- View imported metadata in your OpenMetadata instance\n\n## Troubleshooting\n\n### Common Issues\n\n1. **Version Mismatch Error**\n   - Ensure `openmetadata-ingestion` package version matches your OpenMetadata server version exactly\n   - Check server version: OpenMetadata UI → Settings → About\n   - Update `OPENMETADATA_VERSION` secret in GitHub\n\n2. **Authentication Failed**\n   - Verify JWT token is valid and not expired\n   - Check token has necessary permissions\n\n3. **Service Not Found**\n   - Ensure the `serviceName` in your YAML matches an existing DBT service in OpenMetadata\n   - Create the service in OpenMetadata first if it doesn't exist\n\n4. **Artifacts Not Found**\n   - Verify DBT artifacts exist in your repository\n   - Ensure file paths in configuration match actual locations\n   - Remember: paths are relative to repository root\n\n5. **Connection Timeout**\n   - Verify OpenMetadata server is accessible\n   - Check network/firewall settings\n\n### Debug Mode\n\nEnable debug logging by modifying the configuration:\n\n```yaml\nworkflowConfig:\n  loggerLevel: DEBUG\n```\n\n## Advanced Configuration\n\n### Service Name Configuration\n\nThe `serviceName` must match an existing DBT service in OpenMetadata:\n\n```yaml\nsource:\n  serviceName: your_existing_dbt_service  # Must exist in OpenMetadata\n```\n\nTo create a new DBT service in OpenMetadata:\n1. Navigate to Services → Add Service → DBT\n2. Configure the service with your DBT project details\n3. Use the exact service name in your YAML configuration\n\n### Additional Metadata\n\nYou can enhance the ingestion with additional metadata:\n\n```yaml\nsource:\n  sourceConfig:\n    config:\n      type: DBT\n      dbtUpdateDescriptions: true  # Update descriptions from DBT\n      includeTags: true            # Include DBT tags\n      dbtClassificationName: PII   # Add classification tags\n```\n\n## Security Considerations\n\n- Never commit JWT tokens or sensitive credentials\n- Use GitHub Secrets for all sensitive information\n- Regularly rotate JWT tokens\n- Review GitHub Actions permissions\n\n## Contributing\n\nTo contribute to this demonstration:\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Submit a pull request\n\n## Resources\n\n- [OpenMetadata Documentation](https://docs.open-metadata.org/)\n- [DBT Connector Documentation](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt)\n- [Running DBT Workflow Externally](https://docs.open-metadata.org/latest/connectors/ingestion/workflows/dbt/run-dbt-workflow-externally)\n- [GitHub Actions Documentation](https://docs.github.com/en/actions)\n\n## License\n\nThis demonstration repository is provided as-is for educational purposes.\n\n## Support\n\nFor issues related to:\n- This demo: Open an issue in this repository\n- OpenMetadata: Visit [OpenMetadata Slack](https://slack.open-metadata.org/)\n- DBT: Visit [DBT Community](https://community.getdbt.com/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-metadata%2Fopenmetadata-dbt-action","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-metadata%2Fopenmetadata-dbt-action","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-metadata%2Fopenmetadata-dbt-action/lists"}