{"id":36989778,"url":"https://github.com/anilkulkarni87/digdag-viz","last_synced_at":"2026-01-13T23:31:22.900Z","repository":{"id":327656021,"uuid":"1110242152","full_name":"anilkulkarni87/digdag-viz","owner":"anilkulkarni87","description":null,"archived":false,"fork":false,"pushed_at":"2025-12-05T22:16:17.000Z","size":96,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-08T07:16:10.512Z","etag":null,"topics":["data-engineering","digdag","github-actions","treasure-data","workflow-visualization"],"latest_commit_sha":null,"homepage":"https://anilkulkarni87.github.io/digdag-viz/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anilkulkarni87.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-04T23:03:29.000Z","updated_at":"2025-12-05T22:16:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/anilkulkarni87/digdag-viz","commit_stats":null,"previous_names":["anilkulkarni87/digdag-viz"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/anilkulkarni87/digdag-viz","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anilkulkarni87%2Fdigdag-viz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anilkulkarni87%2Fdigdag-viz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anilkulkarni87%2Fdigdag-viz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anilkulkarni87%2Fdigdag-viz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anilkulkarni87","download_url":"https://codeload.github.com/anilkulkarni87/digdag-viz/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anilkulkarni87%2Fdigdag-viz/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28399508,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-13T14:36:09.778Z","status":"ssl_error","status_checked_at":"2026-01-13T14:35:19.697Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","digdag","github-actions","treasure-data","workflow-visualization"],"created_at":"2026-01-13T23:31:22.033Z","updated_at":"2026-01-13T23:31:22.894Z","avatar_url":"https://github.com/anilkulkarni87.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Digdag Viz\n\n**Turn your Treasure Data workflows into beautiful, interactive visualizations in seconds.**\n\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n\n![Workflow Visualization Example](docs/example-graph.png)\n\n## Why Use This Tool?\n\n- 📊 **Understand Complex Workflows**: See the entire workflow structure at a glance\n- 🔍 **Debug Faster**: Identify task dependencies and execution flow visually\n- 📝 **Auto-Documentation**: Generate always-up-to-date workflow documentation\n- 🎨 **Interactive UI**: Click tasks to view SQL queries, parameters, and schedules\n- 🚀 **Zero Configuration**: Works out of the box with your existing `.dig` files\n\n## Quick Start (30 Seconds)\n\n```bash\n# 1. Install\ngit clone https://github.com/yourname/digdag-viz.git\ncd digdag-viz\npip install -e .\n\n# 2. Visualize your workflows\ndigdag-viz /path/to/your/workflows --outdir output\n\n# 3. Open the result\nopen output/index.html\n```\n\nThat's it! You'll see an interactive dashboard with all your workflows.\n\n## What You Get\n\n### 📊 Interactive Workflow Graphs\n- **Interactive Visualization**: Pan, zoom, and explore complex workflow graphs.\n- **Data Lineage (Experimental)**: Trace table dependencies across your entire data pipeline. [Learn more](wiki/Data-Lineage.md).\n- **Search \u0026 Filter**: Instantly find tasks and filter by status or name.\n- Color-coded tasks by operator type (`td\u003e`, `sh\u003e`, `py\u003e`, etc.)\n- Click any task to see details: SQL queries, parameters, schedules\n- Hover to highlight dependencies (upstream/downstream)\n- Search and filter tasks\n\n### 📅 Schedule Overview\n- All scheduled workflows in one table\n- Filter by project, search by name\n- See cron schedules and timezones at a glance\n\n### 🔗 SQL Query Viewer\n- Syntax-highlighted SQL with proper formatting\n- Direct links from workflow tasks to query files\n- \"Back to workflow\" navigation\n\n### 📁 Project Organization\n- Supports single projects or workspace with multiple projects\n- Automatic project detection and grouping\n- Separate pages for scheduled vs. unscheduled workflows\n\n## Installation\n\n### Prerequisites\n```bash\n# Install Graphviz (required for graph rendering)\n# macOS:\nbrew install graphviz\n\n# Ubuntu/Debian:\nsudo apt-get install graphviz\n\n# Windows:\n# Download from https://graphviz.org/download/\n```\n\n### Option 1: Install from Source (Recommended)\n```bash\ngit clone https://github.com/yourname/digdag-viz.git\ncd digdag-viz\npip install -e .\n```\n\n### Option 2: Install Dependencies Only\n```bash\ngit clone https://github.com/yourname/digdag-viz.git\ncd digdag-viz\npip install -r requirements.txt\n```\n\n### Verify Installation\n```bash\n# Check version\ndigdag-viz --version\n\n# Run example\ndigdag-viz example --outdir output\nopen output/index.html\n```\n\n## Usage Examples\n\n### Visualize a Single Project\n```bash\npython digdag-viz ./my-project --outdir graphs\n```\n\n### Visualize Multiple Projects (Workspace)\n```bash\npython digdag-viz ./projects --outdir graphs\n# Automatically detects all projects in subdirectories\n```\n\n### Custom Output Format\n```bash\n# Generate PNG images instead of SVG\npython digdag-viz ./workflows --outdir graphs --format png\n```\n\n### Exclude Test Workflows\n```bash\npython digdag-viz ./workflows --outdir graphs \\\n  --exclude \"**/test_*.dig\" \\\n  --exclude \"**/.archive/**\"\n```\n\n### Verbose Output (for debugging)\n```bash\npython digdag-viz ./workflows --outdir graphs --verbose\n```\n\n## Command-Line Options\n\n```\nusage: digdag-viz [-h] [--outdir OUTDIR] [--format {svg,png,pdf}]\n                    [--config CONFIG] [--no-schedule]\n                    [--exclude EXCLUDE_PATTERNS] [--include-only INCLUDE_PATTERNS]\n                    [--direction {LR,TB,RL,BT}] [--max-depth MAX_DEPTH]\n                    [--template-dir TEMPLATE_DIR] [--verbose] [--quiet]\n                    path\n\npositional arguments:\n  path                  Path to .dig file or directory containing workflows\n\noptional arguments:\n  --outdir OUTDIR       Output directory (default: graphs)\n  --format {svg,png,pdf}\n                        Graph output format (default: svg)\n  --config CONFIG       Path to configuration file (.digdag-graph.yml)\n  --no-schedule         Skip schedule page generation\n  --exclude EXCLUDE_PATTERNS\n                        Exclude patterns (glob format, can be used multiple times)\n  --include-only INCLUDE_PATTERNS\n                        Include only matching patterns\n  --direction {LR,TB,RL,BT}\n                        Graph direction: LR=Left-Right, TB=Top-Bottom (default: LR)\n  --max-depth MAX_DEPTH\n                        Maximum task nesting depth to visualize\n  --template-dir TEMPLATE_DIR\n                        Custom template directory for HTML pages\n  --verbose, -v         Verbose output for debugging\n  --quiet, -q           Minimal output (errors only)\n```\n\n## Configuration File\n\nCreate `.digdag-graph.yml` in your project root for persistent settings:\n\n```yaml\noutput:\n  directory: graphs\n  format: svg\n\ngraph:\n  direction: LR\n  max_depth: 5\n  include_schedule: true\n\nfilters:\n  exclude_patterns:\n    - \"**/test_*.dig\"\n    - \"**/.archive/**\"\n    - \"**/.backup/**\"\n\nstyling:\n  node_colors:\n    default: \"#e8f0fe\"\n    td\u003e: \"#b2dfdb\"\n    sh\u003e: \"#bbdefb\"\n    py\u003e: \"#ffccbc\"\n```\n\nThen simply run:\n```bash\npython digdag-viz ./workflows\n```\n\n## Documentation\n\nFor detailed guides, see:\n\n- **[Configuration Guide](docs/configuration.md)** - Complete configuration options, environment variables, and examples\n- **[GitHub Actions Integration](docs/github-actions.md)** - Step-by-step guide for CI/CD integration with examples\n\n## Supported Digdag Operators\n\n✅ **Fully Supported:**\n- `td\u003e` - Treasure Data queries (with SQL viewer)\n- `sh\u003e` - Shell commands\n- `py\u003e` - Python scripts\n- `rb\u003e` - Ruby scripts\n- `echo\u003e` - Echo operator\n- `call\u003e` - Call other workflows (with navigation links)\n- `require\u003e` - Require workflows\n- `loop\u003e`, `for_each\u003e`, `for_range\u003e` - Loop constructs\n- `if\u003e` - Conditional execution\n- `_parallel` - Parallel task execution\n\n## Troubleshooting\n\n### \"Command not found: graphviz\"\n**Solution**: Install Graphviz system dependency (see Installation section)\n\n### \"No workflow documents found\"\n**Solution**: Ensure you're pointing to a directory containing `.dig` files\n```bash\n# Check if .dig files exist\nfind /path/to/workflows -name \"*.dig\"\n```\n\n### \"Failed to render graph\"\n**Solution**: Run with `--verbose` to see detailed error messages\n```bash\npython digdag-viz ./workflows --outdir graphs --verbose\n```\n\n### Workflows not showing in scheduled page\n**Solution**: Ensure your workflow has a `_schedule` section:\n```yaml\n_schedule:\n  cron: \"0 0 * * *\"\n  timezone: UTC\n```\n\n## Advanced Features\n\n### CI/CD Integration\n \n #### GitHub Actions\n \n **Option 1: Use the Action (Recommended)**\n ```yaml\n name: Visualize Workflows\n \n on:\n   push:\n     paths: ['**/*.dig']\n \n jobs:\n   visualize:\n     runs-on: ubuntu-latest\n     steps:\n       - uses: actions/checkout@v4\n       \n       - name: Generate workflow graphs\n         uses: yourname/digdag-viz@v2\n         with:\n           workflow-path: '.'\n           output-dir: 'docs/graphs'\n       \n       - name: Deploy to GitHub Pages\n         uses: peaceiris/actions-gh-pages@v3\n         with:\n           github_token: ${{ secrets.GITHUB_TOKEN }}\n           publish_dir: ./docs/graphs\n ```\n \n **Option 2: Manual Installation**\n ```yaml\n - name: Set up Python\n   uses: actions/setup-python@v4\n   with:\n     python-version: '3.11'\n \n - name: Install Graphviz\n   run: sudo apt-get install -y graphviz\n \n - name: Install digdag-viz\n   run: pip install .\n \n - name: Generate graphs\n   run: digdag-viz . --outdir docs/graphs\n ```\n \n #### GitLab CI\n \n Add to `.gitlab-ci.yml`:\n ```yaml\n visualize:\n   image: python:3.11-slim\n   before_script:\n     - apt-get update \u0026\u0026 apt-get install -y graphviz\n     - pip install .\n   script:\n     - digdag-viz . --outdir public\n   artifacts:\n     paths:\n       - public\n ```\n \n ### Docker Support\n \n ```bash\n # Build image\n docker build -t digdag-viz .\n \n # Run\n docker run -v $(pwd)/workflows:/workflows -v $(pwd)/graphs:/output \\\n   digdag-viz /workflows --outdir /output\n ```\n\n## Project Structure\n\n```\nyour-project/\n├── workflows/              # Your .dig files\n│   ├── daily_import.dig\n│   ├── weekly_report.dig\n│   └── queries/\n│       ├── import.sql\n│       └── report.sql\n└── graphs/                 # Generated output\n    ├── index.html          # Main dashboard\n    ├── scheduled_workflows.html\n    ├── unscheduled_workflows.html\n    ├── daily_import.html   # Interactive graph\n    ├── weekly_report.html\n    └── queries/            # SQL viewers\n        ├── import.html\n        └── report.html\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nApache License 2.0 - see [LICENSE](LICENSE) file for details.\n\n## Support\n\n- **Issues**: [GitHub Issues](https://github.com/yourname/digdag-viz/issues)\n- **Documentation**: [Wiki](https://github.com/yourname/digdag-viz/wiki)\n- **Treasure Data Support**: Contact your account team\n\n## Acknowledgments\n\n- Original inspiration: [y-abe/digdag-graph](https://github.com/y-abe/digdag-graph)\n- Built for [Treasure Data](https://www.treasuredata.com/) customers\n- Powered by [Graphviz](https://graphviz.org/) and [Panzoom](https://github.com/timmywil/panzoom)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanilkulkarni87%2Fdigdag-viz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanilkulkarni87%2Fdigdag-viz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanilkulkarni87%2Fdigdag-viz/lists"}